path: root/collectors/ebpf.plugin/README.md

<!--
title: "eBPF monitoring with Netdata"
description: "Use Netdata's extended Berkeley Packet Filter (eBPF) collector to monitor kernel-level metrics about your
complex applications with per-second granularity."
custom_edit_url: https://github.com/netdata/netdata/edit/master/collectors/ebpf.plugin/README.md
sidebar_label: "eBPF"
-->

# eBPF monitoring with Netdata

eBPF consists of a wide toolchain that ultimately outputs a set of bytecode that will run inside the eBPF virtual
machine (VM) which lives inside the Linux kernel. The program in particular is executed in response to a [tracepoint
or kprobe](#probes-and-tracepoints) activation.

Netdata has written many eBPF programs, which, when compiled and integrated into the Netdata Agent, are able to collect
a wide array of data about the host that would otherwise be impossible. The data eBPF programs can collect is truly unique,
which gives the Netdata Agent access to data that is high value but normally hard to capture.

eBPF monitoring can help you troubleshoot and debug how applications interact with the Linux kernel. See
our [guide on troubleshooting apps with eBPF metrics](/docs/guides/troubleshoot/monitor-debug-applications-ebpf.md) for
configuration and troubleshooting tips.

<figure>
  <img src="https://user-images.githubusercontent.com/1153921/74746434-ad6a1e00-5222-11ea-858a-a7882617ae02.png" alt="An example of VFS charts, made possible by the eBPF collector plugin" />
  <figcaption>An example of VFS charts made possible by the eBPF collector plugin.</figcaption>
</figure>

## Probes and Tracepoints

The following two features from the Linux kernel are used by Netdata to run eBPF programs:

- Kprobes and return probes (kretprobe): Probes can insert virtually into any kernel instruction. When eBPF runs in
  `entry` mode, it attaches only `kprobes` for internal functions monitoring calls and some arguments every time a
  function is called. The user can also change configuration to use [`return`](#global) mode, and this will allow users
  to monitor return from these functions and detect possible failures.
- Tracepoints are hooks to call specific functions. Tracepoints are more stable than `kprobes` and are preferred when
  both options are available.

In each case, wherever a normal kprobe, kretprobe, or tracepoint would have run its hook function, an eBPF program is
run instead, performing various collection logic before letting the kernel continue its normal control flow.

There are more methods by which eBPF programs can be triggered but which are not currently supported, such as via uprobes
which allow hooking into arbitrary user-space functions in a similar manner to kprobes.

## Manually enable the collector on Linux

**The eBPF collector is installed and enabled by default on most new installations of the Agent**. The eBPF collector
does not currently work with [static build installations](/packaging/installer/methods/kickstart-64.md) for kernels older
than `4.11`, but improved support is in active development.

eBPF monitoring only works on Linux systems and with specific Linux kernels, including all kernels newer than `4.11.0`,
and all kernels on CentOS 7.6 or later.

If your Agent is v1.22 or older, you may to enable the collector yourself. See the [configuration](#configuration)
section for details.

## Charts

The eBPF collector creates charts on different menus, like System Overview, Memory, MD arrays, Disks, Filesystem,
Mount Points, Networking Stack, systemd Services, and Applications.

The collector stores the actual value inside of its process, but charts only show the difference between the values
collected in the previous and current seconds.

### System overview

Not all charts within the System Overview menu are enabled by default, because they add around 100ns overhead for each
function call, this number is small for a human perspective, but the functions are called many times creating an impact
on host. See the [configuration](#configuration) section for details about how to enable them.

#### Processes

Internally, the Linux kernel treats both processes and threads as `tasks`. To create a thread, the kernel offers a few
system calls: `fork(2)`, `vfork(2)`, and `clone(2)`. To generate this chart, the eBPF
collector uses the following `tracepoints` and `kprobe`:

- `sched/sched_process_fork`: Tracepoint called after a call for `fork (2)`, `vfork (2)` and `clone (2)`.
- `sched/sched_process_exec`: Tracepoint called after a exec-family syscall.
- `kprobe/kernel_clone`: This is the main [`fork()`](https://elixir.bootlin.com/linux/v5.10/source/kernel/fork.c#L2415)
   routine since kernel `5.10.0` was released.
- `kprobe/_do_fork`: Like `kernel_clone`, but this was the main function between kernels `4.2.0` and `5.9.16`
- `kprobe/do_fork`: This was the main function before kernel `4.2.0`.

#### Process Exit

Ending a task requires two steps. The first is a call to the internal function `do_exit`, which notifies the operating
system that the task is finishing its work. The second step is to release the kernel information with the internal
function `release_task`. The difference between the two dimensions can help you discover
[zombie processes](https://en.wikipedia.org/wiki/Zombie_process). To get the metrics, the collector uses:

- `sched/sched_process_exit`: Tracepoint called after a task exits.
- `kprobe/release_task`: This function is called when a process exits, as the kernel still needs to remove the process
  descriptor.

#### Task error

The functions responsible for ending tasks do not return values, so this chart contains information about failures on
process and thread creation only.

#### Swap

Inside the swap submenu the eBPF plugin creates the chart `swapcalls`; this chart is displaying when processes are
calling functions [`swap_readpage` and `swap_writepage`](https://hzliu123.github.io/linux-kernel/Page%20Cache%20in%20Linux%202.6.pdf ),
which are functions responsible for doing IO in swap memory. To collect the exact moment that an access to swap happens,
the collector attaches `kprobes` for cited functions.

#### Soft IRQ

The following `tracepoints` are used to measure time usage for soft IRQs:

- [`irq/softirq_entry`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_softirq_entry): Called
   before softirq handler
- [`irq/softirq_exit`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_softirq_exit): Called when
   softirq handler returns.

#### Hard IRQ

The following tracepoints are used to measure the latency of servicing a
hardware interrupt request (hard IRQ).

- [`irq/irq_handler_entry`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_irq_handler_entry):
  Called immediately before the IRQ action handler.
- [`irq/irq_handler_exit`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_irq_handler_exit):
  Called immediately after the IRQ action handler returns.
- `irq_vectors`: These are traces from `irq_handler_entry` and
  `irq_handler_exit` when an IRQ is handled. The following elements from vector
  are triggered:
    - `irq_vectors/local_timer_entry`
    - `irq_vectors/local_timer_exit`
    - `irq_vectors/reschedule_entry`
    - `irq_vectors/reschedule_exit`
    - `irq_vectors/call_function_entry`
    - `irq_vectors/call_function_exit`
    - `irq_vectors/call_function_single_entry`
    - `irq_vectors/call_function_single_xit`
    - `irq_vectors/irq_work_entry`
    - `irq_vectors/irq_work_exit`
    - `irq_vectors/error_apic_entry`
    - `irq_vectors/error_apic_exit`
    - `irq_vectors/thermal_apic_entry`
    - `irq_vectors/thermal_apic_exit`
    - `irq_vectors/threshold_apic_entry`
    - `irq_vectors/threshold_apic_exit`
    - `irq_vectors/deferred_error_entry`
    - `irq_vectors/deferred_error_exit`
    - `irq_vectors/spurious_apic_entry`
    - `irq_vectors/spurious_apic_exit`
    - `irq_vectors/x86_platform_ipi_entry`
    - `irq_vectors/x86_platform_ipi_exit`

#### IPC shared memory

To monitor shared memory system call counts, the following `kprobes` are used:

- `shmget`: Runs when [`shmget`](https://man7.org/linux/man-pages/man2/shmget.2.html) is called.
- `shmat`: Runs when [`shmat`](https://man7.org/linux/man-pages/man2/shmat.2.html) is called.
- `shmdt`: Runs when [`shmdt`](https://man7.org/linux/man-pages/man2/shmat.2.html) is called.
- `shmctl`: Runs when [`shmctl`](https://man7.org/linux/man-pages/man2/shmctl.2.html) is called.

### Memory

In the memory submenu the eBPF plugin creates two submenus **page cache** and **synchronization** with the following
organization:

* Page Cache
    * Page cache ratio
    * Dirty pages
    * Page cache hits
    * Page cache misses
* Synchronization
    * File sync
    * Memory map sync
    * File system sync
    * File range sync

#### Page cache ratio

The chart `cachestat_ratio` shows how processes are accessing page cache. In a normal scenario, we expect values around
100%, which means that the majority of the work on the machine is processed in memory. To calculate the ratio, Netdata
attaches `kprobes` for kernel functions:

- `add_to_page_cache_lru`: Page addition.
- `mark_page_accessed`: Access to cache.
- `account_page_dirtied`: Dirty (modified) pages.
- `mark_buffer_dirty`: Writes to page cache.

#### Dirty pages

On `cachestat_dirties` Netdata demonstrates the number of pages that were modified. This chart shows the number of calls
to the function `mark_buffer_dirty`.

#### Page cache hits

A page cache hit is when the page cache is successfully accessed with a read operation. We do not count pages that were
added relatively recently.

#### Page cache misses

A page cache miss means that a page was not inside memory when the process tried to access it. This chart shows the
result of the difference for calls between functions `add_to_page_cache_lru` and `account_page_dirtied`.

#### File sync

This chart shows calls to synchronization methods, [`fsync(2)`](https://man7.org/linux/man-pages/man2/fdatasync.2.html)
and [`fdatasync(2)`](https://man7.org/linux/man-pages/man2/fdatasync.2.html), to transfer all modified page caches
for the files on disk devices. These calls block until the disk reports that the transfer has been completed. They flush
data for specific file descriptors.

#### Memory map sync

The chart shows calls to [`msync(2)`](https://man7.org/linux/man-pages/man2/msync.2.html) syscalls. This syscall flushes
changes to a file that was mapped into memory using [`mmap(2)`](https://man7.org/linux/man-pages/man2/mmap.2.html).

#### File system sync

This chart monitors calls demonstrating commits from filesystem caches to disk. Netdata attaches `kprobes` for
[`sync(2)`](https://man7.org/linux/man-pages/man2/sync.2.html), and [`syncfs(2)`](https://man7.org/linux/man-pages/man2/sync.2.html).

#### File range sync

This chart shows calls to [`sync_file_range(2)`](https://man7.org/linux/man-pages/man2/sync_file_range.2.html) which
synchronizes file segments with disk.

> Note: This is the most dangerous syscall to synchronize data, according to its manual.

### Multiple Device (MD) arrays

The eBPF plugin shows multi-device flushes happening in real time. This can be used to explain some spikes happening
in [disk latency](#disk) charts.

By default, MD flush is disabled. To enable it, configure your
`/etc/netdata/ebpf.d.conf` file as:

```conf
[global]
    mdflush = yes
```

#### MD flush

To collect data related to Linux multi-device (MD) flushing, the following kprobe is used:

-  `kprobe/md_flush_request`: called whenever a request for flushing multi-device data is made.

### Disk

The eBPF plugin also shows a chart in the Disk section when the `disk` thread is enabled. This will create the
chart `disk_latency_io` for each disk on the host. The following tracepoints are used:

- [`block/block_rq_issue`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_block_rq_issue):
  IO request operation to a device drive.
- [`block/block_rq_complete`](https://www.kernel.org/doc/html/latest/core-api/tracepoint.html#c.trace_block_rq_complete):
  IO operation completed by device.

### Filesystem

This group has charts demonstrating how applications interact with the Linux
kernel to open and close file descriptors. It also brings latency charts for
several different filesystems.

#### ext4

To measure the latency of executing some actions in an
[ext4](https://elixir.bootlin.com/linux/latest/source/fs/ext4) filesystem, the
collector needs to attach `kprobes` and `kretprobes` for each of the following
functions:

- `ext4_file_read_iter`: Function used to measure read latency.
- `ext4_file_write_iter`: Function used to measure write latency.
- `ext4_file_open`: Function used to measure open latency.
- `ext4_sync_file`: Function used to measure sync latency.

#### ZFS

To measure the latency of executing some actions in a zfs filesystem, the
collector needs to attach `kprobes` and `kretprobes` for each of the following
functions:

- `zpl_iter_read`: Function used to measure read latency.
- `zpl_iter_write`: Function used to measure write latency.
- `zpl_open`: Function used to measure open latency.
- `zpl_fsync`: Function used to measure sync latency.

#### XFS

To measure the latency of executing some actions in an
[xfs](https://elixir.bootlin.com/linux/latest/source/fs/xfs) filesystem, the
collector needs to attach `kprobes` and `kretprobes` for each of the following
functions:

- `xfs_file_read_iter`: Function used to measure read latency.
- `xfs_file_write_iter`: Function used to measure write latency.
- `xfs_file_open`: Function used to measure open latency.
- `xfs_file_fsync`: Function used to measure sync latency.

#### NFS

To measure the latency of executing some actions in an
[nfs](https://elixir.bootlin.com/linux/latest/source/fs/nfs) filesystem, the
collector needs to attach `kprobes` and `kretprobes` for each of the following
functions:

- `nfs_file_read`: Function used to measure read latency.
- `nfs_file_write`: Function used to measure write latency.
- `nfs_file_open`: Functions used to measure open latency.
- `nfs4_file_open`: Functions used to measure open latency for NFS v4.
- `nfs_getattr`: Function used to measure sync latency.

#### btrfs

To measure the latency of executing some actions in a [btrfs](https://elixir.bootlin.com/linux/latest/source/fs/btrfs/file.c)
filesystem, the collector needs to attach `kprobes` and `kretprobes` for each of the following functions:

> Note: We are listing two functions used to measure `read` latency, but we use either `btrfs_file_read_iter`