path: root/collectors/ebpf.plugin/README.md

<!--
title: "Kernel traces/metrics (eBPF) monitoring with Netdata"
description: "Use Netdata's extended Berkeley Packet Filter (eBPF) collector to monitor kernel-level metrics about yourcomplex applications with per-second granularity."
custom_edit_url: "https://github.com/netdata/netdata/edit/master/collectors/ebpf.plugin/README.md"
sidebar_label: "Kernel traces/metrics (eBPF)"
learn_status: "Published"
learn_topic_type: "References"
learn_rel_path: "Integrations/Monitor/System metrics"
-->

# Kernel traces/metrics (eBPF) collector

The Netdata Agent provides many [eBPF](https://ebpf.io/what-is-ebpf/) programs to help you troubleshoot and debug how applications interact with the Linux kernel. The `ebpf.plugin` uses [tracepoints, trampoline, and2 kprobes](#how-netdata-collects-data-using-probes-and-tracepoints) to collect a wide array of high value data about the host that would otherwise be impossible to capture. 

> ❗ eBPF monitoring only works on Linux systems and with specific Linux kernels, including all kernels newer than `4.11.0`, and all kernels on CentOS 7.6 or later. For kernels older than `4.11.0`, improved support is in active development.

This document provides comprehensive details about the `ebpf.plugin`.
For hands-on configuration and troubleshooting tips see our [tutorial on troubleshooting apps with eBPF metrics](https://github.com/netdata/netdata/blob/master/docs/guides/troubleshoot/monitor-debug-applications-ebpf.md).

<figure>
  <img src="https://user-images.githubusercontent.com/1153921/74746434-ad6a1e00-5222-11ea-858a-a7882617ae02.png" alt="An example of VFS charts, made possible by the eBPF collector plugin" />
  <figcaption>An example of virtual file system (VFS) charts made possible by the eBPF collector plugin.</figcaption>
</figure>

## How Netdata collects data using probes and tracepoints

Netdata uses the following features from the Linux kernel to run eBPF programs:

-   Tracepoints are hooks to call specific functions. Tracepoints are more stable than `kprobes` and are preferred when
    both options are available.
-   Trampolines are bridges between kernel functions, and BPF programs. Netdata uses them by default whenever available.
-   Kprobes and return probes (`kretprobe`): Probes can insert virtually into any kernel instruction. When eBPF runs in `entry` mode, it attaches only `kprobes` for internal functions monitoring calls and some arguments every time a function is called. The user can also change configuration to use [`return`](#global-configuration-options) mode, and this will allow users to monitor return from these functions and detect possible failures.

In each case, wherever a normal kprobe, kretprobe, or tracepoint would have run its hook function, an eBPF program is run instead, performing various collection logic before letting the kernel continue its normal control flow.

There are more methods to trigger eBPF programs, such as uprobes, but currently are not supported.

## Configuring ebpf.plugin

The eBPF collector is installed and enabled by default on most new installations of the Agent.
If your Agent is v1.22 or older, you may to enable the collector yourself. 

### Enable the eBPF collector

To enable or disable the entire eBPF collector: 

1.  Navigate to the [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory).
    ```bash
    cd /etc/netdata
    ```

2.  Use the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) script to edit `netdata.conf`.

    ```bash
    ./edit-config netdata.conf
    ```

3.  Enable the collector by scrolling down to the `[plugins]` section. Uncomment the line `ebpf` (not
    `ebpf_process`) and set it to `yes`.

    ```conf
    [plugins]
       ebpf = yes
    ```

### Configure the eBPF collector

You can configure the eBPF collector's behavior to fine-tune which metrics you receive and [optimize performance]\(#performance opimization).

To edit the `ebpf.d.conf`:

1.  Navigate to the [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory).
    ```bash
    cd /etc/netdata
    ```
2.  Use the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) script to edit [`ebpf.d.conf`](https://github.com/netdata/netdata/blob/master/collectors/ebpf.plugin/ebpf.d.conf).

    ```bash
    ./edit-config ebpf.d.conf
    ```

    You can now edit the behavior of the eBPF collector. The following sections describe each configuration option in detail.

### `[global]` configuration options

The `[global]` section defines settings for the whole eBPF collector.

#### eBPF load mode

The collector uses two different eBPF programs. These programs rely on the same functions inside the kernel, but they
monitor, process, and display different kinds of information.

By default, this plugin uses the `entry` mode. Changing this mode can create significant overhead on your operating
system, but also offer valuable information if you are developing or debugging software. The `ebpf load mode` option
accepts the following values:

-   `entry`: This is the default mode. In this mode, the eBPF collector only monitors calls for the functions described in
    the sections above, and does not show charts related to errors.
-   `return`: In the `return` mode, the eBPF collector monitors the same kernel functions as `entry`, but also creates new
    charts for the return of these functions, such as errors. Monitoring function returns can help in debugging software,
    such as failing to close file descriptors or creating zombie processes.

#### Integration with `apps.plugin`

The eBPF collector also creates charts for each running application through an integration with the
[`apps.plugin`](https://github.com/netdata/netdata/blob/master/collectors/apps.plugin/README.md). This integration helps you understand how specific applications
interact with the Linux kernel.

If you want to enable `apps.plugin` integration, change the "apps" setting to "yes".

```conf
[global]
   apps = yes
```

#### Integration with `cgroups.plugin`

The eBPF collector also creates charts for each cgroup through an integration with the
[`cgroups.plugin`](https://github.com/netdata/netdata/blob/master/collectors/cgroups.plugin/README.md). This integration helps you understand how a specific cgroup
interacts with the Linux kernel.

The integration with `cgroups.plugin` is disabled by default to avoid creating overhead on your system. If you want to
_enable_ the integration with `cgroups.plugin`, change the `cgroups` setting to `yes`.

```conf
[global]
   cgroups = yes
```

If you do not need to monitor specific metrics for your `cgroups`, you can enable `cgroups` inside
`ebpf.d.conf`, and then disable the plugin for a specific `thread` by following the steps in the
[Configuration](#configuring-ebpfplugin) section.

#### Maps per Core

When netdata is running on kernels newer than `4.6` users are allowed to modify how the `ebpf.plugin` creates maps (hash or 
array). When `maps per core` is defined as `yes`, plugin will create a map per core on host, on the other hand,
when the value is set as `no` only one hash table will be created, this option will use less memory, but it also can
increase overhead for processes.

#### Collect PID

When one of the previous integrations is enabled, `ebpf.plugin` will use Process Identifier (`PID`) to identify the
process group for which it needs to plot data.

There are different ways to collect PID, and you can select the way `ebpf.plugin` collects data with the following
values:

-   `real parent`: This is the default mode. Collection will aggregate data for the real parent, the thread that creates
     child threads.
-   `parent`: Parent and real parent are the same when a process starts, but this value can be changed during run time.
-   `all`: This option will store all PIDs that run on the host. Note, this method can be expensive for the host,
    because more memory needs to be allocated and parsed.

The threads that have integration with other collectors have an internal clean up wherein they attach either a
`trampoline` or a `kprobe` to `release_task` internal function. To avoid `overload` on this function, `ebpf.plugin`
will only enable these threads integrated with other collectors when the kernel is compiled with
`CONFIG_DEBUG_INFO_BTF`, unless you enable them manually.

#### Collection period

The plugin uses the option `update every` to define the number of seconds used for eBPF to send data for Netdata. The default value
is 5 seconds.

#### PID table size

The option `pid table size` defines the maximum number of PIDs stored inside the application hash table. The default value
is defined according [kernel](https://elixir.bootlin.com/linux/v6.0.19/source/include/linux/threads.h#L28) source code.

#### Integration Dashboard Elements

When an integration is enabled, your dashboard will also show the following cgroups and apps charts using low-level
Linux metrics:

> Note: The parenthetical accompanying each bulleted item provides the chart name.

-   mem
    -   Number of processes killed due out of memory. (`oomkills`)
-   process
    -   Number of processes created with `do_fork`. (`process_create`)
    -   Number of threads created with `do_fork` or `clone (2)`, depending on your system's kernel
        version. (`thread_create`)
    -   Number of times that a process called `do_exit`. (`task_exit`)
    -   Number of times that a process called `release_task`. (`task_close`)
    -   Number of times that an error happened to create thread or process. (`task_error`)
-   swap
    -   Number of calls to `swap_readpage`. (`swap_read_call`)
    -   Number of calls to `swap_writepage`. (`swap_write_call`)
-   network
    -   Number of outbound connections using TCP/IPv4. (`outbound_conn_ipv4`)
    -   Number of outbound connections using TCP/IPv6. (`outbound_conn_ipv6`)
    -   Number of bytes sent. (`total_bandwidth_sent`)
    -   Number of bytes received. (`total_bandwidth_recv`)
    -   Number of calls to `tcp_sendmsg`. (`bandwidth_tcp_send`)
    -   Number of calls to `tcp_cleanup_rbuf`. (`bandwidth_tcp_recv`)
    -   Number of calls to `tcp_retransmit_skb`. (`bandwidth_tcp_retransmit`)
    -   Number of calls to `udp_sendmsg`. (`bandwidth_udp_send`)
    -   Number of calls to `udp_recvmsg`. (`bandwidth_udp_recv`)
-   file access
    -   Number of calls to open files. (`file_open`)
    -   Number of calls to open files that returned errors. (`open_error`)
    -   Number of files closed. (`file_closed`)
    -   Number of calls to close files that returned errors. (`file_error_closed`)
-   vfs
    -   Number of calls to `vfs_unlink`. (`file_deleted`)
    -   Number of calls to `vfs_write`. (`vfs_write_call`)
    -   Number of calls to write a file that returned errors. (`vfs_write_error`)
    -   Number of calls to `vfs_read`. (`vfs_read_call`)
    -   -   Number of calls to read a file that returned errors. (`vfs_read_error`)
    -   Number of bytes written with `vfs_write`. (`vfs_write_bytes`)
    -   Number of bytes read with `vfs_read`. (`vfs_read_bytes`)
    -   Number of calls to `vfs_fsync`. (`vfs_fsync`)
    -   Number of calls to sync file that returned errors. (`vfs_fsync_error`)
    -   Number of calls to `vfs_open`. (`vfs_open`)
    -   Number of calls to open file that returned errors. (`vfs_open_error`)
    -   Number of calls to `vfs_create`. (`vfs_create`)
    -   Number of calls to open file that returned errors. (`vfs_create_error`)
-   page cache
    -   Ratio of pages accessed. (`cachestat_ratio`)
    -   Number of modified pages ("dirty"). (`cachestat_dirties`)
    -   Number of accessed pages. (`cachestat_hits`)
    -   Number of pages brought from disk. (`cachestat_misses`)
-   directory cache
    -   Ratio of files available in directory cache. (