diff options
author | thiagoftsm <thiagoftsm@gmail.com> | 2020-02-17 21:28:33 +0000 |
---|---|---|
committer | GitHub <noreply@github.com> | 2020-02-17 21:28:33 +0000 |
commit | 696264006c1aa4ef111aab2be28f591340575c9e (patch) | |
tree | dd1a01bb2fdee73c232b3639d27f88d202c3e6d3 /collectors | |
parent | 8772cb7a1869e61b3950be48cc05aad90c41b651 (diff) |
eBPF process plugin (#7979)
* syscall_plugin: Compilation
This commit brings the necessaries changes to the compilation files
* syscall_plugin: Collector body
This commit brings the collector body to files.
* syscall_plugin: .gitignore
This commit adds syscall.plugin to .gitignore
* syscall_plugin: Plugin adjust
Fix reference and remove message
* syscall_plugin: Remove limit
Remove call to setrlimit
* syscall: Fix start
This commit fixes problems related with start of the plugin
* syscall_plugin: Bring heartbeat
This commit removes the sleep and changes to heartbeat to avoid plugin receive a SIGTERM
* syscall_plugin: Missing semicolon
* syscall_plugin: Fix dimension
Brings the initial value of chart for the normal dimension of the other values
* syscall_plugin: Fix dimension 2
The previous change did not give the expected results, so I am bringing more a fix
* syscall_plugin: adjust values
Rename function and adjust pid size
* syscall_plugin: Remove Chart and fix var
this commit removes a chart that will not be created and fix an error
when the bytes were calculated
* syscall_plugin: Brings error
This commit brings a new variable that will be used to identify errors
* syscall_plugin: Rename charts
This commit starts to rename the charts properly
* syscall_plugin: Rename plugin
* syscall_plugin: missing changes for rename
* syscall_plugin: fix compilation
* syscall_plugin: bring new charts
* syscall_plugin: Warnings
Remove warnings from compilation time
* vfs_plugin: Fix Error chart plot
There was an error when the chart was being displayed
* vfs_plugin: Change family
This commit changes the family of the VFS plugin
* vfs_plugin: Fix order
This PR fixes the wrong order when creating a chart
* vfs_plugin: Remove path
Remove path from structure
* vfs_plugin: From Perf to HASH
This commit converts the main source a hash table and also split the data collection per chart
* vfs_plugin: Adjusts and exit
This commit brings adjusts to the collect and the complete monitor to exit events
* vfs_plugin: Start process
This commit brings the monitoring of a process start and thread creation to Netdata
* vfs_plugin: Visualization and collection
Adjust variables to show and to collect data
* vfs_plugin: Connection with apps plugin
This commit starts to bring the connection with apps.
* vfs_plugin: Various
This commit brings new label for charts, fix to error chart and adjusts for new charts, I am sorry
* vfs_plugin: basis new chart
This commit brings the basis of the new charts for the plugin
* vfs_plugin: Apps plugin
This commit brings the integration with apps.plugin
* vfs_plugin:fix counter
This commit fixer the difference between apps plugin and counter
* ebpf_plugin: rename charts
This commit renames the charts
* ebpf_plugin: New charts adjusts and log start
* ebpf_plugin: Log thread
Creates the log thread that will be used to store error message
* ebpf_plugin: Rename Web Group
This commit reorganize the charts on dashboard
* ebpf_plugin: Restore
This commit restore the previous status of the collector where we only have a global vision of the problems
* ebpf_plugin: kretprobe
This commit brings the initial changes for the collector works with both eBPF program
* ebpf_plugin: New syscalls
This commit brings the new syscalls that we are monitoring
* ebpf_plugin: New charts
This commit brings new charts to the collector
* ebpf_plugin: Parse config
This commit starts the parser of the file
* ebpf_plugin: collector debug
* ebpf_plugin: Global variables from config
This commit brings the global variable update from the config file
* ebpf_plugin: Clean kprobe_events
This commit brings the clean of kprobe_events and also starts the common library for all eBPF collectors
* ebpf_plugin: Check kernel version
This function brings a check for the kernel version
* ebpf_plugin: Start documentation
This commit brings the initial documentation for the users
* ebpf_plugin: Documentation
This commit brings adjust to code and updates for the documentation
* ebpf_plugin: this commit brings the developer mode to the collector
* ebpf_plugin: Documentation
This commit brings more information to the documentation
* ebpf_plugin: Documentation
This commit brings more information to the documentation
* ebpf_plugin: errno to logs
Brings errno number to logs
* ebpf_plugin: Documentation
This commit brings fixes to the collector documentation
* ebpf_plugin: Move description
This commit move the chart description from the C code to dashboard_info.js
* ebpf_plugin: Rename files
This commit rename files to the final version
* ebpf_plugin: COntinue renaming
This commit continue renaming the files to the final version
* ebpf_plugin: Renaming process
This commit renames the final plugin
* ebpf_plugin: Finish rename
This commit finishes the rename processing
* ebpf_plugin: fix entry charts
This commit removes one chart from mode
* ebpf_plugin: Fix remove
This commit brings a new function to fix the unload of collector when the collector
is running in entry mode
* ebpf_plugin: Rename on old kernels
This commit brings fixes for syscall names
* ebpf_plugin: Timestamp to log
This commit brings the timestamp to the logs
* ebpf_plugin: Remove syscall
With the changes on the backend, we are not monitoring more sys_clone
* ebpf_plugin: The syscall is important for 5.3 or newer, so I am returning
* ebpf_plugin: Remove concurrency
This commit adds variables necessary to interact with the new structor
of the eBPF program
* ebpf_plugin: Ids to dimension
This commit fews the functions name as ids for the dimensions
* ebpf_plugin: Missing chart
This commit brings the missing chart for Netdata
* ebpf_plugin: Remove unecessary message
Remove unecessary error message from the collector
* ebpf_plugin: Rename dimension
This commit renames the dimension for something more meaninful
* ebpf_plugin: Optional log
This commit converts the developer.log in an optional feature
* redirect to stdoou
This commit starts to bring the capability to redirect everything to stdout
* ebpf_plugin: Disable dev mode
This commit removes the possibility to load the dev mode file for while
* ebpf_plugin: Disable eBPF process
By default this plugin won't be enabled
* ebpf_plugin: Update debug message
* ebpf_plugin: this commit adjusts documentation to next release.
* ebpf_plugin: documentation fix.
* ebpf_plugin: Percpu hash
This commit moves from an unique hash table for various to speed up
the collector
* ebpf_plugin: Compatibility
This commit set compatibility version between kernels
Diffstat (limited to 'collectors')
-rw-r--r-- | collectors/Makefile.am | 1 | ||||
-rw-r--r-- | collectors/ebpf_process.plugin/Makefile.am | 12 | ||||
-rw-r--r-- | collectors/ebpf_process.plugin/README.md | 165 | ||||
-rw-r--r-- | collectors/ebpf_process.plugin/ebpf_process.c | 948 | ||||
-rw-r--r-- | collectors/ebpf_process.plugin/ebpf_process.conf | 4 | ||||
-rw-r--r-- | collectors/ebpf_process.plugin/ebpf_process.h | 103 | ||||
-rw-r--r-- | collectors/plugins.d/plugins_d.c | 1 |
7 files changed, 1234 insertions, 0 deletions
diff --git a/collectors/Makefile.am b/collectors/Makefile.am index 7431025704..9bb5295813 100644 --- a/collectors/Makefile.am +++ b/collectors/Makefile.am @@ -24,6 +24,7 @@ SUBDIRS = \ python.d.plugin \ slabinfo.plugin \ statsd.plugin \ + ebpf_process.plugin \ tc.plugin \ $(NULL) diff --git a/collectors/ebpf_process.plugin/Makefile.am b/collectors/ebpf_process.plugin/Makefile.am new file mode 100644 index 0000000000..efb1a97094 --- /dev/null +++ b/collectors/ebpf_process.plugin/Makefile.am @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: GPL-3.0-or-later + +AUTOMAKE_OPTIONS = subdir-objects +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in + +dist_noinst_DATA = \ + README.md \ + $(NULL) + +dist_libconfig_DATA = \ + ebpf_process.conf \ + $(NULL)
\ No newline at end of file diff --git a/collectors/ebpf_process.plugin/README.md b/collectors/ebpf_process.plugin/README.md new file mode 100644 index 0000000000..f2c0efae36 --- /dev/null +++ b/collectors/ebpf_process.plugin/README.md @@ -0,0 +1,165 @@ +# ebpf_process.plugin + +This plugin uses eBPF to monitor system calls inside your operating system's kernel. For now, the main goal of this +plugin is to monitor IO and process management on the host where it is running. + +This plugin has different configuration modes, all of which can be adjusted with its configuration file at +`ebpf_process.conf`. By default, the plugin uses the less expensive `entry` mode. You can learn more about how the +plugin works using `entry` by reading this configuration file. + +You can always edit this file with `edit-config`: + +```bash +cd /etc/netdata/ # Replace with your Netdata configuration directory, if not /etc/netdata/ +./edit-config ebpf_process.conf +``` + +## Enable the plugin on Linux + +Currently, `ebpf_process` only works on Linux systems. + +To enable this plugin and its collector, your operating system's kernel must be more recent than `4.11.0`, and it must +be compiled with the option `CONFIG_KPROBES=y`. You can verify whether your kernel has this option enabled by running +the following commands: + +```bash +# grep CONFIG_KPROBES=y /boot/config-$(uname -r) +# zgrep CONFIG_KPROBES=y /proc/config.gz +``` + +If `Kprobes` is enabled, you will see `CONFIG_KPROBES=y` as the command's output. If you don't see `CONFIG_KPROBES=y` +for any of the commands above, you will have to recompile your kernel to enable it. See the next step, [Recompiling your +kernel](#recompile-your-kernel), for details. + +You also need to have both the `tracefs` and `debugfs` filesystems mounted on your system. + +### Recompile your kernel + +The process of recompiling Linux kernels varies based on your distribution and version. Read the documentation for your +system's distribution to learn more about the specific workflow for + +- [Ubuntu](https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel) +- [Debian](https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#s-common-official) +- [Fedora](https://fedoraproject.org/wiki/Building_a_custom_kernel) +- [CentOS](https://wiki.centos.org/HowTos/Custom_Kernel) +- [Arch Linux](https://wiki.archlinux.org/index.php/Kernel/Traditional_compilation) +- [Slackware](https://docs.slackware.com/howtos:slackware_admin:kernelbuilding) + +### Mount `debugfs` and `tracefs` + +Try mounting the `tracefs` and `debugfs` filesystems using the commands below: + +```bash +# mount -t debugfs nodev /sys/kernel/debug +# mount -t tracefs nodev /sys/kernel/tracing +``` + +If they are already mounted, you will see an error. You can also configure your system's `/etc/fstab` configuration to +mount these filesystems. + +## Enable the eBPF plugin +The plugin is disabled by default because it adds overhead to the system running the Netdata agent. + +To enable it, use `edit-config` to open `netdata.conf` and set `ebpf_process = yes` in the `[plugins]` section. + +```conf +[plugins] + ebpf_process = yes +``` + +## Charts + +The first version of `ebpf_process.plugin` gives a general vision about process running on computer. The charts related +to this plugin are inside the **eBPF** option on dashboard menu and divided in three groups `file`, `vfs`, and +`process`. + +All the collector charts show values per second. The collector retains the total value, but charts only show the +difference between the previous and current metrics collections. + +### File + +This group has two charts to demonstrate how software interacts with the Linux kernel to open and close file +descriptors. + +#### File descriptor + +This chart contain two dimensions that show the number of calls to the functions `do_sys_open` and `__close_fd`. These +functions are not commonly called from software, but they are behind the system cals `open(2)`, `openat(2)`, and +`close(2)`. + +#### File error + +This charts demonstrate the number of times some software tried and failed to open or close a file descriptor. + +### VFS + +A [virtual file system](https://en.wikipedia.org/wiki/Virtual_file_system) (VFS) is an layer on top of regular +filesystems. The functions present inside this API are used for all filesystems, so it's possible the charts in this +group won't show _all_ the actions that occured on your system. + +#### Deleted objects + +This chart monitors calls for `vfs_unlink`. This function is responsible for removing object from the file system. + +#### IO + +This chart shows the number of calls to the functions `vfs_read` and `vfs_write`. + +#### IO bytes + +This chart also monitors `vfs_read` and `vfs_write`, but instead shows the total of bytes read and written with these +functions. + +Netdata displays the number of bytes written as negative, because they are moving down to disk. + +#### IO errors + +Netdata counts and shows the number of instances where a running program experiences a read or write error. + +### Process + +For this group, the eBPF collector monitors process/thread creation and process end, and then displays any errors in the +following charts. + +#### Process thread + +Internally, the Linux kernel treats both process and threads as `tasks`. To create a thread, the kernel offers a few +system calls: `fork(2)`, `vfork(2)` and `clone(2)`. Each of these system calls in turn use the function `_do_fork`. To +generate this chart, Netdata monitors `_do_fork` to populate the `process` dimension, and monitors `sys_clone` to +identify threads + +#### Exit + +Ending a task is actually two steps. The first is a call to the internal function `do_exit`, which notifies the +operating system that the task is finishing its work. The second step is the release of kernel information, which is +done with the internal function `release_task`. The difference between the two dimensions can help you discover [zombie +processes](https://en.wikipedia.org/wiki/Zombie_process). + +#### Task error + +The functions responsible for ending tasks do not return values, so this chart contains information about failures on +process and thread creation. + +## Configuration + +The collector configuration file follows the same structure as `netdata.conf`. It is divided in different sections, with +each one of them having the internal variables. + +### `[global]` + +In this section we define variables applied to the whole collector and the other subsections. + +#### load + +The collector has three different eBPF programs. These programs monitor the same functions inside the kernel, but they +monitor, process, and display different kinds of information. + +By default, this plugin uses the `entry` mode. Changing this mode can create significant overhead on your operating +system, but also offer important information if you are developing or debugging software. The `load` option accepts the +following values: + +- `entry`: This is the default mode. In this mode, Netdata monitors only calls for the functions described in the + sections above. When this mode is selected, Netdata does not show charts related to errors. +- `return`: In this mode, Netdata also monitors the calls to function. In the `entry` mode, Netdata only traces kernel + functions, but with `return`, Netdata also monitors the return of each function. This mode creates more charts, but + also creates an overhead of roughly 110 nanosections for each function call. diff --git a/collectors/ebpf_process.plugin/ebpf_process.c b/collectors/ebpf_process.plugin/ebpf_process.c new file mode 100644 index 0000000000..4a58a54dcc --- /dev/null +++ b/collectors/ebpf_process.plugin/ebpf_process.c @@ -0,0 +1,948 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +#include <sys/time.h> +#include <sys/resource.h> + +#include "ebpf_process.h" + +// callback required by eval() +int health_variable_lookup(const char *variable, uint32_t hash, struct rrdcalc *rc, calculated_number *result) { + (void)variable; + (void)hash; + (void)rc; + (void)result; + return 0; +}; + +void send_statistics( const char *action, const char *action_result, const char *action_data) { + (void) action; + (void) action_result; + (void) action_data; + return; +} + +// callbacks required by popen() +void signals_block(void) {}; +void signals_unblock(void) {}; +void signals_reset(void) {}; + +// required by get_system_cpus() +char *netdata_configured_host_prefix = ""; + +// callback required by fatal() +void netdata_cleanup_and_exit(int ret) { + exit(ret); +} + +// ---------------------------------------------------------------------- +//Netdata eBPF library +void *libnetdata = NULL; +int (*load_bpf_file)(char *, int) = NULL; +int (*set_bpf_perf_event)(int, int); +int (*perf_event_unmap)(struct perf_event_mmap_page *, size_t); +int (*perf_event_mmap_header)(int, struct perf_event_mmap_page **, int); +void (*netdata_perf_loop_multi)(int *, struct perf_event_mmap_page **, int, int *, int (*nsb)(void *, int), int); +int *map_fd = NULL; + +//Perf event variables +static int pmu_fd[NETDATA_MAX_PROCESSOR]; +static struct perf_event_mmap_page *headers[NETDATA_MAX_PROCESSOR]; +int page_cnt = 8; + +//Libbpf (It is necessary to have at least kernel 4.10) +int (*bpf_map_lookup_elem)(int, const void *, void *); + +static char *plugin_dir = PLUGINS_DIR; +static char *user_config_dir = CONFIG_DIR; +static char *stock_config_dir = LIBCONFIG_DIR; +static char *netdata_configured_log_dir = LOG_DIR; + +FILE *developer_log = NULL; + +//Global vectors +netdata_syscall_stat_t *aggregated_data = NULL; +netdata_publish_syscall_t *publish_aggregated = NULL; + +static int update_every = 1; +static int thread_finished = 0; +static int close_plugin = 0; +static int mode = 2; +static int debug_log = 0; +static int use_stdout = 0; +struct config collector_config; +static int mykernel = 0; +static int nprocs; +uint32_t *hash_values; + +pthread_mutex_t lock; + +static char *dimension_names[NETDATA_MAX_MONITOR_VECTOR] = { "open", "close", "delete", "read", "write", "process", "task", "process", "thread" }; +static char *id_names[NETDATA_MAX_MONITOR_VECTOR] = { "do_sys_open", "__close_fd", "vfs_unlink", "vfs_read", "vfs_write", "do_exit", "release_task", "_do_fork", "sys_clone" }; +static char *status[] = { "process", "zombie" }; + +int event_pid = 0; +netdata_ebpf_events_t collector_events[] = { + { .type = 'r', .name = "vfs_write" }, + { .type = 'r', .name = "vfs_writev" }, + { .type = 'r', .name = "vfs_read" }, + { .type = 'r', .name = "vfs_readv" }, + { .type = 'r', .name = "do_sys_open" }, + { .type = 'r', .name = "vfs_unlink" }, + { .type = 'p', .name = "do_exit" }, + { .type = 'p', .name = "release_task" }, + { .type = 'r', .name = "_do_fork" }, + { .type = 'r', .name = "__close_fd" }, + { .type = 'r', .name = "__x64_sys_clone" }, + { .type = 0, .name = NULL } +}; + +void open_developer_log() { + char filename[FILENAME_MAX+1]; + int tot = sprintf(filename, "%s/%s", netdata_configured_log_dir, NETDATA_DEVELOPER_LOG_FILE); + + if(tot > 0) + developer_log = fopen(filename, "a"); +} + +static int unmap_memory() { + int i; + int size = (int)sysconf(_SC_PAGESIZE)*(page_cnt + 1); + for ( i = 0 ; i < nprocs ; i++ ) { + if (perf_event_unmap(headers[i], size) < 0) { + fprintf(stderr,"[EBPF PROCESS] CANNOT unmap headers.\n"); + return -1; + } + + close(pmu_fd[i]); + } + + return 0; +} + +static void int_exit(int sig) +{ + close_plugin = 1; + + //When both threads were not finished case I try to go in front this address, the collector will crash + if (!thread_finished) { + return; + } + + if (aggregated_data) { + free(aggregated_data); + aggregated_data = NULL; + } + + if (publish_aggregated) { + free(publish_aggregated); + publish_aggregated = NULL; + } + + if(mode == 1 && debug_log) { + unmap_memory(); + } + + if (libnetdata) { + dlclose(libnetdata); + libnetdata = NULL; + } + + if (developer_log) { + fclose(developer_log); + developer_log = NULL; + } + + if (hash_values) { + freez(hash_values); + } + + if (event_pid) { + int ret = fork(); + if (ret < 0) //error + error("[EBPF PROCESS] Cannot fork(), so I won't be able to clean %skprobe_events", NETDATA_DEBUGFS); + else if (!ret) { //child + int i; + for ( i=getdtablesize(); i>=0; --i) + close(i); + + int fd = open("/dev/null",O_RDWR, 0); + if (fd != -1) { + dup2 (fd, STDIN_FILENO); + dup2 (fd, STDOUT_FILENO); + dup2 (fd, STDERR_FILENO); + + if (fd > 2) + close (fd); + } + + int sid = setsid(); + if(sid >= 0) { + sleep(1); + if(debug_log) { + open_developer_log(); + } + debug(D_EXIT, "Wait for father %d die", event_pid); + clean_kprobe_events(developer_log, event_pid, collector_events); + } else { + error("Cannot become session id leader, so I won't try to clean kprobe_events.\n"); + } + } else { //parent + exit(0); + } + + if (developer_log) { + fclose(developer_log); + developer_log = NULL; + } + } + + exit(sig); +} + +static inline void netdata_write_chart_cmd(char *type + , char *id + , char *axis + , char *web + , int order) +{ + printf("CHART %s.%s '' '' '%s' '%s' '' line %d 1 ''\n" + , type + , id + , axis + , web + , order); +} + +static void netdata_write_global_dimension(char *dimension, char *name) +{ + printf("DIMENSION %s %s absolute 1 1\n", dimension, name); +} + +static void netdata_create_global_dimension(void *ptr, int end) +{ + netdata_publish_syscall_t *move = ptr; + + int i = 0; + while (move && i < end) { + netdata_write_global_dimension(move->name, move->dimension); + + move = move->next; + i++; + } +} +static inline void netdata_create_chart(char *family + , char *name + , char *axis + , char *web + , int order + , void (*ncd)(void *, int) + , void *move + , int end) +{ + netdata_write_chart_cmd(family, name, axis, web, order); + + ncd(move, end); +} + +static void netdata_create_io_chart(char *family, char *name, char *axis, char *web, int order) { + printf("CHART %s.%s '' '' '%s' '%s' '' line %d 1 ''\n" + , family + , name + , axis + , web + , order); + + printf("DIMENSION %s %s absolute 1 1\n", id_names[3], NETDATA_VFS_DIM_OUT_FILE_BYTES); + printf("DIMENSION %s %s absolute 1 1\n", id_names[4], NETDATA_VFS_DIM_IN_FILE_BYTES); +} + +static void netdata_process_status_chart(char *family, char *name, char *axis, char *web, int order) { + printf("CHART %s.%s '' '' '%s' '%s' '' line %d 1 ''\n" + , family + , name + , axis + , web + , order); + + printf("DIMENSION %s '' absolute 1 1\n", status[0]); + printf("DIMENSION %s '' absolute 1 1\n", status[1]); +} + +static void netdata_global_charts_create() { + netdata_create_chart(NETDATA_EBPF_FAMILY + , NETDATA_FILE_OPEN_CLOSE_COUNT + , "Calls" + , NETDATA_FILE_GROUP + , 970 + , netdata_create_global_dimension + , publish_aggregated + , 2); + + if(mode < 2) { + netdata_create_chart(NETDATA_EBPF_FAMILY + , NETDATA_FILE_OPEN_ERR_COUNT + , "Calls" + , NETDATA_FILE_GROUP + , 971 + , netdata_create_global_dimension + , publish_aggregated + , 2); + } + + netdata_create_chart(NETDATA_EBPF_FAMILY + , NETDATA_VFS_FILE_CLEAN_COUNT + , "Calls" + , NETDATA_VFS_GROUP + , 972 + , netdata_create_global_dimension + , &publish_aggregated[NETDATA_DEL_START] + , 1); + + netdata_create_chart(NETDATA_EBPF_FAMILY + , NETDATA_VFS_FILE_IO_COUNT + , "Calls" + , NETDATA_VFS_GROUP + , 973 + , netdata_create_global_dimension + , &publish_aggregated[NETDATA_IN_START_BYTE] + , 2); + + if(mode < 2) { + netdata_create_io_chart(NETDATA_EBPF_FAMILY + , NETDATA_VFS_IO_FILE_BYTES + , "bytes/s" + , NETDATA_VFS_GROUP + , 974); + + netdata_create_chart(NETDATA_EBPF_FAMILY + , NETDATA_VFS_FILE_ERR_COUNT + , "Calls" + , NETDATA_VFS_GROUP + , 975 + , netdata_create_global_dimension + , &publish_aggregated[2] + , NETDATA_VFS_ERRORS); + + } + + netdata_create_chart(NETDATA_EBPF_FAMILY + , NETDATA_PROCESS_SYSCALL + , "Calls" + , NETDATA_PROCESS_GROUP + , 976 + , netdata_create_global_dimension + , &publish_aggregated[NETDATA_PROCESS_START] + , 2); + + netdata_create_chart(NETDATA_EBPF_FAMILY + , NETDATA_EXIT_SYSCALL + , "Calls" + , NETDATA_PROCESS_GROUP + , 977 + , netdata_create_global_dimension + , &publish_aggregated[NETDATA_EXIT_START] + , 2); + + netdata_process_status_chart(NETDATA_EBPF_FAMILY + , NETDATA_PROCESS_STATUS_NAME + , "Total" + , NETDATA_PROCESS_GROUP + , 978); + + if(mode < 2) { + netdata_create_chart(NETDATA_EBPF_FAMILY + , NETDATA_PROCESS_ERROR_NAME + , "Calls" + , NETDATA_PROCESS_GROUP + , 979 + , netdata_create_global_dimension + , &publish_aggregated[NETDATA_PROCESS_START] + , 2); + } + +} + + +static void netdata_create_charts() { + netdata_global_charts_create(); +} + +static void netdata_update_publish(netdata_publish_syscall_t *publish + , netdata_publish_vfs_common_t *pvc + , netdata_syscall_stat_t *input) { + + netdata_publish_syscall_t *move = publish; + while(move) { + if(input->call != move->pcall) { + //This condition happens to avoid initial values with dimensions higher than normal values. + if(move->pcall) { + move->ncall = (input->call > move->pcall)?input->call - move->pcall: move->pcall - input->call; + move->nbyte = (input->bytes > move->pbyte)?input->bytes - move->pbyte: move->pbyte - input->bytes; + move->nerr = (input->ecall > move->nerr)?input->ecall - move->perr: move->perr - input->ecall; + } else { + move->ncall = 0; + move->nbyte = 0; + move->nerr = 0; + } + + move->pcall = input->call; + move->pbyte = input->bytes; + move->perr = input->ecall; + } else { + move->ncall = 0; + move->nbyte = 0; + move->nerr = 0; + } + + input = input->next; + move = move->next; + } + + pvc->write = -((long)publish[2].nbyte); + pvc->read = (long)publish[3].nbyte; + + pvc->running = (long)publish[7].ncall - (long)publish[8].ncall; + publish[6].ncall = -publish[6].ncall; // release + pvc->zombie = (long)publish[5].ncall + (long)publish[6].ncall; +} + +static inline void write_begin_chart(char *family, char *name) +{ + int ret = printf( "BEGIN %s.%s\n" + , family + , name); + + (void)ret; +} + +static inline void write_chart_dimension(char *dim, long long value) +{ + int ret = printf("SET %s = %lld\n", dim, value); + (void)ret; +} + +static void write_global_count_chart(char *name, char *family, netdata_publish_syscall_t *move, int end) { + write_begin_chart(family, name); + + int i = 0; + while (move && i < end) { + write_chart_dimension(move->name, move->ncall); + + move = move->next; + i++; + } + + printf("END\n"); +} + +static void write_global_err_chart(char *name, char *family, netdata_publish_syscall_t *move, int end) { + write_begin_chart(family, name); + + int i = 0; + while (move && i < end) { + write_chart_dimension(move->name, move->nerr); + + move = move->next; + i++; + } + + printf("END\n"); +} + +static void write_io_chart(char *family, netdata_publish_vfs_common_t *pvc) { + write_begin_chart(family, NETDATA_VFS_IO_FILE_BYTES); + + write_chart_dimension(id_names[3], (long long) pvc->write); + write_chart_dimension(id_names[4], (long long) pvc->read); + + printf("END\n"); +} + +static void write_status_chart(char *family, netdata_publish_vfs_common_t *pvc) { + write_begin_chart(family, NETDATA_PROCESS_STATUS_NAME); + + write_chart_dimension(status[0], (long long) pvc->running); + write_chart_dimension(status[1], (long long) pvc->zombie); + + printf("END\n"); +} + +static void netdata_publish_data() { + netdata_publish_vfs_common_t pvc; + netdata_update_publish(publish_aggregated, &pvc, aggregated_data); + + write_global_count_chart(NETDATA_FILE_OPEN_CLOSE_COUNT, NETDATA_EBPF_FAMILY, publish_aggregated, 2); + write_global_count_chart(NETDATA_VFS_FILE_CLEAN_COUNT, NETDATA_EBPF_FAMILY, &publish_aggregated[NETDATA_DEL_START], 1); + write_global_count_chart(NETDATA_VFS_FILE_IO_COUNT, NETDATA_EBPF_FAMILY, &publish_aggregated[NETDATA_IN_START_BYTE], 2); + write_global_count_chart(NETDATA_EXIT_SYSCALL, NETDATA_EBPF_FAMILY, &publish_aggregated[NETDATA_EXIT_START], 2); + write_global_count_chart(NETDATA_PROCESS_SYSCALL, NETDATA_EBPF_FAMILY, &publish_aggregated[NETDATA_PROCESS_START], 2); + + write_status_chart(NETDATA_EBPF_FAMILY, &pvc); + if(mode < 2) { + write_global_err_chart(NETDATA_FILE_OPEN_ERR_COUNT, NETDATA_EBPF_FAMILY, publish_aggregated, 2); + write_global_err_chart(NETDATA_VFS_FILE_ERR_COUNT, NETDATA_EBPF_FAMILY, &publish_aggregated[2], NETDATA_VFS_ERRORS); + write_global_err_chart(NETDATA_PROCESS_ERROR_NAME, NETDATA_EBPF_FAMILY, &publish_aggregated[NETDATA_PROCESS_START], 2); + + write_io_chart(NETDATA_EBPF_FAMILY, &pvc); + } +} + +void *process_publisher(void *ptr) +{ + (void)ptr; + netdata_create_charts(); + + usec_t step = update_every * USEC_PER_SEC; + heartbeat_t hb; + heartbeat_init(&hb); + while(!close_plugin) { + usec_t dt = heartbeat_next(&hb, step); + (void)dt; + + pthread_mutex_lock(&lock); + netdata_publish_data(); + pthread_mutex_unlock(&lock); + + fflush(stdout); + } + + return NULL; +} + +static void move_from_kernel2user_global() { + uint32_t idx; + uint32_t res[NETDATA_GLOBAL_VECTOR]; + + uint32_t *val = hash_values; + for (idx = 0; idx < NETDATA_GLOBAL_VECTOR; idx++) { + if(!bpf_map_lookup_elem(map_fd[1], &idx, val)) { + uint32_t total = 0; + int i; + int end = (mykernel < 265984)?1:nprocs; + for (i = 0; i < end; i++) + total += val[i]; + + res[idx] = total; + } else { + res[idx] = 0; + } + } + + aggregated_data[0].call = res[0]; //open + aggregated_data[1].call = res[14]; //close + aggregated_data[2].call = res[8]; //unlink + aggregated_data[3].call = res[5] + res[21]; //read + readv + aggregated_data[4].call = res[2] + res[18]; //write + writev + aggregated_data[5].call = res[10]; //exit + aggregated_data[6].call = res[11]; //release + aggregated_data[7].call = res[12]; //fork + aggregated_data[8].call = res[16]; //thread + + aggregated_data[0].ecall = res[1]; //open + aggregated_data[1].ecall = res[15]; //close + aggregated_data[2].ecall = res[9]; //unlink + aggregated_data[3].ecall = res[6] + res[22]; //read + readv + aggregated_data[4].ecall = res[3] + res[19]; //write + writev + aggregated_data[7].ecall = res[13]; //fork + aggregated_data[8].ecall = res[17]; //thread + + aggregated_data[2].bytes = (uint64_t)res[4] + (uint64_t)res[20]; //write + writev + aggregated_data[3].bytes = (uint64_t)res[7] + (uint64_t)res[23];//read + readv +} + +static void move_from_kernel2user() +{ + move_from_kernel2user_global(); +} + +void *process_collector(void *ptr) +{ + (void)ptr; + + usec_t step = 778879ULL; + heartbeat_t hb; + heartbeat_init(&hb); + while(!close_plugin) { + usec_t dt = heartbeat_next(&hb, step); + (void)dt; + + pthread_mutex_lock(&lock); + move_from_kernel2user(); + pthread_mutex_unlock(&lock); + } + + return NULL; +} + +static int netdata_store_bpf(void *data, int size) { + (void)size; + + if (close_plugin) + return 0; + + if(!debug_log) + return -2; //LIBBPF_PERF_EVENT_CONT; + + netdata_error_report_t *e = data; + fprintf(developer_log + ,"%llu %s %u: %s, %d\n" + , now_realtime_usec() ,e->comm, e->pid, dimension_names[e->type], e->err); + fflush(developer_log); + + return -2; //LIBBPF_PERF_EVENT_CONT; +} + +void *process_log(void *ptr) +{ + (void) ptr; + + if (mode == 1 && debug_log) { + netdata_perf_loop_multi(pmu_fd, headers, nprocs, &close_plugin, netdata_store_bpf, page_cnt); + } + + return NULL; +} + +void set_global_labels() { + int i; + + netdata_syscall_stat_t *is = aggregated_data; + netdata_syscall_stat_t *prev = NULL; + + netdata_publish_syscall_t *pio = publish_aggregated; + netdata_publish_syscall_t *publish_prev = NULL; + for (i = 0; i < NETDATA_MAX_MONITOR_VECTOR; i++) { + if(prev) { + prev->next = &is[i]; + } + prev = &is[i]; + + pio[i].dimension = dimension_names[i]; + pio[i].name = id_names[i]; + if(publish_prev) { + publish_prev->next = &pio[i]; + } + publish_prev = &pio[i]; + } +} + +int al |