diff options
-rw-r--r-- | collectors/README.md | 68 | ||||
-rw-r--r-- | collectors/REFERENCE.md | 207 | ||||
-rw-r--r-- | collectors/charts.d.plugin/README.md | 2 | ||||
-rw-r--r-- | collectors/plugins.d/README.md | 6 | ||||
-rw-r--r-- | collectors/python.d.plugin/README.md | 2 | ||||
-rw-r--r-- | docs/a-github-star-is-important.md | 28 | ||||
-rw-r--r-- | docs/collect/enable-configure.md | 72 | ||||
-rw-r--r-- | docs/collect/how-collectors-work.md | 82 | ||||
-rw-r--r-- | docs/guides/step-by-step/step-06.md | 129 | ||||
-rw-r--r-- | docs/guides/step-by-step/step-07.md | 121 | ||||
-rw-r--r-- | docs/guides/step-by-step/step-08.md | 407 | ||||
-rw-r--r-- | docs/guides/step-by-step/step-09.md | 169 | ||||
-rw-r--r-- | docs/guides/step-by-step/step-99.md | 55 |
13 files changed, 139 insertions, 1209 deletions
diff --git a/collectors/README.md b/collectors/README.md index 91a4eeb449..537da6e381 100644 --- a/collectors/README.md +++ b/collectors/README.md @@ -10,45 +10,63 @@ learn_rel_path: "References/Collectors" # Collecting metrics -Netdata can collect metrics from hundreds of different sources, be they internal data created by the system itself, or -external data created by services or applications. To see _all_ of the sources Netdata collects from, view our -[list of supported collectors](https://github.com/netdata/netdata/blob/master/collectors/COLLECTORS.md). +When Netdata starts, and with zero configuration, it auto-detects thousands of data sources and immediately collects +per-second metrics. -There are two essential points to understand about how collecting metrics works in Netdata: +Netdata can immediately collect metrics from these endpoints thanks to 300+ **collectors**, which all come pre-installed +when you [install Netdata](https://github.com/netdata/netdata/blob/master/packaging/installer/README.md). -- All collectors are **installed by default** with every installation of Netdata. You do not need to install - collectors manually to collect metrics from new sources. -- Upon startup, Netdata will **auto-detect** any application or service that has a - [collector](https://github.com/netdata/netdata/blob/master/collectors/COLLECTORS.md), as long as both the collector - and the app/service are configured correctly. +All collectors are **installed by default** with every installation of Netdata. You do not need to install +collectors manually to collect metrics from new sources. See our +[list of supported collectors](https://github.com/netdata/netdata/blob/master/collectors/COLLECTORS.md). -Most users will want to enable a new Netdata collector for their app/service. For those details, see +Upon startup, Netdata will **auto-detect** any application or service that has a collector, as long as both the collector +and the app/service are configured correctly. If you don't see charts for your application, see our [collectors' configuration reference](https://github.com/netdata/netdata/blob/master/collectors/REFERENCE.md). -## Take your next steps with collectors +## How Netdata's metrics collectors work + +Every collector has two primary jobs: + +- Look for exposed metrics at a pre- or user-defined endpoint. +- Gather exposed metrics and use additional logic to build meaningful, interactive visualizations. -[Supported collectors list](https://github.com/netdata/netdata/blob/master/collectors/COLLECTORS.md) +If the collector finds compatible metrics exposed on the configured endpoint, it begins a per-second collection job. The +Netdata Agent gathers these metrics, sends them to the +[database engine for storage](https://github.com/netdata/netdata/blob/master/docs/store/change-metrics-storage.md) +, and immediately +[visualizes them meaningfully](https://github.com/netdata/netdata/blob/master/docs/visualize/interact-dashboards-charts.md) +on dashboards. -[Collectors configuration reference](https://github.com/netdata/netdata/blob/master/collectors/REFERENCE.md) +Each collector comes with a pre-defined configuration that matches the default setup for that application. This endpoint +can be a URL and port, a socket, a file, a web page, and more. The endpoint is user-configurable, as are many other +specifics of what a given collector does. -## Guides +## Collector architecture and terminology -[Monitor Nginx or Apache web server log files with Netdata](https://github.com/netdata/netdata/blob/master/docs/guides/collect-apache-nginx-web-logs.md) +- **Collectors** are the processes/programs that actually gather metrics from various sources. -[Monitor CockroachDB metrics with Netdata](https://github.com/netdata/netdata/blob/master/docs/guides/monitor-cockroachdb.md) +- **Plugins** help manage all the independent data collection processes in a variety of programming languages, based on + their purpose and performance requirements. There are three types of plugins: -[Monitor Unbound DNS servers with Netdata](https://github.com/netdata/netdata/blob/master/docs/guides/collect-unbound-metrics.md) + - **Internal** plugins organize collectors that gather metrics from `/proc`, `/sys` and other Linux kernel sources. + They are written in `C`, and run as threads within the Netdata daemon. -[Monitor a Hadoop cluster with Netdata](https://github.com/netdata/netdata/blob/master/docs/guides/monitor-hadoop-cluster.md) + - **External** plugins organize collectors that gather metrics from external processes, such as a MySQL database or + Nginx web server. They can be written in any language, and the `netdata` daemon spawns them as long-running + independent processes. They communicate with the daemon via pipes. All external plugins are managed by + [plugins.d](https://github.com/netdata/netdata/blob/master/collectors/plugins.d/README.md), which provides additional management options. -## Related features +- **Orchestrators** are external plugins that run and manage one or more modules. They run as independent processes. + The Go orchestrator is in active development. -**[Dashboards](https://github.com/netdata/netdata/blob/master/web/README.md)**: Visualize your newly-collect metrics in -real-time using Netdata's [built-in dashboard](https://github.com/netdata/netdata/blob/master/web/gui/README.md). + - [go.d.plugin](https://github.com/netdata/go.d.plugin/blob/master/README.md): An orchestrator for data + collection modules written in `go`. -**[Exporting](https://github.com/netdata/netdata/blob/master/exporting/README.md)**: Extend our -built-in [database engine](https://github.com/netdata/netdata/blob/master/database/engine/README.md), which supports -long-term metrics storage, by archiving metrics to external databases like Graphite, Prometheus, MongoDB, TimescaleDB, -and more. It can export metrics to multiple databases simultaneously. + - [python.d.plugin](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/README.md): + An orchestrator for data collection modules written in `python` v2/v3. + - [charts.d.plugin](https://github.com/netdata/netdata/blob/master/collectors/charts.d.plugin/README.md): + An orchestrator for data collection modules written in`bash` v4+. +- **Modules** are the individual programs controlled by an orchestrator to collect data from a specific application, or type of endpoint. diff --git a/collectors/REFERENCE.md b/collectors/REFERENCE.md index 270dded29f..8ca4ef7f8e 100644 --- a/collectors/REFERENCE.md +++ b/collectors/REFERENCE.md @@ -9,76 +9,24 @@ learn_rel_path: "Setup" # Collectors configuration reference -Welcome to the collector configuration reference guide. +The list of supported collectors can be found in [the documentation](https://github.com/netdata/netdata/blob/master/collectors/COLLECTORS.md), +and on [our website](https://www.netdata.cloud/integrations). The documentation of each collector provides all the +necessary configuration options and prerequisites for that collector. In most cases, either the charts are automatically generated +without any configuration, or you just fulfil those prerequisites and [configure the collector](#configure-a-collector). -This guide contains detailed information about enabling/disabling plugins or modules, in addition a quick reference to -the internal plugins API. +If the application you are interested in monitoring is not listed in our integrations, +look at our [generic Prometheus collector](https://github.com/netdata/go.d.plugin/blob/master/modules/prometheus/README.md) +or [write a custom collector in Go](https://github.com/netdata/go.d.plugin/blob/master/docs/how-to-write-a-module.md). -## Netdata's collector architecture +If we do support your collector but the charts described in the documentation don't appear on your dashboard, the reason will +be one of the following: -Netdata has an intricate system for organizing and managing its collectors. **Collectors** are the processes/programs -that actually gather metrics from various sources. Collectors are organized by **plugins**, which help manage all the -independent processes in a variety of programming languages based on their purpose and performance requirements. -**Modules** are a type of collector, used primarily to connect to external applications, such as an Nginx web server or -MySQL database, among many others. +- The entire data collection plugin is disabled by default. Read how to [enable and disable plugins](#enable-and-disable-plugins) -For most users, enabling individual collectors for the application/service you're interested in is far more important -than knowing which plugin it uses. See our [collectors list](https://github.com/netdata/netdata/blob/master/collectors/COLLECTORS.md) to see whether your favorite app/service has -a collector, and then read the documentation for that specific collector to figure out how to enable it. +- The data collection plugin is enabled, but a specific data collection module is disabled. Read how to + [enable and disable a specific collection module](#enable-and-disable-a-specific-collection-module). -There are three types of plugins: - -- **Internal** plugins organize collectors that gather metrics from `/proc`, `/sys` and other Linux kernel sources. - They are written in `C`, and run as threads within the Netdata daemon. -- **External** plugins organize collectors that gather metrics from external processes, such as a MySQL database or - Nginx web server. They can be written in any language, and the `netdata` daemon spawns them as long-running - independent processes. They communicate with the daemon via pipes. -- **Plugin orchestrators**, which are external plugins that instead support a number of **modules**. Modules are a - type of collector. We have a few plugin orchestrators available for those who want to develop their own collectors, - but focus most of our efforts on the [Go plugin](https://github.com/netdata/go.d.plugin/blob/master/README.md). - -## Enable, configure, and disable modules - -Most collector modules come with **auto-detection**, configured to work out-of-the-box on popular operating systems with -the default settings. - -However, there are cases that auto-detection fails. Usually, the reason is that the applications to be monitored do not -allow Netdata to connect. In most of the cases, allowing the user `netdata` from `localhost` to connect and collect -metrics, will automatically enable data collection for the application in question (it will require a Netdata restart). - - -## Troubleshoot a collector - -First, navigate to your plugins directory, which is usually at `/usr/libexec/netdata/plugins.d/`. If that's not the case -on your system, open `netdata.conf` and look for the setting `plugins directory`. Once you're in the plugins directory, -switch to the `netdata` user. - -```bash -cd /usr/libexec/netdata/plugins.d/ -sudo su -s /bin/bash netdata -``` - -The next step is based on the collector's orchestrator. You can figure out which orchestrator the collector uses by - -uses either -by viewing the [collectors list](https://github.com/netdata/netdata/blob/master/collectors/COLLECTORS.md) and referencing the _configuration file_ field. For example, if that -field contains `go.d`, that collector uses the Go orchestrator. - -```bash -# Go orchestrator (go.d.plugin) -./go.d.plugin -d -m <MODULE_NAME> - -# Python orchestrator (python.d.plugin) -./python.d.plugin <MODULE_NAME> debug trace - -# Bash orchestrator (bash.d.plugin) -./charts.d.plugin debug 1 <MODULE_NAME> -``` - -The output from the relevant command will provide valuable troubleshooting information. If you can't figure out how to -enable the collector using the details from this output, feel free to [create an issue on our -GitHub](https://github.com/netdata/netdata/issues/new?assignees=&labels=bug%2Cneeds+triage&template=BUG_REPORT.yml) to get some -help from our collectors experts. +- Autodetection failed. Read how to [configure](#configure-a-collector) and [troubleshoot](#troubleshoot-a-collector) a collector. ## Enable and disable plugins @@ -88,87 +36,114 @@ This section features a list of Netdata's plugins, with a boolean setting to ena ```conf [plugins] - # proc = yes - # diskspace = yes # timex = yes - # cgroups = yes - # tc = yes # idlejitter = yes + # netdata monitoring = yes + # tc = yes + # diskspace = yes + # proc = yes + # cgroups = yes # enable running new plugins = yes # check for new plugins every = 60 # slabinfo = no - # ioping = yes # python.d = yes + # perf = yes + # ioping = yes + # fping = yes + # nfacct = yes # go.d = yes # apps = yes - # perf = yes + # ebpf = yes # charts.d = yes + # statsd = yes ``` By default, most plugins are enabled, so you don't need to enable them explicitly to use their collectors. To enable or disable any specific plugin, remove the comment (`#`) and change the boolean setting to `yes` or `no`. -All **external plugins** are managed by [plugins.d](https://github.com/netdata/netdata/blob/master/collectors/plugins.d/README.md), which provides additional management options. +## Enable and disable a specific collection module -## Internal plugins +You can enable/disable of the collection modules supported by `go.d`, `python.d` or `charts.d` individually, using the +configuration file of that orchestrator. For example, you can change the behavior of the Go orchestrator, or any of its +collectors, by editing `go.d.conf`. -Each of the internal plugins runs as a thread inside the `netdata` daemon. Once this thread has started, the plugin may -spawn additional threads according to its design. +Use `edit-config` from your [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) +to open the orchestrator primary configuration file: -### Internal plugins API +```bash +cd /etc/netdata +sudo ./edit-config go.d.conf +``` -The internal data collection API consists of the following calls: +Within this file, you can either disable the orchestrator entirely (`enabled: yes`), or find a specific collector and +enable/disable it with `yes` and `no` settings. Uncomment any line you change to ensure the Netdata daemon reads it on +start. -```c -collect_data() { - // collect data here (one iteration) +After you make your changes, restart the Agent with `sudo systemctl restart netdata`, or the [appropriate +method](https://github.com/netdata/netdata/blob/master/docs/configure/start-stop-restart.md) for your system. - collected_number collected_value = collect_a_value(); +## Configure a collector - // give the metrics to Netdata +Most collector modules come with **auto-detection**, configured to work out-of-the-box on popular operating systems with +the default settings. - static RRDSET *st = NULL; // the chart - static RRDDIM *rd = NULL; // a dimension attached to this chart +However, there are cases that auto-detection fails. Usually, the reason is that the applications to be monitored do not +allow Netdata to connect. In most of the cases, allowing the user `netdata` from `localhost` to connect and collect +metrics, will automatically enable data collection for the application in question (it will require a Netdata restart). - if(unlikely(!st)) { - // we haven't created this chart before - // create it now - st = rrdset_create_localhost( - "type" - , "id" - , "name" - , "family" - , "context" - , "Chart Title" - , "units" - , "plugin-name" - , "module-name" - , priority - , update_every - , chart_type - ); +When Netdata starts up, each collector searches for exposed metrics on the default endpoint established by that service +or application's standard installation procedure. For example, +the [Nginx collector](https://github.com/netdata/go.d.plugin/blob/master/modules/nginx/README.md) searches at +`http://127.0.0.1/stub_status` for exposed metrics in the correct format. If an Nginx web server is running and exposes +metrics on that endpoint, the collector begins gathering them. - // attach a metric to it - rd = rrddim_add(st, "id", "name", multiplier, divider, algorithm); - } +However, not every node or infrastructure uses standard ports, paths, files, or naming conventions. You may need to +enable or configure a collector to gather all available metrics from your systems, containers, or applications. - // give the collected value(s) to the chart - rrddim_set_by_pointer(st, rd, collected_value); +First, [find the collector](https://github.com/netdata/netdata/blob/master/collectors/COLLECTORS.md) you want to edit +and open its documentation. Some software has collectors written in multiple languages. In these cases, you should always +pick the collector written in Go. - // signal Netdata we are done with this iteration - rrdset_done(st); -} +Use `edit-config` from your +[Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) +to open a collector's configuration file. For example, edit the Nginx collector with the following: + +```bash +./edit-config go.d/nginx.conf ``` -Of course, Netdata has a lot of libraries to help you also in collecting the metrics. The best way to find your way -through this, is to examine what other similar plugins do. +Each configuration file describes every available option and offers examples to help you tweak Netdata's settings +according to your needs. In addition, every collector's documentation shows the exact command you need to run to +configure that collector. Uncomment any line you change to ensure the collector's orchestrator or the Netdata daemon +read it on start. + +After you make your changes, restart the Agent with `sudo systemctl restart netdata`, or the [appropriate +method](https://github.com/netdata/netdata/blob/master/docs/configure/start-stop-restart.md) for your system. + +## Troubleshoot a collector -## External Plugins +First, navigate to your plugins directory, which is usually at `/usr/libexec/netdata/plugins.d/`. If that's not the case +on your system, open `netdata.conf` and look for the setting `plugins directory`. Once you're in the plugins directory, +switch to the `netdata` user. -**External plugins** use the API and are managed -by [plugins.d](https://github.com/netdata/netdata/blob/master/collectors/plugins.d/README.md). +```bash +cd /usr/libexec/netdata/plugins.d/ +sudo su -s /bin/bash netdata +``` -## Write a custom collector +The next step is based on the collector's orchestrator. -You can add custom collectors by following the [external plugins documentation](https://github.com/netdata/netdata/blob/master/collectors/plugins.d/README.md). +```bash +# Go orchestrator (go.d.plugin) +./go.d.plugin -d -m <MODULE_NAME> +# Python orchestrator (python.d.plugin) +./python.d.plugin <MODULE_NAME> debug trace + +# Bash orchestrator (bash.d.plugin) +./charts.d.plugin debug 1 <MODULE_NAME> +``` + +The output from the relevant command will provide valuable troubleshooting information. If you can't figure out how to +enable the collector using the details from this output, feel free to [join our Discord server](https://discord.com/invite/mPZ6WZKKG2), +to get help from our experts. diff --git a/collectors/charts.d.plugin/README.md b/collectors/charts.d.plugin/README.md index 092a3f0272..950bc27e12 100644 --- a/collectors/charts.d.plugin/README.md +++ b/collectors/charts.d.plugin/README.md @@ -4,7 +4,7 @@ custom_edit_url: "https://github.com/netdata/netdata/edit/master/collectors/char sidebar_label: "charts.d.plugin" learn_status: "Published" learn_topic_type: "Tasks" -learn_rel_path: "Developers/Collectors" +learn_rel_path: "Developers/External plugins" --> # charts.d.plugin diff --git a/collectors/plugins.d/README.md b/collectors/plugins.d/README.md index 375099b128..cf548c250c 100644 --- a/collectors/plugins.d/README.md +++ b/collectors/plugins.d/README.md @@ -1,13 +1,13 @@ <!-- -title: "External plugins overview" +title: "External plugins" custom_edit_url: "https://github.com/netdata/netdata/edit/master/collectors/plugins.d/README.md" -sidebar_label: "External plugins overview" +sidebar_label: "External plugins" learn_status: "Published" learn_topic_type: "References" learn_rel_path: "Developers" --> -# External plugins overview +# External plugins `plugins.d` is the Netdata internal plugin that collects metrics from external processes, thus allowing Netdata to use **external plugins**. diff --git a/collectors/python.d.plugin/README.md b/collectors/python.d.plugin/README.md index 601a1abc80..62e1bb8b6c 100644 --- a/collectors/python.d.plugin/README.md +++ b/collectors/python.d.plugin/README.md @@ -4,7 +4,7 @@ custom_edit_url: "https://github.com/netdata/netdata/edit/master/collectors/pyth sidebar_label: "python.d.plugin" learn_status: "Published" learn_topic_type: "Tasks" -learn_rel_path: "Developers/Collectors" +learn_rel_path: "Developers/External plugins" --> # python.d.plugin diff --git a/docs/a-github-star-is-important.md b/docs/a-github-star-is-important.md deleted file mode 100644 index 754b317f9a..0000000000 --- a/docs/a-github-star-is-important.md +++ /dev/null @@ -1,28 +0,0 @@ -<!-- -title: "A GitHub star is important" -custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/a-github-star-is-important.md -sidebar_label: "A GitHub star is important" -learn_status: "Published" -learn_topic_type: "References" -learn_rel_path: "Misc" ---> - -# A GitHub star is important - -**GitHub stars** allow Netdata to expand its reach, its community, especially attract people with skills willing to -contribute to it. - -Compared to its first release, Netdata is now **twice as fast**, has all its bugs settled and a lot more functionality. -This happened because a lot of people find it useful, use it daily at home and work, **rely on it** and **contribute to -it**. - -**GitHub stars** also **motivate** us. They state that you find our work **useful**. They give us strength to continue, -to work **harder** to make it even **better**. - -So, give Netdata a **GitHub star**, at the top right of this page. - -Thank you! - -Costa Tsaousis - - diff --git a/docs/collect/enable-configure.md b/docs/collect/enable-configure.md deleted file mode 100644 index cd8960ac1a..0000000000 --- a/docs/collect/enable-configure.md +++ /dev/null @@ -1,72 +0,0 @@ -<!-- -title: "Enable or configure a collector" -description: "Every collector is highly configurable, allowing them to collect metrics from any node and any infrastructure." -custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/collect/enable-configure.md" -sidebar_label: "Enable or configure a collector" -learn_status: "Published" -learn_topic_type: "Tasks" -learn_rel_path: "Setup" ---> - -# Enable or configure a collector - -When Netdata starts up, each collector searches for exposed metrics on the default endpoint established by that service -or application's standard installation procedure. For example, the [Nginx -collector](https://github.com/netdata/go.d.plugin/blob/master/modules/nginx/README.md) searches at -`http://127.0.0.1/stub_status` for exposed metrics in the correct format. If an Nginx web server is running and exposes -metrics on that endpoint, the collector begins gathering them. - -However, not every node or infrastructure uses standard ports, paths, files, or naming conventions. You may need to -enable or configure a collector to gather all available metrics from your systems, containers, or applications. - -## Enable a collector or its orchestrator - -You can enable/disable collectors individually, or enable/disable entire orchestrators, using their configuration files. -For example, you can change the behavior of the Go orchestrator, or any of its collectors, by editing `go.d.conf`. - -Use `edit-config` from your [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) to open -the orchestrator primary configuration file: - -```bash -cd /etc/netdata -sudo ./edit-config go.d.conf -``` - -Within this file, you can either disable the orchestrator entirely (`enabled: yes`), or find a specific collector and -enable/disable it with `yes` and `no` settings. Uncomment any line you change to ensure the Netdata daemon reads it on -start. - -After you make your changes, restart the Agent with `sudo systemctl restart netdata`, or the [appropriate -method](https://github.com/netdata/netdata/blob/master/docs/configure/start-stop-restart.md) for your system. - -## Configure a collector - -First, [find the collector](https://github.com/netdata/netdata/blob/master/collectors/COLLECTORS.md) you want to edit and open its documentation. Some software has -collectors written in multiple languages. In these cases, you should always pick the collector written in Go. - -Use `edit-config` from your [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) to open a -collector's configuration file. For example, edit the Nginx collector with the following: - -```bash -./edit-config go.d/nginx.conf -``` - -Each configuration file describes every available option and offers examples to help you tweak Netdata's settings -according to your needs. In addition, every collector's documentation shows the exact command you need to run to -configure that collector. Uncomment any line you change to ensure the collector's orchestrator or the Netdata daemon -read it on start. - -After you make your changes, restart the Agent with `sudo systemctl restart netdata`, or the [appropriate -method](https://github.com/netdata/netdata/blob/master/docs/configure/start-stop-restart.md) for your system. - -## What's next? - -Read high-level overviews on how Netdata collects [system metrics](https://github.com/netdata/netdata/blob/master/docs/collect/system-metrics.md), [container -metrics](https://github.com/netdata/netdata/blob/master/docs/collect/container-metrics.md), and [application metrics](https://github.com/netdata/netdata/blob/master/docs/collect/application-metrics.md). - -If you're already collecting all metrics from your systems, containers, and applications, it's time to move into -Netdata's visualization features. [See an overview of your infrastructure](https://github.com/netdata/netdata/blob/master/docs/visualize/overview-infrastructure.md) -using Netdata Cloud, or learn how to [interact with dashboards and -charts](https://github.com/netdata/netdata/blob/master/docs/visualize/interact-dashboards-charts.md). - - diff --git a/docs/collect/how-collectors-work.md b/docs/collect/how-collectors-work.md deleted file mode 100644 index 382d4ccc6f..0000000000 --- a/docs/collect/how-collectors-work.md +++ /dev/null @@ -1,82 +0,0 @@ -<!-- -title: "How Netdata's metrics collectors work" -description: "When Netdata starts, and with zero configuration, it auto-detects thousands of data sources and immediately collects per-second metrics." -custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/collect/how-collectors-work.md" -sidebar_label: "How Netdata's metrics collectors work" -learn_status: "Published" -learn_topic_type: "Concepts" -learn_rel_path: "Concepts" ---> - -# How Netdata's metrics collectors work - -When Netdata starts, and with zero configuration, it auto-detects thousands of data sources and immediately collects -per-second metrics. - -Netdata can immediately collect metrics from these endpoints thanks to 300+ **collectors**, which all come pre-installed -when you [install Netdata](https://github.com/netdata/netdata/blob/master/docs/get-started.mdx). - -Every collector has two primary jobs: - -- Look for exposed metrics at a pre- or user-defined endpoint. -- Gather exposed metrics and use additional logic to build meaningful, interactive visualizations. - -If the collector finds compatible metrics exposed on the configured endpoint, it begins a per-second collection job. The -Netdata Agent gathers these metrics, sends them to the [database engine for -storage](https://github.com/netdata/netdata/blob/master/docs/store/change-metrics-storage.md), and immediately [visualizes them -meaningfully](https://github.com/netdata/netdata/blob/master/docs/visualize/interact-dashboards-charts.md) on dashboards. - -Each collector comes with a pre-defined configuration that matches the default setup for that application. This endpoint -can be a URL and port, a socket, a file, a web page, and more. - -For example, the [Nginx collector](https://github.com/netdata/go.d.plugin/blob/master/modules/nginx/README.md) searches -at `http://127.0.0.1/stub_status`, which is the default endpoint for exposing Nginx metrics. The [web log collector for -Nginx or Apache](https://github.com/netdata/go.d.plugin/blob/master/README.mdmodules/weblog) searches at -`/var/log/nginx/access.log` and `/var/log/apache2/access.log`, respectively, both of which are standard locations for -access log files on Linux systems. - -The endpoint is user-configurable, as are many other specifics of what a given collector does. - -## What can Netdata collect? - -To quickly find your answer, see our [list of supported collectors](https://github.com/netdata/netdata/blob/master/collectors/COLLECTORS.md). - -Generally, Netdata's collectors can be grouped into three types: - -- [Systems](https://github.com/netdata/netdata/blob/master/docs/collect/system-metrics.md): Monitor CPU, memory, disk, networking, systemd, eBPF, and much more. - Every metric exposed by `/proc`, `/sys`, and other Linux kernel sources. -- [Containers](https://github.com/netdata/netdata/blob/master/docs/collect/container-metrics.md): Gather metrics from container agents, like `dockerd` or `kubectl`, - along with the resource usage of containers and the applications they run. -- [Applications](https://github.com/netdata/netdata/blob/master/docs/collect/application-metrics.md): Collect per-second metrics from web servers, databases, logs, - message brokers, APM tools, email servers, and much more. - -## Collector architecture and terminology - -**Collector** is a catch-all term for any Netdata process that gathers metrics from an endpoint. - -While we use _collector_ most often in documentation, release notes, and educational content, you may encounter other -terms related to collecting metrics. - -- **Modules** are a type of collector. -- **Orchestrators** are external plugins that run and manage one or more modules. They run as independent processes. - The Go orchestrator is in active development. - - [go.d.plugin](https://github.com/netdata/go.d.plugin/blob/master/README.md): An orchestrator for data - collection modules written in `go`. - - [python.d.plugin](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/README.md): An orchestrator for data collection modules written in - `python` v2/v3. - - [charts.d.plugin](https://github.com/netdata/netdata/blob/master/collectors/charts.d.plugin/README.md): An orchestrator for data collection modules written in - `bash` v4+. -- **External plugins** gather metrics from external processes, such as a webserver or database, and run as independent - processes that communicate with the Netdata daemon via pipes. -- **Internal plugins** gather metrics from `/proc`, `/sys`, and other Linux kernel sources. They are written in `C`, - and run as threads within the Netdata daemon. - -## What's next? - -[Enable or configure a collector](https://github.com/netdata/netdata/blob/master/docs/collect/enable-configure.md) if the default settings are not compatible with -your infrastructure. - -See our [collectors reference](https://github.com/netdata/netdata/blob/master/collectors/REFERENCE.md) for detailed information on Netdata's collector architecture, -troubleshooting a collector, developing a custom collector, and more. - - diff --git a/docs/guides/step-by-step/step-06.md b/docs/guides/step-by-step/step-06.md deleted file mode 100644 index 19f533ce07..0000000000 --- a/docs/guides/step-by-step/step-06.md +++ /dev/null @@ -1,129 +0,0 @@ -<!-- -title: "Step 6. Collect metrics from more services and apps" -sidebar_label: "Step 6. Collect metrics from more services and apps" -custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/guides/step-by-step/step-06.md -learn_status: "Published" -learn_topic_type: "Tasks" -learn_rel_path: "Guides/Step by step" ---> - -# Step 6. Collect metrics from more services and apps - -When Netdata _starts_, it auto-detects dozens of **data sources**, such as database servers, web servers, and more. - -To auto-detect and collect metrics from a source you just installed, you need to restart Netdata using `sudo systemctl -restart netdata`, or the [appropriate method](https://github.com/netdata/netdata/blob/master/docs/configure/start-stop-restart.md) for your system. - -However, auto-detection only works if you installed the source using its standard installation -procedure. If Netdata isn't collecting metrics after a restart, your source probably isn't configured -correctly. - -Check out the [collectors that come pre-installed with Netdata](https://github.com/netdata/netdata/blob/master/collectors/COLLECTORS.md) to find the module for the -source you want to monitor. - -## What you'll learn in this step - -We'll begin with an overview on Netdata's collector architecture, and then dive into the following: - -- [Step 6. Collect metrics from more services and apps](#step-6-collect-metrics-from-more-services-and-apps) - - [What you'll learn in this step](#what-youll-learn-in-this-step) - - [Netdata's collector architecture](#netdatas-collector-architecture) - - [Enable and disable plugins](#enable-and-disable-plugins) - - [Example: Enable the Nginx collector](#example-enable-the-nginx-collector) - - [What's next?](#whats-next) - -## Netdata's collector architecture - -Many Netdata users never have to configure collector or worry about which plugin orchestrator they want to use. - -But, if you want to configure collector or write a collector for your custom source, it's important to understand the -underlying architecture. - -By default, Netdata collects a lot of metrics every second using any number of discrete collector. Collectors, in turn, -are organized and manged by plugins. **Internal** plugins collect system metrics, **external** plugins collect -non-system metrics, and **orchestrator** plugins group individual collectors together based on the programming language -they were built in. - -These modules are primarily written in [Go](https://github.com/netdata/go.d.plugin/blob/master/README.md) (`go.d`) and -[Python](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/README.md), although some use [Bash](https://github.com/netdata/netdata/blob/master/collectors/charts.d.plugin/README.md) -(`charts.d`). - -## Enable and disable plugins - -You don't need to explicitly enable plugins to auto-detect properly configured sources, but it's useful to know how to -enable or disable them. - -One reason you might want to _disable_ plugins is to improve Netdata's performance on low-resource systems, like -ephemeral nodes or edge devices. Disabling orchestrator plugins like `python.d` can save significant resources if you're -not using any of its data collector modules. - -You can enable or disable plugins in the `[plugin]` section of `netdata.conf`. This section features a list of all the -plugins with a boolean setting (`yes` or `no`) to enable or disable them. Be sure to uncomment the line by removing the -hash (`#`)! - -Enabled: - -```conf -[plugins] - # python.d = yes -``` - -Disabled:< |