diff options
author | vkalintiris <vasilis@netdata.cloud> | 2024-02-05 11:16:59 +0200 |
---|---|---|
committer | GitHub <noreply@github.com> | 2024-02-05 11:16:59 +0200 |
commit | 8121b18d1e59c1a28fb54844a01677fe9f5d29e2 (patch) | |
tree | 2bd25d8ddebb5838823dabf912810259b7fddb7f /exporting | |
parent | c5579a186f22aba0edd7da844c6186c688625e3c (diff) |
Move exporting/ under src/ (#16913)
Diffstat (limited to 'exporting')
86 files changed, 0 insertions, 14730 deletions
diff --git a/exporting/README.md b/exporting/README.md deleted file mode 100644 index 8a52968eed..0000000000 --- a/exporting/README.md +++ /dev/null @@ -1,327 +0,0 @@ -<!-- -title: "Exporting reference" -description: "With the exporting engine, you can archive your Netdata metrics to multiple external databases for long-term storage or further analysis." -sidebar_label: "Export" -custom_edit_url: "https://github.com/netdata/netdata/edit/master/exporting/README.md" -learn_status: "Published" -learn_rel_path: "Integrations/Export" -learn_doc_purpose: "Explain the exporting engine options and all of our the exporting connectors options" ---> - -# Exporting reference - -Welcome to the exporting engine reference guide. This guide contains comprehensive information about enabling, -configuring, and monitoring Netdata's exporting engine, which allows you to send metrics to external time-series -databases. - -For a quick introduction to the exporting engine's features, read our doc on [exporting metrics to time-series -databases](https://github.com/netdata/netdata/blob/master/docs/export/external-databases.md), or jump in to [enabling a connector](https://github.com/netdata/netdata/blob/master/docs/export/enable-connector.md). - -The exporting engine has a modular structure and supports metric exporting via multiple exporting connector instances at -the same time. You can have different update intervals and filters configured for every exporting connector instance. - -When you enable the exporting engine and a connector, the Netdata Agent exports metrics _beginning from the time you -restart its process_, not the entire [database of long-term metrics](https://github.com/netdata/netdata/blob/master/docs/store/change-metrics-storage.md). - -Since Netdata collects thousands of metrics per server per second, which would easily congest any database server when -several Netdata servers are sending data to it, Netdata allows sending metrics at a lower frequency, by resampling them. - -So, although Netdata collects metrics every second, it can send to the external database servers averages or sums every -X seconds (though, it can send them per second if you need it to). - -## Features - -### Integration - -The exporting engine uses a number of connectors to send Netdata metrics to external time-series databases. See our -[list of supported databases](https://github.com/netdata/netdata/blob/master/docs/export/external-databases.md#supported-databases) for information on which -connector to enable and configure for your database of choice. - -- [**AWS Kinesis Data Streams**](https://github.com/netdata/netdata/blob/master/exporting/aws_kinesis/README.md): Metrics are sent to the service in `JSON` - format. -- [**Google Cloud Pub/Sub Service**](https://github.com/netdata/netdata/blob/master/exporting/pubsub/README.md): Metrics are sent to the service in `JSON` - format. -- [**Graphite**](https://github.com/netdata/netdata/blob/master/exporting/graphite/README.md): A plaintext interface. Metrics are sent to the database server as - `prefix.hostname.chart.dimension`. `prefix` is configured below, `hostname` is the hostname of the machine (can - also be configured). Learn more in our guide to [export and visualize Netdata metrics in - Graphite](https://github.com/netdata/netdata/blob/master/exporting/graphite/README.md). -- [**JSON** document databases](https://github.com/netdata/netdata/blob/master/exporting/json/README.md) -- [**OpenTSDB**](https://github.com/netdata/netdata/blob/master/exporting/opentsdb/README.md): Use a plaintext or HTTP interfaces. Metrics are sent to - OpenTSDB as `prefix.chart.dimension` with tag `host=hostname`. -- [**MongoDB**](https://github.com/netdata/netdata/blob/master/exporting/mongodb/README.md): Metrics are sent to the database in `JSON` format. -- [**Prometheus**](https://github.com/netdata/netdata/blob/master/exporting/prometheus/README.md): Use an existing Prometheus installation to scrape metrics - from node using the Netdata API. -- [**Prometheus remote write**](https://github.com/netdata/netdata/blob/master/exporting/prometheus/remote_write/README.md). A binary snappy-compressed protocol - buffer encoding over HTTP. Supports many [storage - providers](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage). -- [**TimescaleDB**](https://github.com/netdata/netdata/blob/master/exporting/TIMESCALE.md): Use a community-built connector that takes JSON streams from a - Netdata client and writes them to a TimescaleDB table. - -### Chart filtering - -Netdata can filter metrics, to send only a subset of the collected metrics. You can use the -configuration file - -```txt -[prometheus:exporter] - send charts matching = system.* -``` - -or the URL parameter `filter` in the `allmetrics` API call. - -```txt -http://localhost:19999/api/v1/allmetrics?format=shell&filter=system.* -``` - -### Operation modes - -Netdata supports three modes of operation for all exporting connectors: - -- `as-collected` sends to external databases the metrics as they are collected, in the units they are collected. - So, counters are sent as counters and gauges are sent as gauges, much like all data collectors do. For example, - to calculate CPU utilization in this format, you need to know how to convert kernel ticks to percentage. - -- `average` sends to external databases normalized metrics from the Netdata database. In this mode, all metrics - are sent as gauges, in the units Netdata uses. This abstracts data collection and simplifies visualization, but - you will not be able to copy and paste queries from other sources to convert units. For example, CPU utilization - percentage is calculated by Netdata, so Netdata will convert ticks to percentage and send the average percentage - to the external database. - -- `sum` or `volume`: the sum of the interpolated values shown on the Netdata graphs is sent to the external - database. So, if Netdata is configured to send data to the database every 10 seconds, the sum of the 10 values - shown on the Netdata charts will be used. - -Time-series databases suggest to collect the raw values (`as-collected`). If you plan to invest on building your -monitoring around a time-series database and you already know (or you will invest in learning) how to convert units -and normalize the metrics in Grafana or other visualization tools, we suggest to use `as-collected`. - -If, on the other hand, you just need long term archiving of Netdata metrics and you plan to mainly work with -Netdata, we suggest to use `average`. It decouples visualization from data collection, so it will generally be a lot -simpler. Furthermore, if you use `average`, the charts shown in the external service will match exactly what you -see in Netdata, which is not necessarily true for the other modes of operation. - -### Independent operation - -This code is smart enough, not to slow down Netdata, independently of the speed of the external database server. - -> ❗ You should keep in mind though that many exporting connector instances can consume a lot of CPU resources if they -> run their batches at the same time. You can set different update intervals for every exporting connector instance, -> but even in that case they can occasionally synchronize their batches for a moment. - -## Configuration - -Here are the configuration blocks for every supported connector. Your current `exporting.conf` file may look a little -different. - -You can configure each connector individually using the available [options](#options). The -`[graphite:my_graphite_instance]` block contains examples of some of these additional options in action. - -```conf -[exporting:global] - enabled = yes - send configured labels = no - send automatic labels = no - update every = 10 - -[prometheus:exporter] - send names instead of ids = yes - send configured labels = yes - send automatic labels = no - send charts matching = * - send hosts matching = localhost * - prefix = netdata - -[graphite:my_graphite_instance] - enabled = yes - destination = localhost:2003 - data source = average - prefix = Netdata - hostname = my-name - update every = 10 - buffer on failures = 10 - timeout ms = 20000 - send charts matching = * - send hosts matching = localhost * - send names instead of ids = yes - send configured labels = yes - send automatic labels = yes - -[prometheus_remote_write:my_prometheus_remote_write_instance] - enabled = yes - destination = localhost - remote write URL path = /receive - -[kinesis:my_kinesis_instance] - enabled = yes - destination = us-east-1 - stream name = netdata - aws_access_key_id = my_access_key_id - aws_secret_access_key = my_aws_secret_access_key - -[pubsub:my_pubsub_instance] - enabled = yes - destination = pubsub.googleapis.com - credentials file = /etc/netdata/pubsub_credentials.json - project id = my_project - topic id = my_topic - -[mongodb:my_mongodb_instance] - enabled = yes - destination = localhost - database = my_database - collection = my_collection - -[json:my_json_instance] - enabled = yes - destination = localhost:5448 - -[opentsdb:my_opentsdb_plaintext_instance] - enabled = yes - destination = localhost:4242 - -[opentsdb:http:my_opentsdb_http_instance] - enabled = yes - destination = localhost:4242 - username = my_username - password = my_password - -[opentsdb:https:my_opentsdb_https_instance] - enabled = yes - destination = localhost:8082 -``` - -### Sections - -- `[exporting:global]` is a section where you can set your defaults for all exporting connectors -- `[prometheus:exporter]` defines settings for Prometheus exporter API queries (e.g.: - `http://NODE:19999/api/v1/allmetrics?format=prometheus&help=yes&source=as-collected`). -- `[<type>:<name>]` keeps settings for a particular exporting connector instance, where: - - `type` selects the exporting connector type: graphite | opentsdb:telnet | opentsdb:http | - prometheus_remote_write | json | kinesis | pubsub | mongodb. For graphite, opentsdb, - json, and prometheus_remote_write connectors you can also use `:http` or `:https` modifiers - (e.g.: `opentsdb:https`). - - `name` can be arbitrary instance name you chose. - -### Options - -Configure individual connectors and override any global settings with the following options. - -- `enabled = yes | no`, enables or disables an exporting connector instance - -- `destination = host1 host2 host3 ...`, accepts **a space separated list** of hostnames, IPs (IPv4 and IPv6) and - ports to connect to. Netdata will use the **first available** to send the metrics. - - The format of each item in this list, is: `[PROTOCOL:]IP[:PORT]`. - - `PROTOCOL` can be `udp` or `tcp`. `tcp` is the default and only supported by the current exporting engine. - - `IP` can be `XX.XX.XX.XX` (IPv4), or `[XX:XX...XX:XX]` |