diff options
Diffstat (limited to 'docs/export')
| -rw-r--r-- | docs/export/README.md | 101 | ||||
| -rw-r--r-- | docs/export/enable-connector.md | 90 | ||||
| -rw-r--r-- | docs/export/external-databases.md | 89 |
3 files changed, 179 insertions, 101 deletions
diff --git a/docs/export/README.md b/docs/export/README.md deleted file mode 100644 index 03d642e1b0..0000000000 --- a/docs/export/README.md +++ /dev/null @@ -1,101 +0,0 @@ -<!-- -title: Export metrics -description: "Archive your Netdata metrics to multiple external time series databases for long-term storage or further analysis." -custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/export/README.md ---> - -# Export metrics - -One of Netdata's pillars is interoperability with other monitoring and visualization solutions. To this end, you can use -the Agent's [exporting engine](/exporting/README.md) to send metrics to multiple external databases/services in -parallel. Once you connect Netdata metrics to other solutions, you can apply machine learning analysis or correlation -with other tools, such as application tracing. - -The exporting engine supports a number of connectors, including AWS Kinesis Data Streams, Graphite, JSON, MongoDB, -OpenTSDB, Prometheus remote write, and more, via exporting **connectors**. These connectors help you seamlessly send -Netdata metrics to more than 20 different endpoints, including every [service that -supports](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage) Prometheus remote write. See -the [exporting reference guide](/exporting/README.md) for the full list. - -## Exporting quickstart - -Let's cover the process of enabling an exporting connector, using the Graphite connector as an example. These steps can -be applied to other connectors as well. - -> If you are migrating from the deprecated backends system, this quickstart will also help you update your configuration -> to the new format. For the most part, the configurations are identical, but there are two exceptions. First, -> `exporting.conf` uses a new `[<type>:<name>]` format for defining connector instances. Second, the `host tags` setting -> is deprecated. Instead, use [host labels](/docs/guides/using-host-labels.md) to tag exported metrics. - -Open the `exporting.conf` file with `edit-config`. - -```bash -cd /etc/netdata # Replace this path with your Netdata config directory -sudo ./edit-config exporting.conf -``` - -### Enable the exporting engine - -Enable the exporting engine by setting `enabled` to `yes`: - -```conf -[exporting:global] - enabled = yes -``` - -### Change how often the exporting engine sends metrics - -By default, the exporting engine only sends metrics to external databases every 10 seconds to avoid congesting the -destination with thousands of per-second metrics. - -You can change this frequency for all connectors based on how you use exported metrics or the resources you can allocate -to long-term storage. Use the `update every` setting to change the frequency in seconds. - -```conf -[exporting:global] - update every = 10 -``` - -### Enable a connector (Graphite) - -To enable the Graphite connector, find the `[graphite:my_graphite_instance]` example section in `exporting.conf`. You -can use this (or the respective example for the connector you want to use) as a framework for your configration. - -`[graphite:my_graphite_instance]` is an example of the new `[<type>:<name>]` format for defining connector instances. - -Uncomment the section itself and replace `my_graphite_instance` with a name of your choice. Then set `enabled` to `yes` -and uncomment the line. - -```conf -[graphite:my_graphite_instance] - enabled = yes - # destination = localhost:2003 - # data source = average - # prefix = netdata - # hostname = my_hostname - # update every = 10 - # buffer on failures = 10 - # timeout ms = 20000 - # send names instead of ids = yes - # send charts matching = * - # send hosts matching = localhost * -``` - -Next, edit and uncomment any other lines necessary to connect the exporting engine to your endpoint. If migrating from -backends, port your settings over and uncomment any lines you change. You must edit the `destination` setting in most -situations. - -For details on all the configuration options, see the [exporting reference](/exporting/README.md#configuration). - -Restart your Agent to begin exporting to the destination of your choice. Because the Agent exports metrics as they're -collected, you should start seeing data in your external database after only a few seconds. - -## Exporting reference, guides, and related features - -- [Exporting reference guide](/exporting/README.md) -- [Guide: Export and visualize Netdata metrics in Graphite](/docs/guides/export/export-netdata-metrics-graphite.md) -- [Guide: Use host labels to organize systems, metrics, and alarms](/docs/guides/using-host-labels.md) -- [Guide: Change how long Netdata stores metrics (long-term storage)](/docs/guides/longer-metrics-storage.md) -- [Backends (deprecated)](/backends/README.md) - -[](<>) diff --git a/docs/export/enable-connector.md b/docs/export/enable-connector.md new file mode 100644 index 0000000000..c287276086 --- /dev/null +++ b/docs/export/enable-connector.md @@ -0,0 +1,90 @@ +<!-- +title: "Enable an exporting connector" +description: "Learn how to enable and configure any connector using examples to start exporting metrics to external time-series databases in minutes." +custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/export/enable-connector.md +--> + +# Enable an exporting connector + +Now that you found the right connector for your [external time-series +database](/docs/export/external-databases.md#supported-databases), you can now enable the exporting engine and the +connector itself. We'll walk through the process of enabling the exporting engine itself, followed by two examples using +the OpenTSDB and Graphite connectors. + +Once you understand the process of enabling a connector, you can translate that knowledge to any other connector. + +## Enable the exporting engine + +Use `edit-config` from your [Netdata config directory](/docs/configure/nodes.md#the-netdata-config-directory) to open +`exporting.conf`: + +```bash +sudo ./edit-config exporting.conf +``` + +Enable the exporting engine itself by setting `enabled` to `yes`: + +```conf +[exporting:global] + enabled = yes +``` + +Save the file but keep it open, as you will edit it again to enable specific connectors. + +## Example: Enable the OpenTSDB connector + +Use the following configuration as a starting point. Copy and paste it into `exporting.conf`. + +```conf +[opentsdb:http:my_opentsdb_http_instance] + enabled = yes + destination = localhost:4242 +``` + +Replace `my_opentsdb_http_instance` with an instance name of your choice, and change the `destination` setting to the IP +address or hostname of your OpenTSDB database. + +Restart your Agent with `service netdata restart` to begin exporting to your OpenTSDB database. Because the +Agent exports metrics as they're collected, you should start seeing data in your external database after only a few +seconds. + +Any further configuration is optional, based on your needs and the configuration of your OpenTSDB database. See the +[OpenTSDB connector doc](/exporting/opentsdb/README.md) and [exporting engine +reference](/exporting/README.md#configuration) for details. + +## Example: Enable the Graphite connector + +Use the following configuration as a starting point. Copy and paste it into `exporting.conf`. + +```conf +[graphite:my_graphite_instance] + enabled = yes + destination = 203.0.113.0:2003 +``` + +Replace `my_graphite_instance` with an instance name of your choice, and change the `destination` setting to the IP +address or hostname of your Graphite-supported database. + +Restart your Agent with `service netdata restart` to begin exporting to your Graphite-supported database. Because the +Agent exports metrics as they're collected, you should start seeing data in your external database after only a few +seconds. + +Any further configuration is optional, based on your needs and the configuration of your Graphite-supported database. +See [exporting engine reference](/exporting/README.md#configuration) for details. + +## What's next? + +If you want to further configure your exporting connectors, see the [exporting engine +reference](/exporting/README.md#configuration). + +For a comprehensive example of using the Graphite connector, read our guide: [_Export and visualize Netdata metrics in +Graphite_](/docs/guides/export/export-netdata-metrics-graphite.md). Or, start [using host +labels](/docs/guides/using-host-labels.md) on exported metrics. + +### Related reference documentation + +- [Exporting engine reference](/exporting/README.md) +- [OpenTSDB connector](/exporting/opentsdb/README.md) +- [Graphite connector](/exporting/graphite/README.md) + +[](<>) diff --git a/docs/export/external-databases.md b/docs/export/external-databases.md new file mode 100644 index 0000000000..e1759f4041 --- /dev/null +++ b/docs/export/external-databases.md @@ -0,0 +1,89 @@ +<!-- +title: "Export metrics to external time-series databases" +description: "Use the exporting engine to send Netdata metrics to popular external time series databases for long-term storage or further analysis." +custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/export/external-databases.md +--> + +# Export metrics to external time-series databases + +Netdata allows you to export metrics to external time-series databases with the [exporting +engine](/exporting/README.md). This system uses a number of **connectors** to intiate connections to [more than +thirty](#supported-databases) supported databases, including InfluxDB, Prometheus, Graphite, ElasticSearch, and much +more. + +The exporting engine resamples Netdata's thousands of per-second metrics at a user-configurable interval, and can export +metrics to multiple time-series databases simultaneously. + +Based on your needs and resources you allocated to your external time-series database, you can configure the interval +that metrics are exported or export only certain charts with filtering. You can also choose whether metrics are exported +as-collected, a normalized average, or the sum/volume of metrics values over the configured interval. + +Exporting is an important part of Netdata's effort to be [interoperable](/docs/overview/netdata-monitoring-stack.md) +with other monitoring software. You can use an external time-series database for long-term metrics retention, further +analysis, or correlation with other tools, such as application tracing. + +## Supported databases + +Netdata supports exporting metrics to the following databases through several +[connectors](/exporting/README.md#features). Once you find the connector that works for your database, open its +documentation and the [enabling a connector](/docs/export/enable-connector.md) doc for details on enabling it. + +- **AppOptics**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **AWS Kinesis**: [AWS Kinesis Data Streams](/exporting/aws_kinesis/README.md) +- **Azure Data Explorer**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **Azure Event Hubs**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **Blueflood**: [Graphite](/exporting/graphite/README.md) +- **Chronix**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **Cortex**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **CrateDB**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **ElasticSearch**: [Graphite](/exporting/graphite/README.md), [Prometheus remote + write](/exporting/prometheus/remote_write/README.md) +- **Gnocchi**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **Google BigQuery**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **Google Cloud Pub/Sub**: [Google Cloud Pub/Sub Service](/exporting/pubsub/README.md) +- **Graphite**: [Graphite](/exporting/graphite/README.md), [Prometheus remote + write](/exporting/prometheus/remote_write/README.md) +- **InfluxDB**: [Graphite](/exporting/graphite/README.md), [Prometheus remote + write](/exporting/prometheus/remote_write/README.md) +- **IRONdb**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **JSON**: [JSON document databases](/exporting/json/README.md) +- **Kafka**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **KairosDB**: [Graphite](/exporting/graphite/README.md), [OpenTSDB](/exporting/opentsdb/README.md) +- **M3DB**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **MetricFire**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **MongoDB**: [MongoDB](/exporting/mongodb/) +- **New Relic**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **OpenTSDB**: [OpenTSDB](/exporting/opentsdb/README.md), [Prometheus remote + write](/exporting/prometheus/remote_write/README.md) +- **PostgreSQL**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **Prometheus**: [Prometheus scraper](/exporting/prometheus/README.md) +- **TimescaleDB**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md), + [netdata-timescale-relay](/exporting/TIMESCALE.md) +- **QuasarDB**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **SignalFx**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **Splunk**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **TiKV**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **Thanos**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **VictoriaMetrics**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) +- **Wavefront**: [Prometheus remote write](/exporting/prometheus/remote_write/README.md) + +Can't find your preferred external time-series database? Ask our [community](https://community.netdata.cloud/) for +solutions, or file an [issue on +GitHub](https://github.com/netdata/netdata/issues/new?labels=bug%2C+needs+triage&template=bug_report.md). + +## What's next? + +We recommend you read our document on [enabling a connector](/docs/export/enable-connector.md) to learn about the +process and discover important configuration options. If you would rather skip ahead, click on any of the above links to +connectors for their reference documentation, which outline any prerequisites to install for that connector, along with +connector-specific configuration options. + +Read about one possible use case for exporting metrics in our guide: [_Export and visualize Netdata metrics in +Graphite_](/docs/guides/export/export-netdata-metrics-graphite.md). + +### Related reference documentation + +- [Exporting engine reference](/exporting/README.md) +- [Backends reference (deprecated)](/backends/README.md) + +[](<>) |