summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorJoel Hans <joel@netdata.cloud>2020-06-02 11:08:39 -0700
committerGitHub <noreply@github.com>2020-06-02 11:08:39 -0700
commitf3d762437a01d5f7d2d5829b9120a20ccb69ff97 (patch)
tree725a150037f934a20262d1c02ab39881e26ab3aa
parent57c916530c7ce9b7ca7905bfc06d676d34dd2f76 (diff)
Add new exporting "home base" document (#9246)
* Init new export doc * Touch of cleanup * A few additional fixes * Fix broken links * Fix for Thiago * Fixes for Vlad
-rw-r--r--backends/README.md3
-rw-r--r--docs/export/README.md101
-rw-r--r--exporting/README.md45
-rw-r--r--exporting/mongodb/README.md8
4 files changed, 133 insertions, 24 deletions
diff --git a/backends/README.md b/backends/README.md
index 10571bbb3f..4c8f897558 100644
--- a/backends/README.md
+++ b/backends/README.md
@@ -7,6 +7,9 @@ custom_edit_url: https://github.com/netdata/netdata/edit/master/backends/README.
# Metrics long term archiving
+> ⚠️ The backends system is now deprecated in favor of the [exporting engine](/exporting/README.md). Please see the
+> [migration guide](/docs/export/) for details on how to get started with exporting.
+
Netdata supports backends for archiving the metrics, or providing long term dashboards, using Grafana or other tools,
like this:
diff --git a/docs/export/README.md b/docs/export/README.md
new file mode 100644
index 0000000000..9702174865
--- /dev/null
+++ b/docs/export/README.md
@@ -0,0 +1,101 @@
+<!--
+title: Export metrics
+description: "Archive your Netdata metrics to multiple external time series databases for long-term storage or further analysis."
+custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/export/README.md
+-->
+
+# Export metrics
+
+One of Netdata's pillars is interoperability with other monitoring and visualization solutions. To this end, you can use
+the Agent's [exporting engine](/exporting/README.md) to send metrics to multiple external databases/services in
+parallel. Once you connect Netdata metrics to other solutions, you can apply machine learning analysis or correlation
+with other tools, such as application tracing.
+
+The exporting engine supports a number of connectors, including AWS Kinesis Data Streams, Graphite, JSON, MongoDB,
+OpenTSDB, Prometheus remote write, and more, via exporting **connectors**. These connectors help you seamlessly send
+Netdata metrics to more than 20 different endpoints, including every [service that
+supports](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage) Prometheus remote write. See
+the [exporting reference guide](/exporting/README.md) for the full list.
+
+## Exporting quickstart
+
+Let's cover the process of enabling an exporting connector, using the Graphite connector as an example. These steps can
+be applied to other connectors as well.
+
+> If you are migrating from the deprecated backends system, this quickstart will also help you update your configuration
+> to the new format. For the most part, the configurations are identical, but there are two exceptions. First,
+> `exporting.conf` uses a new `[<type>:<name>]` format for defining connector instances. Second, the `host tags` setting
+> is deprecated. Instead, use [host labels](/docs/tutorials/using-host-labels.md) to tag exported metrics.
+
+Open the `exporting.conf` file with `edit-config`.
+
+```bash
+cd /etc/netdata # Replace this path with your Netdata config directory
+sudo ./edit-config exporting.conf
+```
+
+### Enable the exporting engine
+
+Enable the exporting engine by setting `enabled` to `yes`:
+
+```conf
+[exporting:global]
+ enabled = yes
+```
+
+### Change how often the exporting engine sends metrics
+
+By default, the exporting engine only sends metrics to external databases every 10 seconds to avoid congesting the
+destination with thousands of per-second metrics.
+
+You can change this frequency for all connectors based on how you use exported metrics or the resources you can allocate
+to long-term storage. Use the `update every` setting to change the frequency in seconds.
+
+```conf
+[exporting:global]
+ update every = 10
+```
+
+### Enable a connector (Graphite)
+
+To enable the Graphite connector, find the `[graphite:my_graphite_instance]` example section in `exporting.conf`. You
+can use this (or the respective example for the connector you want to use) as a framework for your configration.
+
+`[graphite:my_graphite_instance]` is an example of the new `[<type>:<name>]` format for defining connector instances.
+
+Uncomment the section itself and replace `my_graphite_instance` with a name of your choice. Then set `enabled` to `yes`
+and uncomment the line.
+
+```conf
+[graphite:my_graphite_instance]
+ enabled = yes
+ # destination = localhost:2003
+ # data source = average
+ # prefix = netdata
+ # hostname = my_hostname
+ # update every = 10
+ # buffer on failures = 10
+ # timeout ms = 20000
+ # send names instead of ids = yes
+ # send charts matching = *
+ # send hosts matching = localhost *
+```
+
+Next, edit and uncomment any other lines necessary to connect the exporting engine to your endpoint. If migrating from
+backends, port your settings over and uncomment any lines you change. You must edit the `destination` setting in most
+situations.
+
+For details on all the configuration options, see the [exporting reference](/exporting/README.md#configuration).
+
+Restart your Agent to begin exporting to the destination of your choice. Because the Agent exports metrics as they're
+collected, you should start seeing data in your external database after only a few seconds.
+
+## Exporting reference and related features
+
+- [Exporting reference guide](/exporting/README.md)
+- [Backends (deprecated)](/backends/README.md)
+- [Use host labels to organize systems, metrics, and alarms](/docs/tutorials/using-host-labels.md)
+- [Database engine](/database/engine/README.md)
+- [Change how long Netdata stores metrics (long-term storage)](/docs/tutorials/longer-metrics-storage.md)
+
+[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fdocs%2Fexporting%2FREADME.md&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>)
diff --git a/exporting/README.md b/exporting/README.md
index 6802011039..ba2c51c357 100644
--- a/exporting/README.md
+++ b/exporting/README.md
@@ -1,25 +1,29 @@
<!--
-title: "Export metrics to external databases"
+title: "Exporting engine reference"
description: "With the exporting engine, you can archive your Netdata metrics to multiple external databases for long-term storage or further analysis."
+sidebar_label: Reference guide
custom_edit_url: https://github.com/netdata/netdata/edit/master/exporting/README.md
-->
-# Exporting metrics to external databases
+# Exporting engine reference
-The exporting engine is an update for the former [backends](/backends/README.md) which is deprecated and will be deleted
-soon. It has a modular structure and supports metric exporting via multiple exporting connector instances at the same
-time. You can have different update intervals and filters configured for every exporting connector instance. The
-exporting engine has its own configuration file `exporting.conf`. Configuration is almost similar to
-[backends](/backends/README.md#configuration). The most important difference is that type of a connector should be
-specified in a section name before a colon and an instance name after the colon. Also, you can't use `host tags`
-anymore. Set your labels using the [`[host labels]`](/docs/tutorials/using-host-labels.md) section in `netdata.conf`.
+Welcome to the exporting engine reference guide.
+
+This guide contains comprehensive information about enabling, configuring, and monitoring Netdata's exporting engine,
+which allows you to send metrics to more than 20 external time series databases.
-# Metrics long term archiving
+To learn the basics of locating and editing health configuration files, read up on [how to export
+metrics](/docs/export/README.md), and follow the [exporting
+quickstart](/docs/export/README.md#exporting-quickstart).
-Netdata supports external databases and services for archiving the metrics, or providing long term dashboards, using
-Grafana or other tools, like this:
+The exporting engine is an update for the former [backends](/backends/README.md), which is deprecated and will be
+deleted soon. It has a modular structure and supports metric exporting via multiple exporting connector instances at the
+same time. You can have different update intervals and filters configured for every exporting connector instance.
-![image](https://cloud.githubusercontent.com/assets/2662304/20649711/29f182ba-b4ce-11e6-97c8-ab2c0ab59833.png)
+The exporting engine has its own configuration file `exporting.conf`. Configuration is almost similar to
+[backends](/backends/README.md#configuration). The most important difference is that type of a connector should be
+specified in a section name before a colon and an instance name after the colon. Also, you can't use `host tags`
+anymore. Set your labels using the [`[host labels]`](/docs/tutorials/using-host-labels.md) section in `netdata.conf`.
Since Netdata collects thousands of metrics per server per second, which would easily congest any database server when
several Netdata servers are sending data to it, Netdata allows sending metrics at a lower frequency, by resampling them.
@@ -27,7 +31,7 @@ several Netdata servers are sending data to it, Netdata allows sending metrics a
So, although Netdata collects metrics every second, it can send to the external database servers averages or sums every
X seconds (though, it can send them per second if you need it to).
-## features
+## Features
1. Supported databases and services
@@ -104,7 +108,7 @@ X seconds (though, it can send them per second if you need it to).
their batches at the same time. You can set different update intervals for every exporting connector instance, but
even in that case they can occasionally synchronize their batches for a moment.
-## configuration
+## Configuration
In `/etc/netdata/exporting.conf` you should have something like this:
@@ -252,9 +256,10 @@ Options:
> You can check how the host tags were parsed using the /api/v1/info API call. But, keep in mind that backends subsystem
> is deprecated and will be deleted soon. Please move your existing tags to the `[host labels]` section.
-## monitoring operation
+## Exporting engine monitoring
-Netdata provides 5 charts:
+Netdata creates five charts in the dashboard, under the **Netdata Monitoring** section, to help you monitor the health
+and performance of the exporting engine itself:
1. **Buffered metrics**, the number of metrics Netdata added to the buffer for dispatching them to the
external database server.
@@ -263,12 +268,12 @@ Netdata provides 5 charts:
3. **Exporting operations**, the number of operations performed by Netdata.
-4. **Exporting thread CPU usage**, the CPU resources consumed by the Netdata thread, that is responsible for sending the
- metrics to the external database server.
+4. **Exporting thread CPU usage**, the CPU resources consumed by the Netdata thread, that is responsible for sending
+ the metrics to the external database server.
![image](https://cloud.githubusercontent.com/assets/2662304/20463536/eb196084-af3d-11e6-8ee5-ddbd3b4d8449.png)
-## alarms
+## Exporting engine alarms
Netdata adds 3 alarms:
diff --git a/exporting/mongodb/README.md b/exporting/mongodb/README.md
index 6f8d0dc327..2934f38c52 100644
--- a/exporting/mongodb/README.md
+++ b/exporting/mongodb/README.md
@@ -7,8 +7,8 @@ sidebar_label: MongoDB
# Export metrics to MongoDB
-You can use the MongoDB connector and the experimental [exporting engine](../README.md) to archive your agent's metrics
-to a MongoDB database for long-term storage, further analysis, or correlation with data from other sources.
+You can use the MongoDB connector for the [exporting engine](/exporting/README.md) to archive your agent's metrics to a
+MongoDB database for long-term storage, further analysis, or correlation with data from other sources.
## Prerequisites
@@ -18,8 +18,8 @@ from the source, which detects that the required library is now available.
## Configuration
-To enable data exporting to a MongoDB database, run `./edit-config exporting.conf`
-in the Netdata configuration directory and set the following options:
+To enable data exporting to a MongoDB database, run `./edit-config exporting.conf` in the Netdata configuration
+directory and set the following options:
```conf
[mongodb:my_instance]