Update the exporting documentation (#9066)

9 files changed, 744 insertions, 37 deletions

diff --git a/exporting/Makefile.am b/exporting/Makefile.am
index e982c8b15f..ece9ebb71f 100644
--- a/exporting/Makefile.am
+++ b/exporting/Makefile.am
@@ -14,6 +14,12 @@ SUBDIRS = \
     mongodb \
     $(NULL)
 
+dist_libconfig_DATA = \
+    exporting.conf \
+    $(NULL)
+
 dist_noinst_DATA = \
     README.md \
+    TIMESCALE.md \
+    WALKTHROUGH.md \
     $(NULL)
diff --git a/exporting/README.md b/exporting/README.md
index bc4003aa69..b37387277f 100644
--- a/exporting/README.md
+++ b/exporting/README.md
@@ -4,48 +4,278 @@ description: "With the exporting engine, you can archive your Netdata metrics to
 custom_edit_url: https://github.com/netdata/netdata/edit/master/exporting/README.md
 -->
 
-# Exporting metrics to external databases (experimental)
+# Exporting metrics to external databases
 
-The exporting engine is an update for the former [backends](/backends/README.md). It's still work in progress. It has a
-modular structure and supports metric exporting via multiple exporting connector instances at the same time. You can
-have different update intervals and filters configured for every exporting connector instance. The exporting engine has
-its own configuration file `exporting.conf`. Configuration is almost similar to [backends](/backends/README.md#configuration).
-The only difference is that the type of a connector should be specified in a section name before a colon and a name after
-the colon. At the moment only four types of connectors are supported: `graphite`, `json`, `opentsdb`, `opentsdb:http`.
+The exporting engine is an update for the former [backends](/backends/README.md) which is deprecated and will be deleted
+soon. It has a modular structure and supports metric exporting via multiple exporting connector instances at the same
+time. You can have different update intervals and filters configured for every exporting connector instance. The
+exporting engine has its own configuration file `exporting.conf`. Configuration is almost similar to
+[backends](/backends/README.md#configuration). The most important difference is that type of a connector should be
+specified in a section name before a colon and an instance name after the colon. Also, you can't use `host tags`
+anymore. Set your labels using the [`[host labels]`](/docs/tutorials/using-host-labels.md) section in `netdata.conf`.
+
+# Metrics long term archiving
+
+Netdata supports external databases and services for archiving the metrics, or providing long term dashboards, using
+Grafana or other tools, like this:
+
+![image](https://cloud.githubusercontent.com/assets/2662304/20649711/29f182ba-b4ce-11e6-97c8-ab2c0ab59833.png)
+
+Since Netdata collects thousands of metrics per server per second, which would easily congest any database server when
+several Netdata servers are sending data to it, Netdata allows sending metrics at a lower frequency, by resampling them.
+
+So, although Netdata collects metrics every second, it can send to the external database servers averages or sums every
+X seconds (though, it can send them per second if you need it to).
+
+## features
+
+1.  Supported databases and services
+
+    -   **graphite** (`plaintext interface`, used by **Graphite**, **InfluxDB**, **KairosDB**, **Blueflood**,
+        **ElasticSearch** via logstash tcp input and the graphite codec, etc)
+
+        metrics are sent to the database server as `prefix.hostname.chart.dimension`. `prefix` is configured below,
+        `hostname` is the hostname of the machine (can also be configured).
+
+    -   **opentsdb** (`telnet or HTTP interfaces`, used by **OpenTSDB**, **InfluxDB**, **KairosDB**, etc)
+
+        metrics are sent to OpenTSDB as `prefix.chart.dimension` with tag `host=hostname`.
+
+    -   **json** document DBs
+
+        metrics are sent to a document DB, `JSON` formatted.
+
+    -   **prometheus** is described at [prometheus page](/exporting/prometheus/README.md) since it pulls data from
+        Netdata.
+
+    -   **prometheus remote write** (a binary snappy-compressed protocol buffer encoding over HTTP used by
+        **Elasticsearch**, **Gnocchi**, **Graphite**, **InfluxDB**, **Kafka**, **OpenTSDB**, **PostgreSQL/TimescaleDB**,
+        **Splunk**, **VictoriaMetrics**, and a lot of other [storage
+        providers](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage))
+
+        metrics are labeled in the format, which is used by Netdata for the [plaintext prometheus
+        protocol](/exporting/prometheus/README.md). Notes on using the remote write connector are
+        [here](/exporting/prometheus/remote_write/README.md).
+
+    -   **TimescaleDB** via [community-built connector](/exporting/TIMESCALE.md) that takes JSON streams from a Netdata
+        client and writes them to a TimescaleDB table.
+
+    -   **AWS Kinesis Data Streams**
+
+        metrics are sent to the service in `JSON` format.
+
+    -   **Google Cloud Pub/Sub Service**
+
+        metrics are sent to the service in `JSON` format.
+
+    -   **MongoDB**
+
+        metrics are sent to the database in `JSON` format.
+
+2.  Netdata can filter metrics (at the chart level), to send only a subset of the collected metrics.
+
+3.  Netdata supports three modes of operation for all exporting connectors:
+
+    -   `as-collected` sends to external databases the metrics as they are collected, in the units they are collected.
+        So, counters are sent as counters and gauges are sent as gauges, much like all data collectors do. For example,
+        to calculate CPU utilization in this format, you need to know how to convert kernel ticks to percentage.
+
+    -   `average` sends to external databases normalized metrics from the Netdata database. In this mode, all metrics
+        are sent as gauges, in the units Netdata uses. This abstracts data collection and simplifies visualization, but
+        you will not be able to copy and paste queries from other sources to convert units. For example, CPU utilization
+        percentage is calculated by Netdata, so Netdata will convert ticks to percentage and send the average percentage
+        to the external database.
+
+    -   `sum` or `volume`: the sum of the interpolated values shown on the Netdata graphs is sent to the external
+        database. So, if Netdata is configured to send data to the database every 10 seconds, the sum of the 10 values
+        shown on the Netdata charts will be used.
+
+    Time-series databases suggest to collect the raw values (`as-collected`). If you plan to invest on building your
+    monitoring around a time-series database and you already know (or you will invest in learning) how to convert units
+    and normalize the metrics in Grafana or other visualization tools, we suggest to use `as-collected`.
+
+    If, on the other hand, you just need long term archiving of Netdata metrics and you plan to mainly work with
+    Netdata, we suggest to use `average`. It decouples visualization from data collection, so it will generally be a lot
+    simpler. Furthermore, if you use `average`, the charts shown in the external service will match exactly what you
+    see in Netdata, which is not necessarily true for the other modes of operation.
+
+4.  This code is smart enough, not to slow down Netdata, independently of the speed of the external database server. You
+    should keep in mind though that many exporting connector instances can consume a lot of CPU resources if they run
+    their batches at the same time. You can set different update intervals for every exporting connector instance, but
+    even in that case they can occasionally synchronize their batches for a moment.
+
+## configuration
+
+In `/etc/netdata/exporting.conf` you should have something like this:
 
-An example configuration:
 ```conf
 [exporting:global]
-enabled = yes
+    enabled = yes
+    send configured labels = no
+    send automatic labels = no
+    update every = 10
+
+[prometheus:exporter]
+    send charts matching = system.processes
 
-[graphite:my_instance1]
-enabled = yes
-destination = localhost:2003
-data source = sum
-update every = 5
-send charts matching = system.load
+[graphite:my_instance_1]
+    enabled = yes
+    destination = localhost:2003
+    data source = average
+    prefix = Netdata
+    hostname = my-name
+    update every = 10
+    buffer on failures = 10
+    timeout ms = 20000
+    send charts matching = *
+    send hosts matching = localhost *
+    send names instead of ids = yes
 
 [json:my_instance2]
-enabled = yes
-destination = localhost:5448
-data source = as collected
-update every = 2
-send charts matching = system.active_processes
+    enabled = yes
+    destination = localhost:5448
+    data source = as collected
+    update every = 2
+    send charts matching = system.active_processes
 
 [opentsdb:my_instance3]
-enabled = yes
-destination = localhost:4242
-data source = sum
-update every = 10
-send charts matching = system.cpu
+    enabled = yes
+    destination = localhost:4242
+    data source = sum
+    update every = 10
+    send charts matching = system.cpu
 
 [opentsdb:http:my_instance4]
-enabled = yes
-destination = localhost:4243
-data source = average
-update every = 3
-send charts matching = system.active_processes
+    enabled = yes
+    destination = localhost:4243
+    data source = average
+    update every = 3
+    send charts matching = system.active_processes
+```
+
+Sections:
+- `[exporting:global]` is a section where you can set your defaults for all exporting connectors
+- `[prometheus:exporter]` defines settings for Prometheus exporter API queries (e.g.:
+  `http://your.netdata.ip:19999/api/v1/allmetrics?format=prometheus&help=yes&source=as-collected`).
+- `[<type>:<name>]` keeps settings for a particular exporting connector instance, where:
+  - `type` selects the exporting connector type: graphite | opentsdb:telnet | opentsdb:http | opentsdb:https |
+    prometheus_remote_write | json | kinesis | pubsub | mongodb
+  - `name` can be arbitrary instance name you chose.
+
+Options:
+-   `enabled = yes | no`, enables or disables an exporting connector instance
+
+-   `destination = host1 host2 host3 ...`, accepts **a space separated list** of hostnames, IPs (IPv4 and IPv6) and
+     ports to connect to. Netdata will use the **first available** to send the metrics.
 
+     The format of each item in this list, is: `[PROTOCOL:]IP[:PORT]`.
+
+     `PROTOCOL` can be `udp` or `tcp`. `tcp` is the default and only supported by the current exporting engine.
+
+     `IP` can be `XX.XX.XX.XX` (IPv4), or `[XX:XX...XX:XX]` (IPv6). For IPv6 you can to enclose the IP in `[]` to
+     separate it from the port.
+
+     `PORT` can be a number of a service name. If omitted, the default port for the exporting connector will be used
+     (graphite = 2003, opentsdb = 4242).
+
+     Example IPv4:
+
+```conf
+   destination = 10.11.14.2:4242 10.11.14.3:4242 10.11.14.4:4242
+```
+
+   Example IPv6 and IPv4 together:
+
+```conf
+   destination = [ffff:...:0001]:2003 10.11.12.1:2003
 ```
 
+   When multiple servers are defined, Netdata will try the next one when the previous one fails.
+
+   Netdata also ships `nc-exporting.sh`, a script that can be used as a fallback exporting connector to save the
+   metrics to disk and push them to the time-series database when it becomes available again. It can also be used to
+   monitor / trace / debug the metrics Netdata generates.
+
+   For the Kinesis exporting connector `destination` should be set to an AWS region (for example, `us-east-1`).
+
+   For the MongoDB exporting connector `destination` should be set to a
+   [MongoDB URI](https://docs.mongodb.com/manual/reference/connection-string/).
+
+   For the Pub/Sub exporting connector `destination` can be set to a specific service endpoint.
+
+-   `data source = as collected`, or `data source = average`, or `data source = sum`, selects the kind of data that will
+     be sent to the external database.
+
+-   `hostname = my-name`, is the hostname to be used for sending data to the external database server. By default this
+    is `[global].hostname`.
+
+-   `prefix = Netdata`, is the prefix to add to all metrics.
+
+-   `update every = 10`, is the number of seconds between sending data to the external datanase. Netdata will add some
+    randomness to this number, to prevent stressing the external server when many Netdata servers send data to the same
+    database. This randomness does not affect the quality of the data, only the time they are sent.
+
+-   `buffer on failures = 10`, is the number of iterations (each iteration is `update every` seconds) to buffer data,
+    when the external database server is not available. If the server fails to receive the data after that many
+    failures, data loss on the connector instance is expected (Netdata will also log it).
+
+-   `timeout ms = 20000`, is the timeout in milliseconds to wait for the external database server to process the data.
+    By default this is `2 * update_every * 1000`.
+
+-   `send hosts matching = localhost *` includes one or more space separated patterns, using `*` as wildcard (any number
+    of times within each pattern). The patterns are checked against the hostname (the localhost is always checked as
+    `localhost`), allowing us to filter which hosts will be sent to the external database when this Netdata is a central
+    Netdata aggregating multiple hosts. A pattern starting with `!` gives a negative match. So to match all hosts named
+    `*db*` except hosts containing `*slave*`, use `!*slave* *db*` (so, the order is important: the first pattern
+    matching the hostname will be used - positive or negative).
+
+-   `send charts matching = *` includes one or more space separated patterns, using `*` as wildcard (any number of times
+    within each pattern). The patterns are checked against both chart id and chart name. A pattern starting with `!`
+    gives a negative match. So to match all charts named `apps.*` except charts ending in `*reads`, use `!*reads
+    apps.*` (so, the order is important: the first pattern matching the chart id or the chart name will be used -
+    positive or negative).
+
+-   `send names instead of ids = yes | no` controls the metric names Netdata should send to the external database.
+    Netdata supports names and IDs for charts and dimensions. Usually IDs are unique identifiers as read by the system
+    and names are human friendly labels (also unique). Most charts and metrics have the same ID and name, but in several
+    cases they are different: disks with device-mapper, interrupts, QoS classes, statsd synthetic charts, etc.
+
+-   `send configured labels = yes | no` controls if labels defined in the `[host labels]` section in `netdata.conf`
+    should be sent to the external database
+
+-   `send automatic labels = yes | no` controls if automatially created labels, like `_os_name` or `_architecture`
+    should be sent to the external database
+
+> Starting from Netdata v1.20 the host tags (defined in the `[backend]` section of `netdata.conf`) are parsed in
+> accordance with a configured backend type and stored as host labels so that they can be reused in API responses and
+> exporting connectors. The parsing is supported for graphite, json, opentsdb, and prometheus (default) backend types.
+> You can check how the host tags were parsed using the /api/v1/info API call. But, keep in mind that backends subsystem
+> is deprecated and will be deleted soon. Please move your existing tags to the `[host labels]` section.
+
+## monitoring operation
+
+Netdata provides 5 charts:
+
+1.  **Buffered metrics**, the number of metrics Netdata added to the buffer for dispatching them to the
+    external database server.
+
+2.  **Exporting data size**, the amount of data (in KB) Netdata added the buffer.
+
+3.  **Exporting operations**, the number of operations performed by Netdata.
+
+4.  **Exporting thread CPU usage**, the CPU resources consumed by the Netdata thread, that is responsible for sending the
+    metrics to the external database server.
+
+![image](https://cloud.githubusercontent.com/assets/2662304/20463536/eb196084-af3d-11e6-8ee5-ddbd3b4d8449.png)
+
+## alarms
+
+Netdata adds 3 alarms:
+
+1.  `exporting_last_buffering`, number of seconds since the last successful buffering of exported data
+2.  `exporting_metrics_sent`, percentage of metrics sent to the external database server
+3.  `exporting_metrics_lost`, number of metrics lost due to repeating failures to contact the external database server
+
+![image](https://cloud.githubusercontent.com/assets/2662304/20463779/a46ed1c2-af43-11e6-91a5-07ca4533cac3.png)
+
 [![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fexporting%2FREADME&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>)
diff --git a/exporting/TIMESCALE.md b/exporting/TIMESCALE.md
new file mode 100644
index 0000000000..8a074b5836
--- /dev/null
+++ b/exporting/TIMESCALE.md
@@ -0,0 +1,69 @@
+<!--
+title: "Writing metrics to TimescaleDB"
+description: "Send Netdata metrics to TimescaleDB for long-term archiving and further analysis."
+custom_edit_url: https://github.com/netdata/netdata/edit/master/exporting/TIMESCALE.md
+sidebar_label: Writing metrics to TimescaleDB
+-->
+
+# Writing metrics to TimescaleDB
+
+Thanks to Netdata's community of developers and system administrators, and Mahlon Smith
+([GitHub](https://github.com/mahlonsmith)/[Website](http://www.martini.nu/)) in particular, Netdata now supports
+archiving metrics directly to TimescaleDB.
+
+What's TimescaleDB? Here's how their team defines the project on their [GitHub page](https://github.com/timescale/timescaledb):
+
+> TimescaleDB is an open-source database designed to make SQL scalable for time-series data. It is engineered up from
+> PostgreSQL, providing automatic partitioning across time and space (partitioning key), as well as full SQL support.
+
+## Quickstart
+
+To get started archiving metrics to TimescaleDB right away, check out Mahlon's [`netdata-timescale-relay`
+repository](https://github.com/mahlonsmith/netdata-timescale-relay) on GitHub. Please be aware that backends subsystem
+is deprecated and Netdata configuration should be moved to the new `exporting conf` configuration file. Use
+```conf
+[json:my_instance]
+```
+in `exporting.conf` instead of
+```conf
+[backend]
+    type = json
+```
+in `netdata.conf`.
+
+This small program takes JSON streams from a Netdata client and writes them to a PostgreSQL (aka TimescaleDB) table.
+You'll run this program in parallel with Netdata, and after a short [configuration
+process](https://github.com/mahlonsmith/netdata-timescale-relay#configuration), your metrics should start populating
+TimescaleDB.
+
+Finally, another member of Netdata's community has built a project that quickly launches Netdata, TimescaleDB, and
+Grafana in easy-to-manage Docker containers. Rune Juhl Jacobsen's
+[project](https://github.com/runejuhl/grafana-timescaledb) uses a `Makefile` to create everything, which makes it
+perferct for testing and experimentation.
+
+## Netdata&#8596;TimescaleDB in action
+
+Aside from creating incredible contributions to Netdata, Mahlon works at [LAIKA](https://www.laika.com/), an
+Oregon-based animation studio that's helped create acclaimed films like _Coraline_ and _Kubo and the Two Strings_.
+
+As part of his work to maintain the company's infrastructure of render farms, workstations, and virtual machines, he's
+using Netdata, `netdata-timescale-relay`, and TimescaleDB to store Netdata metrics alongside other data from other
+sources.
+
+> LAIKA is a long-time PostgreSQL user and added TimescaleDB to their infrastructure in 2018 to help manage and store
+> their IT metrics and time-series data. So far, the tool has been in production at LAIKA for over a year and helps them
+> with their use case of time-based logging, where they record over 8 million metrics an hour for netdata content alone.
+
+By archiving Netdata metrics to a database like TimescaleDB, LAIKA can consolidate metrics data from distributed
+machines efficiently. Mahlon can then correlate Netdata metrics with other sources directly in TimescaleDB.
+
+And, because LAIKA will soon be storing years worth of Netdata metrics data in TimescaleDB, they can analyze long-term
+metrics as their films move from concept to final cut.
+
+Read the full blog post from LAIKA at the [TimescaleDB
+blog](https://blog.timescale.com/blog/writing-it-metrics-from-netdata-to-timescaledb/amp/).
+
+Thank you to Mahlon, Rune, TimescaleDB, and the members of the Netdata community that requested and then built this
+exporting connection between Netdata and TimescaleDB!
+
+[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fexporting%2FTIMESCALE&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>)
diff --git a/exporting/WALKTHROUGH.md b/exporting/WALKTHROUGH.md
new file mode 100644
index 0000000000..ac17129164
--- /dev/null
+++ b/exporting/WALKTHROUGH.md
@@ -0,0 +1,259 @@
+<!--
+title: "Exporting to Netdata, Prometheus, Grafana stack"
+description: "Using Netdata in conjunction with Prometheus and Grafana."
+custom_edit_url: https://github.com/netdata/netdata/edit/master/exporting/WALKTHROUGH.md
+sidebar_label: Netdata, Prometheus, Grafana stack
+-->
+
+# Netdata, Prometheus, Grafana stack
+
+## Intro
+
+In this article I will walk you through the basics of getting Netdata, Prometheus and Grafana all working together and
+monitoring your application servers. This article will be using docker on your local workstation. We will be working
+with docker in an ad-hoc way, launching containers that run `/bin/bash` and attaching a TTY to them. I use docker here
+in a purely academic fashion and do not condone running Netdata in a container. I pick this method so individuals
+without cloud accounts or access to VMs can try this out and for it's speed of deployment.
+
+## Why Netdata, Prometheus, and Grafana
+
+Some time ago I was introduced to Netdata by a coworker. We were attempting to troubleshoot python code which seemed to
+be bottlenecked. I was instantly impressed by the amount of metrics Netdata exposes to you. I quickly added Netdata to
+my set of go-to tools when troubleshooting systems performance.
+
+Some time ago, even later, I was introduced to Prometheus. Prometheus is a monitoring application which flips the normal
+architecture around and polls rest endpoints for its metrics. This architectural change greatly simplifies and decreases
+the time necessary to begin monitoring your applications. Compared to current monitoring solutions the time spent on
+designing the infrastructure is greatly reduced. Running a single Prometheus server per application becomes feasible
+with the help of Grafana.
+
+Grafana has been the go to graphing tool for… some time now. It's awesome, anyone that has used it knows it's awesome.
+We can point Grafana at Prometheus and use Prometheus as a data source. This allows a pretty simple overall monitoring
+architecture: Install Netdata on your application servers, point Prometheus at Netdata, and then point Grafana at
+Prometheus.
+
+I'm omitting an important ingredient in this stack in order to keep this tutorial simple and that is service discovery.
+My personal preference is to use Consul. Prometheus can plug into consul and automatically begin to scrape new hosts
+that register a Netdata client with Consul.
+
+At the end of this tutorial you will understand how each technology fits together to create a modern monitoring stack.
+This stack will offer you visibility into your application and systems performance.
+
+## Getting Started - Netdata
+
+To begin let's create our container which we will install Netdata on. We need to run a container, forward the necessary
+port that Netdata listens on, and attach a tty so we can interact with the bash shell on the container. But before we do
+this we want name resolution between the two containers to work. In order to accomplish this we will create a
+user-defined network and attach both containers to this network. The first command we should run is: 
+
+```sh
+docker network create --driver bridge netdata-tutorial
+```
+
+With this user-defined network created we can now launch our container we will install Netdata on and point it to this
+network.
+
+```sh
+docker run -it --name netdata --hostname netdata --network=netdata-tutorial -p 19999:19999  centos:latest '/bin/bash'
+```
+
+This command creates an interactive tty session (`-it`), gives the container both a name in relation to the docker
+daemon and a hostname (this is so you know what container is which when working in the shells and docker maps hostname
+resolution to this container), forwards the local port 19999 to the container's port 19999 (`-p 19999:19999`), sets the
+command to run (`/bin/bash`) and then chooses the base container images (`centos:latest`). After running this you should
+be sitting inside the shell of the container.
+
+After we have entered the shell we can install Netdata. This process could not be easier. If you take a look at [this
+link](/packaging/installer/README.md), the Netdata devs give us several one-liners to install Netdata. I have not had
+any issues with these one liners and their bootstrapping scripts so far (If you guys run into anything do share). Run
+the following command in your container.
+
+```sh
+bash <(curl -Ss https://my-netdata.io/kickstart.sh) --dont-wait
+```
+
+After the install completes you should be able to hit the Netdata dashboard at <http://localhost:19999/> (replace
+localhost if you're doing this on a VM or have the docker container hosted on a machine not on your local system). If
+this is your first time using Netdata I suggest you take a look around. The amount of time I've spent digging through
+`/proc` and calculating my own metrics has been greatly reduced by this tool. Take it all in.
+
+Next I want to draw your attention to a particular endpoint. Navigate to
+<http://localhost:19999/api/v1/allmetrics?format=prometheus&help=yes> In your browser. This is the endpoint which
+publishes all the metrics in a format which Prometheus understands. Let's take a look at one of these metrics.
+`netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="system"} 0.0831255 1501271696000` This
+metric is representing several things which I will go in more details in the section on Prometheus. For now understand
+that this metric: `netdata_system_cpu_percentage_average` has several labels: (`chart`, `family`, `dimension`). This
+corresponds with the first cpu chart you see on the Netdata dashboard.
+
+![](https://github.com/ldelossa/NetdataTutorial/raw/master/Screen%20Shot%202017-07-28%20at%204.00.45%20PM.png)
+
+This CHART is called `system.cpu`, The FAMILY is `cpu`, and the DIMENSION we are observing is `system`. You can begin to
+draw links between the charts in Netdata to the Prometheus metrics format in this manner.
+
+## Prometheus
+
+We will be installing Prometheus in a container for purpose of demonstration. While Prometheus does have an official
+container I would like to walk through the install process and setup on a fresh container. This will allow anyone
+reading to migrate this tutorial to a VM or Server of any sort.
+
+Let's start another container in the same fashion as we did the Netdata container. 
+
+```sh
+docker run -it --name prometheus --hostname prometheus
+--network=netdata-tutorial -p 9090:9090  centos:latest '/bin/bash'
+``` 
+
+This should drop you into a shell once again. Once there quickly install your favorite editor as we will be editing
+files later in this tutorial. 
+
+```sh
+yum install vim -y
+```
+
+Prometheus provides a tarball of their latest stable versions [here](https://prometheus.io/download/).
+
+Let's download the latest version and install into your container.
+
+```sh
+cd /tmp && curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest \
+| grep "browser_download_url.*linux-amd64.tar.gz" \
+| cut -d '"' -f 4 \
+| wget -qi -
+
+mkdir /opt/prometheus
+
+sudo tar -xvf /tmp/prometheus-*linux-amd64.tar.gz -C /opt/prometheus --strip=1
+```
+
+This should get Prometheus installed into the container. Let's test that we can run Prometheus and connect to it's web
+interface.
+
+```sh
+/opt/prometheus/prometheus
+```
+
+Now attempt to go to <http://localhost:9090/>. You should be presented with the Prometheus homepage. This is a good
+point to talk about Prometheus's data model which can be viewed here: <https://prometheus.io/docs/concepts/data_model/>
+As explained we have two key elements in Prometheus metrics. We have the _metric_ and its _labels_. Labels allow for
+granularity between metrics. Let's use our previous example to further explain.
+
+```conf
+netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="system"} 0.0831255 1501271696000
+```
+
+Here our metric is `netdata_system_cpu_percentage_average` and our labels are `chart`, `family`, and `dimension`. The
+last two values constitute the actual metric value for the metric type (gauge, counter, etc…). We can begin graphing
+system metrics with this information, but first we need to hook up Prometheus to poll Netdata stats.
+
+Let's move our attention to Prometheus's configuration. Prometheus gets it config from the file located (in our example)
+at `/opt/prometheus/prometheus.yml`. I won't spend an extensive amount of time going over the configuration values
+documented here: <https://prometheus.io/docs/operating/configuration/>. We will be adding a new job under the
+`scrape_configs`. Let's make the `scrape_configs` section look like this (we can use the DNS name Netdata due to the
+custom user-defined network we created in docker beforehand).
+
+```yaml
+scrape_configs:
+  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
+  - job_name: 'prometheus'
+
+    # metrics_path defaults to '/metrics'
+    # scheme defaults to 'http'.
+
+    static_configs:
+      - targets: ['localhost:9090']
+
+  - job_name: 'netdata'
+
+    metrics_path: /api/v1/allmetrics
+    params:
+      format: [ prometheus ]
+
+    static_configs:
+      - targets: ['netdata:19999']
+```
+
+Let's start Prometheus once again by running `/opt/prometheus/prometheus`. If we now navigate to Prometheus at
+<http://localhost:9090/targets> we should see our target being successfully scraped. If we now go back to the
+Prometheus's homepage and begin to type `netdata\_` Prometheus should auto complete metrics it is now scraping.
+
+![](https://github.com/ldelossa/NetdataTutorial/raw/master/Screen%20Shot%202017-07-28%20at%205.13.43%20PM.png)
+
+Let's now start exploring how we can graph some metrics. Back in our NetData container lets get the CPU spinning with a
+pointless busy loop. On the shell do the following:
+
+```sh
+[root@netdata /]# while true; do echo "HOT HOT HOT CPU"; done
+```
+
+Our NetData cpu graph should be showing some activity. Let's represent this in Prometheus. In order to do this let's
+keep our metrics page open for reference: <http://localhost:19999/api/v1/allmetrics?format=prometheus&help=yes>. We are
+setting out to graph the data in the CPU chart so let's search for `system.cpu` in the metrics page above. We come
+across a section of metrics with the first comments  `# COMMENT homogeneous chart "system.cpu", context "system.cpu",
+family "cpu", units "percentage"` followed by the metrics. This is a good start now let us drill down to the specific
+metric we would like to graph.
+
+```conf
+# COMMENT
+netdata_system_cpu_percentage_average: dimension "system", value is percentage, gauge, dt 1501275951 to 1501275951 inclusive
+netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="system"} 0.0000000 1501275951000
+```
+
+Here we learn that the metric name we care about is `netdata_system_cpu_percentage_average` so throw this into
+Prometheus and see what we get. We should see something similar to this (I shut off my busy loop)
+
+![](https://github.com/ldelossa/NetdataTutorial/raw/master/Screen%20Shot%202017-07-28%20at%205.47.53%20PM.png)
+
+This is a good step toward what we want. Also make note that Prometheus will tag on an `instance` label for us which
+corresponds to our statically defined job in the configuration file. This allows us to tailor our queries to specific
+instances. Now we need to isolate the dimension we want in our query. To do this let us refine the query slightly. Let's
+query the dimension also. Place this into our query text box.
+`netdata_system_cpu_percentage_average{dimension="system"}` We now wind up with the following graph.
+
+![](https://github.com/ldelossa/NetdataTutorial/raw/master/Screen%20Shot%202017-07-28%20at%205.54.40%20PM.png)
+
+Awesome, this is exactly what we wanted. If you haven't caught on yet we can emulate entire charts from NetData by using
+the `chart` dimension. If you'd like you can combine the `chart` and `instance` dimension to create per-instance charts.
+Let's give this a try: `netdata_system_cpu_percentage_average{chart="system.cpu", instance="netdata:19999"}`
+
+This is the basics of using Prometheus to query NetData. I'd advise everyone at this point to read [this
+page](/exporting/prometheus/#using-netdata-with-prometheus). The key point here is that NetData can export metrics from
+its internal DB or can send metrics _as-collected_ by specifying the `source=as-collected` URL parameter like so.
+<http://localhost:19999/api/v1/allmetrics?format=prometheus&help=yes&types=yes&source=as-collected> If you choose to use
+this method you will need to use Prometheus's set of functions here: <https://prometheus.io/docs/querying/functions/> to
+obtain useful metrics as you are now dealing with raw counters from the system. For example you will have to use the
+`irate()` function over a counter to get that metric's rate per second. If your graphing needs are met by using the
+metrics returned by NetData's internal database (not specifying any source= URL parameter) then use that. If you find
+limitations then consider re-writing your queries using the raw data and using Prometheus functions to get the desired
+chart.
+
+## Grafana
+
+Finally we make it to grafana. This is the easiest part in my opinion. This time we will actually run the official
+grafana docker container as all configuration we need to do is done via the GUI. Let's run the following command:
+
+```sh
+docker run -i -p 3000:3000 --network=netdata-tutorial grafana/grafana
+```
+
+This will get grafana running at <http://localhost:3000/>. Let's go there and
+login using the credentials Admin:Admin.
+
+The first thing we want to do is click "Add data source". Let's make it look like the following screenshot
+
+![](https://github.com/ldelossa/NetdataTutorial/raw/master/Screen%20Shot%202017-07-28%20at%206.36.55%20PM.png)
+
+With this completed let's graph! Create a new Dashboard by clicking on the top left Grafana Icon and create a new graph
+in that dashboard. Fill in the query like we did above and save.
+
+![](https://github.com/ldelossa/NetdataTutorial/raw/master/Screen%20Shot%202017-07-28%20at%206.39.38%20PM.png)
+
+## Conclusion
+
+There you have it, a complete systems monitoring stack which is very easy to deploy. From here I would begin to
+understand how Prometheus and a service discovery mechanism such as Consul can play together nicely. My current prod
+deployments automatically register Netdata services into Consul and Prometheus automatically begins to scrape them. Once
+achieved you do not have to think about the monitoring system until Prometheus cannot keep up with your scale. Once this
+happens there are options presented in the Prometheus documentation for solving this. Hope this was helpful, happy
+monitoring.
+
+[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2Fexporting%2FWALKTHROUGH&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>)
diff --git a/exporting/exporting.conf b/exporting/exporting.conf
new file mode 100644
index 0000000000..9c1e7ffb75
--- /dev/null
+++ b/exporting/exporting.conf
@@ -0,0 +1,88 @@
+[exporting:global]
+    enabled = no
+    # send configured labels = yes
+    # send automatic labels = no
+    # update every = 10
+
+[prometheus:exporter]
+    # send names instead of ids = yes
+    # send configured labels = yes
+    # send automatic labels = no
+    # send charts matching = *
+    # send hosts matching = localhost *
+    # prefix = netdata
+
+# An example configuration for graphite, json, opentsdb exporting connectors
+# [graphite:my_graphite_instance]
+    #