1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
|
<!--
title: "Export metrics to Prometheus"
description: "Export Netdata metrics to Prometheus for archiving and further analysis."
custom_edit_url: https://github.com/netdata/netdata/edit/master/exporting/prometheus/README.md
sidebar_label: "Using Netdata with Prometheus"
-->
import { OneLineInstallWget, OneLineInstallCurl } from '../../../src/components/OneLineInstall/'
# Using Netdata with Prometheus
Prometheus is a distributed monitoring system which offers a very simple setup along with a robust data model. Recently
Netdata added support for Prometheus. I'm going to quickly show you how to install both Netdata and Prometheus on the
same server. We can then use Grafana pointed at Prometheus to obtain long term metrics Netdata offers. I'm assuming we
are starting at a fresh ubuntu shell (whether you'd like to follow along in a VM or a cloud instance is up to you).
## Installing Netdata and Prometheus
### Installing Netdata
There are number of ways to install Netdata according to [Installation](/packaging/installer/README.md). The suggested way
of installing the latest Netdata and keep it upgrade automatically.
<!-- candidate for reuse -->
To install Netdata, run the following as your normal user:
<OneLineInstallWget/>
Or, if you have cURL but not wget (such as on macOS):
<OneLineInstallCurl/>
At this point we should have Netdata listening on port 19999. Attempt to take your browser here:
```sh
http://your.netdata.ip:19999
```
_(replace `your.netdata.ip` with the IP or hostname of the server running Netdata)_
### Installing Prometheus
In order to install Prometheus we are going to introduce our own systemd startup script along with an example of
prometheus.yaml configuration. Prometheus needs to be pointed to your server at a specific target url for it to scrape
Netdata's api. Prometheus is always a pull model meaning Netdata is the passive client within this architecture.
Prometheus always initiates the connection with Netdata.
#### Download Prometheus
```sh
cd /tmp && curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest \
| grep "browser_download_url.*linux-amd64.tar.gz" \
| cut -d '"' -f 4 \
| wget -qi -
```
#### Create prometheus system user
```sh
sudo useradd -r prometheus
```
#### Create prometheus directory
```sh
sudo mkdir /opt/prometheus
sudo chown prometheus:prometheus /opt/prometheus
```
#### Untar prometheus directory
```sh
sudo tar -xvf /tmp/prometheus-*linux-amd64.tar.gz -C /opt/prometheus --strip=1
```
#### Install prometheus.yml
We will use the following `prometheus.yml` file. Save it at `/opt/prometheus/prometheus.yml`.
Make sure to replace `your.netdata.ip` with the IP or hostname of the host running Netdata.
```yaml
# my global config
global:
scrape_interval: 5s # Set the scrape interval to every 5 seconds. Default is every 1 minute.
evaluation_interval: 5s # Evaluate rules every 5 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: 'codelab-monitor'
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first.rules"
# - "second.rules"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['0.0.0.0:9090']
- job_name: 'netdata-scrape'
metrics_path: '/api/v1/allmetrics'
params:
# format: prometheus | prometheus_all_hosts
# You can use `prometheus_all_hosts` if you want Prometheus to set the `instance` to your hostname instead of IP
format: [prometheus]
#
# sources: as-collected | raw | average | sum | volume
# default is: average
#source: [as-collected]
#
# server name for this prometheus - the default is the client IP
# for Netdata to uniquely identify it
#server: ['prometheus1']
honor_labels: true
static_configs:
- targets: ['{your.netdata.ip}:19999']
```
#### Install nodes.yml
The following is completely optional, it will enable Prometheus to generate alerts from some Netdata sources. Tweak the
values to your own needs. We will use the following `nodes.yml` file below. Save it at `/opt/prometheus/nodes.yml`, and
add a _- "nodes.yml"_ entry under the _rule_files:_ section in the example prometheus.yml file above.
```yaml
groups:
- name: nodes
rules:
- alert: node_high_cpu_usage_70
expr: sum(sum_over_time(netdata_system_cpu_percentage_average{dimension=~"(user|system|softirq|irq|guest)"}[10m])) by (job) / sum(count_over_time(netdata_system_cpu_percentage_average{dimension="idle"}[10m])) by (job) > 70
for: 1m
annotations:
description: '{{ $labels.job }} on ''{{ $labels.job }}'' CPU usage is at {{ humanize $value }}%.'
summary: CPU alert for container node '{{ $labels.job }}'
- alert: node_high_memory_usage_70
expr: 100 / sum(netdata_system_ram_MB_average) by (job)
* sum(netdata_system_ram_MB_average{dimension=~"free|cached"}) by (job) < 30
for: 1m
annotations:
description: '{{ $labels.job }} memory usage is {{ humanize $value}}%.'
summary: Memory alert for container node '{{ $labels.job }}'
- alert: node_low_root_filesystem_space_20
expr: 100 / sum(netdata_disk_space_GB_average{family="/"}) by (job)
* sum(netdata_disk_space_GB_average{family="/",dimension=~"avail|cached"}) by (job) < 20
for: 1m
annotations:
description: '{{ $labels.job }} root filesystem space is {{ humanize $value}}%.'
summary: Root filesystem alert for container node '{{ $labels.job }}'
- alert: node_root_filesystem_fill_rate_6h
expr: predict_linear(netdata_disk_space_GB_average{family="/",dimension=~"avail|cached"}[1h], 6 * 3600) < 0
for: 1h
labels:
severity: critical
annotations:
description: Container node {{ $labels.job }} root filesystem is going to fill up in 6h.
summary: Disk fill alert for Swarm node '{{ $labels.job }}'
```
#### Install prometheus.service
Save this service file as `/etc/systemd/system/prometheus.service`:
```sh
[Unit]
Description=Prometheus Server
AssertPathExists=/opt/prometheus
[Service]
Type=simple
WorkingDirectory=/opt/prometheus
User=prometheus
Group=prometheus
ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml --log.level=info
ExecReload=/bin/kill -SIGHUP $MAINPID
ExecStop=/bin/kill -SIGINT $MAINPID
[Install]
WantedBy=multi-user.target
```
##### Start Prometheus
```sh
sudo systemctl start prometheus
sudo systemctl enable prometheus
```
Prometheus should now start and listen on port 9090. Attempt to head there with your browser.
If everything is working correctly when you fetch `http://your.prometheus.ip:9090` you will see a 'Status' tab. Click
this and click on 'targets' We should see the Netdata host as a scraped target.
---
## Netdata support for Prometheus
Before explaining the changes, we have to understand the key differences between Netdata and Prometheus.
### understanding Netdata metrics
#### charts
Each chart in Netdata has several properties (common to all its metrics):
- `chart_id` - uniquely identifies a chart.
- `chart_name` - a more human friendly name for `chart_id`, also unique.
- `context` - this is the template of the chart. All disk I/O charts have the same context, all mysql requests charts
have the same context, etc. This is used for alarm templates to match all the charts they should be attached to.
- `family` groups a set of charts together. It is used as the submenu of the dashboard.
- `units` is the units for all the metrics attached to the chart.
#### dimensions
Then each Netdata chart contains metrics called `dimensions`. All the dimensions of a chart have the same units of
measurement, and are contextually in the same category (ie. the metrics for disk bandwidth are `read` and `write` and
they are both in the same chart).
### Netdata data source
Netdata can send metrics to Prometheus from 3 data sources:
- `as collected` or `raw` - this data source sends the metrics to Prometheus as they are collected. No conversion is
done by Netdata. The latest value for each metric is just given to Prometheus. This is the most preferred method by
Prometheus, but it is also the harder to work with. To work with this data source, you will need to understand how
to get meaningful values out of them.
The format of the metrics is: `CONTEXT{chart="CHART",family="FAMILY",dimension="DIMENSION"}`.
If the metric is a counter (`incremental` in Netdata lingo), `_total` is appended the context.
Unlike Prometheus, Netdata allows each dimension of a chart to have a different algorithm and conversion constants
(`multiplier` and `divisor`). In this case, that the dimensions of a charts are heterogeneous, Netdata will use this
format: `CONTEXT_DIMENSION{chart="CHART",family="FAMILY"}`
- `average` - this data source uses the Netdata database to send the metrics to Prometheus as they are presented on
the Netdata dashboard. So, all the metrics are sent as gauges, at the units they are presented in the Netdata
dashboard charts. This is the easiest to work with.
The format of the metrics is: `CONTEXT_UNITS_average{chart="CHART",family="FAMILY",dimension="DIMENSION"}`.
When this source is used, Netdata keeps track of the last access time for each Prometheus server fetching the
metrics. This last access time is used at the subsequent queries of the same Prometheus server to identify the
time-frame the `average` will be calculated.
So, no matter how frequently Prometheus scrapes Netdata, it will get all the database data.
To identify each Prometheus server, Netdata uses by default the IP of the client fetching the metrics.
If there are multiple Prometheus servers fetching data from the same Netdata, using the same IP, each Prometheus
server can append `server=NAME` to the URL. Netdata will use this `NAME` to uniquely identify the Prometheus server.
- `sum` or `volume`, is like `average` but instead of averaging the values, it sums them.
The format of the metrics is: `CONTEXT_UNITS_sum{chart="CHART",family="FAMILY",dimension="DIMENSION"}`. All the
other operations are the same with `average`.
To change the data source to `sum` or `as-collected` you need to provide the `source` parameter in the request URL.
e.g.: `http://your.netdata.ip:19999/api/v1/allmetrics?format=prometheus&help=yes&source=as-collected`
Keep in mind that early versions of Netdata were sending the metrics as: `CHART_DIMENSION{}`.
### Querying Metrics
Fetch with your web browser this URL:
`http://your.netdata.ip:19999/api/v1/allmetrics?format=prometheus&help=yes`
_(replace `your.netdata.ip` with the ip or hostname of your Netdata server)_
Netdata will respond with all the metrics it sends to Prometheus.
If you search that page for `"system.cpu"` you will find all the metrics Netdata is exporting to Prometheus for this
chart. `system.cpu` is the chart name on the Netdata dashboard (on the Netdata dashboard all charts have a text heading
such as : `Total CPU utilization (system.cpu)`. What we are interested here in the chart name: `system.cpu`).
Searching for `"system.cpu"` reveals:
```sh
# COMMENT homogeneous chart "system.cpu", context "system.cpu", family "cpu", units "percentage"
# COMMENT netdata_system_cpu_percentage_average: dimension "guest_nice", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive
netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="guest_nice"} 0.0000000 1500066662000
# COMMENT netdata_system_cpu_percentage_average: dimension "guest", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive
netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="guest"} 1.7837326 1500066662000
# COMMENT netdata_system_cpu_percentage_average: dimension "steal", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive
netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="steal"} 0.0000000 1500066662000
# COMMENT netdata_system_cpu_percentage_average: dimension "softirq", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive
netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="softirq"} 0.5275442 1500066662000
# COMMENT netdata_system_cpu_percentage_average: dimension "irq", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive
netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="irq"} 0.2260836 1500066662000
# COMMENT netdata_system_cpu_percentage_average: dimension "user", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive
netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="user"} 2.3362762 1500066662000
# COMMENT netdata_system_cpu_percentage_average: dimension "system", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive
netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="system"} 1.7961062 1500066662000
# COMMENT netdata_system_cpu_percentage_average: dimension "nice", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive
netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="nice"} 0.0000000 1500066662000
# COMMENT netdata_system_cpu_percentage_average: dimension "iowait", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive
netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="iowait"} 0.9671802 1500066662000
# COMMENT netdata_system_cpu_percentage_average: dimension "idle", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive
netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="idle"} 92.3630770 1500066662000
```
_(Netdata response for `system.cpu` with source=`average`)_
In `average` or `sum` data sources, all values are normalized and are reported to Prometheus as gauges. Now, use the
'expression' text form in Prometheus. Begin to type the metrics we are looking for: `netdata_system_cpu`. You should see
that the text form begins to auto-fill as Prometheus knows about this metric.
If the data source was `as collected`, the response would be:
```sh
# COMMENT homogeneous chart "system.cpu", context "system.cpu", family "cpu", units "percentage"
# COMMENT netdata_system_cpu_total: chart "system.cpu", context "system.cpu", family "cpu", dimension "guest_nice", value * 1 / 1 delta gives percentage (counter)
netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="guest_nice"} 0 1500066716438
# COMMENT netdata_system_cpu_total: chart "system.cpu", context "system.cpu", family "cpu", dimension "guest", value * 1 / 1 delta gives percentage (counter)
netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="guest"} 63945 1500066716438
# COMMENT netdata_system_cpu_total: chart "system.cpu", context "system.cpu", family "cpu", dimension "steal", value * 1 / 1 delta gives percentage (counter)
netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="steal"} 0 1500066716438
# COMMENT netdata_system_cpu_total: chart "system.cpu", context "system.cpu", family "cpu", dimension "softirq", value * 1 / 1 delta gives percentage (counter)
netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="softirq"} 8295 1500066716438
# COMMENT netdata_system_cpu_total: chart "system.cpu", context "system.cpu", family "cpu", dimension "irq", value * 1 / 1 delta gives percentage (counter)
netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="irq"} 4079 1500066716438
# COMMENT netdata_system_cpu_total: chart "system.cpu", context "system.cpu", family "cpu", dimension "user", value * 1 / 1 delta gives percentage (counter)
netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="user"} 116488 1500066716438
# COMMENT netdata_system_cpu_total: chart "system.cpu", context "system.cpu", family "cpu", dimension "system", value * 1 / 1 delta gives percentage (counter)
netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="system"} 35084 1500066716438
# COMMENT netdata_system_cpu_total: chart "system.cpu", context "system.cpu", family "cpu", dimension "nice", value * 1 / 1 delta gives percentage (counter)
netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="nice"} 505 1500066716438
# COMMENT netdata_system_cpu_total: chart "system.cpu", context "system.cpu", family "cpu", dimension "iowait", value * 1 / 1 delta gives percentage (counter)
netdata_system_cpu_t
|