diff options
author | Joel Hans <joel@netdata.cloud> | 2020-02-20 20:46:35 -0700 |
---|---|---|
committer | GitHub <noreply@github.com> | 2020-02-20 19:46:35 -0800 |
commit | 19ef3f93792d382992afbdce74743d868eaa3074 (patch) | |
tree | 239376c6e805e270803f9efc409775b86b70b2e1 /docs | |
parent | 89519bcb51155c56bd5aadfe55f18ca9a885a3f8 (diff) |
Tutorials to support v1.20 release (#7943)
* Add draft of CockroachDB tutorial
* Fixed and new images
* Support figures for images
* Change border color
* Change job
* Initialize eBPF tutorial
* Very very rough draft of host labels tutorial
* Add a few mentions of tutorial
* Fix for Thiago
* Simplify health entities
* Fixes for Thiago
* Fixes and add tutorials to collectors README
* Fixes to cockroachBD
* Remove ebpf tutorial
* remove link
* Updates for Patti and Thiago
* Add streaming security note
* Straightaway
Diffstat (limited to 'docs')
-rw-r--r-- | docs/configuration-guide.md | 7 | ||||
-rw-r--r-- | docs/generator/custom/css/netdata.css | 17 | ||||
-rw-r--r-- | docs/tutorials/monitor-cockroachdb.md | 128 | ||||
-rw-r--r-- | docs/tutorials/using-host-labels.md | 203 |
4 files changed, 353 insertions, 2 deletions
diff --git a/docs/configuration-guide.md b/docs/configuration-guide.md index 0c6d934e49..a30149cb2a 100644 --- a/docs/configuration-guide.md +++ b/docs/configuration-guide.md @@ -187,7 +187,12 @@ So, Netdata supports [simple patterns](../libnetdata/simple_pattern/). ## Netdata labels -Since version 1.20, Netdata accepts user defined labels for host. The labels are defined in the section `[host labels]`. +Beginning with 1.20, Netdata accepts user-defined **host labels**. These labels are defined in the section `[host +labels]`. + +Read more about how these labels work and why they're an effective way to organize complex infrasturctures in our +tutorial: [Use host labels to organize systems, metrics, and alarms](tutorials/using-host-labels.md). + To define a label inside this section, some rules needs to be followed, or Netdata will reject the label. The following restrictions are applied for label names: diff --git a/docs/generator/custom/css/netdata.css b/docs/generator/custom/css/netdata.css index 7b1c2e2ef9..9aa751b0b4 100644 --- a/docs/generator/custom/css/netdata.css +++ b/docs/generator/custom/css/netdata.css @@ -429,6 +429,21 @@ html [data-md-color-primary="blue-grey"] .md-nav--primary .md-nav__title--site { font-size: 0.75rem; } +/* Support figures and figcaptions with some styling. */ +.md-typeset figure img { + border-radius: 3px 3px 0 0; +} + +.md-typeset figure figcaption { + font-size: 0.75rem; + font-style: italic; + margin-top: -8px; + padding: 0.5rem; + border-width: 0 1px 1px 1px; + border-color: #AEB3B7; + border-style: solid; +} + /* Installer grid */ @@ -443,7 +458,7 @@ html [data-md-color-primary="blue-grey"] .md-nav--primary .md-nav__title--site { .grid-item { grid-column: span 2; border-radius: 2px; - border: 1px solid black; + border: 1px solid #AEB3B7; padding: 1rem; } diff --git a/docs/tutorials/monitor-cockroachdb.md b/docs/tutorials/monitor-cockroachdb.md new file mode 100644 index 0000000000..7850a693f4 --- /dev/null +++ b/docs/tutorials/monitor-cockroachdb.md @@ -0,0 +1,128 @@ +# Monitor CockroachDB metrics with Netdata + +[CockroachDB](https://github.com/cockroachdb/cockroach) is an open-source project that brings SQL databases into +scalable, disaster-resilient cloud deployments. Thanks to a [new CockroachDB +collector](https://docs.netdata.cloud/collectors/go.d.plugin/modules/cockroachdb/) released in +[v1.20](https://blog.netdata.cloud/posts/release-1.20/), you can now monitor any number of CockroachDB databases with +maximum granularity using Netdata. Collect more than 50 unique metrics and put them on interactive visualizations +designed for better visual anomaly detection. + +Netdata itself uses CockroachDB as part of its Netdata Cloud infrastructure, so we're happy to introduce this new +collector and help others get started with it straightaway. + +Let's dive in and walk through the process of monitoring CockroachDB metrics with Netdata. + +## What's in this guide + +- [Configure the CockroachDB collector](#configure-the-cockroachdb-collector) + - [Manual setup for a local CockroachDB database](#manual-setup-for-a-local-cockroachdb-database) +- [Tweak CockroachDB alarms](#tweak-cockroachdb-alarms) + +## Configure the CockroachDB collector + +Because _all_ of Netdata's collectors can auto-detect the services they monitor, you _shouldn't_ need to worry about +configuring CockroachDB. Netdata only needs to regularly query the database's `_status/vars` page to gather metrics and +display them on the dashboard. + +If your CockroachDB instance is accessible through `http://localhost:8080/` or `http://127.0.0.1:8080`, your setup is +complete. Restart Netdata with `service netdata restart`, or use the [appropriate +method](../getting-started.md#start-stop-and-restart-netdata) for your system, and refresh your browser. You should see +CockroachDB metrics in your Netdata dashboard! + +<figure> + <img src="https://user-images.githubusercontent.com/1153921/73564467-d7e36b00-441c-11ea-9ec9-b5d5ea7277d4.png" alt="CPU utilization charts from a CockroachDB database monitored by Netdata"> + <figcaption>CPU utilization charts from a CockroachDB database monitored by Netdata</figcaption> +</figure> + +> Note: Netdata collects metrics from CockroachDB every 10 seconds, instead of our usual 1 second, because CockroachDB +> only updates `_status/vars` every 10 seconds. You can't change this setting in CockroachDB. + +If you don't see CockroachDB charts, you may need to configure the collector manually. + +### Manual setup for a local CockroachDB database + +To configure Netdata's CockroachDB collector, navigate to your Netdata configuration directory (typically at +`/etc/netdata/`) and use `edit-config` to initialize and edit your CockroachDB configuration file. + +```bash +cd /etc/netdata/ # Replace with your Netdata configuration directory, if not /etc/netdata/ +./edit-config go.d/cockroachdb.conf +``` + +Scroll down to the `[JOBS]` section at the bottom of the file. You will see the two default jobs there, which you can +edit, or create a new job with any of the parameters listed above in the file. Both the `name` and `url` values are +required, and everything else is optional. + +For a production cluster, you'll use either an IP address or the system's hostname. Be sure that your remote system +allows TCP communication on port 8080, or whichever port you have configured CockroachDB's [Admin +UI](https://www.cockroachlabs.com/docs/stable/monitoring-and-alerting.html#prometheus-endpoint) to listen on. + +```yaml +# [ JOBS ] +jobs: + - name: remote + url: http://203.0.113.0:8080/_status/vars + + - name: remote_hostname + url: http://cockroachdb.example.com:8080/_status/vars +``` + +For a secure cluster, use `https` in the `url` field instead. + +```yaml +# [ JOBS ] +jobs: + - name: remote + url: https://203.0.113.0:8080/_status/vars + tls_skip_verify: yes # If your certificate is self-signed + + - name: remote_hostname + url: https://cockroachdb.example.com:8080/_status/vars + tls_skip_verify: yes # If your certificate is self-signed +``` + +You can add as many jobs as you'd like based on how many CockroachDB databases you have—Netdata will create separate +charts for each job. Once you've edited `cockroachdb.conf` according to the needs of your infrastructure, restart +Netdata to see your new charts. + +<figure> + <img src="https://user-images.githubusercontent.com/1153921/73564469-d7e36b00-441c-11ea-8333-02ba0e1c294c.png" alt="Charts showing a node failure during a simulated test"> + <figcaption>Charts showing a node failure during a simulated test</figcaption> +</figure> + +## Tweak CockroachDB alarms + +This release also includes eight pre-configured alarms for live nodes, such as whether the node is live, storage +capacity, issues with replication, and the number of SQL connections/statements. See [health.d/cockroachdb.conf on +GitHub](https://raw.githubusercontent.com/netdata/netdata/master/health/health.d/cockroachdb.conf) for details. + +You can also edit these files directly with `edit-config`: + +```bash +cd /etc/netdata/ # Replace with your Netdata configuration directory, if not /etc/netdata/ +./edit-config health.d/cockroachdb.conf # You may need to use `sudo` for write privileges +``` + +For more information about editing the defaults or writing new alarm entities, see our health monitoring [quickstart +guide](../../health/QUICKSTART.md). + +## What's next? + +Now that you're collecting metrics from your CockroachDB databases, let us know how it's working for you! There's always +room for improvement or refinement based on real-world use cases. Feel free to [file an +issue](https://github.com/netdata/netdata/issues/new?labels=bug%2C+needs+triage&template=bug_report.md) with your +thoughts. + +Also, be sure to check out these useful resources: + +- [Netdata's CockroachDB documentation](https://docs.netdata.cloud/collectors/go.d.plugin/modules/cockroachdb/) +- [Netdata's CockroachDB + configuration](https://github.com/netdata/go.d.plugin/blob/master/config/go.d/cockroachdb.conf) +- [Netdata's CockroachDB + alarms](https://github.com/netdata/netdata/blob/29d9b5e51603792ee27ef5a21f1de0ba8e130158/health/health.d/cockroachdb.conf) +- [CockroachDB homepage](https://www.cockroachlabs.com/product/) +- [CockroachDB documentation](https://www.cockroachlabs.com/docs/stable/) +- [`_status/vars` endpoint + docs](https://www.cockroachlabs.com/docs/stable/monitoring-and-alerting.html#prometheus-endpoint) +- [Monitor CockroachDB with + Prometheus](https://www.cockroachlabs.com/docs/stable/monitor-cockroachdb-with-prometheus.html) diff --git a/docs/tutorials/using-host-labels.md b/docs/tutorials/using-host-labels.md new file mode 100644 index 0000000000..4e8152e9f4 --- /dev/null +++ b/docs/tutorials/using-host-labels.md @@ -0,0 +1,203 @@ +# Use host labels to organize systems, metrics, and alarms + +When you use Netdata to monitor and troubleshoot an entire infrastructure, whether that's dozens or hundreds of systems, +you need sophisticated ways of keeping everything organized. You need alarms that adapt to the system's purpose, or +whether the `master` or `slave` in a streaming setup. You need properly-labeled metrics archiving so you can sort, +correlate, and mash-up your data to your heart's content. You need to keep tabs on ephemeral Docker containers in a +Kubernetes cluster. + +You need **host labels**: a powerful new way of organizing your Netdata-monitored systems. We introduced host labels in +[v1.20 of Netdata](https://blog.netdata.cloud/posts/release-1.20/), and they come pre-configured out of the box. + +Let's take a peek into how to create host labels and apply them across a few of Netdata's features to give you more +organization power over your infrastructure. + +## Create unique host labels + +Host labels are defined in `netdata.conf`. To create host labels, open that file using `edit-config`. + +```bash +cd /etc/netdata # Replace this path with your Netdata config directory, if different +sudo ./edit-config netdata.conf +``` + +Create a new `[host labels]` section defining a new host label and its value for the system in question. Make sure not +to violate any of the [host label naming rules](../configuration-guide.md#netdata-labels). + +```conf +[host labels] + type = webserver + location = us-seattle + installed = 20200218 +``` + +Once you've written a few host labels, you need to enable them. Instead of restarting the entire Netdata service, you +can reload labels using the helpful `netdatacli` tool: + +```bash +netdatacli reload-labels +``` + +Your host labels will now be enabled. You can double-check these by using `curl http://HOST-IP:19999/api/v1/info` to +read the status of your agent. For example, from a VPS system running Debian 10: + +```json +{ + ... + "host_labels": { + "_is_master": "false", + "_virt_detection": "systemd-detect-virt", + "_container_detection": "none", + "_container": "unknown", + "_virtualization": "kvm", + "_architecture": "x86_64", + "_kernel_version": "4.19.0-6-amd64", + "_os_version": "10 (buster)", + "_os_name": "Debian GNU/Linux", + "type": "webserver", + "location": "seattle", + "installed": "20200218" + }, + ... +} +``` + +You may have noticed a handful of labels that begin with an underscore (`_`). These are automatic labels. + +### Automatic labels + +When Netdata starts, it captures relevant information about the system and converts them into automatically-generated +host labels. You can use these to logically organize your systems via health entities, exporting metrics, +streaming/master status, and more. + +They capture the following: + +- Kernel version +- Operating system name and version +- CPU architecture, system cores, CPU frequency, RAM, and disk space +- Whether Netdata is running inside of a container, and if so, the OS and hardware details about the container's host +- What virtualization layer the system runs on top of, if any +- Whether the system is a streaming master or slave + +If you want to organize your systems without manually creating host tags, try the automatic labels in some of the +features below. + +## Host labels in streaming + +You may have noticed the `_is_master` and `_is_slave` automatic labels from above. Host labels are also now streamed +from a slave to its master agent, which concentrates an entire infrastructure's OS, hardware, container, and +virtualization information in one place: the master. + +Now, if you'd like to remind yourself of how much RAM a certain slave system has, you can simply access +`http://localhost:19999/host/SLAVE_NAME/api/v1/info` and reference the automatically-generated host labels from the +slave system. It's a vastly simplified way of accessing critical information about your infrastructure. + +> ⚠️ Because automatic labels for slave nodes are accessible via API calls, and contain sensitive information like +> kernel and operating system versions, you should secure streaming connections with SSL. See the [streaming +> documentation](../..//streaming/README.md#securing-streaming-communications) for details. You may also want to use +> [access lists](../../web/server/README.md#access-lists) or [expose the API only to LAN/localhost +> connections](../netdata-security.md#expose-netdata-only-in-a-private-lan). + +You can also use `_is_master`, `_is_slave`, and any other host labels in both health entities and metrics exporting. +Speaking of which... + +## Host labels in health entities + +You can use host labels to logically organize your systems by their type, purpose, or location, and then apply specific +alarms to them. + +For example, let's use configuration example from earlier: + +```conf +[host labels] + type = webserver + location = us-seattle + installed = 20200218 +``` + +You could now create a new health entity (checking if disk space will run out soon) that applies only to any host +labeled `webserver`: + +```yaml + template: disk_fill_rate + on: disk.space + lookup: max -1s at -30m unaligned of avail + calc: ($this - $avail) / (30 * 60) + every: 15s + host labels: type = webserver +``` + +Or, by using one of the automatic labels, for only webserver systems running a specific OS: + +```yaml + host labels: _os_name = Debian* +``` + +In a streaming configuration where a master agent is triggering alarms for its slaves, you could create health entities +that apply only to slaves: + +```yaml + host labels: _is_slave = true +``` + +Or when ephemeral Docker nodes are involved: + +```yaml + host labels: _container = docker +``` + +Of course, there are many more possibilities for intuitively organizing your systems with host labels. See the [health +documentation](../../health/REFERENCE.md#alarm-line-host-labels) for more details, and then get creative! + +## Host labels in metrics exporting + +If you have enabled any metrics exporting via our experimental [exporters](../../exporting/README.md), any new host +labels you created manually are sent to the destination database alongside metrics. You can change this behavior by +editing `exporting.conf`, and you can even send automatically-generated labels on with exported metrics. + +```conf +[exporting:global] +enabled = yes +send configured labels = yes +send automatic labels = no +``` + +You can also change this behavior per exporting connection: + +```conf +[opentsdb:my_instance3] +enabled = yes +destination = localhost:4242 +data source = sum +update every = 10 +send charts matching = system.cpu +send configured labels = no +send automatic labels = yes +``` + +By applying labels to exported metrics, you can more easily parse historical metrics with the labels applied. To learn +more about exporting, read the [documentation](../../exporting/README.md). + +## What's next? + +Host labels are a brand-new feature to Netdata, and yet they've already propagated deeply into some of its core +functionality. We're just getting started with labels, and will keep the community apprised of additional functionality +as it's made available. You can also track [issue #6503](https://github.com/netdata/netdata/issues/6503), which is where +the Netdata team first kicked off this work. + +It should be noted that while the Netdata dashboard does not expose either user-configured or automatic host labels, API +queries _do_ showcase this information. As always, we recommend you secure Netdata + +- [Expose Netdata only in a private LAN](../netdata-security.md#expose-netdata-only-in-a-private-lan) +- [Enable TLS/SSL for web/API requests](../../web/server/README.md#enabling-tls-support) +- Put Netdata behind a proxy + - [Use an authenticating web server in proxy + mode](../netdata-security.md#use-an-authenticating-web-server-in-proxy-mode) + - [Nginx proxy](../Running-behind-nginx.md) + - [Apache proxy](../Running-behind-apache.md) + - [Lighttpd](../Running-behind-lighttpd.md) + - [Caddy](../Running-behind-caddy.md) + +If you have issues or questions around using host labels, don't hesitate to [file an +issue](https://github.com/netdata/netdata/issues/new?labels=bug%2C+needs+triage&template=bug_report.md) on GitHub. We're +excited to make host labels even more valuable to our users, which we can only do with your input. |