diff options
author | Joel Hans <joel@netdata.cloud> | 2020-08-26 07:21:29 -0700 |
---|---|---|
committer | GitHub <noreply@github.com> | 2020-08-26 07:21:29 -0700 |
commit | a305ee79d6342b8ece27bf0b43f8d88025769353 (patch) | |
tree | e2c424327aa7fc81d3663b2d192c57cea8fcb080 | |
parent | ab7ff3131f3698710e0bd9fa3c66d31a3a194725 (diff) |
Improve dbengine docs and add new multihost setting (#9817)
* Clean up dbengine doc and metrics guide
* Tweaks to the text
* Fixes for Markos
-rw-r--r-- | database/engine/README.md | 55 | ||||
-rw-r--r-- | docs/guides/longer-metrics-storage.md | 13 |
2 files changed, 33 insertions, 35 deletions
diff --git a/database/engine/README.md b/database/engine/README.md index eea3c6de87..d50653943b 100644 --- a/database/engine/README.md +++ b/database/engine/README.md @@ -1,6 +1,6 @@ <!-- title: "Database engine" -description: "The highly-efficient database engine stores per-second metrics in RAM and then spills historical metrics to disk long-term storage." +description: "Netdata's highly-efficient database engine use both RAM and disk for distributed, long-term storage of per-second metrics." custom_edit_url: https://github.com/netdata/netdata/edit/master/database/engine/README.md --> @@ -26,8 +26,8 @@ To use the database engine, open `netdata.conf` and set `memory mode` to `dbengi memory mode = dbengine ``` -To configure the database engine, look for the `page cache size` and `dbengine disk space` settings in the `[global]` -section of your `netdata.conf`. The Agent ignores the `history` setting when using the database engine. +To configure the database engine, look for the `page cache size` and `dbengine multihost disk space` settings in the +`[global]` section of your `netdata.conf`. The Agent ignores the `history` setting when using the database engine. ```conf [global] @@ -43,51 +43,48 @@ actual page cache size will be slightly larger than this figure—see the [memor section for details. The `dbengine multihost disk space` option determines the amount of disk space in **MiB** that is dedicated to storing -Netdata metric values and all related metadata describing them. +Netdata metric values and all related metadata describing them. You can use the [**database engine +calculator**](https://learn.netdata.cloud/docs/agent/database/calculator) to correctly set `dbengine multihost disk +space` based on your metrics retention policy. The calculator gives an accurate estimate based on how many child nodes +you have, how many metrics your Agent collects, and more. ### Legacy configuration The deprecated `dbengine disk space` option determines the amount of disk space in **MiB** that is dedicated to storing -Netdata metric values per legacy database engine instance (see [below](#Streaming-metrics-to-the-database-engine)). +Netdata metric values per legacy database engine instance (see [details on the legacy mode](#legacy-mode) below). ```conf [global] dbengine disk space = 256 ``` -Use the [**database engine calculator**](https://learn.netdata.cloud/docs/agent/database/calculator) to correctly set -`dbengine disk space`(**deprecated**) based on your needs. The calculator gives an accurate estimate based on how many -child nodes you have, how many metrics your Agent collects, and more. - ### Streaming metrics to the database engine -##### Legacy mode - -When streaming metrics, the Agent on the parent node used to create one instance (legacy, version <= 1.23.2) of the -database engine for itself, and another instance for every child node it receives metrics from. If you had four -streaming nodes, you would have five instances in total (`1 parent + 4 child nodes = 5 instances`). +When using the multihost database engine, all parent and child nodes share the same `page cache size` and `dbengine +multihost disk space` in a single dbengine instance. The [**database engine +calculator**](https://learn.netdata.cloud/docs/agent/database/calculator) helps you properly set `page cache size` and +`dbengine multihost disk space` on your parent node to allocate enough resources based on your metrics retention policy +and how many child nodes you have. -The Agent allocated resources for each instance separately using the `dbengine disk space`(**deprecated**) setting. If -`dbengine disk space`(**deprecated**) is set to the default `256`, each instance is given 256 MiB in disk space, which -means the total disk space required to store all instances is, roughly, `256 MiB * 1 parent * 4 child nodes = 1280 MiB`. +#### Legacy mode -See the [database engine calculator](https://learn.netdata.cloud/docs/agent/database/calculator) to help you correctly -set `dbengine disk space`(**deprecated**) and understand the total disk space required based on your streaming setup. +_For Netdata Agents earlier than v1.23.2_, the Agent on the parent node uses one dbengine instance for itself, and +another instance for every child node it receives metrics from. If you had four streaming nodes, you would have five +instances in total (`1 parent + 4 child nodes = 5 instances`). -##### Multi host DB mode - -In the newer agent versions the parent and child nodes all share `page cache size` and `dbengine multihost disk space` -in a single dbengine multi-host instance. +The Agent allocates resources for each instance separately using the `dbengine disk space` (**deprecated**) setting. If +`dbengine disk space`(**deprecated**) is set to the default `256`, each instance is given 256 MiB in disk space, which +means the total disk space required to store all instances is, roughly, `256 MiB * 1 parent * 4 child nodes = 1280 MiB`. -##### Backward compatibility +#### Backward compatibility All existing metrics belonging to child nodes are automatically converted to legacy dbengine instances and the localhost -metrics are transferred to the multi-host dbengine instance. +metrics are transferred to the multihost dbengine instance. -All new child nodes are automatically transferred to the mult-host dbengine instance and share its page cache and disk -space. If you want to migrate a child node from its legacy dbengine instance to the multi-host dbengine instance you -must delete the instance's directory located in `/var/cache/netdata/MACHINE_GUID/dbengine` after stopping the netdata -agent. +All new child nodes are automatically transferred to the multihost dbengine instance and share its page cache and disk +space. If you want to migrate a child node from its legacy dbengine instance to the multihost dbengine instance, you +must delete the instance's directory, which is located in `/var/cache/netdata/MACHINE_GUID/dbengine`, after stopping the +Agent. ##### Information diff --git a/docs/guides/longer-metrics-storage.md b/docs/guides/longer-metrics-storage.md index 328b724019..5bf335c5cd 100644 --- a/docs/guides/longer-metrics-storage.md +++ b/docs/guides/longer-metrics-storage.md @@ -43,21 +43,22 @@ When the Netdata dashboard queries for historical metrics, the database engine w return relevant metrics for visualization in charts. Now, given that the database engine uses _both_ RAM and disk, there are two other settings to consider: `page cache -size` and `dbengine disk space`. +size` and `dbengine multihost disk space`. ```conf [global] page cache size = 32 - dbengine disk space = 256 + dbengine multihost disk space = 256 ``` `page cache size` sets the maximum amount of RAM (in MiB) the database engine will use for caching and indexing. -`dbengine disk space` sets the maximum disk space (again, in MiB) the database engine will use for storing compressed -metrics. The default settings retain about two day's worth of metris on a system collecting 2,000 metrics every second. +`dbengine multihost disk space` sets the maximum disk space (again, in MiB) the database engine will use for storing +compressed metrics. The default settings retain about two day's worth of metris on a system collecting 2,000 metrics +every second. [**See our database engine calculator**](https://learn.netdata.cloud/docs/agent/database/calculator) to help you -correctly set `dbengine disk space` based on your needs. The calculator gives an accurate estimate based on how many -child nodes you have, how many metrics your Agent collects, and more. +correctly set `dbengine multihost disk space` based on your needs. The calculator gives an accurate estimate based on +how many child nodes you have, how many metrics your Agent collects, and more. With the database engine active, you can back up your `/var/cache/netdata/dbengine/` folder to another location for redundancy. |