summaryrefslogtreecommitdiffstats
path: root/database/engine
diff options
context:
space:
mode:
authorPromise Akpan <akpanpromise@hotmail.com>2019-09-29 08:48:18 +0100
committerChris Akritidis <43294513+cakrit@users.noreply.github.com>2019-09-29 09:48:18 +0200
commit8982b9968e9763567ce1a20acca49c794dc91f9d (patch)
tree1dac77d3581f3811828936a7755b945a31b5755a /database/engine
parent0063c2126d02374928e6ae9d25926c2bf21a50f6 (diff)
Fix Remark Lint for READMEs in Database (#6942)
* fix remark lint Database engine * fix remark lint of database README * rewrap dbengine readme for consistency * rewrap database README * make character limit to 120 not 80
Diffstat (limited to 'database/engine')
-rw-r--r--database/engine/README.md131
1 files changed, 61 insertions, 70 deletions
diff --git a/database/engine/README.md b/database/engine/README.md
index 7791a549f8..12c22a92c2 100644
--- a/database/engine/README.md
+++ b/database/engine/README.md
@@ -1,18 +1,17 @@
# Database engine
-The Database Engine works like a traditional
-database. There is some amount of RAM dedicated to data caching and indexing and the rest of
-the data reside compressed on disk. The number of history entries is not fixed in this case,
-but depends on the configured disk space and the effective compression ratio of the data stored.
-This is the **only mode** that supports changing the data collection update frequency
-(`update_every`) **without losing** the previously stored metrics.
+The Database Engine works like a traditional database. There is some amount of RAM dedicated to data caching and
+indexing and the rest of the data reside compressed on disk. The number of history entries is not fixed in this case,
+but depends on the configured disk space and the effective compression ratio of the data stored. This is the **only
+mode** that supports changing the data collection update frequency (`update_every`) **without losing** the previously
+stored metrics.
## Files
-With the DB engine memory mode the metric data are stored in database files. These files are
-organized in pairs, the datafiles and their corresponding journalfiles, e.g.:
+With the DB engine memory mode the metric data are stored in database files. These files are organized in pairs, the
+datafiles and their corresponding journalfiles, e.g.:
-```
+```sh
datafile-1-0000000001.ndf
journalfile-1-0000000001.njf
datafile-1-0000000002.ndf
@@ -22,21 +21,19 @@ journalfile-1-0000000003.njf
...
```
-They are located under their host's cache directory in the directory `./dbengine`
-(e.g. for localhost the default location is `/var/cache/netdata/dbengine/*`). The higher
-numbered filenames contain more recent metric data. The user can safely delete some pairs
-of files when Netdata is stopped to manually free up some space.
+They are located under their host's cache directory in the directory `./dbengine` (e.g. for localhost the default
+location is `/var/cache/netdata/dbengine/*`). The higher numbered filenames contain more recent metric data. The user
+can safely delete some pairs of files when Netdata is stopped to manually free up some space.
_Users should_ **back up** _their `./dbengine` folders if they consider this data to be important._
## Configuration
-There is one DB engine instance per Netdata host/node. That is, there is one `./dbengine` folder
-per node, and all charts of `dbengine` memory mode in such a host share the same storage space
-and DB engine instance memory state. You can select the memory mode for localhost by editing
-netdata.conf and setting:
+There is one DB engine instance per Netdata host/node. That is, there is one `./dbengine` folder per node, and all
+charts of `dbengine` memory mode in such a host share the same storage space and DB engine instance memory state. You
+can select the memory mode for localhost by editing netdata.conf and setting:
-```
+```conf
[global]
memory mode = dbengine
```
@@ -44,57 +41,52 @@ netdata.conf and setting:
For setting the memory mode for the rest of the nodes you should look at
[streaming](../../streaming/).
-The `history` configuration option is meaningless for `memory mode = dbengine` and is ignored
-for any metrics being stored in the DB engine.
+The `history` configuration option is meaningless for `memory mode = dbengine` and is ignored for any metrics being
+stored in the DB engine.
-All DB engine instances, for localhost and all other streaming recipient nodes inherit their
-configuration from `netdata.conf`:
+All DB engine instances, for localhost and all other streaming recipient nodes inherit their configuration from
+`netdata.conf`:
-```
+```conf
[global]
page cache size = 32
dbengine disk space = 256
```
-The above values are the default and minimum values for Page Cache size and DB engine disk space
-quota. Both numbers are in **MiB**. All DB engine instances will allocate the configured resources
-separately.
+The above values are the default and minimum values for Page Cache size and DB engine disk space quota. Both numbers are
+in **MiB**. All DB engine instances will allocate the configured resources separately.
-The `page cache size` option determines the amount of RAM in **MiB** that is dedicated to caching
-Netdata metric values themselves.
+The `page cache size` option determines the amount of RAM in **MiB** that is dedicated to caching Netdata metric values
+themselves.
-The `dbengine disk space` option determines the amount of disk space in **MiB** that is dedicated
-to storing Netdata metric values and all related metadata describing them.
+The `dbengine disk space` option determines the amount of disk space in **MiB** that is dedicated to storing Netdata
+metric values and all related metadata describing them.
## Operation
-The DB engine stores chart metric values in 4096-byte pages in memory. Each chart dimension gets
-its own page to store consecutive values generated from the data collectors. Those pages comprise
-the **Page Cache**.
+The DB engine stores chart metric values in 4096-byte pages in memory. Each chart dimension gets its own page to store
+consecutive values generated from the data collectors. Those pages comprise the **Page Cache**.
-When those pages fill up they are slowly compressed and flushed to disk.
-It can take `4096 / 4 = 1024 seconds = 17 minutes`, for a chart dimension that is being collected
-every 1 second, to fill a page. Pages can be cut short when we stop Netdata or the DB engine
-instance so as to not lose the data. When we query the DB engine for data we trigger disk read
-I/O requests that fill the Page Cache with the requested pages and potentially evict cold
-(not recently used) pages.
+When those pages fill up they are slowly compressed and flushed to disk. It can take `4096 / 4 = 1024 seconds = 17
+minutes`, for a chart dimension that is being collected every 1 second, to fill a page. Pages can be cut short when we
+stop Netdata or the DB engine instance so as to not lose the data. When we query the DB engine for data we trigger disk
+read I/O requests that fill the Page Cache with the requested pages and potentially evict cold (not recently used)
+pages.
-When the disk quota is exceeded the oldest values are removed from the DB engine at real time, by
-automatically deleting the oldest datafile and journalfile pair. Any corresponding pages residing
-in the Page Cache will also be invalidated and removed. The DB engine logic will try to maintain
-between 10 and 20 file pairs at any point in time.
+When the disk quota is exceeded the oldest values are removed from the DB engine at real time, by automatically deleting
+the oldest datafile and journalfile pair. Any corresponding pages residing in the Page Cache will also be invalidated
+and removed. The DB engine logic will try to maintain between 10 and 20 file pairs at any point in time.
-The Database Engine uses direct I/O to avoid polluting the OS filesystem caches and does not
-generate excessive I/O traffic so as to create the minimum possible interference with other
-applications.
+The Database Engine uses direct I/O to avoid polluting the OS filesystem caches and does not generate excessive I/O
+traffic so as to create the minimum possible interference with other applications.
## Memory requirements
-Using memory mode `dbengine` we can overcome most memory restrictions and store a dataset that
-is much larger than the available memory.
+Using memory mode `dbengine` we can overcome most memory restrictions and store a dataset that is much larger than the
+available memory.
-There are explicit memory requirements **per** DB engine **instance**, meaning **per** Netdata
-**node** (e.g. localhost and streaming recipient nodes):
+There are explicit memory requirements **per** DB engine **instance**, meaning **per** Netdata **node** (e.g. localhost
+and streaming recipient nodes):
- `page cache size` must be at least `#dimensions-being-collected x 4096 x 2` bytes.
@@ -102,48 +94,47 @@ There are explicit memory requirements **per** DB engine **instance**, meaning *
- roughly speaking this is 3% of the uncompressed disk space taken by the DB files.
- - for very highly compressible data (compression ratio > 90%) this RAM overhead
- is comparable to the disk space footprint.
+ - for very highly compressible data (compression ratio > 90%) this RAM overhead is comparable to the disk space
+ footprint.
-An important observation is that RAM usage depends on both the `page cache size` and the
-`dbengine disk space` options.
+An important observation is that RAM usage depends on both the `page cache size` and the `dbengine disk space` options.
## File descriptor requirements
-The Database Engine may keep a **significant** amount of files open per instance (e.g. per streaming
-slave or master server). When configuring your system you should make sure there are at least 50
-file descriptors available per `dbengine` instance.
+The Database Engine may keep a **significant** amount of files open per instance (e.g. per streaming slave or master
+server). When configuring your system you should make sure there are at least 50 file descriptors available per
+`dbengine` instance.
-Netdata allocates 25% of the available file descriptors to its Database Engine instances. This means that only 25%
-of the file descriptors that are available to the Netdata service are accessible by dbengine instances.
-You should take that into account when configuring your service
-or system-wide file descriptor limits. You can roughly estimate that the Netdata service needs 2048 file
-descriptors for every 10 streaming slave hosts when streaming is configured to use `memory mode = dbengine`.
+Netdata allocates 25% of the available file descriptors to its Database Engine instances. This means that only 25% of
+the file descriptors that are available to the Netdata service are accessible by dbengine instances. You should take
+that into account when configuring your service or system-wide file descriptor limits. You can roughly estimate that the
+Netdata service needs 2048 file descriptors for every 10 streaming slave hosts when streaming is configured to use
+`memory mode = dbengine`.
-If for example one wants to allocate 65536 file descriptors to the Netdata service on a systemd system
-one needs to override the Netdata service by running `sudo systemctl edit netdata` and creating a
-file with contents:
+If for example one wants to allocate 65536 file descriptors to the Netdata service on a systemd system one needs to
+override the Netdata service by running `sudo systemctl edit netdata` and creating a file with contents:
-```
+```sh
[Service]
LimitNOFILE=65536
```
For other types of services one can add the line:
-```
+```sh
ulimit -n 65536
```
-at the beginning of the service file. Alternatively you can change the system-wide limits of the kernel by changing `/etc/sysctl.conf`. For linux that would be:
+at the beginning of the service file. Alternatively you can change the system-wide limits of the kernel by changing
+ `/etc/sysctl.conf`. For linux that would be:
-```
+```conf
fs.file-max = 65536
```
In FreeBSD and OS X you change the lines like this:
-```
+```conf
kern.maxfilesperproc=65536
kern.maxfiles=65536
```