Database engine (#5282)

* Database engine prototype version 0

* Database engine initial integration with netdata POC

* Scalable database engine with file and memory management.

* Database engine integration with netdata

* Added MIN MAX definitions to fix alpine build of travis CI

* Bugfix for backends and new DB engine, remove useless rrdset_time2slot() calls and erroneous checks

* DB engine disk protocol correction

* Moved DB engine storage file location to /var/cache/netdata/{host}/dbengine

* Fix configure to require openSSL for DB engine

* Fix netdata daemon health not holding read lock when iterating chart dimensions

* Optimized query API for new DB engine and old netdata DB fallback code-path

* netdata database internal query API improvements and cleanup

* Bugfix for DB engine queries returning empty values

* Added netdata internal check for data queries for old and new DB

* Added statistics to DB engine and fixed memory corruption bug

* Added preliminary charts for DB engine statistics

* Changed DB engine ratio statistics to incremental

* Added netdata statistics charts for DB engine internal statistics

* Fix for netdata not compiling successfully when missing dbengine dependencies

* Added DB engine functional test to netdata unittest command parameter

* Implemented DB engine dataset generator based on example.random chart

* Fix build error in CI

* Support older versions of libuv1

* Fixes segmentation fault when using multiple DB engine instances concurrently

* Fix memory corruption bug

* Fixed createdataset advanced option not exiting

* Fix for DB engine not working on FreeBSD

* Support FreeBSD library paths of new dependencies

* Workaround for unsupported O_DIRECT in OS X

* Fix unittest crashing during cleanup

* Disable DB engine FS caching in Apple OS X since O_DIRECT is not available

* Fix segfault when unittest and DB engine dataset generator don't have permissions to create temporary host

* Modified DB engine dataset generator to create multiple files

* Toned down overzealous page cache prefetcher

* Reduce internal memory fragmentation for page-cache data pages

* Added documentation describing the DB engine

* Documentation bugfixes

* Fixed unit tests compilation errors since last rebase

* Added note to back-up the DB engine files in documentation

* Added codacy fix.

* Support old gcc versions for atomic counters in DB engine

2 files changed, 11 insertions, 6 deletions

diff --git a/streaming/README.md b/streaming/README.md
index e97411b31a..0ad9d7e2eb 100644
--- a/streaming/README.md
+++ b/streaming/README.md
@@ -73,7 +73,7 @@ These are options that affect the operation of netdata in this area:
 
 ```
 [global]
-    memory mode = none | ram | save | map
+    memory mode = none | ram | save | map | dbengine
 ```
 
 `[global].memory mode = none` disables the database at this host. This also disables health
@@ -170,6 +170,10 @@ the unique id the netdata generating the metrics (i.e. the netdata that original
 them `/var/lib/netdata/registry/netdata.unique.id`). So, metrics for netdata `A` that pass through
 any number of other netdata, will have the same `MACHINE_GUID`.
 
+You can also use `default memory mode = dbengine` for an API key or `memory mode = dbengine` for
+ a single host. The additional `page cache size` and `dbengine disk space` configuration options
+ are inherited from the global netdata configuration.
+
 ##### allow from
 
 `allow from` settings are [netdata simple patterns](../libnetdata/simple_pattern): string matches
diff --git a/streaming/stream.conf b/streaming/stream.conf
index 493eba37ca..d0d02a7c87 100644
--- a/streaming/stream.conf
+++ b/streaming/stream.conf
@@ -103,10 +103,11 @@
     # You can also set it per host below.
     # If you don't set it here, the memory mode of netdata.conf will be used.
     # Valid modes:
-    #    save    save on exit, load on start
-    #    map     like swap (continuously syncing to disks - you need SSD)
-    #    ram     keep it in RAM, don't touch the disk
-    #    none    no database at all (use this on headless proxies)
+    #    save     save on exit, load on start
+    #    map      like swap (continuously syncing to disks - you need SSD)
+    #    ram      keep it in RAM, don't touch the disk
+    #    none     no database at all (use this on headless proxies)
+    #    dbengine like a traditional database
     default memory mode = ram
 
     # Shall we enable health monitoring for the hosts using this API key?
@@ -167,7 +168,7 @@
     # The number of entries in the database
     history = 3600
 
-    # The memory mode of the database: save | map | ram | none
+    # The memory mode of the database: save | map | ram | none | dbengine
     memory mode = save
 
     # Health / alarms control: yes | no | auto