Database engine (#5282)

* Database engine prototype version 0

* Database engine initial integration with netdata POC

* Scalable database engine with file and memory management.

* Database engine integration with netdata

* Added MIN MAX definitions to fix alpine build of travis CI

* Bugfix for backends and new DB engine, remove useless rrdset_time2slot() calls and erroneous checks

* DB engine disk protocol correction

* Moved DB engine storage file location to /var/cache/netdata/{host}/dbengine

* Fix configure to require openSSL for DB engine

* Fix netdata daemon health not holding read lock when iterating chart dimensions

* Optimized query API for new DB engine and old netdata DB fallback code-path

* netdata database internal query API improvements and cleanup

* Bugfix for DB engine queries returning empty values

* Added netdata internal check for data queries for old and new DB

* Added statistics to DB engine and fixed memory corruption bug

* Added preliminary charts for DB engine statistics

* Changed DB engine ratio statistics to incremental

* Added netdata statistics charts for DB engine internal statistics

* Fix for netdata not compiling successfully when missing dbengine dependencies

* Added DB engine functional test to netdata unittest command parameter

* Implemented DB engine dataset generator based on example.random chart

* Fix build error in CI

* Support older versions of libuv1

* Fixes segmentation fault when using multiple DB engine instances concurrently

* Fix memory corruption bug

* Fixed createdataset advanced option not exiting

* Fix for DB engine not working on FreeBSD

* Support FreeBSD library paths of new dependencies

* Workaround for unsupported O_DIRECT in OS X

* Fix unittest crashing during cleanup

* Disable DB engine FS caching in Apple OS X since O_DIRECT is not available

* Fix segfault when unittest and DB engine dataset generator don't have permissions to create temporary host

* Modified DB engine dataset generator to create multiple files

* Toned down overzealous page cache prefetcher

* Reduce internal memory fragmentation for page-cache data pages

* Added documentation describing the DB engine

* Documentation bugfixes

* Fixed unit tests compilation errors since last rebase

* Added note to back-up the DB engine files in documentation

* Added codacy fix.

* Support old gcc versions for atomic counters in DB engine

6 files changed, 473 insertions, 2 deletions

diff --git a/daemon/README.md b/daemon/README.md
index 858394c770..62cc8c3b2b 100644
--- a/daemon/README.md
+++ b/daemon/README.md
@@ -164,6 +164,8 @@ The command line options of the netdata 1.10.0 version are the following:
 
   -W unittest              Run internal unittests and exit.
 
+  -W createdataset=N       Create a DB engine dataset of N seconds and exit.
+
   -W set section option value
                            set netdata.conf option from the command line.
 
diff --git a/daemon/config/README.md b/daemon/config/README.md
index 44bca2b2df..0508a19d99 100644
--- a/daemon/config/README.md
+++ b/daemon/config/README.md
@@ -57,7 +57,7 @@ cache directory | `/var/cache/netdata` | The directory the memory database will
 lib directory | `/var/lib/netdata` | Contains the alarm log and the netdata instance guid.
 home directory | `/var/cache/netdata` | Contains the db files for the collected metrics
 plugins directory | `"/usr/libexec/netdata/plugins.d" "/etc/netdata/custom-plugins.d"` | The directory plugin programs are kept. This setting supports multiple directories, space separated. If any directory path contains spaces, enclose it in single or double quotes.
-memory mode | `save` | When set to `save` netdata will save its round robin database on exit and load it on startup. When set to `map` the cache files will be updated in real time (check `man mmap` - do not set this on systems with heavy load or slow disks - the disks will continuously sync the in-memory database of netdata). When set to `ram` the round robin database will be temporary and it will be lost when netdata exits. `none` disables the database at this host. This also disables health monitoring (there cannot be health monitoring without a database). host access prefix | | This is used in docker environments where /proc, /sys, etc have to be accessed via another path. You may also have to set SYS_PTRACE capability on the docker for this work. Check [issue 43](https://github.com/netdata/netdata/issues/43).
+memory mode | `save` | When set to `save` netdata will save its round robin database on exit and load it on startup. When set to `map` the cache files will be updated in real time (check `man mmap` - do not set this on systems with heavy load or slow disks - the disks will continuously sync the in-memory database of netdata). When set to `dbengine` it behaves similarly to `map` but with much better disk and memory efficiency, however, with higher overhead. When set to `ram` the round robin database will be temporary and it will be lost when netdata exits. `none` disables the database at this host. This also disables health monitoring (there cannot be health monitoring without a database). host access prefix | | This is used in docker environments where /proc, /sys, etc have to be accessed via another path. You may also have to set SYS_PTRACE capability on the docker for this work. Check [issue 43](https://github.com/netdata/netdata/issues/43).
 memory deduplication (ksm) | `yes` | When set to `yes`, netdata will offer its in-memory round robin database to kernel same page merging (KSM) for deduplication. For more information check [Memory Deduplication - Kernel Same Page Merging - KSM](../../database/#ksm)
 TZ environment variable | `:/etc/localtime` | Where to find the timezone
 timezone | auto-detected | The timezone retrieved from the environment variable
diff --git a/daemon/global_statistics.c b/daemon/global_statistics.c
index 9933d0dfb2..9cc05abb7b 100644
--- a/daemon/global_statistics.c
+++ b/daemon/global_statistics.c
@@ -530,4 +530,223 @@ void global_statistics_charts(void) {
 
         rrdset_done(st_rrdr_points);
     }
+
+    // ----------------------------------------------------------------
+
+#ifdef ENABLE_DBENGINE
+    if (localhost->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) {
+        unsigned long long stats_array[27];
+
+        /* get localhost's DB engine's statistics */
+        rrdeng_get_27_statistics(localhost->rrdeng_ctx, stats_array);
+
+        // ----------------------------------------------------------------
+
+        {
+            static RRDSET *st_compression = NULL;
+            static RRDDIM *rd_savings = NULL;
+
+            if (unlikely(!st_compression)) {
+                st_compression = rrdset_create_localhost(
+                        "netdata"
+                        , "dbengine_compression_ratio"
+                        , NULL
+                        , "dbengine"
+                        , NULL
+                        , "NetData DB engine data extents' compression savings ratio"
+                        , "percentage"
+                        , "netdata"
+                        , "stats"
+                        , 130502
+                        , localhost->rrd_update_every
+                        , RRDSET_TYPE_LINE
+                );
+
+                rd_savings = rrddim_add(st_compression, "savings", NULL, 1, 1000, RRD_ALGORITHM_ABSOLUTE);
+            }
+            else
+                rrdset_next(st_compression);
+
+            unsigned long long ratio;
+            unsigned long long compressed_content_size = stats_array[12];
+            unsigned long long content_size = stats_array[11];
+
+            if (content_size) {
+                // allow negative savings
+                ratio = ((content_size - compressed_content_size) * 100 * 1000) / content_size;
+            } else {
+                ratio = 0;
+            }
+            rrddim_set_by_pointer(st_compression, rd_savings, ratio);
+
+            rrdset_done(st_compression);
+        }
+
+        // ----------------------------------------------------------------
+
+        {
+            static RRDSET *st_pg_cache_hit_ratio = NULL;
+            static RRDDIM *rd_hit_ratio = NULL;
+
+            if (unlikely(!st_pg_cache_hit_ratio)) {
+                st_pg_cache_hit_ratio = rrdset_create_localhost(
+                        "netdata"
+                        , "page_cache_hit_ratio"
+                        , NULL
+                        , "dbengine"
+                        , NULL
+                        , "NetData DB engine page cache hit ratio"
+                        , "percentage"
+                        , "netdata"
+                        , "stats"
+                        , 130503
+                        , localhost->rrd_update_every
+                        , RRDSET_TYPE_LINE
+                );
+
+                rd_hit_ratio = rrddim_add(st_pg_cache_hit_ratio, "ratio", NULL, 1, 1000, RRD_ALGORITHM_ABSOLUTE);
+            }
+            else
+                rrdset_next(st_pg_cache_hit_ratio);
+
+            static unsigned long long old_hits = 0;
+            static unsigned long long old_misses = 0;
+            unsigned long long hits = stats_array[7];
+            unsigned long long misses = stats_array[8];
+            unsigned long long hits_delta;
+            unsigned long long misses_delta;
+            unsigned long long ratio;
+
+            hits_delta = hits - old_hits;
+            misses_delta = misses - old_misses;
+            old_hits = hits;
+            old_misses = misses;
+
+            if (hits_delta + misses_delta) {
+                // allow negative savings
+                ratio = (hits_delta * 100 * 1000) / (hits_delta + misses_delta);
+            } else {
+                ratio = 0;
+            }
+            rrddim_set_by_pointer(st_pg_cache_hit_ratio, rd_hit_ratio, ratio);
+
+            rrdset_done(st_pg_cache_hit_ratio);
+        }
+
+        // ----------------------------------------------------------------
+
+        {
+            static RRDSET *st_pg_cache_pages = NULL;
+            static RRDDIM *rd_populated = NULL;
+            static RRDDIM *rd_commited = NULL;
+            static RRDDIM *rd_insertions = NULL;
+            static RRDDIM *rd_deletions = NULL;
+            static RRDDIM *rd_backfills = NULL;
+            static RRDDIM *rd_evictions = NULL;
+
+            if (unlikely(!st_pg_cache_pages)) {
+                st_pg_cache_pages = rrdset_create_localhost(
+                        "netdata"
+                        , "page_cache_stats"
+                        , NULL
+                        , "dbengine"
+                        , NULL
+                        , "NetData DB engine page statistics"
+                        , "pages"
+                        , "netdata"
+                        , "stats"
+                        , 130504
+                        , localhost->rrd_update_every
+                        , RRDSET_TYPE_LINE
+                );
+
+                rd_populated = rrddim_add(st_pg_cache_pages, "populated", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
+                rd_commited = rrddim_add(st_pg_cache_pages, "commited", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
+                rd_insertions = rrddim_add(st_pg_cache_pages, "insertions", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
+                rd_deletions = rrddim_add(st_pg_cache_pages, "deletions", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
+                rd_backfills = rrddim_add(st_pg_cache_pages, "backfills", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
+                rd_evictions = rrddim_add(st_pg_cache_pages, "evictions", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
+            }
+            else
+                rrdset_next(st_pg_cache_pages);
+
+            rrddim_set_by_pointer(st_pg_cache_pages, rd_populated, (collected_number)stats_array[3]);
+            rrddim_set_by_pointer(st_pg_cache_pages, rd_commited, (collected_number)stats_array[4]);
+            rrddim_set_by_pointer(st_pg_cache_pages, rd_insertions, (collected_number)stats_array[5]);
+            rrddim_set_by_pointer(st_pg_cache_pages, rd_deletions, (collected_number)stats_array[6]);
+            rrddim_set_by_pointer(st_pg_cache_pages, rd_backfills, (collected_number)stats_array[9]);
+            rrddim_set_by_pointer(st_pg_cache_pages, rd_evictions, (collected_number)stats_array[10]);
+            rrdset_done(st_pg_cache_pages);
+        }
+
+        // ----------------------------------------------------------------
+
+        {
+            static RRDSET *st_io_stats = NULL;
+            static RRDDIM *rd_reads = NULL;
+            static RRDDIM *rd_writes = NULL;
+
+            if (unlikely(!st_io_stats)) {
+                st_io_stats = rrdset_create_localhost(
+                "netdata"
+                , "dbengine_io_throughput"
+                , NULL
+                , "dbengine"
+                , NULL
+                , "NetData DB engine I/O throughput"
+                , "MiB/s"
+                , "netdata"
+                , "stats"
+                , 130505
+                , localhost->rrd_update_every
+                , RRDSET_TYPE_LINE
+                );
+
+                rd_reads = rrddim_add(st_io_stats, "reads", NULL, 1, 1024 * 1024, RRD_ALGORITHM_INCREMENTAL);
+                rd_writes = rrddim_add(st_io_stats, "writes", NULL, -1, 1024 * 1024, RRD_ALGORITHM_INCREMENTAL);
+            }
+            else
+                rrdset_next(st_io_stats);
+
+            rrddim_set_by_pointer(st_io_stats, rd_reads, (collected_number)stats_array[17]);
+            rrddim_set_by_pointer(st_io_stats, rd_writes, (collected_number)stats_array[15]);
+            rrdset_done(st_io_stats);
+        }
+
+        // ----------------------------------------------------------------
+
+        {
+            static RRDSET *st_io_stats = NULL;
+            static RRDDIM *rd_reads = NULL;
+            static RRDDIM *rd_writes = NULL;
+
+            if (unlikely(!st_io_stats)) {
+                st_io_stats = rrdset_create_localhost(
+                        "netdata"
+                        , "dbengine_io_operations"
+                        , NULL
+                        , "dbengine"
+                        , NULL
+                        , "NetData DB engine I/O operations"
+                        , "operations/s"
+                        , "netdata"
+                        , "stats"
+                        , 130506
+                        , localhost->rrd_update_every
+                        , RRDSET_TYPE_LINE
+                );
+
+                rd_reads = rrddim_add(st_io_stats, "reads", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
+                rd_writes = rrddim_add(st_io_stats, "writes", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
+            }
+            else
+                rrdset_next(st_io_stats);
+
+            rrddim_set_by_pointer(st_io_stats, rd_reads, (collected_number)stats_array[18]);
+            rrddim_set_by_pointer(st_io_stats, rd_writes, (collected_number)stats_array[16]);
+            rrdset_done(st_io_stats);
+        }
+    }
+#endif
+
 }
diff --git a/daemon/main.c b/daemon/main.c
index fe9e9963c1..a1577fb937 100644
--- a/daemon/main.c
+++ b/daemon/main.c
@@ -301,6 +301,7 @@ int help(int exitcode) {
             "  -W stacksize=N           Set the stacksize (in bytes).\n\n"
             "  -W debug_flags=N         Set runtime tracing to debug.log.\n\n"
             "  -W unittest              Run internal unittests and exit.\n\n"
+            "  -W createdataset=N       Create a DB engine dataset of N seconds and exit.\n\n"
             "  -W set section option value\n"
             "                           set netdata.conf option from the command line.\n\n"
             "  -W simple-pattern pattern string\n"
@@ -471,6 +472,25 @@ static void get_netdata_configured_variables() {
 
     default_rrd_memory_mode = rrd_memory_mode_id(config_get(CONFIG_SECTION_GLOBAL, "memory mode", rrd_memory_mode_name(default_rrd_memory_mode)));
 
+#ifdef ENABLE_DBENGINE
+    // ------------------------------------------------------------------------
+    // get default Database Engine page cache size in MiB
+
+    default_rrdeng_page_cache_mb = (int) config_get_number(CONFIG_SECTION_GLOBAL, "page cache size", default_rrdeng_page_cache_mb);
+    if(default_rrdeng_page_cache_mb < RRDENG_MIN_PAGE_CACHE_SIZE_MB) {
+        error("Invalid page cache size %d given. Defaulting to %d.", default_rrdeng_page_cache_mb, RRDENG_MIN_PAGE_CACHE_SIZE_MB);
+        default_rrdeng_page_cache_mb = RRDENG_MIN_PAGE_CACHE_SIZE_MB;
+    }
+
+    // ------------------------------------------------------------------------
+    // get default Database Engine disk space quota in MiB
+
+    default_rrdeng_disk_quota_mb = (int) config_get_number(CONFIG_SECTION_GLOBAL, "dbengine disk space", default_rrdeng_disk_quota_mb);
+    if(default_rrdeng_disk_quota_mb < RRDENG_MIN_DISK_SPACE_MB) {
+        error("Invalid dbengine disk space %d given. Defaulting to %d.", default_rrdeng_disk_quota_mb, RRDENG_MIN_DISK_SPACE_MB);
+        default_rrdeng_disk_quota_mb = RRDENG_MIN_DISK_SPACE_MB;
+    }
+#endif
     // ------------------------------------------------------------------------
 
     netdata_configured_host_prefix = config_get(CONFIG_SECTION_GLOBAL, "host access prefix", "");
@@ -841,6 +861,7 @@ int main(int argc, char **argv) {
                     {
                         char* stacksize_string = "stacksize=";
                         char* debug_flags_string = "debug_flags=";
+                        char* createdataset_string = "createdataset=";
 
                         if(strcmp(optarg, "unittest") == 0) {
                             if(unit_test_buffer()) return 1;
@@ -853,9 +874,23 @@ int main(int argc, char **argv) {
                             default_rrdpush_enabled = 0;
                             if(run_all_mockup_tests()) return 1;
                             if(unit_test_storage()) return 1;
+#ifdef ENABLE_DBENGINE
+                            if(test_dbengine()) return 1;
+#endif
                             fprintf(stderr, "\n\nALL TESTS PASSED\n\n");
                             return 0;
                         }
+                        else if(strncmp(optarg, createdataset_string, strlen(createdataset_string)) == 0) {
+                            unsigned history_seconds;
+
+                            optarg += strlen(createdataset_string);
+                            history_seconds = (unsigned )strtoull(optarg, NULL, 0);
+
+#ifdef ENABLE_DBENGINE
+                            generate_dbengine_dataset(history_seconds);
+#endif
+                            return 0;
+                        }
                         else if(strcmp(optarg, "simple-pattern") == 0) {
                             if(optind + 2 > argc) {
                                 fprintf(stderr, "%s", "\nUSAGE: -W simple-pattern 'pattern' 'string'\n\n"
@@ -1138,7 +1173,6 @@ int main(int argc, char **argv) {
 
     rrd_init(netdata_configured_hostname, system_info);
     rrdhost_system_info_free(system_info);
-
     // ------------------------------------------------------------------------
     // enable log flood protection
 
diff --git a/daemon/unit_test.c b/daemon/unit_test.c
index a92a50a111..f9b58ce6b8 100644
--- a/daemon/unit_test.c
+++ b/daemon/unit_test.c
@@ -1566,3 +1566,215 @@ int unit_test(long delay, long shift)
 
     return ret;
 }
+
+#ifdef ENABLE_DBENGINE
+static inline void rrddim_set_by_pointer_fake_time(RRDDIM *rd, collected_number value, time_t now)
+{
+    rd->last_collected_time.tv_sec = now;
+    rd->last_collected_time.tv_usec = 0;
+    rd->collected_value = value;
+    rd->updated = 1;
+
+    rd->collections_counter++;
+
+    collected_number v = (value >= 0) ? value : -value;
+    if(unlikely(v > rd->collected_value_max)) rd->collected_value_max = v;
+}
+
+int test_dbengine(void)
+{
+    const int CHARTS = 128;
+    const int DIMS = 16; /* That gives us 2048 metrics */
+    const int POINTS = 16384; /* This produces 128MiB of metric data */
+    const int QUERY_BATCH = 4096;
+    uint8_t same;
+    int i, j, k, c, errors;
+    RRDHOST *host = NULL;
+    RRDSET *st[CHARTS];
+    RRDDIM *rd[CHARTS][DIMS];
+    char name[101];
+    time_t time_now;
+    collected_number last;
+    struct rrddim_query_handle handle;
+    calculated_number value, expected;
+    storage_number n;
+
+    error_log_limit_unlimited();
+    fprintf(stderr, "\nRunning DB-engine test\n");
+
+    default_rrd_memory_mode = RRD_MEMORY_MODE_DBENGINE;
+
+    debug(D_RRDHOST, "Initializing localhost with hostname 'unittest-dbengine'");
+    host = rrdhost_find_or_create(
+            "unittest-dbengine"
+            , "unittest-dbengine"
+            , "unittest-dbengine"
+            , os_type
+            , netdata_configured_timezone
+            , config_get(CONFIG_SECTION_BACKEND, "host tags", "")
+            , program_name
+            , program_version
+            , default_rrd_update_every
+            , default_rrd_history_entries
+            , RRD_MEMORY_MODE_DBENGINE
+            , default_health_enabled
+            , default_rrdpush_enabled
+            , default_rrdpush_destination
+            , default_rrdpush_api_key
+            , default_rrdpush_send_charts_matching
+            , NULL
+    );
+    if (NULL == host)
+        return 1;
+
+    for (i = 0 ; i < CHARTS ; ++i) {
+        snprintfz(name, 100, "dbengine-chart-%d", i);
+
+        // create the chart
+        st[i] = rrdset_create(host, "netdata", name, name, "netdata", NULL, "Unit Testing", "a value", "unittest",
+                NULL, 1, 1, RRDSET_TYPE_LINE);
+        rrdset_flag_set(st[i], RRDSET_FLAG_DEBUG);
+        rrdset_flag_set(st[i], RRDSET_FLAG_STORE_FIRST);
+        for (j = 0 ; j < DIMS ; ++j) {
+            snprintfz(name, 100, "dim-%d", j);
+
+            rd[i][j] = rrddim_add(st[i], name, NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
+        }
+    }
+
+    // feed it with the test data
+    time_now = 1;
+    last = 0;
+    for (i = 0 ; i < CHARTS ; ++i) {
+        for (j = 0 ; j < DIMS ; ++j) {
+            rd[i][j]->last_collected_time.tv_sec =
+            st[i]->last_collected_time.tv_sec = st[i]->last_updated.tv_sec = time_now;
+            rd[i][j]->last_collected_time.tv_usec =
+                    st[i]->last_collected_time.tv_usec = st[i]->last_updated.tv_usec = 0;
+        }
+    }
+    for(c = 0; c < POINTS ; ++c) {
+        ++time_now; // time_now = c + 2
+        for (i = 0 ; i < CHARTS ; ++i) {
+            st[i]->usec_since_last_update = USEC_PER_SEC;
+
+            for (j = 0; j < DIMS; ++j) {
+                last = i * DIMS * POINTS + j * POINTS + c;
+                rrddim_set_by_pointer_fake_time(rd[i][j], last, time_now);
+            }
+            rrdset_done(st[i]);
+        }
+    }
+
+    // check the result
+    errors = 0;
+
+    for(c = 0; c < POINTS ; c += QUERY_BATCH) {
+        time_now = c + 2;
+        for (i = 0 ; i < CHARTS ; ++i) {
+            for (j = 0; j < DIMS; ++j) {
+                rd[i][j]->state->query_ops.init(rd[i][j], &handle, time_now, time_now + QUERY_BATCH);
+                for (k = 0; k < QUERY_BATCH; ++k) {
+                    last = i * DIMS * POINTS + j * POINTS + c + k;
+                    expected = unpack_storage_number(pack_storage_number((calculated_number)last, SN_EXISTS));
+
+                    n = rd[i][j]->state->query_ops.next_metric(&handle);
+                    value = unpack_storage_number(n);
+
+                    same = (calculated_number_round(value * 10000000.0) == calculated_number_round(expected * 10000000.0)) ? 1 : 0;
+                    if(!same) {
+                        fprintf(stderr, "    DB-engine unittest %s/%s: at %lu secs, expecting value "
+                                        CALCULATED_NUMBER_FORMAT ", found " CALCULATED_NUMBER_FORMAT ", ### E R R O R ###\n",
+                                st[i]->name, rd[i][j]->name, (unsigned long)time_now + k, expected, value);
+                        errors++;
+                    }
+                }
+                rd[i][j]->state->query_ops.finalize(&handle);
+            }
+        }
+    }
+
+    rrdeng_exit(host->rrdeng_ctx);
+    rrd_wrlock();
+    rrdhost_delete_charts(host);
+    rrd_unlock();
+
+    return errors;
+}
+
+void generate_dbengine_dataset(unsigned history_seconds)
+{
+    const int DIMS = 128;
+    const uint64_t EXPECTED_COMPRESSION_RATIO = 94;
+    int j;
+    RRDHOST *host = NULL;
+    RRDSET *st;
+    RRDDIM *rd[DIMS];
+    char name[101];
+    time_t time_current, time_present;
+
+    default_rrd_memory_mode = RRD_MEMORY_MODE_DBENGINE;
+    default_rrdeng_page_cache_mb = 128;
+    /* Worst case for uncompressible data */
+    default_rrdeng_disk_quota_mb = (((uint64_t)DIMS) * sizeof(storage_number) * history_seconds) / (1024 * 1024);
+    default_rrdeng_disk_quota_mb -= default_rrdeng_disk_quota_mb * EXPECTED_COMPRESSION_RATIO / 100;
+
+    error_log_limit_unlimited();
+    debug(D_RRDHOST, "Initializing localhost with hostname 'dbengine-dataset'");
+
+    host = rrdhost_find_or_create(
+            "dbengine-dataset"
+            , "dbengine-dataset"
+            , "dbengine-dataset"
+            , os_type
+            , netdata_configured_timezone
+            , config_get(CONFIG_SECTION_BACKEND, "host tags", "")
+            , program_name
+            , program_version
+            , default_rrd_update_every
+            , default_rrd_history_entries
+            , RRD_MEMORY_MODE_DBENGINE
+            , default_health_enabled
+            , default_rrdpush_enabled
+            , default_rrdpush_destination
+            , default_rrdpush_api_key
+            , default_rrdpush_send_charts_matching
+            , NULL
+    );
+    if (NULL == host)
+        return;
+
+    fprintf(stderr, "\nRunning DB-engine workload generator\n");
+
+    // create the chart
+    st = rrdset_create(host, "example", "random", "random", "example", NULL, "random", "random", "random",
+                          NULL, 1, 1, RRDSET_TYPE_LINE);
+    for (j = 0 ; j < DIMS ; ++j) {
+        snprintfz(name, 100, "random%d", j);
+
+        rd[j] = rrddim_add(st, name, NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
+    }
+
+    time_present = now_realtime_sec();
+    // feed it with the test data
+    time_current = time_present - history_seconds;
+    for (j = 0 ; j < DIMS ; ++j) {
+        rd[j]->last_collected_time.tv_sec =
+        st->last_collected_time.tv_sec = st->last_updated.tv_sec = time_current;
+        rd[j]->last_collected_time.tv_usec =
+        st->last_collected_time.tv_usec = st->last_updated.tv_usec = 0;
+    }
+    for( ; time_current < time_present; ++time_current) {
+        st->usec_since_last_update = USEC_PER_SEC;
+
+        for (j = 0; j < DIMS; ++j) {
+            rrddim_set_by_pointer_fake_time(rd[j], (time_current + j) % 128, time_current);
+        }
+        rrdset_done(st);
+    }
+    rrd_wrlock();
+    rrdhost_free(host);
+    rrd_unlock();
+
+}
+#endif
\ No newline at end of file
diff --git a/daemon/unit_test.h b/daemon/unit_test.h
index 0023c8de97..fd3e801705 100644
--- a/daemon/unit_test.h
+++ b/daemon/unit_test.h
@@ -8,5 +8,9 @@ extern int unit_test(long delay, long shift);
 extern int run_all_mockup_tests(void);
 extern int unit_test_str2ld(void);
 extern int unit_test_buffer(void);
+#ifdef ENABLE_DBENGINE
+extern int test_dbengine(void);
+extern void generate_dbengine_dataset(unsigned history_seconds);
+#endif
 
 #endif /* NETDATA_UNIT_TEST_H */