Implemented multihost database (#9556)

* Hard code a node for non-legacy multidb test
Skip dbengine initialization for new incoming children
Add code to switch to multidb ctx when accessing the dbengine

* When a non-legacy streaming connection is detected, use the multidb metadata log context

* Clear the superblock memory to avoid random data written in the metadata log

* Activate the host detection during compaction
Activate the host detection during metadata log chart updates
Keep the host in the user object during replay of the HOST command

* Add defaults for health / rrdpush on HOST metadata replay
Check for legacy status on host creation by checking is_archived and if not conclusive, call is_legacy_child()

Use defaults from the stream.conf

* Count hosts only if not archived
When host switches from archived to active update rrd_hosts_available
Remove archived hosts from charts and info

* Change parameter from "multidb disk space" to "dbengine multihost disk space"
Remove unused variables
Fix compilation error when dbengine is disabled
Fix condition for machine_guid directory creation under cache_dir

* Enable multidb disk space file creation.

* Stop deleting dimensions when rotating archived metrics if the dimension is active in a different database engine.

* Fix old bug in the code that confused obsolete hosts with orphan hosts.

* Do not delete multi-host DB host files.

* Discard dbengine state when a legacy memory mode instantiates to avoid inconsistencies.

* Identify metadata that collide with non-dbengine memory mode hosts and ignore them.

* Handle non-dbengine localhost with dbengine archived charts in localhost and streaming.

* Ignore archived hosts in streaming.

* Add documentation before merging to master.

Co-authored-by: Markos Fountoulakis <markos.fountoulakis.senior@gmail.com>

13 files changed, 239 insertions, 101 deletions

diff --git a/database/engine/README.md b/database/engine/README.md
index 3a74b58541..6f5cc6a15d 100644
--- a/database/engine/README.md
+++ b/database/engine/README.md
@@ -32,35 +32,64 @@ section of your `netdata.conf`. The Agent ignores the `history` setting when usi
 ```conf
 [global]
     page cache size = 32
-    dbengine disk space = 256
+    dbengine multihost disk space = 256
 ```
 
-The above values are the default and minimum values for Page Cache size and DB engine disk space quota. Both numbers are
+The above values are the default values for Page Cache size and DB engine disk space quota. Both numbers are
 in **MiB**.
 
 The `page cache size` option determines the amount of RAM in **MiB** dedicated to caching Netdata metric values. The
 actual page cache size will be slightly larger than this figure—see the [memory requirements](#memory-requirements)
 section for details.
 
-The `dbengine disk space` option determines the amount of disk space in **MiB** that is dedicated to storing Netdata
-metric values and all related metadata describing them.
+The `dbengine multihost disk space` option determines the amount of disk space in **MiB** that is dedicated to storing
+Netdata metric values and all related metadata describing them.
+
+### Legacy configuration
+
+The deprecated `dbengine disk space` option determines the amount of disk space in **MiB** that is dedicated to storing
+Netdata metric values per legacy database engine instance (see [below](#Streaming-metrics-to-the-database-engine)).
+
+```conf
+[global]
+    dbengine disk space = 256
+```
 
 Use the  [**database engine calculator**](https://learn.netdata.cloud/docs/agent/database/calculator) to correctly set
-`dbengine disk space` based on your needs. The calculator gives an accurate estimate based on how many child nodes
-you have, how many metrics your Agent collects, and more.
+`dbengine disk space`(**deprecated**) based on your needs. The calculator gives an accurate estimate based on how many
+child nodes you have, how many metrics your Agent collects, and more.
 
 ### Streaming metrics to the database engine
 
-When streaming metrics, the Agent on the parent node creates one instance of the database engine for itself, and another
-instance for every child node it receives metrics from. If you have four streaming nodes, you will have five instances
-in total (`1 parent + 4 child nodes = 5 instances`).
+##### Legacy mode
 
-The Agent allocates resources for each instance separately using the `dbengine disk space` setting. If `dbengine disk
-space` is set to the default `256`, each instance is given 256 MiB in disk space, which means the total disk space
-required to store all instances is, roughly, `256 MiB * 1 parent * 4 child nodes = 1280 MiB`. 
+When streaming metrics, the Agent on the parent node used to create one instance (legacy, version <= 1.23.2) of the
+database engine for itself, and another instance for every child node it receives metrics from. If you had four
+streaming nodes, you would have five instances in total (`1 parent + 4 child nodes = 5 instances`).
+
+The Agent allocated resources for each instance separately using the `dbengine disk space`(**deprecated**) setting. If
+`dbengine disk space`(**deprecated**) is set to the default `256`, each instance is given 256 MiB in disk space, which
+means the total disk space required to store all instances is, roughly, `256 MiB * 1 parent * 4 child nodes = 1280 MiB`. 
 
 See the [database engine calculator](https://learn.netdata.cloud/docs/agent/database/calculator) to help you correctly
-set `dbengine disk space` and undertand the toal disk space required based on your streaming setup.
+set `dbengine disk space`(**deprecated**) and understand the total disk space required based on your streaming setup.
+
+##### Multi host DB mode
+
+In the newer agent versions the parent and child nodes all share `page cache size` and `dbengine multihost disk space`
+in a single dbengine multi-host instance.
+
+##### Backward compatibility
+
+All existing metrics belonging to child nodes are automatically converted to legacy dbengine instances and the localhost
+metrics are transferred to the multi-host dbengine instance.
+
+All new child nodes are automatically transferred to the mult-host dbengine instance and share its page cache and disk
+space. If you want to migrate a child node from its legacy dbengine instance to the multi-host dbengine instance you
+must delete the instance's directory located in `/var/cache/netdata/MACHINE_GUID/dbengine` after stopping the netdata
+agent.
+
+##### Information
 
 For more information about setting `memory mode` on your nodes, in addition to other streaming configurations, see
 [streaming](/streaming/README.md).
@@ -70,8 +99,7 @@ For more information about setting `memory mode` on your nodes, in addition to o
 Using memory mode `dbengine` we can overcome most memory restrictions and store a dataset that is much larger than the
 available memory.
 
-There are explicit memory requirements **per** DB engine **instance**, meaning **per** Netdata **node** (e.g. localhost
-and streaming recipient nodes):
+There are explicit memory requirements **per** DB engine **instance**:
 
 -   The total page cache memory footprint will be an additional `#dimensions-being-collected x 4096 x 2` bytes over what
     the user configured with `page cache size`.
@@ -83,10 +111,11 @@ and streaming recipient nodes):
     -   for very highly compressible data (compression ratio > 90%) this RAM overhead is comparable to the disk space
         footprint.
 
-An important observation is that RAM usage depends on both the `page cache size` and the `dbengine disk space` options.
+An important observation is that RAM usage depends on both the `page cache size` and the `dbengine multihost disk space`
+options.
 
 You can use our [database engine calculator](https://learn.netdata.cloud/docs/agent/database/calculator) to
-validate the memory requirements for your particular system(s) and configuration.
+validate the memory requirements for your particular system(s) and configuration (**out-of-date**).
 
 ### File descriptor requirements
 
diff --git a/database/engine/metadata_log/compaction.c b/database/engine/metadata_log/compaction.c
index 8bdfc47049..c261cd9454 100644
--- a/database/engine/metadata_log/compaction.c
+++ b/database/engine/metadata_log/compaction.c
@@ -82,7 +82,7 @@ static void compact_record_by_uuid(struct metalog_instance *ctx, uuid_t *uuid)
     RRDSET *st;
     RRDDIM *rd;
     BUFFER *buffer;
-    RRDHOST *host = ctx->rrdeng_ctx->host;
+    RRDHOST *host = NULL;
 
     ret = find_object_by_guid(uuid, NULL, 0);
     switch (ret) {
@@ -130,8 +130,9 @@ static void compact_record_by_uuid(struct metalog_instance *ctx, uuid_t *uuid)
             }
             break;
         case GUID_TYPE_HOST:
-            //TODO: will be enabled when multidb is activated
-            //RRDHOST *host = metalog_get_host_from_uuid(ctx, uuid);
+            host = metalog_get_host_from_uuid(ctx, uuid);
+            if (unlikely(!host))
+                break;
             if (ctx->current_compaction_id > host->compaction_id) {
                 host->compaction_id = ctx->current_compaction_id;
                 buffer = metalog_update_host_buffer(host);
diff --git a/database/engine/metadata_log/logfile.c b/database/engine/metadata_log/logfile.c
index 2a025b16bb..fdee93da10 100644
--- a/database/engine/metadata_log/logfile.c
+++ b/database/engine/metadata_log/logfile.c
@@ -330,6 +330,7 @@ int create_metadata_logfile(struct metadata_logfile *metalogfile)
     if (unlikely(ret)) {
         fatal("posix_memalign:%s", strerror(ret));
     }
+    memset(superblock, 0, sizeof(*superblock));
     (void) strncpy(superblock->magic_number, RRDENG_METALOG_MAGIC, RRDENG_MAGIC_SZ);
     superblock->version = RRDENG_METALOG_VER;
 
diff --git a/database/engine/metadata_log/metadatalog.c b/database/engine/metadata_log/metadatalog.c
index 8be938f3d5..6e648f3da6 100644
--- a/database/engine/metadata_log/metadatalog.c
+++ b/database/engine/metadata_log/metadatalog.c
@@ -157,9 +157,9 @@ void metalog_test_quota(struct metalog_worker_config *wc)
     metalogfile = ctx->metadata_logfiles.last;
     only_one_metalogfile = (metalogfile == ctx->metadata_logfiles.first) ? 1 : 0;
     debug(D_METADATALOG, "records=%lu objects=%lu", (long unsigned)ctx->records_nr,
-          (long unsigned)ctx->rrdeng_ctx->host->objects_nr);
+           (long unsigned)ctx->objects_nr);
     if (unlikely(!only_one_metalogfile &&
-                 ctx->records_nr > (ctx->rrdeng_ctx->host->objects_nr * (uint64_t)MAX_DUPLICATION_PERCENTAGE) / 100) &&
+                 ctx->records_nr > (ctx->objects_nr * (uint64_t)MAX_DUPLICATION_PERCENTAGE) / 100) &&
                 NO_QUIESCE == ctx->quiesce) {
         metalog_do_compaction(wc);
     }
diff --git a/database/engine/metadata_log/metadatalog.h b/database/engine/metadata_log/metadatalog.h
index bd2b0d9914..85607627b9 100644
--- a/database/engine/metadata_log/metadatalog.h
+++ b/database/engine/metadata_log/metadatalog.h
@@ -112,6 +112,8 @@ struct metalog_instance {
     uint32_t current_compaction_id; /* Every compaction run increments this by 1 */
     unsigned long disk_space;
     unsigned long records_nr;
+    unsigned long objects_nr; /* total objects (hosts, charts, dimensions) monitored in this context */
+    uint8_t initialized; /* set to 1 to mark context initialized */
     unsigned last_fileno; /* newest index of metadata log file */
 
     uint8_t quiesce; /*
diff --git a/database/engine/metadata_log/metadatalogapi.c b/database/engine/metadata_log/metadatalogapi.c
index 7f45a44db4..307f18b8ac 100755
--- a/database/engine/metadata_log/metadatalogapi.c
+++ b/database/engine/metadata_log/metadatalogapi.c
@@ -3,6 +3,14 @@
 
 #include "metadatalog.h"
 
+static inline struct metalog_instance *get_metalog_ctx(RRDHOST *host)
+{
+    if (host->rrdeng_ctx)
+        return host->rrdeng_ctx->metalog_ctx;
+
+    return NULL;
+}
+
 static inline int metalog_is_initialized(struct metalog_instance *ctx)
 {
     return ctx->rrdeng_ctx->metalog_ctx != NULL;
@@ -18,6 +26,16 @@ static inline void metalog_commit_deletion_record(struct metalog_instance *ctx,
     metalog_commit_record(ctx, buffer, METALOG_COMMIT_DELETION_RECORD, NULL, 0);
 }
 
+void metalog_upd_objcount(RRDHOST *host, int count)
+{
+    struct metalog_instance *ctx = get_metalog_ctx(host);
+
+    if (unlikely(!ctx))
+        return;
+
+    rrd_atomic_fetch_add(&ctx->objects_nr, count);
+}
+
 BUFFER *metalog_update_host_buffer(RRDHOST *host)
 {
     BUFFER *buffer;
@@ -62,11 +80,10 @@ void metalog_commit_update_host(RRDHOST *host)
     BUFFER *buffer;
 
     /* Metadata are only available with dbengine */
-    if (!host->rrdeng_ctx)
+    ctx = get_metalog_ctx(host);
+    if (!ctx)
         return;
-
-    ctx = host->rrdeng_ctx->metalog_ctx;
-    if (!ctx) /* metadata log has not been initialized yet */
+    if (!ctx->initialized) /* metadata log has not been initialized yet */
         return;
 
     buffer = metalog_update_host_buffer(host);
@@ -157,14 +174,15 @@ void metalog_commit_update_chart(RRDSET *st)
 {
     struct metalog_instance *ctx;
     BUFFER *buffer;
-    RRDHOST *host = st->rrdhost;
 
     /* Metadata are only available with dbengine */
-    if (!host->rrdeng_ctx || RRD_MEMORY_MODE_DBENGINE != st->rrd_memory_mode)
+    if (RRD_MEMORY_MODE_DBENGINE != st->rrd_memory_mode)
         return;
 
-    ctx = host->rrdeng_ctx->metalog_ctx;
-    if (!ctx) /* metadata log has not been initialized yet */
+    ctx = get_metalog_ctx(st->rrdhost);
+    if (!ctx)
+        return;
+    if (!ctx->initialized) /* metadata log has not been initialized yet */
         return;
 
     buffer = metalog_update_chart_buffer(st, 0);
@@ -176,15 +194,16 @@ void metalog_commit_delete_chart(RRDSET *st)
 {
     struct metalog_instance *ctx;
     BUFFER *buffer;
-    RRDHOST *host = st->rrdhost;
     char uuid_str[37];
 
     /* Metadata are only available with dbengine */
-    if (!host->rrdeng_ctx || RRD_MEMORY_MODE_DBENGINE != st->rrd_memory_mode)
+    if (RRD_MEMORY_MODE_DBENGINE != st->rrd_memory_mode)
         return;
 
-    ctx = host->rrdeng_ctx->metalog_ctx;
-    if (!ctx) /* metadata log has not been initialized yet */
+    ctx = get_metalog_ctx(st->rrdhost);
+    if (!ctx)
+        return;
+    if (!ctx->initialized) /* metadata log has not been initialized yet */
         return;
     buffer = buffer_create(64); /* This will be freed after it has been committed to the metadata log buffer */
 
@@ -228,14 +247,15 @@ void metalog_commit_update_dimension(RRDDIM *rd)
     struct metalog_instance *ctx;
     BUFFER *buffer;
     RRDSET *st = rd->rrdset;
-    RRDHOST *host = st->rrdhost;
 
     /* Metadata are only available with dbengine */
-    if (!host->rrdeng_ctx || RRD_MEMORY_MODE_DBENGINE != st->rrd_memory_mode)
+    if (RRD_MEMORY_MODE_DBENGINE != st->rrd_memory_mode)
         return;
 
-    ctx = host->rrdeng_ctx->metalog_ctx;
-    if (!ctx) /* metadata log has not been initialized yet */
+    ctx = get_metalog_ctx(st->rrdhost);
+    if (!ctx)
+        return;
+    if (!ctx->initialized) /* metadata log has not been initialized yet */
         return;
 
     buffer = metalog_update_dimension_buffer(rd);
@@ -248,15 +268,16 @@ void metalog_commit_delete_dimension(RRDDIM *rd)
     struct metalog_instance *ctx;
     BUFFER *buffer;
     RRDSET *st = rd->rrdset;
-    RRDHOST *host = st->rrdhost;
     char uuid_str[37];
 
     /* Metadata are only available with dbengine */
-    if (!host->rrdeng_ctx || RRD_MEMORY_MODE_DBENGINE != st->rrd_memory_mode)
+    if (RRD_MEMORY_MODE_DBENGINE != st->rrd_memory_mode)
         return;
 
-    ctx = host->rrdeng_ctx->metalog_ctx;
-    if (!ctx) /* metadata log has not been initialized yet */
+    ctx = get_metalog_ctx(st->rrdhost);
+    if (!ctx)
+        return;
+    if (!ctx->initialized) /* metadata log has not been initialized yet */
         return;
     buffer = buffer_create(64); /* This will be freed after it has been committed to the metadata log buffer */
 
@@ -273,12 +294,15 @@ RRDHOST *metalog_get_host_from_uuid(struct metalog_instance *ctx, uuid_t *host_g
     char machine_guid[37];
 
     uuid_unparse_lower(*host_guid, machine_guid);
+    RRDHOST *host = rrdhost_find_by_guid(machine_guid, 0);
     ret = find_object_by_guid(host_guid, NULL, 0);
     if (unlikely(GUID_TYPE_HOST != ret)) {
-        error("Host with GUID %s not found in the global map", machine_guid);
-        return NULL;
+        errno = 0;
+        if (unlikely(!host))
+            error("Host with GUID %s not found in the global map or in the list of hosts", machine_guid);
+        else
+            error("Host with GUID %s not found in the global map", machine_guid);
     }
-    RRDHOST *host = rrdhost_find_by_guid(machine_guid, 0);
     return host;
 }
 
@@ -292,9 +316,12 @@ RRDSET *metalog_get_chart_from_uuid(struct metalog_instance *ctx, uuid_t *chart_
     if (unlikely(GUID_TYPE_CHART != ret))
         return NULL;
 
-    machine_guid = (uuid_t *)chart_object;
-    RRDHOST *host = ctx->rrdeng_ctx->host;
+    machine_guid = (uuid_t  *)chart_object;
+    RRDHOST *host = metalog_get_host_from_uuid(ctx, machine_guid);
+    if (unlikely(!host))
+        return NULL;
     if (unlikely(uuid_compare(host->host_uuid, *machine_guid))) {
+        errno = 0;
         error("Metadata host machine GUID does not match the one assosiated with the chart");
         return NULL;
     }
@@ -311,6 +338,8 @@ RRDSET *metalog_get_chart_from_uuid(struct metalog_instance *ctx, uuid_t *chart_
 
 RRDDIM *metalog_get_dimension_from_uuid(struct metalog_instance *ctx, uuid_t *metric_uuid)
 {
+    UNUSED(ctx);
+
     GUID_TYPE ret;
     char dim_object[49], chart_object[33], id_str[PLUGINSD_LINE_MAX], chart_fullid[RRD_ID_LENGTH_MAX + 1];
     uuid_t *machine_guid, *chart_guid, *chart_char_guid, *dim_char_guid;
@@ -320,8 +349,12 @@ RRDDIM *metalog_get_dimension_from_uuid(struct metalog_instance *ctx, uuid_t *me
         return NULL;
 
     machine_guid = (uuid_t *)dim_object;
-    RRDHOST *host = ctx->rrdeng_ctx->host;
+
+    RRDHOST *host = metalog_get_host_from_uuid(ctx, machine_guid);
+    if (unlikely(!host))
+        return NULL;
     if (unlikely(uuid_compare(host->host_uuid, *machine_guid))) {
+        errno = 0;
         error("Metadata host machine GUID does not match the one assosiated with the dimension");
         return NULL;
     }
@@ -359,10 +392,11 @@ void metalog_delete_dimension_by_uuid(struct metalog_instance *ctx, uuid_t *metr
     uint8_t empty_chart;
 
     rd = metalog_get_dimension_from_uuid(ctx, metric_uuid);
-    if (!rd) { /* in the case of legacy UUID convert to multihost and try again */
+    if (!rd) { /* in 8the case of legacy UUID convert to multihost and try again */
+        // TODO: Check what to do since we have no host
         uuid_t multihost_uuid;
 
-        rrdeng_convert_legacy_uuid_to_multihost(ctx->rrdeng_ctx->host->machine_guid, metric_uuid, &multihost_uuid);
+        rrdeng_convert_legacy_uuid_to_multihost(ctx->rrdeng_ctx->machine_guid, metric_uuid, &multihost_uuid);
         rd = metalog_get_dimension_from_uuid(ctx, &multihost_uuid);
     }
     if(!rd) {
@@ -372,6 +406,10 @@ void metalog_delete_dimension_by_uuid(struct metalog_instance *ctx, uuid_t *metr
     st = rd->rrdset;
     host = st->rrdhost;
 
+    /* In case there are active metrics in a different database engine do not delete the dimension object */
+    if (unlikely(host->rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE))
+        return;
+
     /* Since the metric has no writer it will not be commited to the metadata log by rrddim_free_custom().
      * It must be commited explicitly before calling rrddim_free_custom(). */
     metalog_commit_delete_dimension(rd);
@@ -401,8 +439,11 @@ int metalog_init(struct rrdengine_instance *rrdeng_parent_ctx)
 
     ctx = callocz(1, sizeof(*ctx));
     ctx->records_nr = 0;
+    ctx->objects_nr = 0;
     ctx->current_compaction_id = 0;
     ctx->quiesce = NO_QUIESCE;
+    ctx->initialized = 0;
+    rrdeng_parent_ctx->metalog_ctx = ctx;
 
     memset(&ctx->worker_config, 0, sizeof(ctx->worker_config));
     ctx->rrdeng_ctx = rrdeng_parent_ctx;
@@ -422,7 +463,7 @@ int metalog_init(struct rrdengine_instance *rrdeng_parent_ctx)
     if (ctx->worker_config.error) {
         goto error_after_rrdeng_worker;
     }
-    rrdeng_parent_ctx->metalog_ctx = ctx; /* notify dbengine that the metadata log has finished initializing */
+    ctx->initialized = 1; /* notify dbengine that the metadata log has finished initializing */
     return 0;
 
 error_after_rrdeng_worker:
diff --git a/database/engine/metadata_log/metadatalogapi.h b/database/engine/metadata_log/metadatalogapi.h
index 3808ab6ce3..5395844ba5 100644
--- a/database/engine/metadata_log/metadatalogapi.h
+++ b/database/engine/metadata_log/metadatalogapi.h
@@ -13,6 +13,7 @@ extern void metalog_commit_delete_chart(RRDSET *st);
 extern BUFFER *metalog_update_dimension_buffer(RRDDIM *rd);
 extern void metalog_commit_update_dimension(RRDDIM *rd);
 extern void metalog_commit_delete_dimension(RRDDIM *rd);
+extern void metalog_upd_objcount(RRDHOST *host, int count);
 
 extern RRDSET *metalog_get_chart_from_uuid(struct metalog_instance *ctx, uuid_t *chart_uuid);
 extern RRDDIM *metalog_get_dimension_from_uuid(struct metalog_instance *ctx, uuid_t *metric_uuid);
diff --git a/database/engine/metadata_log/metalogpluginsd.c b/database/engine/metadata_log/metalogpluginsd.c
index 4e8ba63628..25af24143b 100755
--- a/database/engine/metadata_log/metalogpluginsd.c
+++ b/database/engine/metadata_log/metalogpluginsd.c
@@ -4,15 +4,33 @@
 #include "metadatalog.h"
 #include "metalogpluginsd.h"
 
+extern struct config stream_config;
+
 PARSER_RC metalog_pluginsd_host_action(
     void *user, char *machine_guid, char *hostname, char *registry_hostname, int update_every, char *os, char *timezone,
     char *tags)
 {
+    int history = 5;
+    RRD_MEMORY_MODE mode = RRD_MEMORY_MODE_DBENGINE;
+    int health_enabled = default_health_enabled;
+    int rrdpush_enabled = default_rrdpush_enabled;
+    char *rrdpush_destination = default_rrdpush_destination;
+    char *rrdpush_api_key = default_rrdpush_api_key;
+    char *rrdpush_send_charts_matching = default_rrdpush_send_charts_matching;
+
     struct metalog_pluginsd_state *state = ((PARSER_USER_OBJECT *)user)->private;
 
     RRDHOST *host = rrdhost_find_by_guid(machine_guid, 0);
-    if (host)
+    if (host) {
+        if (unlikely(host->rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE)) {
+            error("Archived host '%s' has memory mode '%s', but the archived one is '%s'. Ignoring archived state.",
+                  host->hostname, rrd_memory_mode_name(host->rrd_memory_mode),
+                  rrd_memory_mode_name(RRD_MEMORY_MODE_DBENGINE));
+            ((PARSER_USER_OBJECT *) user)->host = NULL; /* Ignore objects if memory mode is not dbengine */
+            return PARSER_RC_OK;
+        }
         goto write_replay;
+    }
 
     if (strcmp(machine_guid, registry_get_this_machine_guid()) == 0) {
         struct metalog_record record;
@@ -20,12 +38,32 @@ PARSER_RC metalog_pluginsd_host_action(
 
         uuid_parse(machine_guid, record.uuid);
         mlf_record_insert(metalogfile, &record);
+        if (localhost->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE)
+            ((PARSER_USER_OBJECT *) user)->host = localhost;
+        else
+            ((PARSER_USER_OBJECT *) user)->host = NULL;
         return PARSER_RC_OK;
     }
 
-    // Ignore HOST command for now
-    // TODO: Remove when the next task is completed ie. accept new children in the lcoalhost / multidb
-    return PARSER_RC_OK;
+    // Fetch configuration options from streaming config
+    update_every = (int)appconfig_get_number(&stream_config, machine_guid, "update every", update_every);
+    if(update_every < 0) update_every = 1;
+
+    //health_enabled = appconfig_get_boolean_ondemand(&stream_config, rpt->key, "health enabled by default", health_enabled);
+    health_enabled = appconfig_get_boolean_ondemand(&stream_config, machine_guid, "health enabled", health_enabled);
+
+    //rrdpush_enabled = appconfig_get_boolean(&stream_config, rpt->key, "default proxy enabled", rrdpush_enabled);
+    rrdpush_enabled = appconfig_get_boolean(&stream_config, machine_guid, "proxy enabled", rrdpush_enabled);
+
+    //rrdpush_destination = appconfig_get(&stream_config, rpt->key, "default proxy destination", rrdpush_destination);
+    rrdpush_destination = appconfig_get(&stream_config, machine_guid, "proxy destination", rrdpush_destination);
+
+    //rrdpush_api_key = appconfig_get(&stream_config, rpt->key, "default proxy api key", rrdpush_api_key);
+    rrdpush_api_key = appconfig_get(&stream_config, machine_guid, "proxy api key", rrdpush_api_key);
+
+    //rrdpush_send_charts_matching = appconfig_get(&stream_config, rpt->key, "default proxy send charts matching", rrdpush_send_charts_matching);
+    rrdpush_send_charts_matching = appconfig_get(&stream_config, machine_guid, "proxy send charts matching", rrdpush_send_charts_matching);
+
 
     host = rrdhost_create(
         hostname
@@ -37,13 +75,13 @@ PARSER_RC metalog_pluginsd_host_action(
         , NULL
         , NULL
         , update_every
-        , 3600
-        , RRD_MEMORY_MODE_DBENGINE
-        , 0   // health enabled
-        , 0   // Push enabled
-        , NULL
-        , NULL
-        , NULL
+        , history   // entries
+        , mode
+        , health_enabled    // health enabled
+        , rrdpush_enabled   // Push enabled
+        , rrdpush_destination  //destination
+        , rrdpush_api_key  // api key
+        , rrdpush_send_charts_matching  // charts matching
         , callocz(1, sizeof(struct rrdhost_system_info))
         , 0     // localhost
         , 1     // archived
@@ -57,7 +95,7 @@ write_replay:
         uuid_copy(record.uuid, host->host_uuid);
         mlf_record_insert(metalogfile, &record);
     }
-
+    ((PARSER_USER_OBJECT *) user)->host = host;
     return PARSER_RC_OK;
 }
 
@@ -70,6 +108,10 @@ PARSER_RC metalog_pluginsd_chart_action(void *user, char *type, char *id, char *
     RRDHOST *host = ((PARSER_USER_OBJECT *) user)->host;
     uuid_t *chart_uuid;
 
+    if (unlikely(!host)) {
+        debug(D_METADATALOG, "Ignoring chart belonging to missing or ignored host.");
+        return PARSER_RC_OK;
+    }
     chart_uuid = uuid_is_null(state->uuid) ? NULL : &state->uuid;
     st = rrdset_create_custom(
         host, type, id, name, family, context, title, units,
@@ -117,6 +159,10 @@ PARSER_RC metalog_pluginsd_dimension_action(void *user, RRDSET *st, char *id, ch
     UNUSED(algorithm);
     uuid_t *dim_uuid;
 
+    if (unlikely(!st)) {
+        debug(D_METADATALOG, "Ignoring dimension belonging to missing or ignored chart.");
+        return PARSER_RC_OK;
+    }
     dim_uuid = uuid_is_null(state->uuid) ? NULL : &state->uuid;
 
     RRDDIM *rd = rrddim_add_custom(st, id, name, multiplier, divisor, algorithm_type, RRD_MEMORY_MODE_DBENGINE, 1,
@@ -155,8 +201,8 @@ PARSER_RC metalog_pluginsd_guid_action(void *user, uuid_t *uuid)
 PARSER_RC metalog_pluginsd_context_action(void *user, uuid_t *uuid)
 {
     GUID_TYPE ret;
-    struct metalog_pluginsd_state *state = ((PARSER_USER_OBJECT *)user)->private;
-    struct metalog_instance *ctx = state->ctx;
+    //struct metalog_pluginsd_state *state = ((PARSER_USER_OBJECT *)user)->private;
+    //struct metalog_instance *ctx = state->ctx;
     char object[49], chart_object[33], id_str[1024];
     uuid_t *chart_guid, *chart_char_guid;
     RRDHOST *host;
@@ -164,15 +210,16 @@ PARSER_RC metalog_pluginsd_context_action(void *user, uuid_t *uuid)
     ret = find_object_by_guid(uuid, object, 49);
     switch (ret) {
         case GUID_TYPE_NOTFOUND:
-            if (unlikely(ctx->rrdeng_ctx->host && uuid_compare(ctx->rrdeng_ctx->host->host_uuid, *uuid)))
-                error_with_guid(uuid, "Failed to find valid context");
+            error_with_guid(uuid, "Failed to find valid context");
             break;
         case GUID_TYPE_CHAR:
             error_with_guid(uuid, "Ignoring unexpected type GUID_TYPE_CHAR");
             break;
         case GUID_TYPE_CHART:
         case GUID_TYPE_DIMENSION:
-            host = ctx->rrdeng_ctx->host;
+            host = metalog_get_host_from_uuid(NULL, (uuid_t *) &object);
+            if (unlikely(!host))
+                break;
             switch (ret) {
                 case GUID_TYPE_CHART:
                     chart_char_guid = (uuid_t *)(object + 16);
@@ -204,7 +251,7 @@ PARSER_RC metalog_pluginsd_context_action(void *user, uuid_t *uuid)
             }
             break;
         case GUID_TYPE_HOST:
-            /* Ignore for now */
+            ((PARSER_USER_OBJECT *)user)->host = metalog_get_host_from_uuid(NULL, (uuid_t *) &object);
             break;
         case GUID_TYPE_NOSPACE:
             error_with_guid(uuid, "Not enough space for object retrieval");
@@ -222,7 +269,7 @@ PARSER_RC metalog_pluginsd_tombstone_action(void *user, uuid_t *uuid)
     GUID_TYPE ret;
     struct metalog_pluginsd_state *state = ((PARSER_USER_OBJECT *)user)->private;
     struct metalog_instance *ctx = state->ctx;
-    RRDHOST *host = ctx->rrdeng_ctx->host;
+    RRDHOST *host = NULL;
     RRDSET *st;
     RRDDIM *rd;
 
@@ -234,6 +281,7 @@ PARSER_RC metalog_pluginsd_tombstone_action(void *user, uuid_t *uuid)
         case GUID_TYPE_CHART:
             st = metalog_get_chart_from_uuid(ctx, uuid);
             if (st) {
+                host = st->rrdhost;
                 rrdhost_wrlock(host);
                 rrdset_free(st);
                 rrdhost_unlock(host);
diff --git a/database/engine/rrdengine.c b/database/engine/rrdengine.c
index 000bf88a6a..abd97952a8 100644
--- a/database/engine/rrdengine.c
+++ b/database/engine/rrdengine.c
@@ -644,7 +644,7 @@ static void delete_old_data(void *arg)
         for (i = 0 ; i < count ; ++i) {
             descr = extent->pages[i];
             can_delete_metric = pg_cache_punch_hole(ctx, descr, 0, 0, &metric_id);
-            if (unlikely(can_delete_metric && ctx->metalog_ctx)) {
+            if (unlikely(can_delete_metric && ctx->metalog_ctx->initialized)) {
                 /*
                  * If the metric is empty, has no active writers and if the metadata log has been initialized then
                  * attempt to delete the corresponding netdata dimension.
@@ -821,7 +821,7 @@ void timer_cb(uv_timer_t* handle)
 
     uv_stop(handle->loop);
     uv_update_time(handle->loop);
-    if (unlikely(!ctx->metalog_ctx))
+    if (unlikely(!ctx->metalog_ctx->initialized))
         return; /* Wait for the metadata log to initialize */
     rrdeng_test_quota(wc);
     debug(D_RRDENGINE, "%s: timeout reached.", __func__);
diff --git a/database/engine/rrdengine.h b/database/engine/rrdengine.h
index e5d474a74b..eb13d8474e 100644
--- a/database/engine/rrdengine.h
+++ b/database/engine/rrdengine.h
@@ -176,7 +176,6 @@ extern rrdeng_stats_t global_flushing_pressure_page_deletions; /* number of dele
 #define QUIESCED    (2) /* is set after all threads have finished running */
 
 struct rrdengine_instance {
-    RRDHOST *host;
     struct metalog_instance *metalog_ctx;
     struct rrdengine_worker_config worker_config;
     struct completion rrdengine_completion;
@@ -185,7 +184,9 @@ struct rrdengine_instance {
     uint8_t global_compress_alg;
     struct transaction_commit_log commit_log;
     struct rrdengine_datafile_list datafiles;
-    char dbfiles_path[FILENAME_MAX+1];
+    RRDHOST *host; /* the legacy host, or NULL for multi-host DB */
+    char dbfiles_path[FILENAME_MAX + 1];
+    char machine_guid[GUID_LEN + 1]; /* the unique ID of the corresponding host, or localhost for multihost DB */
     uint64_t disk_space;
     uint64_t max_disk_space;
     unsigned last_fileno; /* newest index of datafile and journalfile */
diff --git a/database/engine/rrdengineapi.c b/database/engine/rrdengineapi.c
index a309b85f60..64f4f3c0ec 100755
--- a/database/engine/rrdengineapi.c
+++ b/database/engine/rrdengineapi.c
@@ -2,7 +2,7 @@
 #include "rrdengine.h"
 
 /* Default global database instance */
-static struct rrdengine_instance default_global_ctx;
+struct rrdengine_instance multidb_ctx;
 
 int default_rrdeng_page_cache_mb = 32;
 int default_rrdeng_disk_quota_mb = 256;
@@ -10,6 +10,11 @@ int default_multidb_disk_quota_mb = 256;
 /* Default behaviour is to unblock data collection if the page cache is full of dirty pages by dropping metrics */
 uint8_t rrdeng_drop_metrics_under_page_cache_pressure = 1;
 
+static inline struct rrdengine_instance *get_rrdeng_ctx_from_host(RRDHOST *host)
+{
+    return host->rrdeng_ctx;
+}
+
 /* This UUID is not unique across hosts */
 void rrdeng_generate_legacy_uuid(const char *dim_id, char *chart_id, uuid_t *ret_uuid)
 {