RRD structures managed by dictionaries (#13646)

* rrdset - in progress

* rrdset optimal constructor; rrdset conflict

* rrdset final touches

* re-organization of rrdset object members

* prevent use-after-free

* dictionary dfe supports also counting of iterations

* rrddim managed by dictionary

* rrd.h cleanup

* DICTIONARY_ITEM now is referencing actual dictionary items in the code

* removed rrdset linked list

* Revert "removed rrdset linked list"

This reverts commit 690d6a588b4b99619c2c5e10f84e8f868ae6def5.

* removed rrdset linked list

* added comments

* Switch chart uuid to static allocation in rrdset
Remove unused functions

* rrdset_archive() and friends...

* always create rrdfamily

* enable ml_free_dimension

* rrddim_foreach done with dfe

* most custom rrddim loops replaced with rrddim_foreach

* removed accesses to rrddim->dimensions

* removed locks that are no longer needed

* rrdsetvar is now managed by the dictionary

* set rrdset is rrdsetvar, fixes https://github.com/netdata/netdata/pull/13646#issuecomment-1242574853

* conflict callback of rrdsetvar now properly checks if it has to reset the variable

* dictionary registered callbacks accept as first parameter the DICTIONARY_ITEM

* dictionary dfe now uses internal counter to report; avoided excess variables defined with dfe

* dictionary walkthrough callbacks get dictionary acquired items

* dictionary reference counters that can be dupped from zero

* added advanced functions for get and del

* rrdvar managed by dictionaries

* thread safety for rrdsetvar

* faster rrdvar initialization

* rrdvar string lengths should match in all add, del, get functions

* rrdvar internals hidden from the rest of the world

* rrdvar is now acquired throughout netdata

* hide the internal structures of rrdsetvar

* rrdsetvar is now acquired through out netdata

* rrddimvar managed by dictionary; rrddimvar linked list removed; rrddimvar structures hidden from the rest of netdata

* better error handling

* dont create variables if not initialized for health

* dont create variables if not initialized for health again

* rrdfamily is now managed by dictionaries; references of it are acquired dictionary items

* type checking on acquired objects

* rrdcalc renaming of functions

* type checking for rrdfamily_acquired

* rrdcalc managed by dictionaries

* rrdcalc double free fix

* host rrdvars is always needed

* attempt to fix deadlock 1

* attempt to fix deadlock 2

* Remove unused variable

* attempt to fix deadlock 3

* snprintfz

* rrdcalc index in rrdset fix

* Stop storing active charts and computing chart hashes

* Remove store active chart function

* Remove compute chart hash function

* Remove sql_store_chart_hash function

* Remove store_active_dimension function

* dictionary delayed destruction

* formatting and cleanup

* zero dictionary base on rrdsetvar

* added internal error to log delayed destructions of dictionaries

* typo in rrddimvar

* added debugging info to dictionary

* debug info

* fix for rrdcalc keys being empty

* remove forgotten unlock

* remove deadlock

* Switch to metadata version 5 and drop
  chart_hash
  chart_hash_map
  chart_active
  dimension_active
  v_chart_hash

* SQL cosmetic changes

* do not busy wait while destroying a referenced dictionary

* remove deadlock

* code cleanup; re-organization;

* fast cleanup and flushing of dictionaries

* number formatting fixes

* do not delete configured alerts when archiving a chart

* rrddim obsolete linked list management outside dictionaries

* removed duplicate contexts call

* fix crash when rrdfamily is not initialized

* dont keep rrddimvar referenced

* properly cleanup rrdvar

* removed some locks

* Do not attempt to cleanup chart_hash / chart_hash_map

* rrdcalctemplate managed by dictionary

* register callbacks on the right dictionary

* removed some more locks

* rrdcalc secondary index replaced with linked-list; rrdcalc labels updates are now executed by health thread

* when looking up for an alarm look using both chart id and chart name

* host initialization a bit more modular

* init rrdlabels on host update

* preparation for dictionary views

* improved comment

* unused variables without internal checks

* service threads isolation and worker info

* more worker info in service thread

* thread cancelability debugging with internal checks

* strings data races addressed; fixes https://github.com/netdata/netdata/issues/13647

* dictionary modularization

* Remove unused SQL statement definition

* unit-tested thread safety of dictionaries; removed data race conditions on dictionaries and strings; dictionaries now can detect if the caller is holds a write lock and automatically all the calls become their unsafe versions; all direct calls to unsafe version is eliminated

* remove worker_is_idle() from the exit of service functions, because we lose the lock time between loops

* rewritten dictionary to have 2 separate locks, one for indexing and another for traversal

* Update collectors/cgroups.plugin/sys_fs_cgroup.c

Co-authored-by: Vladimir Kobal <vlad@prokk.net>

* Update collectors/cgroups.plugin/sys_fs_cgroup.c

Co-authored-by: Vladimir Kobal <vlad@prokk.net>

* Update collectors/proc.plugin/proc_net_dev.c

Co-authored-by: Vladimir Kobal <vlad@prokk.net>

* fix memory leak in rrdset cache_dir

* minor dictionary changes

* dont use index locks in single threaded

* obsolete dict option

* rrddim options and flags separation; rrdset_done() optimization to keep array of reference pointers to rrddim;

* fix jump on uninitialized value in dictionary; remove double free of cache_dir

* addressed codacy findings

* removed debugging code

* use the private refcount on dictionaries

* make dictionary item desctructors work on dictionary destruction; strictier control on dictionary API; proper cleanup sequence on rrddim;

* more dictionary statistics

* global statistics about dictionary operations, memory, items, callbacks

* dictionary support for views - missing the public API

* removed warning about unused parameter

* chart and context name for cloud

* chart and context name for cloud, again

* dictionary statistics fixed; first implementation of dictionary views - not currently used

* only the master can globally delete an item

* context needs netdata prefix

* fix context and chart it of spins

* fix for host variables when health is not enabled

* run garbage collector on item insert too

* Fix info message; remove extra "using"

* update dict unittest for new placement of garbage collector

* we need RRDHOST->rrdvars for maintaining custom host variables

* Health initialization needs the host->host_uuid

* split STRING to its own files; no code changes other than that

* initialize health unconditionally

* unit tests do not pollute the global scope with their variables

* Skip initialization when creating archived hosts on startup. When a child connects it will initialize properly

Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
Co-authored-by: Vladimir Kobal <vlad@prokk.net>

5 files changed, 22 insertions, 38 deletions

diff --git a/exporting/json/json.c b/exporting/json/json.c
index 06100ba5e2..dc2d5aae99 100644
--- a/exporting/json/json.c
+++ b/exporting/json/json.c
@@ -197,7 +197,7 @@ int format_dimension_collected_json_plaintext(struct instance *instance, RRDDIM
         rrdset_name(st),
         rrdset_family(st),
         rrdset_context(st),
-        rrdset_type(st),
+        rrdset_parts_type(st),
         rrdset_units(st),
         rrddim_id(rd),
         rrddim_name(rd),
@@ -281,7 +281,7 @@ int format_dimension_stored_json_plaintext(struct instance *instance, RRDDIM *rd
         rrdset_name(st),
         rrdset_family(st),
         rrdset_context(st),
-        rrdset_type(st),
+        rrdset_parts_type(st),
         rrdset_units(st),
         rrddim_id(rd),
         rrddim_name(rd),
diff --git a/exporting/process_data.c b/exporting/process_data.c
index e1e939c51d..05cf0a98bb 100644
--- a/exporting/process_data.c
+++ b/exporting/process_data.c
@@ -338,26 +338,22 @@ void prepare_buffers(struct engine *engine)
 
     rrd_rdlock();
     RRDHOST *host;
-    rrdhost_foreach_read(host)
-    {
-        rrdhost_rdlock(host);
+    rrdhost_foreach_read(host) {
         start_host_formatting(engine, host);
         RRDSET *st;
-        rrdset_foreach_read(st, host)
-        {
-            rrdset_rdlock(st);
+        rrdset_foreach_read(st, host) {
             start_chart_formatting(engine, st);
 
             RRDDIM *rd;
             rrddim_foreach_read(rd, st)
                 metric_formatting(engine, rd);
+            rrddim_foreach_done(rd);
 
             end_chart_formatting(engine, st);
-            rrdset_unlock(st);
         }
+        rrdset_foreach_done(st);
         variables_formatting(engine, host);
         end_host_formatting(engine, host);
-        rrdhost_unlock(host);
     }
     rrd_unlock();
     netdata_thread_enable_cancelability();
diff --git a/exporting/prometheus/prometheus.c b/exporting/prometheus/prometheus.c
index 26c0811d6f..5719db4187 100644
--- a/exporting/prometheus/prometheus.c
+++ b/exporting/prometheus/prometheus.c
@@ -349,12 +349,12 @@ struct host_variables_callback_options {
  * @param data callback options.
  * @return Returns 1 if the chart can be sent, 0 otherwise.
  */
-static int print_host_variables(const char *name __maybe_unused, void *rv_ptr, void *data) {
-    RRDVAR *rv = rv_ptr;
+static int print_host_variables_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rv_ptr __maybe_unused, void *data) {
+    const RRDVAR_ACQUIRED *rv = (const RRDVAR_ACQUIRED *)item;
 
     struct host_variables_callback_options *opts = data;
 
-    if (rv->options & (RRDVAR_OPTION_CUSTOM_HOST_VAR | RRDVAR_OPTION_CUSTOM_CHART_VAR)) {
+    if (rrdvar_flags(rv) & (RRDVAR_FLAG_CUSTOM_HOST_VAR | RRDVAR_FLAG_CUSTOM_CHART_VAR)) {
         if (!opts->host_header_printed) {
             opts->host_header_printed = 1;
 
@@ -519,7 +519,6 @@ static void rrd_stats_api_v1_charts_allmetrics_prometheus(
     PROMETHEUS_OUTPUT_OPTIONS output_options)
 {
     SIMPLE_PATTERN *filter = simple_pattern_create(filter_string, NULL, SIMPLE_PATTERN_EXACT);
-    rrdhost_rdlock(host);
 
     char hostname[PROMETHEUS_ELEMENT_MAX + 1];
     prometheus_label_copy(hostname, rrdhost_hostname(host), PROMETHEUS_ELEMENT_MAX);
@@ -564,17 +563,14 @@ static void rrd_stats_api_v1_charts_allmetrics_prometheus(
             .host_header_printed = 0
         };
 
-        rrdvar_walkthrough_read(host->rrdvar_root_index, print_host_variables, &opts);
+        rrdvar_walkthrough_read(host->rrdvars, print_host_variables_callback, &opts);
     }
 
     // for each chart
     RRDSET *st;
-    rrdset_foreach_read(st, host)
-    {
+    rrdset_foreach_read(st, host) {
 
         if (likely(can_send_rrdset(instance, st, filter))) {
-            rrdset_rdlock(st);
-
             char chart[PROMETHEUS_ELEMENT_MAX + 1];
             char context[PROMETHEUS_ELEMENT_MAX + 1];
             char family[PROMETHEUS_ELEMENT_MAX + 1];
@@ -615,8 +611,7 @@ static void rrd_stats_api_v1_charts_allmetrics_prometheus(
 
             // for each dimension
             RRDDIM *rd;
-            rrddim_foreach_read(rd, st)
-            {
+            rrddim_foreach_read(rd, st) {
                 if (rd->collections_counter && !rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE)) {
                     char dimension[PROMETHEUS_ELEMENT_MAX + 1];
                     char *suffix = "";
@@ -665,7 +660,8 @@ static void rrd_stats_api_v1_charts_allmetrics_prometheus(
                                 buffer_sprintf(wb, "# TYPE %s_%s%s %s\n", prefix, context, suffix, p.type);
 
                             generate_as_collected_prom_metric(wb, &p, homogeneous, prometheus_collector);
-                        } else {
+                        }
+                        else {
                             // the dimensions of the chart, do not have the same algorithm, multiplier or divisor
                             // we create a metric per dimension
 
@@ -683,7 +679,8 @@ static void rrd_stats_api_v1_charts_allmetrics_prometheus(
 
                             generate_as_collected_prom_metric(wb, &p, homogeneous, prometheus_collector);
                         }
-                    } else {
+                    }
+                    else {
                         // we need average or sum of the data
 
                         time_t first_time = instance->after;
@@ -750,12 +747,11 @@ static void rrd_stats_api_v1_charts_allmetrics_prometheus(
                     }
                 }
             }
-
-            rrdset_unlock(st);
+            rrddim_foreach_done(rd);
         }
     }
+    rrdset_foreach_done(st);
 
-    rrdhost_unlock(host);
     simple_pattern_free(filter);
 }
 
diff --git a/exporting/prometheus/remote_write/remote_write.c b/exporting/prometheus/remote_write/remote_write.c
index ad337513eb..15f5e41036 100644
--- a/exporting/prometheus/remote_write/remote_write.c
+++ b/exporting/prometheus/remote_write/remote_write.c
@@ -321,12 +321,12 @@ int format_dimension_prometheus_remote_write(struct instance *instance, RRDDIM *
     return 0;
 }
 
-static int format_variable_prometheus_remote_write_callback(const char *name_txt __maybe_unused, void *rv_ptr, void *data) {
-    RRDVAR *rv = rv_ptr;
+static int format_variable_prometheus_remote_write_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rv_ptr __maybe_unused, void *data) {
+    const RRDVAR_ACQUIRED *rv = (const RRDVAR_ACQUIRED *)item;
 
     struct prometheus_remote_write_variables_callback_options *opts = data;
 
-    if (rv->options & (RRDVAR_OPTION_CUSTOM_HOST_VAR | RRDVAR_OPTION_CUSTOM_CHART_VAR)) {        
+    if (rrdvar_flags(rv) & (RRDVAR_FLAG_CUSTOM_HOST_VAR | RRDVAR_FLAG_CUSTOM_CHART_VAR)) {
         RRDHOST *host = opts->host;
         struct instance *instance = opts->instance;
         struct simple_connector_data *simple_connector_data =
@@ -363,7 +363,7 @@ int format_variables_prometheus_remote_write(struct instance *instance, RRDHOST
         .now = now_realtime_usec(),
     };
 
-    return rrdvar_walkthrough_read(host->rrdvar_root_index, format_variable_prometheus_remote_write_callback, &opt);
+    return rrdvar_walkthrough_read(host->rrdvars, format_variable_prometheus_remote_write_callback, &opt);
 }
 
 /**
diff --git a/exporting/tests/netdata_doubles.c b/exporting/tests/netdata_doubles.c
index 735925d478..f83a474f9b 100644
--- a/exporting/tests/netdata_doubles.c
+++ b/exporting/tests/netdata_doubles.c
@@ -242,14 +242,6 @@ void __mock_rrddim_query_finalize(struct rrddim_query_handle *handle)
     function_called();
 }
 
-void sql_store_chart_label(uuid_t *chart_uuid, int source_type, char *label, char *value)
-{
-    (void)chart_uuid;
-    (void)source_type;
-    (void)label;
-    (void)value;
-}
-
 void rrdcalc_update_rrdlabels(RRDSET *st)
 {
     (void)st;