RRD structures managed by dictionaries (#13646)

* rrdset - in progress

* rrdset optimal constructor; rrdset conflict

* rrdset final touches

* re-organization of rrdset object members

* prevent use-after-free

* dictionary dfe supports also counting of iterations

* rrddim managed by dictionary

* rrd.h cleanup

* DICTIONARY_ITEM now is referencing actual dictionary items in the code

* removed rrdset linked list

* Revert "removed rrdset linked list"

This reverts commit 690d6a588b4b99619c2c5e10f84e8f868ae6def5.

* removed rrdset linked list

* added comments

* Switch chart uuid to static allocation in rrdset
Remove unused functions

* rrdset_archive() and friends...

* always create rrdfamily

* enable ml_free_dimension

* rrddim_foreach done with dfe

* most custom rrddim loops replaced with rrddim_foreach

* removed accesses to rrddim->dimensions

* removed locks that are no longer needed

* rrdsetvar is now managed by the dictionary

* set rrdset is rrdsetvar, fixes https://github.com/netdata/netdata/pull/13646#issuecomment-1242574853

* conflict callback of rrdsetvar now properly checks if it has to reset the variable

* dictionary registered callbacks accept as first parameter the DICTIONARY_ITEM

* dictionary dfe now uses internal counter to report; avoided excess variables defined with dfe

* dictionary walkthrough callbacks get dictionary acquired items

* dictionary reference counters that can be dupped from zero

* added advanced functions for get and del

* rrdvar managed by dictionaries

* thread safety for rrdsetvar

* faster rrdvar initialization

* rrdvar string lengths should match in all add, del, get functions

* rrdvar internals hidden from the rest of the world

* rrdvar is now acquired throughout netdata

* hide the internal structures of rrdsetvar

* rrdsetvar is now acquired through out netdata

* rrddimvar managed by dictionary; rrddimvar linked list removed; rrddimvar structures hidden from the rest of netdata

* better error handling

* dont create variables if not initialized for health

* dont create variables if not initialized for health again

* rrdfamily is now managed by dictionaries; references of it are acquired dictionary items

* type checking on acquired objects

* rrdcalc renaming of functions

* type checking for rrdfamily_acquired

* rrdcalc managed by dictionaries

* rrdcalc double free fix

* host rrdvars is always needed

* attempt to fix deadlock 1

* attempt to fix deadlock 2

* Remove unused variable

* attempt to fix deadlock 3

* snprintfz

* rrdcalc index in rrdset fix

* Stop storing active charts and computing chart hashes

* Remove store active chart function

* Remove compute chart hash function

* Remove sql_store_chart_hash function

* Remove store_active_dimension function

* dictionary delayed destruction

* formatting and cleanup

* zero dictionary base on rrdsetvar

* added internal error to log delayed destructions of dictionaries

* typo in rrddimvar

* added debugging info to dictionary

* debug info

* fix for rrdcalc keys being empty

* remove forgotten unlock

* remove deadlock

* Switch to metadata version 5 and drop
  chart_hash
  chart_hash_map
  chart_active
  dimension_active
  v_chart_hash

* SQL cosmetic changes

* do not busy wait while destroying a referenced dictionary

* remove deadlock

* code cleanup; re-organization;

* fast cleanup and flushing of dictionaries

* number formatting fixes

* do not delete configured alerts when archiving a chart

* rrddim obsolete linked list management outside dictionaries

* removed duplicate contexts call

* fix crash when rrdfamily is not initialized

* dont keep rrddimvar referenced

* properly cleanup rrdvar

* removed some locks

* Do not attempt to cleanup chart_hash / chart_hash_map

* rrdcalctemplate managed by dictionary

* register callbacks on the right dictionary

* removed some more locks

* rrdcalc secondary index replaced with linked-list; rrdcalc labels updates are now executed by health thread

* when looking up for an alarm look using both chart id and chart name

* host initialization a bit more modular

* init rrdlabels on host update

* preparation for dictionary views

* improved comment

* unused variables without internal checks

* service threads isolation and worker info

* more worker info in service thread

* thread cancelability debugging with internal checks

* strings data races addressed; fixes https://github.com/netdata/netdata/issues/13647

* dictionary modularization

* Remove unused SQL statement definition

* unit-tested thread safety of dictionaries; removed data race conditions on dictionaries and strings; dictionaries now can detect if the caller is holds a write lock and automatically all the calls become their unsafe versions; all direct calls to unsafe version is eliminated

* remove worker_is_idle() from the exit of service functions, because we lose the lock time between loops

* rewritten dictionary to have 2 separate locks, one for indexing and another for traversal

* Update collectors/cgroups.plugin/sys_fs_cgroup.c

Co-authored-by: Vladimir Kobal <vlad@prokk.net>

* Update collectors/cgroups.plugin/sys_fs_cgroup.c

Co-authored-by: Vladimir Kobal <vlad@prokk.net>

* Update collectors/proc.plugin/proc_net_dev.c

Co-authored-by: Vladimir Kobal <vlad@prokk.net>

* fix memory leak in rrdset cache_dir

* minor dictionary changes

* dont use index locks in single threaded

* obsolete dict option

* rrddim options and flags separation; rrdset_done() optimization to keep array of reference pointers to rrddim;

* fix jump on uninitialized value in dictionary; remove double free of cache_dir

* addressed codacy findings

* removed debugging code

* use the private refcount on dictionaries

* make dictionary item desctructors work on dictionary destruction; strictier control on dictionary API; proper cleanup sequence on rrddim;

* more dictionary statistics

* global statistics about dictionary operations, memory, items, callbacks

* dictionary support for views - missing the public API

* removed warning about unused parameter

* chart and context name for cloud

* chart and context name for cloud, again

* dictionary statistics fixed; first implementation of dictionary views - not currently used

* only the master can globally delete an item

* context needs netdata prefix

* fix context and chart it of spins

* fix for host variables when health is not enabled

* run garbage collector on item insert too

* Fix info message; remove extra "using"

* update dict unittest for new placement of garbage collector

* we need RRDHOST->rrdvars for maintaining custom host variables

* Health initialization needs the host->host_uuid

* split STRING to its own files; no code changes other than that

* initialize health unconditionally

* unit tests do not pollute the global scope with their variables

* Skip initialization when creating archived hosts on startup. When a child connects it will initialize properly

Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
Co-authored-by: Vladimir Kobal <vlad@prokk.net>

1 files changed, 57 insertions, 40 deletions

diff --git a/database/rrdlabels.c b/database/rrdlabels.c
index 6ab1ea66b4..6472743122 100644
--- a/database/rrdlabels.c
+++ b/database/rrdlabels.c
@@ -478,9 +478,7 @@ typedef struct rrdlabel {
     RRDLABEL_SRC label_source;
 } RRDLABEL;
 
-static void rrdlabel_insert_callback(const char *name, void *value, void *data) {
-    (void)name;
-    DICTIONARY *dict = (DICTIONARY *)data; (void)dict;
+static void rrdlabel_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *dict_ptr __maybe_unused) {
     RRDLABEL *lb = (RRDLABEL *)value;
 
     // label_value is already allocated by the STRING
@@ -488,42 +486,43 @@ static void rrdlabel_insert_callback(const char *name, void *value, void *data)
     lb->label_source &= ~RRDLABEL_FLAG_OLD;
 }
 
-static void rrdlabel_delete_callback(const char *name, void *value, void *data) {
-    (void)name;
-    DICTIONARY *dict = (DICTIONARY *)data; (void)dict;
+static void rrdlabel_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *dict_ptr __maybe_unused) {
     RRDLABEL *lb = (RRDLABEL *)value;
 
     string_freez(lb->label_value);
     lb->label_value = NULL;
 }
 
-static void rrdlabel_conflict_callback(const char *name, void *oldvalue, void *newvalue, void *data) {
-    (void)name;
-    DICTIONARY *dict = (DICTIONARY *)data; (void)dict;
+static bool rrdlabel_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *oldvalue, void *newvalue, void *dict_ptr __maybe_unused) {
     RRDLABEL *lbold = (RRDLABEL *)oldvalue;
     RRDLABEL *lbnew = (RRDLABEL *)newvalue;
 
     if(lbold->label_value == lbnew->label_value) {
         // they are the same
+
         lbold->label_source |=  lbnew->label_source;
         lbold->label_source |=  RRDLABEL_FLAG_OLD;
         lbold->label_source &= ~RRDLABEL_FLAG_NEW;
 
         // free the new one
         string_freez(lbnew->label_value);
+
+        return false;
     }
-    else {
-        // they are different
-        string_freez(lbold->label_value);
-        lbold->label_value  =   lbnew->label_value;
-        lbold->label_source =   lbnew->label_source;
-        lbold->label_source |=  RRDLABEL_FLAG_NEW;
-        lbold->label_source &= ~RRDLABEL_FLAG_OLD;
-    }
+
+    // they are different
+
+    string_freez(lbold->label_value);
+    lbold->label_value  =   lbnew->label_value;
+    lbold->label_source =   lbnew->label_source;
+    lbold->label_source |=  RRDLABEL_FLAG_NEW;
+    lbold->label_source &= ~RRDLABEL_FLAG_OLD;
+
+    return true;
 }
 
 DICTIONARY *rrdlabels_create(void) {
-    DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
+    DICTIONARY *dict = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
     dictionary_register_insert_callback(dict, rrdlabel_insert_callback, dict);
     dictionary_register_delete_callback(dict, rrdlabel_delete_callback, dict);
     dictionary_register_conflict_callback(dict, rrdlabel_conflict_callback, dict);
@@ -623,7 +622,7 @@ void rrdlabels_add_pair(DICTIONARY *dict, const char *string, RRDLABEL_SRC ls) {
 // rrdlabels_get_value_to_buffer_or_null()
 
 void rrdlabels_get_value_to_buffer_or_null(DICTIONARY *labels, BUFFER *wb, const char *key, const char *quote, const char *null) {
-    DICTIONARY_ITEM *acquired_item = dictionary_get_and_acquire_item(labels, key);
+    const DICTIONARY_ITEM *acquired_item = dictionary_get_and_acquire_item(labels, key);
     RRDLABEL *lb = dictionary_acquired_item_value(acquired_item);
 
     if(lb && lb->label_value)
@@ -638,7 +637,7 @@ void rrdlabels_get_value_to_buffer_or_null(DICTIONARY *labels, BUFFER *wb, const
 // rrdlabels_get_value_to_char_or_null()
 
 void rrdlabels_get_value_to_char_or_null(DICTIONARY *labels, char **value, const char *key) {
-    DICTIONARY_ITEM *acquired_item = dictionary_get_and_acquire_item(labels, key);
+    const DICTIONARY_ITEM *acquired_item = dictionary_get_and_acquire_item(labels, key);
     RRDLABEL *lb = dictionary_acquired_item_value(acquired_item);
 
     *value = (lb && lb->label_value) ? strdupz(string2str(lb->label_value)) : NULL;
@@ -650,10 +649,7 @@ void rrdlabels_get_value_to_char_or_null(DICTIONARY *labels, char **value, const
 // rrdlabels_unmark_all()
 // remove labels RRDLABEL_FLAG_OLD and RRDLABEL_FLAG_NEW from all dictionary items
 
-static int remove_flags_old_new(const char *name, void *value, void *data) {
-    (void)name;
-    (void)data;
-
+static int remove_flags_old_new(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data __maybe_unused) {
     RRDLABEL *lb = (RRDLABEL *)value;
 
     if(lb->label_source & RRDLABEL_FLAG_OLD) lb->label_source &= ~RRDLABEL_FLAG_OLD;
@@ -671,12 +667,13 @@ void rrdlabels_unmark_all(DICTIONARY *labels) {
 // rrdlabels_remove_all_unmarked()
 // remove dictionary items that are neither old, nor new
 
-static int remove_not_old_not_new_callback(const char *name, void *value, void *data) {
+static int remove_not_old_not_new_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
+    const char *name = dictionary_acquired_item_name(item);
     DICTIONARY *dict = (DICTIONARY *)data;
     RRDLABEL *lb = (RRDLABEL *)value;
 
     if(!(lb->label_source & (RRDLABEL_FLAG_OLD | RRDLABEL_FLAG_NEW | RRDLABEL_FLAG_PERMANENT))) {
-        dictionary_del_having_write_lock(dict, name);
+        dictionary_del(dict, name);
         return 1;
     }
 
@@ -696,7 +693,8 @@ struct labels_walkthrough {
     void *data;
 };
 
-static int labels_walkthrough_callback(const char *name, void *value, void *data) {
+static int labels_walkthrough_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
+    const char *name = dictionary_acquired_item_name(item);
     struct labels_walkthrough *d = (struct labels_walkthrough *)data;
     RRDLABEL *lb = (RRDLABEL *)value;
 
@@ -728,7 +726,8 @@ int rrdlabels_sorted_walkthrough_read(DICTIONARY *labels, int (*callback)(const
 // rrdlabels_migrate_to_these()
 // migrate an existing label list to a new list, INPLACE
 
-static int copy_label_to_dictionary_callback(const char *name, void *value, void *data) {
+static int copy_label_to_dictionary_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
+    const char *name = dictionary_acquired_item_name(item);
     DICTIONARY *dst = (DICTIONARY *)data;
     RRDLABEL *lb = (RRDLABEL *)value;
     labels_add_already_sanitized(dst, name, string2str(lb->label_value), lb->label_source);
@@ -765,7 +764,8 @@ struct simple_pattern_match_name_value {
     char equal;
 };
 
-static int simple_pattern_match_name_only_callback(const char *name, void *value, void *data) {
+static int simple_pattern_match_name_only_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
+    const char *name = dictionary_acquired_item_name(item);
     struct simple_pattern_match_name_value *t = (struct simple_pattern_match_name_value *)data;
     (void)value;
 
@@ -775,7 +775,8 @@ static int simple_pattern_match_name_only_callback(const char *name, void *value
     return 0;
 }
 
-static int simple_pattern_match_name_and_value_callback(const char *name, void *value, void *data) {
+static int simple_pattern_match_name_and_value_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
+    const char *name = dictionary_acquired_item_name(item);
     struct simple_pattern_match_name_value *t = (struct simple_pattern_match_name_value *)data;
     RRDLABEL *lb = (RRDLABEL *)value;
 
@@ -841,7 +842,9 @@ bool rrdlabels_match_simple_pattern(DICTIONARY *labels, const char *simple_patte
 // ----------------------------------------------------------------------------
 // Log all labels
 
-static int rrdlabels_log_label_to_buffer_callback(const char *name, void *value, void *data) {
+static int rrdlabels_log_label_to_buffer_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
+    const char *name = dictionary_acquired_item_name(item);
+
     BUFFER *wb = (BUFFER *)data;
     RRDLABEL *lb = (RRDLABEL *)value;
 
@@ -891,7 +894,8 @@ struct labels_to_buffer {
     size_t count;
 };
 
-static int label_to_buffer_callback(const char *name, void *value, void *data) {
+static int label_to_buffer_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
+    const char *name = dictionary_acquired_item_name(item);
     struct labels_to_buffer *t = (struct labels_to_buffer *)data;
     RRDLABEL *lb = (RRDLABEL *)value;
 
@@ -961,17 +965,30 @@ void rrdset_update_rrdlabels(RRDSET *st, DICTIONARY *new_rrdlabels) {
     if (new_rrdlabels)
         rrdlabels_migrate_to_these(st->rrdlabels, new_rrdlabels);
 
-    rrdcalc_update_rrdlabels(st);
+    rrdset_save_rrdlabels_to_sql(st);
+}
+
+void rrdset_save_rrdlabels_to_sql(RRDSET *st) {
+    if(!st->rrdlabels) return;
+
+    size_t old_version = st->rrdlabels_last_saved_version;
+    size_t new_version = dictionary_version(st->rrdlabels);
+
+    if(new_version != old_version) {
+        // TODO - we should also cleanup sqlite from old new_rrdlabels that have been removed
 
-    // TODO - we should also cleanup sqlite from old new_rrdlabels that have been removed
-    BUFFER  *sql_buf = buffer_create(1024);
-    struct label_str tmp = {.sql = sql_buf, .count = 0 };
-    uuid_unparse_lower(*st->chart_uuid, tmp.uuid_str);
-    rrdlabels_walkthrough_read(st->rrdlabels, chart_label_store_to_sql_callback, &tmp);
-    db_execute(buffer_tostring(sql_buf));
-    buffer_free(sql_buf);
+        BUFFER *sql_buf = buffer_create(1024);
+        struct label_str tmp = {.sql = sql_buf, .count = 0};
+        uuid_unparse_lower(st->chart_uuid, tmp.uuid_str);
+        rrdlabels_walkthrough_read(st->rrdlabels, chart_label_store_to_sql_callback, &tmp);
+        db_execute(buffer_tostring(sql_buf));
+        buffer_free(sql_buf);
+
+        st->rrdlabels_last_saved_version = new_version;
+    }
 }
 
+
 // ----------------------------------------------------------------------------
 // rrdlabels unit test