RRD structures managed by dictionaries (#13646)

* rrdset - in progress

* rrdset optimal constructor; rrdset conflict

* rrdset final touches

* re-organization of rrdset object members

* prevent use-after-free

* dictionary dfe supports also counting of iterations

* rrddim managed by dictionary

* rrd.h cleanup

* DICTIONARY_ITEM now is referencing actual dictionary items in the code

* removed rrdset linked list

* Revert "removed rrdset linked list"

This reverts commit 690d6a588b4b99619c2c5e10f84e8f868ae6def5.

* removed rrdset linked list

* added comments

* Switch chart uuid to static allocation in rrdset
Remove unused functions

* rrdset_archive() and friends...

* always create rrdfamily

* enable ml_free_dimension

* rrddim_foreach done with dfe

* most custom rrddim loops replaced with rrddim_foreach

* removed accesses to rrddim->dimensions

* removed locks that are no longer needed

* rrdsetvar is now managed by the dictionary

* set rrdset is rrdsetvar, fixes https://github.com/netdata/netdata/pull/13646#issuecomment-1242574853

* conflict callback of rrdsetvar now properly checks if it has to reset the variable

* dictionary registered callbacks accept as first parameter the DICTIONARY_ITEM

* dictionary dfe now uses internal counter to report; avoided excess variables defined with dfe

* dictionary walkthrough callbacks get dictionary acquired items

* dictionary reference counters that can be dupped from zero

* added advanced functions for get and del

* rrdvar managed by dictionaries

* thread safety for rrdsetvar

* faster rrdvar initialization

* rrdvar string lengths should match in all add, del, get functions

* rrdvar internals hidden from the rest of the world

* rrdvar is now acquired throughout netdata

* hide the internal structures of rrdsetvar

* rrdsetvar is now acquired through out netdata

* rrddimvar managed by dictionary; rrddimvar linked list removed; rrddimvar structures hidden from the rest of netdata

* better error handling

* dont create variables if not initialized for health

* dont create variables if not initialized for health again

* rrdfamily is now managed by dictionaries; references of it are acquired dictionary items

* type checking on acquired objects

* rrdcalc renaming of functions

* type checking for rrdfamily_acquired

* rrdcalc managed by dictionaries

* rrdcalc double free fix

* host rrdvars is always needed

* attempt to fix deadlock 1

* attempt to fix deadlock 2

* Remove unused variable

* attempt to fix deadlock 3

* snprintfz

* rrdcalc index in rrdset fix

* Stop storing active charts and computing chart hashes

* Remove store active chart function

* Remove compute chart hash function

* Remove sql_store_chart_hash function

* Remove store_active_dimension function

* dictionary delayed destruction

* formatting and cleanup

* zero dictionary base on rrdsetvar

* added internal error to log delayed destructions of dictionaries

* typo in rrddimvar

* added debugging info to dictionary

* debug info

* fix for rrdcalc keys being empty

* remove forgotten unlock

* remove deadlock

* Switch to metadata version 5 and drop
  chart_hash
  chart_hash_map
  chart_active
  dimension_active
  v_chart_hash

* SQL cosmetic changes

* do not busy wait while destroying a referenced dictionary

* remove deadlock

* code cleanup; re-organization;

* fast cleanup and flushing of dictionaries

* number formatting fixes

* do not delete configured alerts when archiving a chart

* rrddim obsolete linked list management outside dictionaries

* removed duplicate contexts call

* fix crash when rrdfamily is not initialized

* dont keep rrddimvar referenced

* properly cleanup rrdvar

* removed some locks

* Do not attempt to cleanup chart_hash / chart_hash_map

* rrdcalctemplate managed by dictionary

* register callbacks on the right dictionary

* removed some more locks

* rrdcalc secondary index replaced with linked-list; rrdcalc labels updates are now executed by health thread

* when looking up for an alarm look using both chart id and chart name

* host initialization a bit more modular

* init rrdlabels on host update

* preparation for dictionary views

* improved comment

* unused variables without internal checks

* service threads isolation and worker info

* more worker info in service thread

* thread cancelability debugging with internal checks

* strings data races addressed; fixes https://github.com/netdata/netdata/issues/13647

* dictionary modularization

* Remove unused SQL statement definition

* unit-tested thread safety of dictionaries; removed data race conditions on dictionaries and strings; dictionaries now can detect if the caller is holds a write lock and automatically all the calls become their unsafe versions; all direct calls to unsafe version is eliminated

* remove worker_is_idle() from the exit of service functions, because we lose the lock time between loops

* rewritten dictionary to have 2 separate locks, one for indexing and another for traversal

* Update collectors/cgroups.plugin/sys_fs_cgroup.c

Co-authored-by: Vladimir Kobal <vlad@prokk.net>

* Update collectors/cgroups.plugin/sys_fs_cgroup.c

Co-authored-by: Vladimir Kobal <vlad@prokk.net>

* Update collectors/proc.plugin/proc_net_dev.c

Co-authored-by: Vladimir Kobal <vlad@prokk.net>

* fix memory leak in rrdset cache_dir

* minor dictionary changes

* dont use index locks in single threaded

* obsolete dict option

* rrddim options and flags separation; rrdset_done() optimization to keep array of reference pointers to rrddim;

* fix jump on uninitialized value in dictionary; remove double free of cache_dir

* addressed codacy findings

* removed debugging code

* use the private refcount on dictionaries

* make dictionary item desctructors work on dictionary destruction; strictier control on dictionary API; proper cleanup sequence on rrddim;

* more dictionary statistics

* global statistics about dictionary operations, memory, items, callbacks

* dictionary support for views - missing the public API

* removed warning about unused parameter

* chart and context name for cloud

* chart and context name for cloud, again

* dictionary statistics fixed; first implementation of dictionary views - not currently used

* only the master can globally delete an item

* context needs netdata prefix

* fix context and chart it of spins

* fix for host variables when health is not enabled

* run garbage collector on item insert too

* Fix info message; remove extra "using"

* update dict unittest for new placement of garbage collector

* we need RRDHOST->rrdvars for maintaining custom host variables

* Health initialization needs the host->host_uuid

* split STRING to its own files; no code changes other than that

* initialize health unconditionally

* unit tests do not pollute the global scope with their variables

* Skip initialization when creating archived hosts on startup. When a child connects it will initialize properly

Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
Co-authored-by: Vladimir Kobal <vlad@prokk.net>

5 files changed, 17 insertions, 32 deletions

diff --git a/registry/registry.c b/registry/registry.c
index afacdfebe1..bc196b5cfa 100644
--- a/registry/registry.c
+++ b/registry/registry.c
@@ -108,9 +108,7 @@ static int registry_json_person_url_callback(void *entry, void *data) {
 }
 
 // callback for rendering MACHINE_URLs
-static int registry_json_machine_url_callback(const char *name, void *entry, void *data) {
-    (void)name;
-
+static int registry_json_machine_url_callback(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *data) {
     REGISTRY_MACHINE_URL *mu = (REGISTRY_MACHINE_URL *)entry;
     struct registry_json_walk_person_urls_callback *c = (struct registry_json_walk_person_urls_callback *)data;
     struct web_client *w = c->w;
@@ -443,8 +441,8 @@ void registry_statistics(void) {
     }
     else rrdset_next(stm);
 
-    rrddim_set(stm, "persons",       registry.persons_memory + dictionary_stats_allocated_memory(registry.persons));
-    rrddim_set(stm, "machines",      registry.machines_memory + dictionary_stats_allocated_memory(registry.machines));
+    rrddim_set(stm, "persons",       registry.persons_memory + dictionary_stats_for_registry(registry.persons));
+    rrddim_set(stm, "machines",      registry.machines_memory + dictionary_stats_for_registry(registry.machines));
     rrddim_set(stm, "urls",          registry.urls_memory);
     rrddim_set(stm, "persons_urls",  registry.persons_urls_memory);
     rrddim_set(stm, "machines_urls", registry.machines_urls_memory);
diff --git a/registry/registry_db.c b/registry/registry_db.c
index db53ff7e00..ae74aa5304 100644
--- a/registry/registry_db.c
+++ b/registry/registry_db.c
@@ -11,9 +11,7 @@ int registry_db_should_be_saved(void) {
 // ----------------------------------------------------------------------------
 // INTERNAL FUNCTIONS FOR SAVING REGISTRY OBJECTS
 
-static int registry_machine_save_url(const char *name, void *entry, void *file) {
-    (void)name;
-
+static int registry_machine_save_url(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *file) {
     REGISTRY_MACHINE_URL *mu = entry;
     FILE *fp = file;
 
@@ -32,8 +30,7 @@ static int registry_machine_save_url(const char *name, void *entry, void *file)
     return ret;
 }
 
-static int registry_machine_save(const char *name, void *entry, void *file) {
-    (void)name;
+static int registry_machine_save(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *file) {
 
     REGISTRY_MACHINE *m = entry;
     FILE *fp = file;
@@ -79,9 +76,7 @@ static inline int registry_person_save_url(void *entry, void *file) {
     return ret;
 }
 
-static inline int registry_person_save(const char *name, void *entry, void *file) {
-    (void)name;
-
+static inline int registry_person_save(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *file) {
     REGISTRY_PERSON *p = entry;
     FILE *fp = file;
 
diff --git a/registry/registry_init.c b/registry/registry_init.c
index bae2ac5c57..ba4250ef38 100644
--- a/registry/registry_init.c
+++ b/registry/registry_init.c
@@ -76,8 +76,8 @@ int registry_init(void) {
     netdata_mutex_init(&registry.lock);
 
     // create dictionaries
-    registry.persons = dictionary_create(REGISTRY_DICTIONARY_FLAGS);
-    registry.machines = dictionary_create(REGISTRY_DICTIONARY_FLAGS);
+    registry.persons = dictionary_create(REGISTRY_DICTIONARY_OPTIONS);
+    registry.machines = dictionary_create(REGISTRY_DICTIONARY_OPTIONS);
     avl_init(&registry.registry_urls_root_index, registry_url_compare);
 
     // load the registry database
@@ -93,9 +93,7 @@ int registry_init(void) {
     return 0;
 }
 
-static int machine_urls_delete_callback(const char *name, void *entry, void *data) {
-    (void)name;
-
+static int machine_urls_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *data) {
     REGISTRY_MACHINE *m = (REGISTRY_MACHINE *)data;
     (void)m;
 
@@ -110,10 +108,7 @@ static int machine_urls_delete_callback(const char *name, void *entry, void *dat
     return 1;
 }
 
-static int machine_delete_callback(const char *name, void *entry, void *data) {
-    (void)name;
-    (void)data;
-
+static int machine_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *data __maybe_unused) {
     REGISTRY_MACHINE *m = (REGISTRY_MACHINE *)entry;
     int ret = dictionary_walkthrough_read(m->machine_urls, machine_urls_delete_callback, m);
 
@@ -122,10 +117,7 @@ static int machine_delete_callback(const char *name, void *entry, void *data) {
 
     return ret + 1;
 }
-static int registry_person_del_callback(const char *name, void *entry, void *d) {
-    (void)name;
-    (void)d;
-
+static int registry_person_del_callback(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *d __maybe_unused) {
     REGISTRY_PERSON *p = (REGISTRY_PERSON *)entry;
 
     debug(D_REGISTRY, "Registry: registry_person_del('%s'): deleting person", p->guid);
diff --git a/registry/registry_internals.h b/registry/registry_internals.h
index 9e0f114775..a49b4a4c3d 100644
--- a/registry/registry_internals.h
+++ b/registry/registry_internals.h
@@ -8,7 +8,7 @@
 #define REGISTRY_URL_FLAGS_DEFAULT 0x00
 #define REGISTRY_URL_FLAGS_EXPIRED 0x01
 
-#define REGISTRY_DICTIONARY_FLAGS (DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE | DICTIONARY_FLAG_NAME_LINK_DONT_CLONE | DICTIONARY_FLAG_SINGLE_THREADED)
+#define REGISTRY_DICTIONARY_OPTIONS (DICT_OPTION_VALUE_LINK_DONT_CLONE | DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_SINGLE_THREADED)
 
 // ----------------------------------------------------------------------------
 // COMMON structures
diff --git a/registry/registry_machine.c b/registry/registry_machine.c
index fb345aea22..414cd16d99 100644
--- a/registry/registry_machine.c
+++ b/registry/registry_machine.c
@@ -25,9 +25,9 @@ REGISTRY_MACHINE_URL *registry_machine_url_allocate(REGISTRY_MACHINE *m, REGISTR
 
     debug(D_REGISTRY, "registry_machine_url_allocate('%s', '%s'): indexing URL in machine", m->guid, u->url);
 
-    registry.machines_urls_memory -= dictionary_stats_allocated_memory(m->machine_urls);
+    registry.machines_urls_memory -= dictionary_stats_for_registry(m->machine_urls);
     dictionary_set(m->machine_urls, u->url, mu, sizeof(REGISTRY_MACHINE_URL));
-    registry.machines_urls_memory += dictionary_stats_allocated_memory(m->machine_urls);
+    registry.machines_urls_memory += dictionary_stats_for_registry(m->machine_urls);
 
     registry_url_link(u);
 
@@ -42,7 +42,7 @@ REGISTRY_MACHINE *registry_machine_allocate(const char *machine_guid, time_t whe
     strncpyz(m->guid, machine_guid, GUID_LEN);
 
     debug(D_REGISTRY, "Registry: registry_machine_allocate('%s'): creating dictionary of urls", machine_guid);
-    m->machine_urls = dictionary_create(REGISTRY_DICTIONARY_FLAGS);
+    m->machine_urls = dictionary_create(REGISTRY_DICTIONARY_OPTIONS);
 
     m->first_t = m->last_t = (uint32_t)when;
     m->usages = 0;
@@ -50,9 +50,9 @@ REGISTRY_MACHINE *registry_machine_allocate(const char *machine_guid, time_t whe
     registry.machines_memory += sizeof(REGISTRY_MACHINE);
     registry.machines_count++;
 
-    registry.machines_urls_memory -= dictionary_stats_allocated_memory(m->machine_urls);
+    registry.machines_urls_memory -= dictionary_stats_for_registry(m->machine_urls);
     dictionary_set(registry.machines, m->guid, m, sizeof(REGISTRY_MACHINE));
-    registry.machines_urls_memory += dictionary_stats_allocated_memory(m->machine_urls);
+    registry.machines_urls_memory += dictionary_stats_for_registry(m->machine_urls);
 
     return m;
 }