Rrdcontext (#13335)

* type checking on dictionary return values

* first STRING implementation, used by DICTIONARY and RRDLABEL

* enable AVL compilation of STRING

* Initial functions to store context info

* Call simple test functions

* Add host_id when getting charts

* Allow host to be null and in this case it will process the localhost

* Simplify init
Do not use strdupz - link directly to sqlite result set

* Init the database during startup

* make it compile - no functionality yet

* intermediate commit

* intermidiate

* first interface to sql

* loading instances

* check if we need to update cloud

* comparison of rrdcontext on conflict

* merge context titles

* rrdcontext public interface; statistics on STRING; scratchpad on DICTIONARY

* dictionaries maintain version numbers; rrdcontext api

* cascading changes

* first operational cleanup

* string unittest

* proper cleanup of referenced dictionaries

* added rrdmetrics

* rrdmetric starting retention

* Add fields to context
Adjuct context creation and delete

* Memory cleanup

* Fix get context list
Fix memory double free in tests
Store context with two hosts

* calculated retention

* rrdcontext retention with collection

* Persist database and shutdown

* loading all from sql

* Get chart list and dimension list changes

* fully working attempt 1

* fully working attempt 2

* missing archived flag from log

* fixed archived / collected

* operational

* proper cleanup

* cleanup - implemented all interface functions - dictionary react callback triggers after the dictionary is unlocked

* track all reasons for changes

* proper tracking of reasons of changes

* fully working thread

* better versioning of contexts

* fix string indexing with AVL

* running version per context vs hub version; ifdef dbengine

* added option to disable rrdmetrics

* release old context when a chart changes context

* cleanup properly

* renamed config

* cleanup contexts; general cleanup;

* deletion inline with dequeue; lots of cleanup; child connected/disconnected

* ml should start after rrdcontext

* added missing NULL to ri->rrdset; rrdcontext flags are now only changed under a mutex lock

* fix buggy STRING under AVL

* Rework database initialization
Add migration logic to the context database

* fix data race conditions during context deletion

* added version hash algorithm

* fix string over AVL

* update aclk-schemas

* compile new ctx related protos

* add ctx stream message utils

* add context messages

* add dummy rx message handlers

* add the new topics

* add ctx capability

* add helper functions to send the new messages

* update cmake build to not fail

* update topic names

* handle rrdcontext_enabled

* add more functions

* fatal on OOM cases instead of return NULL

* silence unknown query type error

* fully working attempt 1

* fully working attempt 2

* allow compiling without ACLK

* added family to the context

* removed excess character in UUID

* smarter merging of titles and families

* Database migration code to add family
Add family to SQL_CHART_DATA and VERSIONED_CONTEXT_DATA

* add family to context message

* enable ctx in communication

* hardcoded enabled contexts

* Add hard code for CTX

* add update node collectors to json

* add context message log

* fix log about last_time_t

* fix collected flags for queued items

* prevent crash on charts cleanup

* fix bug in AVL indexing of dictionaries; make sure react callback of dictionaries has a reference counter, which is acquired while the dictionary is locked

* fixed dictionary unittest

* strict policy to cleanup and garbage collector

* fix db rotation and garbage collection timings

* remove deadlock

* proper garbage collection - a lot faster retention recalculation

* Added not NULL in database columns
Remove migration code for context -- we will ship with version 1 of the table schema
Added define for query in tests to detect localhost

* Use UUID_STR_LEN instead of GUID_LEN + 1
Use realistic timestamps when adding test data in the database

* Add NULL checks for passed parameters

* Log deleted context when compiled with NETDATA_INTERNAL_CHECKS

* Error checking for null host id

* add missing ContextsCheckpoint log convertor

* Fix spelling in VACCUM

* Hold additional information for host -- prepare to load archived hosts on startup

* Make sure claim id is valid

* is_get_claimed is actually get the current claim id

* Simplify ctx get chart list query

* remove env negotiation

* fix string unittest when there are some strings already in the index

* propagate live-retention flag upstream; cleanup all update reasons; updated instances logging; automated attaching started/stopped collecting flags;

* first implementation of /api/v1/contexts

* full contexts API; updated swagger

* disabled debugging; rrdcontext enabled by default

* final cleanup and renaming of global variables

* return current time on currently collected contexts, charts and dimensions

* added option "deepscan" to the API to have the server refresh the retention and recalculate the contexts on the fly

* fixed identation of yaml

* Add constrains to the host table

* host->node_id may not be available

* new capabilities

* lock the context while rendering json

* update aclk-schemas

* added permanent labels to all charts about plugin, module and family; added labels to all proc plugin modules

* always add the labels

* allow merging of families down to [x]

* dont show uuids by default, added option to enable them; response is now accepting after,before to show only data for a specific timeframe; deleted items are only shown when "deleted" is requested; hub version is now shown when "queue" is requested

* Use the localhost claim id

* Fix to handle host constrains better

* cgroups: add "k8s." prefix to chart context in k8s

* Improve sqlite metadata version migration check

* empty values set to "[none]"; fix labels unit test to reflect that

* Check if we reached the version we want first (address CODACY report re: Array index 'i' is used before limits check)

* Rewrite condition to address CODACY report (Redundant condition: t->filter_callback. '!A || (A && B)' is equivalent to '!A || B')

* Properly unlock context

* fixed memory leak on rrdcontexts - it was not freeing all dictionaries in rrdhost; added wait of up to 100ms on dictionary_destroy() to give time to dictionaries to release their items before destroying them

* fixed memory leak on rrdlabels not freed on rrdinstances

* fixed leak when dimensions and charts are redefined

* Mark entries for charts and dimensions as submitted to the cloud 3600 seconds after their creation
Mark entries for charts and dimensions as updated (confirmed by the cloud) 1800 seconds after their submission

* renamed struct string

* update cgroups alarms

* fixed codacy suggestions

* update dashboard info

* fix k8s_cgroup_10s_received_packets_storm alarm

* added filtering options to /api/v1/contexts and /api/v1/context

* fix eslint

* fix eslint

* Fix pointer binding for host / chart uuids

* Fix cgroups unit tests

* fixed non-retention updates not propagated upstream

* removed non-fatal fatals

* Remove context from 2 way string merge.

* Move string_2way_merge to dictionary.c

* Add 2-way string merge tests.

* split long lines

* fix indentation in netdata-swagger.yaml

* update netdata-swagger.json

* yamllint please

* remove the deleted flag when a context is collected

* fix yaml warning in swagger

* removed non-fatal fatals

* charts should now be able to switch contexts

* allow deletion of unused metrics, instances and contexts

* keep the queued flag

* cleanup old rrdinstance labels

* dont hide objects when there is no filter; mark objects as deleted when there are no sub-objects

* delete old instances once they changed context

* delete all instances and contexts that do not have sub-objects

* more precise transitions

* Load archived hosts on startup (part 1)

* update the queued time every time

* disable by default; dedup deleted dimensions after snapshot

* Load archived hosts on startup (part 2)

* delayed processing of events until charts are being collected

* remove dont-trigger flag when object is collected

* polish all triggers given the new dont_process flag

* Remove always true condition
Enums for readbility / create_host_callback only if ACLK is enabled (for now)

* Skip retention message if context streaming is enabled
Add messages in the access log if context streaming is enabled

* Check for node id being a UUID that can be parsed
Improve error check / reporting when loading archived hosts and creating ACLK sync threads

* collected, archived, deleted are now mutually exclusive

* Enable the "orphan" handling for now
Remove dead code
Fix memory leak on free host

* Queue charts and dimensions will be no-op if host is set to stream contexts

* removed unused parameter and made sure flags are set on rrdcontext insert

* make the rrdcontext thread abort mid-work when exiting

* Skip chart hash computation and storage if contexts streaming is enabled

Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
Co-authored-by: Timo <timotej@netdata.cloud>
Co-authored-by: ilyam8 <ilya@netdata.cloud>
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
Co-authored-by: Vasilis Kalintiris <vasilis@netdata.cloud>

3 files changed, 732 insertions, 161 deletions

diff --git a/libnetdata/dictionary/README.md b/libnetdata/dictionary/README.md
index e5bf3ba6ee..879c1bcc1c 100644
--- a/libnetdata/dictionary/README.md
+++ b/libnetdata/dictionary/README.md
@@ -97,7 +97,7 @@ This call is used to get the value of an item, given its name. It utilizes the `
 
 For **multi-threaded** operation, the `dictionary_get()` call gets a shared read lock on the dictionary.
 
-In clone mode, the value returned is not guaranteed to be valid, as any other thread may delete the item from the dictionary at any time. To ensure the value will be available, use `dictionary_acquire_item()`, which uses a reference counter to defer deletes until the item is released.
+In clone mode, the value returned is not guaranteed to be valid, as any other thread may delete the item from the dictionary at any time. To ensure the value will be available, use `dictionary_get_and_acquire_item()`, which uses a reference counter to defer deletes until the item is released.
 
 The format is:
 
@@ -133,7 +133,7 @@ Where:
 
 > **IMPORTANT**<br/>There is also an **unsafe** version (without locks) of this call. This is to be used when traversing the dictionary, to delete the current item. It should never be called without an active lock on the dictionary, which can only be acquired while traversing.
 
-### dictionary_acquire_item()
+### dictionary_get_and_acquire_item()
 
 This call can be used the search and get a dictionary item, while ensuring that it will be available for use, until `dictionary_acquired_item_release()` is called.
 
@@ -149,7 +149,7 @@ DICTIONARY *dict = dictionary_create(DICTIONARY_FLAGS_NONE);
 dictionary_set(dict, "name", "value", 6);
 
 // find the item we added and acquire it
-void *item = dictionary_acquire_item(dict, "name");
+void *item = dictionary_get_and_acquire_item(dict, "name");
 
 // extract its value
 char *value = (char *)dictionary_acquired_item_value(dict, item);
diff --git a/libnetdata/dictionary/dictionary.c b/libnetdata/dictionary/dictionary.c
index c15fb60184..c1325ecb54 100644
--- a/libnetdata/dictionary/dictionary.c
+++ b/libnetdata/dictionary/dictionary.c
@@ -26,7 +26,11 @@ typedef struct dictionary DICTIONARY;
 
 typedef enum name_value_flags {
     NAME_VALUE_FLAG_NONE                   = 0,
-    NAME_VALUE_FLAG_DELETED                = (1 << 0), // this item is deleted
+    NAME_VALUE_FLAG_NAME_IS_ALLOCATED      = (1 << 0), // the name pointer is a STRING
+    NAME_VALUE_FLAG_DELETED                = (1 << 1), // this item is deleted, so it is not available for traversal
+    NAME_VALUE_FLAG_NEW_OR_UPDATED         = (1 << 2), // this item is new or just updated (used by the react callback)
+
+    // IMPORTANT: IF YOU ADD ANOTHER FLAG, YOU NEED TO ALLOCATE ANOTHER BIT TO FLAGS IN NAME_VALUE !!!
 } NAME_VALUE_FLAGS;
 
 /*
@@ -38,20 +42,31 @@ typedef struct name_value {
     avl_t avl_node;
 #endif
 
+#ifdef NETDATA_INTERNAL_CHECKS
+    DICTIONARY *dict;
+#endif
+
     struct name_value *next;    // a double linked list to allow fast insertions and deletions
     struct name_value *prev;
 
-    size_t name_len;            // the size of the name, including the terminating zero
-    size_t value_len;           // the size of the value (assumed binary)
+    uint32_t refcount;          // the reference counter
+    uint32_t value_len:29;      // the size of the value (assumed binary)
+    uint8_t flags:3;            // the flags for this item
 
     void *value;                // the value of the dictionary item
-    char *name;                 // the name of the dictionary item
-
-    int refcount;               // the reference counter
-    NAME_VALUE_FLAGS flags;     // the flags for this item
+    union {
+        STRING *string_name;    // the name of the dictionary item
+        char *caller_name;      // the user supplied string pointer
+    };
 } NAME_VALUE;
 
 struct dictionary {
+#ifdef NETDATA_INTERNAL_CHECKS
+    const char *creation_function;
+    const char *creation_file;
+    size_t creation_line;
+#endif
+
     DICTIONARY_FLAGS flags;             // the flags of the dictionary
 
     NAME_VALUE *first_item;             // the double linked list base pointers
@@ -60,23 +75,28 @@ struct dictionary {
 #ifdef DICTIONARY_WITH_AVL
     avl_tree_type values_index;
     NAME_VALUE *hash_base;
+    void *(*get_thread_static_name_value)(const char *name);
 #endif
 
 #ifdef DICTIONARY_WITH_JUDYHS
     Pvoid_t JudyHSArray;                // the hash table
 #endif
 
-    netdata_rwlock_t *rwlock;           // the r/w lock when DICTIONARY_FLAG_SINGLE_THREADED is not set
+    netdata_rwlock_t rwlock;            // the r/w lock when DICTIONARY_FLAG_SINGLE_THREADED is not set
 
     void (*ins_callback)(const char *name, void *value, void *data);
     void *ins_callback_data;
 
+    void (*react_callback)(const char *name, void *value, void *data);
+    void *react_callback_data;
+
     void (*del_callback)(const char *name, void *value, void *data);
     void *del_callback_data;
 
     void (*conflict_callback)(const char *name, void *old_value, void *new_value, void *data);
     void *conflict_callback_data;
 
+    size_t version;                   // the current version of the dictionary
     size_t inserts;                   // how many index insertions have been performed
     size_t deletes;                   // how many index deletions have been performed
     size_t searches;                  // how many index searches have been performed
@@ -88,10 +108,14 @@ struct dictionary {
     long int pending_deletion_items;  // how many items of the dictionary have been deleted, but have not been removed yet
     int readers;                      // how many readers are currently using the dictionary
     int writers;                      // how many writers are currently using the dictionary
+
+    size_t scratchpad_size;           // the size of the scratchpad in bytes
+    uint8_t scratchpad[];             // variable size scratchpad requested by the caller
 };
 
 static inline void linkedlist_namevalue_unlink_unsafe(DICTIONARY *dict, NAME_VALUE *nv);
 static size_t namevalue_destroy_unsafe(DICTIONARY *dict, NAME_VALUE *nv);
+static inline const char *namevalue_get_name(NAME_VALUE *nv);
 
 // ----------------------------------------------------------------------------
 // callbacks registration
@@ -111,6 +135,11 @@ void dictionary_register_conflict_callback(DICTIONARY *dict, void (*conflict_cal
     dict->conflict_callback_data = data;
 }
 
+void dictionary_register_react_callback(DICTIONARY *dict, void (*react_callback)(const char *name, void *value, void *data), void *data) {
+    dict->react_callback = react_callback;
+    dict->react_callback_data = data;
+}
+
 // ----------------------------------------------------------------------------
 // dictionary statistics maintenance
 
@@ -120,6 +149,9 @@ long int dictionary_stats_allocated_memory(DICTIONARY *dict) {
 long int dictionary_stats_entries(DICTIONARY *dict) {
     return dict->entries;
 }
+size_t dictionary_stats_version(DICTIONARY *dict) {
+    return dict->version;
+}
 size_t dictionary_stats_searches(DICTIONARY *dict) {
     return dict->searches;
 }
@@ -135,6 +167,9 @@ size_t dictionary_stats_resets(DICTIONARY *dict) {
 size_t dictionary_stats_walkthroughs(DICTIONARY *dict) {
     return dict->walkthroughs;
 }
+size_t dictionary_stats_referenced_items(DICTIONARY *dict) {
+    return __atomic_load_n(&dict->referenced_items, __ATOMIC_SEQ_CST);
+}
 
 static inline void DICTIONARY_STATS_SEARCHES_PLUS1(DICTIONARY *dict) {
     if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS) {
@@ -146,11 +181,13 @@ static inline void DICTIONARY_STATS_SEARCHES_PLUS1(DICTIONARY *dict) {
 }
 static inline void DICTIONARY_STATS_ENTRIES_PLUS1(DICTIONARY *dict, size_t size) {
     if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS) {
+        dict->version++;
         dict->inserts++;
         dict->entries++;
         dict->memory += (long)size;
     }
     else {
+        __atomic_fetch_add(&dict->version, 1, __ATOMIC_SEQ_CST);
         __atomic_fetch_add(&dict->inserts, 1, __ATOMIC_RELAXED);
         __atomic_fetch_add(&dict->entries, 1, __ATOMIC_RELAXED);
         __atomic_fetch_add(&dict->memory, (long)size, __ATOMIC_RELAXED);
@@ -158,10 +195,12 @@ static inline void DICTIONARY_STATS_ENTRIES_PLUS1(DICTIONARY *dict, size_t size)
 }
 static inline void DICTIONARY_STATS_ENTRIES_MINUS1(DICTIONARY *dict) {
     if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS) {
+        dict->version++;
         dict->deletes++;
         dict->entries--;
     }
     else {
+        __atomic_fetch_add(&dict->version, 1, __ATOMIC_SEQ_CST);
         __atomic_fetch_add(&dict->deletes, 1, __ATOMIC_RELAXED);
         __atomic_fetch_sub(&dict->entries, 1, __ATOMIC_RELAXED);
     }
@@ -176,11 +215,13 @@ static inline void DICTIONARY_STATS_ENTRIES_MINUS_MEMORY(DICTIONARY *dict, size_
 }
 static inline void DICTIONARY_STATS_VALUE_RESETS_PLUS1(DICTIONARY *dict, size_t oldsize, size_t newsize) {
     if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS) {
+        dict->version++;
         dict->resets++;
         dict->memory += (long)newsize;
         dict->memory -= (long)oldsize;
     }
     else {
+        __atomic_fetch_add(&dict->version, 1, __ATOMIC_SEQ_CST);
         __atomic_fetch_add(&dict->resets, 1, __ATOMIC_RELAXED);
         __atomic_fetch_add(&dict->memory, (long)newsize, __ATOMIC_RELAXED);
         __atomic_fetch_sub(&dict->memory, (long)oldsize, __ATOMIC_RELAXED);
@@ -231,7 +272,7 @@ static void garbage_collect_pending_deletes_unsafe(DICTIONARY *dict) {
 
     NAME_VALUE *nv = dict->first_item;
     while(nv) {
-        if(nv->flags & NAME_VALUE_FLAG_DELETED && DICTIONARY_NAME_VALUE_REFCOUNT_GET(nv) == 0) {
+        if((nv->flags & NAME_VALUE_FLAG_DELETED) && DICTIONARY_NAME_VALUE_REFCOUNT_GET(nv) == 0) {
             NAME_VALUE *nv_next = nv->next;
 
             linkedlist_namevalue_unlink_unsafe(dict, nv);
@@ -252,31 +293,30 @@ static void garbage_collect_pending_deletes_unsafe(DICTIONARY *dict) {
 
 static inline size_t dictionary_lock_init(DICTIONARY *dict) {
     if(likely(!(dict->flags & DICTIONARY_FLAG_SINGLE_THREADED))) {
-            dict->rwlock = mallocz(sizeof(netdata_rwlock_t));
-            netdata_rwlock_init(dict->rwlock);
+        netdata_rwlock_init(&dict->rwlock);
 
-            if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS)
-                dict->flags &= ~DICTIONARY_FLAG_EXCLUSIVE_ACCESS;
+        if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS)
+            dict->flags &= ~DICTIONARY_FLAG_EXCLUSIVE_ACCESS;
 
-            return sizeof(netdata_rwlock_t);
+        return 0;
     }
 
     // we are single threaded
     dict->flags |= DICTIONARY_FLAG_EXCLUSIVE_ACCESS;
-    dict->rwlock = NULL;
     return 0;
 }
 
 static inline size_t dictionary_lock_free(DICTIONARY *dict) {
     if(likely(!(dict->flags & DICTIONARY_FLAG_SINGLE_THREADED))) {
-        netdata_rwlock_destroy(dict->rwlock);
-        freez(dict->rwlock);
-        return sizeof(netdata_rwlock_t);
+        netdata_rwlock_destroy(&dict->rwlock);
+        return 0;
     }
     return 0;
 }
 
 static void dictionary_lock(DICTIONARY *dict, char rw) {
+    if(rw == 'u' || rw == 'U') return;
+
     if(rw == 'r' || rw == 'R') {
         // read lock
         __atomic_add_fetch(&dict->readers, 1, __ATOMIC_RELAXED);
@@ -291,22 +331,24 @@ static void dictionary_lock(DICTIONARY *dict, char rw) {
 
     if(rw == 'r' || rw == 'R') {
         // read lock
-        netdata_rwlock_rdlock(dict->rwlock);
+        netdata_rwlock_rdlock(&dict->rwlock);
 
         if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS) {
-            internal_error(true, "DICTIONARY: left-over exclusive access to dictionary found");
+            internal_error(true, "DICTIONARY: left-over exclusive access to dictionary created by %s (%zu@%s) found", dict->creation_function, dict->creation_line, dict->creation_file);
             dict->flags &= ~DICTIONARY_FLAG_EXCLUSIVE_ACCESS;
         }
     }
     else {
         // write lock
-        netdata_rwlock_wrlock(dict->rwlock);
+        netdata_rwlock_wrlock(&dict->rwlock);
 
         dict->flags |= DICTIONARY_FLAG_EXCLUSIVE_ACCESS;
     }
 }
 
 static void dictionary_unlock(DICTIONARY *dict, char rw) {
+    if(rw == 'u' || rw == 'U') return;
+
     if(rw == 'r' || rw == 'R') {
         // read unlock
         __atomic_sub_fetch(&dict->readers, 1, __ATOMIC_RELAXED);
@@ -323,7 +365,7 @@ static void dictionary_unlock(DICTIONARY *dict, char rw) {
     if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS)
         dict->flags &= ~DICTIONARY_FLAG_EXCLUSIVE_ACCESS;
 
-    netdata_rwlock_unlock(dict->rwlock);
+    netdata_rwlock_unlock(&dict->rwlock);
 }
 
 // ----------------------------------------------------------------------------
@@ -371,6 +413,13 @@ static inline size_t reference_counter_free(DICTIONARY *dict) {
     return 0;
 }
 
+static int reference_counter_increase(NAME_VALUE *nv) {
+    int refcount = __atomic_add_fetch(&nv->refcount, 1, __ATOMIC_SEQ_CST);
+    if(refcount == 1)
+        fatal("DICTIONARY: request to dup item '%s' but its reference counter was zero", namevalue_get_name(nv));
+    return refcount;
+}
+
 static int reference_counter_acquire(DICTIONARY *dict, NAME_VALUE *nv) {
     int refcount;
     if(likely(dict->flags & DICTIONARY_FLAG_SINGLE_THREADED))
@@ -392,18 +441,23 @@ static int reference_counter_acquire(DICTIONARY *dict, NAME_VALUE *nv) {
     return refcount;
 }
 
-static int reference_counter_release(DICTIONARY *dict, NAME_VALUE *nv, bool can_get_write_lock) {
+static uint32_t reference_counter_release(DICTIONARY *dict, NAME_VALUE *nv, bool can_get_write_lock) {
     // this function may be called without any lock on the dictionary
     // or even when someone else has a write lock on the dictionary
     // so, we cannot check for EXCLUSIVE ACCESS
 
-    int refcount;
+    uint32_t refcount;
     if(likely(dict->flags & DICTIONARY_FLAG_SINGLE_THREADED))
-        refcount = --nv->refcount;
+        refcount = nv->refcount--;
     else
-        refcount = __atomic_sub_fetch(&nv->refcount, 1, __ATOMIC_SEQ_CST);
+        refcount = __atomic_fetch_sub(&nv->refcount, 1, __ATOMIC_SEQ_CST);
 
     if(refcount == 0) {
+        internal_error(true, "DICTIONARY: attempted to release item without references: '%s' on dictionary created by %s() (%zu@%s)", namevalue_get_name(nv), dict->creation_function, dict->creation_line, dict->creation_file);
+        fatal("DICTIONARY: attempted to release item without references: '%s'", namevalue_get_name(nv));
+    }
+
+    if(refcount == 1) {
         if((nv->flags & NAME_VALUE_FLAG_DELETED))
             DICTIONARY_STATS_PENDING_DELETES_PLUS1(dict);
 
@@ -415,9 +469,9 @@ static int reference_counter_release(DICTIONARY *dict, NAME_VALUE *nv, bool can_
     if(can_get_write_lock && DICTIONARY_STATS_PENDING_DELETES_GET(dict)) {
         // we can garbage collect now
 
-        dictionary_lock(dict, 'w');
+        dictionary_lock(dict, DICTIONARY_LOCK_WRITE);
         garbage_collect_pending_deletes_unsafe(dict);
-        dictionary_unlock(dict, 'w');
+        dictionary_unlock(dict, DICTIONARY_LOCK_WRITE);
     }
 
     return refcount;
@@ -427,12 +481,22 @@ static int reference_counter_release(DICTIONARY *dict, NAME_VALUE *nv, bool can_
 // hash table
 
 #ifdef DICTIONARY_WITH_AVL
+static inline const char *namevalue_get_name(NAME_VALUE *nv);
+
 static int name_value_compare(void* a, void* b) {
-    return strcmp(((NAME_VALUE *)a)->name, ((NAME_VALUE *)b)->name);
+    return strcmp(namevalue_get_name((NAME_VALUE *)a), namevalue_get_name((NAME_VALUE *)b));
+}
+
+static void *get_thread_static_name_value(const char *name) {
+    static __thread NAME_VALUE tmp = { 0 };
+    tmp.flags = NAME_VALUE_FLAG_NONE;
+    tmp.caller_name = (char *)name;
+    return &tmp;
 }
 
 static void hashtable_init_unsafe(DICTIONARY *dict) {
     avl_init(&dict->values_index, name_value_compare);
+    dict->get_thread_static_name_value = get_thread_static_name_value;
 }
 
 static size_t hashtable_destroy_unsafe(DICTIONARY *dict) {
@@ -440,7 +504,7 @@ static size_t hashtable_destroy_unsafe(DICTIONARY *dict) {
     return 0;
 }
 
-static inline int hashtable_delete_unsafe(DICTIONARY *dict, const char *name, size_t name_len, NAME_VALUE *nv) {
+static inline int hashtable_delete_unsafe(DICTIONARY *dict, const char *name, size_t name_len, void *nv) {
     (void)name;
     (void)name_len;
 
@@ -453,9 +517,8 @@ static inline int hashtable_delete_unsafe(DICTIONARY *dict, const char *name, si
 static inline NAME_VALUE *hashtable_get_unsafe(DICTIONARY *dict, const char *name, size_t name_len) {
     (void)name_len;
 
-    NAME_VALUE tmp;
-    tmp.name = (char *)name;
-    return (NAME_VALUE *)avl_search(&(dict->values_index), (avl_t *) &tmp);
+    void *tmp = dict->get_thread_static_name_value(name);
+    return (NAME_VALUE *)avl_search(&(dict->values_index), (avl_t *)tmp);
 }
 
 static inline NAME_VALUE **hashtable_insert_unsafe(DICTIONARY *dict, const char *name, size_t name_len) {
@@ -469,13 +532,10 @@ static inline NAME_VALUE **hashtable_insert_unsafe(DICTIONARY *dict, const char
     return &dict->hash_base;
 }
 
-static inline void hashtable_inserted_name_value_unsafe(DICTIONARY *dict, const char *name, size_t name_len, NAME_VALUE *nv) {
+static inline void hashtable_inserted_name_value_unsafe(DICTIONARY *dict, void *nv) {
     // we have our new NAME_VALUE object.
     // Let's index it.
 
-    (void)name;
-    (void)name_len;
-
     if(unlikely(avl_insert(&((dict)->values_index), (avl_t *)(nv)) != (avl_t *)nv))
         error("dictionary: INTERNAL ERROR: duplicate insertion to dictionary.");
 }
@@ -503,7 +563,7 @@ static size_t hashtable_destroy_unsafe(DICTIONARY *dict) {
 }
 
 static inline NAME_VALUE **hashtable_insert_unsafe(DICTIONARY *dict, const char *name, size_t name_len) {
-    internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: inserting to the index without exclusive access to the dictionary.");
+    internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: inserting item from the index without exclusive access to the dictionary created by %s() (%zu@%s)", dict->creation_function, dict->creation_line, dict->creation_file);
 
     JError_t J_Error;
     Pvoid_t *Rc = JudyHSIns(&dict->JudyHSArray, (void *)name, name_len, &J_Error);
@@ -522,8 +582,8 @@ static inline NAME_VALUE **hashtable_insert_unsafe(DICTIONARY *dict, const char
     return (NAME_VALUE **)Rc;
 }
 
-static inline int hashtable_delete_unsafe(DICTIONARY *dict, const char *name, size_t name_len, NAME_VALUE *nv) {
-    internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: deleting from the index without exclusive access to the dictionary.");
+static inline int hashtable_delete_unsafe(DICTIONARY *dict, const char *name, size_t name_len, void *nv) {
+    internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: deleting item from the index without exclusive access to the dictionary created by %s() (%zu@%s)", dict->creation_function, dict->creation_line, dict->creation_file);
 
     (void)nv;
     if(unlikely(!dict->JudyHSArray)) return 0;
@@ -566,10 +626,8 @@ static inline NAME_VALUE *hashtable_get_unsafe(DICTIONARY *dict, const char *nam
     }
 }
 
-static inline void hashtable_inserted_name_value_unsafe(DICTIONARY *dict, const char *name, size_t name_len, NAME_VALUE *nv) {
+static inline void hashtable_inserted_name_value_unsafe(DICTIONARY *dict, void *nv) {
     (void)dict;
-    (void)name;
-    (void)name_len;
     (void)nv;
     ;
 }
@@ -580,7 +638,7 @@ static inline void hashtable_inserted_name_value_unsafe(DICTIONARY *dict, const
 // linked list management
 
 static inline void linkedlist_namevalue_link_unsafe(DICTIONARY *dict, NAME_VALUE *nv) {
-    internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: adding item to the linked-list without exclusive access to the dictionary.");
+    internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: adding item to the linked-list without exclusive access to the dictionary created by %s() (%zu@%s)", dict->creation_function, dict->creation_line, dict->creation_file);
 
     if (unlikely(!dict->first_item)) {
         // we are the only ones here
@@ -609,7 +667,7 @@ static inline void linkedlist_namevalue_link_unsafe(DICTIONARY *dict, NAME_VALUE
 }
 
 static inline void linkedlist_namevalue_unlink_unsafe(DICTIONARY *dict, NAME_VALUE *nv) {
-    internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: removing item from the linked-list without exclusive access to the dictionary.");
+    internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: removing item from the linked-list without exclusive access to the dictionary created by %s() (%zu@%s)", dict->creation_function, dict->creation_line, dict->creation_file);
 
     if(nv->next) nv->next->prev = nv->prev;
     if(nv->prev) nv->prev->next = nv->next;
@@ -620,6 +678,31 @@ static inline void linkedlist_namevalue_unlink_unsafe(DICTIONARY *dict, NAME_VAL
 // ----------------------------------------------------------------------------
 // NAME_VALUE methods
 
+static inline size_t namevalue_set_name(DICTIONARY *dict, NAME_VALUE *nv, const char *name, size_t name_len) {
+    if(likely(dict->flags & DICTIONARY_FLAG_NAME_LINK_DONT_CLONE)) {
+        nv->caller_name = (char *)name;
+        return 0;
+    }
+
+    nv->string_name = string_strdupz(name);
+    nv->flags |= NAME_VALUE_FLAG_NAME_IS_ALLOCATED;
+    return name_len;
+}
+
+static inline size_t namevalue_free_name(DICTIONARY *dict, NAME_VALUE *nv) {
+    if(unlikely(!(dict->flags & DICTIONARY_FLAG_NAME_LINK_DONT_CLONE)))
+        string_freez(nv->string_name);
+
+    return 0;
+}
+
+static inline const char *namevalue_get_name(NAME_VALUE *nv) {
+    if(nv->flags & NAME_VALUE_FLAG_NAME_IS_ALLOCATED)
+        return string2str(nv->string_name);
+    else
+        return nv->caller_name;
+}
+
 static NAME_VALUE *namevalue_create_unsafe(DICTIONARY *dict, const char *name, size_t name_len, void *value, size_t value_len) {
     debug(D_DICTIONARY, "Creating name value entry for name '%s'.", name);
 
@@ -627,18 +710,15 @@ static NAME_VALUE *namevalue_create_unsafe(DICTIONARY *dict, const char *name, s
     NAME_VALUE *nv = mallocz(size);
     size_t allocated = size;
 
+#ifdef NETDATA_INTERNAL_CHECKS
+    nv->dict = dict;
+#endif
+
     nv->refcount = 0;
     nv->flags = NAME_VALUE_FLAG_NONE;
-    nv->name_len = name_len;
     nv->value_len = value_len;
 
-    if(likely(dict->flags & DICTIONARY_FLAG_NAME_LINK_DONT_CLONE))
-        nv->name = (char *)name;
-    else {
-        nv->name = mallocz(name_len);
-        memcpy(nv->name, name, name_len);
-        allocated += name_len;
-    }
+    allocated += namevalue_set_name(dict, nv, name, name_len);
 
     if(likely(dict->flags & DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE))
         nv->value = value;
@@ -667,26 +747,26 @@ static NAME_VALUE *namevalue_create_unsafe(DICTIONARY *dict, const char *name, s
     DICTIONARY_STATS_ENTRIES_PLUS1(dict, allocated);
 
     if(dict->ins_callback)
-        dict->ins_callback(nv->name, nv->value, dict->ins_callback_data);
+        dict->ins_callback(namevalue_get_name(nv), nv->value, dict->ins_callback_data);
 
     return nv;
 }
 
 static void namevalue_reset_unsafe(DICTIONARY *dict, NAME_VALUE *nv, void *value, size_t value_len) {
-    debug(D_DICTIONARY, "Dictionary entry with name '%s' found. Changing its value.", nv->name);
+    debug(D_DICTIONARY, "Dictionary entry with name '%s' found. Changing its value.", namevalue_get_name(nv));
 
     DICTIONARY_STATS_VALUE_RESETS_PLUS1(dict, nv->value_len, value_len);
 
     if(dict->del_callback)
-        dict->del_callback(nv->name, nv->value, dict->del_callback_data);
+        dict->del_callback(namevalue_get_name(nv), nv->value, dict->del_callback_data);
 
     if(likely(dict->flags & DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE)) {
-        debug(D_DICTIONARY, "Dictionary: linking value to '%s'", nv->name);
+        debug(D_DICTIONARY, "Dictionary: linking value to '%s'", namevalue_get_name(nv));
         nv->value = value;
         nv->value_len = value_len;
     }
     else {
-        debug(D_DICTIONARY, "Dictionary: cloning value to '%s'", nv->name);
+        debug(D_DICTIONARY, "Dictionary: cloning value to '%s'", namevalue_get_name(nv));
 
         void *oldvalue = nv->value;
         void *newvalue = NULL;
@@ -698,32 +778,31 @@ static void namevalue_reset_unsafe(DICTIONARY *dict, NAME_VALUE *nv, void *value
         nv->value = newvalue;
         nv->value_len = value_len;
 
-        debug(D_DICTIONARY, "Dictionary: freeing old value of '%s'", nv->name);
+        debug(D_DICTIONARY, "Dictionary: freeing old value of '%s'", namevalue_get_name(nv));
         freez(oldvalue);
     }
 
     if(dict->ins_callback)
-        dict->ins_callback(nv->name, nv->value, dict->ins_callback_data);
+        dict->ins_callback(namevalue_get_name(nv), nv->value, dict->ins_callback_data);
 }
 
 static size_t namevalue_destroy_unsafe(DICTIONARY *dict, NAME_VALUE *nv) {
-    debug(D_DICTIONARY, "Destroying name value entry for name '%s'.", nv->name);
+    debug(D_DICTIONARY, "Destroying name value entry for name '%s'.", namevalue_get_name(nv));
 
     if(dict->del_callback)
-        dict->del_callback(nv->name, nv->value, dict->del_callback_data);
+        dict->del_callback(namevalue_get_name(nv), nv->value, dict->del_callback_data);
 
     size_t freed = 0;
 
     if(unlikely(!(dict->flags & DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE))) {
-        debug(D_DICTIONARY, "Dictionary freeing value of '%s'", nv->name);
+        debug(D_DICTIONARY, "Dictionary freeing value of '%s'", namevalue_get_name(nv));
         freez(nv->value);
         freed += nv->value_len;
     }
 
     if(unlikely(!(dict->flags & DICTIONARY_FLAG_NAME_LINK_DONT_CLONE))) {
-        debug(D_DICTIONARY, "Dictionary freeing name '%s'", nv->name);
-        freez(nv->name);
-        freed += nv->name_len;
+        debug(D_DICTIONARY, "Dictionary freeing name '%s'", namevalue_get_name(nv));
+        freed += namevalue_free_name(dict, nv);
     }
 
     freez(nv);
@@ -747,16 +826,20 @@ static bool name_value_can_be_deleted(DICTIONARY *dict, NAME_VALUE *nv) {
 
 // ----------------------------------------------------------------------------
 // API - dictionary management
-
-DICTIONARY *dictionary_create(DICTIONARY_FLAGS flags) {
+#ifdef NETDATA_INTERNAL_CHECKS
+DICTIONARY *dictionary_create_advanced_with_trace(DICTIONARY_FLAGS flags, size_t scratchpad_size, const char *function, size_t line, const char *file) {
+#else
+DICTIONARY *dictionary_create_advanced(DICTIONARY_FLAGS flags, size_t scratchpad_size) {
+#endif
     debug(D_DICTIONARY, "Creating dictionary.");
 
     if(unlikely(flags & DICTIONARY_FLAGS_RESERVED))
         flags &= ~DICTIONARY_FLAGS_RESERVED;
 
-    DICTIONARY *dict = callocz(1, sizeof(DICTIONARY));
-    size_t allocated = sizeof(DICTIONARY);
+    DICTIONARY *dict = callocz(1, sizeof(DICTIONARY) + scratchpad_size);
+    size_t allocated = sizeof(DICTIONARY) + scratchpad_size;
 
+    dict->scratchpad_size = scratchpad_size;
     dict->flags = flags;
     dict->first_item = dict->last_item = NULL;
 
@@ -765,29 +848,87 @@ DICTIONARY *dictionary_create(DICTIONARY_FLAGS flags) {
     dict->memory = (long)allocated;
 
     hashtable_init_unsafe(dict);
+
+#ifdef NETDATA_INTERNAL_CHECKS
+    dict->creation_function = function;
+    dict->creation_file = file;
+    dict->creation_line = line;
+#endif
+
     return (DICTIONARY *)dict;
 }
 
+void *dictionary_scratchpad(DICTIONARY *dict) {
+    return &dict->scratchpad;
+}
+
 size_t dictionary_destroy(DICTIONARY *dict) {
     if(!dict) return 0;
 
-    if(dict->referenced_items) {
+    NAME_VALUE *nv;
+
+    debug(D_DICTIONARY, "Destroying dictionary.");
+
+    long referenced_items = 0;
+    size_t retries = 0;
+    do {
+        referenced_items = __atomic_load_n(&dict->referenced_items, __ATOMIC_SEQ_CST);
+        if (referenced_items) {
+            dictionary_lock(dict, DICTIONARY_LOCK_WRITE);
+
+            // there are referenced items
+            // delete all items individually, so that only the referenced will remain
+            NAME_VALUE *nv_next;
+            for (nv = dict->first_item; nv; nv = nv_next) {
+                nv_next = nv->next;
+                size_t refcount = DICTIONARY_NAME_VALUE_REFCOUNT_GET(nv);
+                if (!refcount && !(nv->flags & NAME_VALUE_FLAG_DELETED))
+                    dictionary_del_unsafe(dict, namevalue_get_name(nv));
+            }
+
+            internal_error(
+                retries == 0,
+                "DICTIONARY: waiting (try %zu) for destruction of dictionary created from %s() %zu@%s, because it has %ld referenced items in it (%ld total).",
+                retries + 1,
+                dict->creation_function,
+                dict->creation_line,
+                dict->creation_file,
+                referenced_items,