Deduplicate all netdata strings (#13570)

* rrdfamily

* rrddim

* rrdset plugin and module names

* rrdset units

* rrdset type

* rrdset family

* rrdset title

* rrdset title more

* rrdset context

* rrdcalctemplate context and removal of context hash from rrdset

* strings statistics

* rrdset name

* rearranged members of rrdset

* eliminate rrdset name hash; rrdcalc chart converted to STRING

* rrdset id, eliminated rrdset hash

* rrdcalc, alarm_entry, alert_config and some of rrdcalctemplate

* rrdcalctemplate

* rrdvar

* eval_variable

* rrddimvar and rrdsetvar

* rrdhost hostname, os and tags

* fix master commits

* added thread cache; implemented string_dup without locks

* faster thread cache

* rrdset and rrddim now use dictionaries for indexing

* rrdhost now uses dictionary

* rrdfamily now uses DICTIONARY

* rrdvar using dictionary instead of AVL

* allocate the right size to rrdvar flag members

* rrdhost remaining char * members to STRING *

* better error handling on indexing

* strings now use a read/write lock to allow parallel searches to the index

* removed AVL support from dictionaries; implemented STRING with native Judy calls

* string releases should be negative

* only 31 bits are allowed for enum flags

* proper locking on strings

* string threading unittest and fixes

* fix lgtm finding

* fixed naming

* stream chart/dimension definitions at the beginning of a streaming session

* thread stack variable is undefined on thread cancel

* rrdcontext garbage collect per host on startup

* worker control in garbage collection

* relaxed deletion of rrdmetrics

* type checking on dictfe

* netdata chart to monitor rrdcontext triggers

* Group chart label updates

* rrdcontext better handling of collected rrdsets

* rrdpush incremental transmition of definitions should use as much buffer as possible

* require 1MB per chart

* empty the sender buffer before enabling metrics streaming

* fill up to 50% of buffer

* reset signaling metrics sending

* use the shared variable for status

* use separate host flag for enabling streaming of metrics

* make sure the flag is clear

* add logging for streaming

* add logging for streaming on buffer overflow

* circular_buffer proper sizing

* removed obsolete logs

* do not execute worker jobs if not necessary

* better messages about compression disabling

* proper use of flags and updating rrdset last access time every time the obsoletion flag is flipped

* monitor stream sender used buffer ratio

* Update exporting unit tests

* no need to compare label value with strcmp

* streaming send workers now monitor bandwidth

* workers now use strings

* streaming receiver monitors incoming bandwidth

* parser shift of worker ids

* minor fixes

* Group chart label updates

* Populate context with dimensions that have data

* Fix chart id

* better shift of parser worker ids

* fix for streaming compression

* properly count received bytes

* ensure LZ4 compression ring buffer does not wrap prematurely

* do not stream empty charts; do not process empty instances in rrdcontext

* need_to_send_chart_definition() does not need an rrdset lock any more

* rrdcontext objects are collected, after data have been written to the db

* better logging of RRDCONTEXT transitions

* always set all variables needed by the worker utilization charts

* implemented double linked list for most objects; eliminated alarm indexes from rrdhost; and many more fixes

* lockless strings design - string_dup() and string_freez() are totally lockless when they dont need to touch Judy - only Judy is protected with a read/write lock

* STRING code re-organization for clarity

* thread_cache improvements; double numbers precision on worker threads

* STRING_ENTRY now shadown STRING, so no duplicate definition is required; string_length() renamed to string_strlen() to follow the paradigm of all other functions, STRING internal statistics are now only compiled with NETDATA_INTERNAL_CHECKS

* rrdhost index by hostname now cleans up; aclk queries of archieved hosts do not index hosts

* Add index to speed up database context searches

* Removed last_updated optimization (was also buggy after latest merge with master)

Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
Co-authored-by: Vladimir Kobal <vlad@prokk.net>

1 files changed, 53 insertions, 66 deletions

diff --git a/database/rrddimvar.c b/database/rrddimvar.c
index 3c2ed75e57..1726612645 100644
--- a/database/rrddimvar.c
+++ b/database/rrddimvar.c
@@ -16,64 +16,64 @@ static inline void rrddimvar_free_variables(RRDDIMVAR *rs) {
 
     // CHART VARIABLES FOR THIS DIMENSION
 
-    rrdvar_free(host, &st->rrdvar_root_index, rs->var_local_id);
+    rrdvar_free(host, st->rrdvar_root_index, rs->var_local_id);
     rs->var_local_id = NULL;
 
-    rrdvar_free(host, &st->rrdvar_root_index, rs->var_local_name);
+    rrdvar_free(host, st->rrdvar_root_index, rs->var_local_name);
     rs->var_local_name = NULL;
 
     // FAMILY VARIABLES FOR THIS DIMENSION
 
-    rrdvar_free(host, &st->rrdfamily->rrdvar_root_index, rs->var_family_id);
+    rrdvar_free(host, st->rrdfamily->rrdvar_root_index, rs->var_family_id);
     rs->var_family_id = NULL;
 
-    rrdvar_free(host, &st->rrdfamily->rrdvar_root_index, rs->var_family_name);
+    rrdvar_free(host, st->rrdfamily->rrdvar_root_index, rs->var_family_name);
     rs->var_family_name = NULL;
 
-    rrdvar_free(host, &st->rrdfamily->rrdvar_root_index, rs->var_family_contextid);
+    rrdvar_free(host, st->rrdfamily->rrdvar_root_index, rs->var_family_contextid);
     rs->var_family_contextid = NULL;
 
-    rrdvar_free(host, &st->rrdfamily->rrdvar_root_index, rs->var_family_contextname);
+    rrdvar_free(host, st->rrdfamily->rrdvar_root_index, rs->var_family_contextname);
     rs->var_family_contextname = NULL;
 
     // HOST VARIABLES FOR THIS DIMENSION
 
-    rrdvar_free(host, &host->rrdvar_root_index, rs->var_host_chartidid);
+    rrdvar_free(host, host->rrdvar_root_index, rs->var_host_chartidid);
     rs->var_host_chartidid = NULL;
 
-    rrdvar_free(host, &host->rrdvar_root_index, rs->var_host_chartidname);
+    rrdvar_free(host, host->rrdvar_root_index, rs->var_host_chartidname);
     rs->var_host_chartidname = NULL;
 
-    rrdvar_free(host, &host->rrdvar_root_index, rs->var_host_chartnameid);
+    rrdvar_free(host, host->rrdvar_root_index, rs->var_host_chartnameid);
     rs->var_host_chartnameid = NULL;
 
-    rrdvar_free(host, &host->rrdvar_root_index, rs->var_host_chartnamename);
+    rrdvar_free(host, host->rrdvar_root_index, rs->var_host_chartnamename);
     rs->var_host_chartnamename = NULL;
 
     // KEYS
 
-    freez(rs->key_id);
+    string_freez(rs->key_id);
     rs->key_id = NULL;
 
-    freez(rs->key_name);
+    string_freez(rs->key_name);
     rs->key_name = NULL;
 
-    freez(rs->key_fullidid);
+    string_freez(rs->key_fullidid);
     rs->key_fullidid = NULL;
 
-    freez(rs->key_fullidname);
+    string_freez(rs->key_fullidname);
     rs->key_fullidname = NULL;
 
-    freez(rs->key_contextid);
+    string_freez(rs->key_contextid);
     rs->key_contextid = NULL;
 
-    freez(rs->key_contextname);
+    string_freez(rs->key_contextname);
     rs->key_contextname = NULL;
 
-    freez(rs->key_fullnameid);
+    string_freez(rs->key_fullnameid);
     rs->key_fullnameid = NULL;
 
-    freez(rs->key_fullnamename);
+    string_freez(rs->key_fullnamename);
     rs->key_fullnamename = NULL;
 }
 
@@ -88,29 +88,29 @@ static inline void rrddimvar_create_variables(RRDDIMVAR *rs) {
 
     // KEYS
 
-    snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s%s%s", rs->prefix, rd->id, rs->suffix);
-    rs->key_id = strdupz(buffer);
+    snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s%s%s", string2str(rs->prefix), rrddim_id(rd), string2str(rs->suffix));
+    rs->key_id = string_strdupz(buffer);
 
-    snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s%s%s", rs->prefix, rd->name, rs->suffix);
-    rs->key_name = strdupz(buffer);
+    snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s%s%s", string2str(rs->prefix), rrddim_name(rd), string2str(rs->suffix));
+    rs->key_name = string_strdupz(buffer);
 
-    snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", st->id, rs->key_id);
-    rs->key_fullidid = strdupz(buffer);
+    snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_id(st), string2str(rs->key_id));
+    rs->key_fullidid = string_strdupz(buffer);
 
-    snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", st->id, rs->key_name);
-    rs->key_fullidname = strdupz(buffer);
+    snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_id(st), string2str(rs->key_name));
+    rs->key_fullidname = string_strdupz(buffer);
 
-    snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", st->context, rs->key_id);
-    rs->key_contextid = strdupz(buffer);
+    snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_context(st), string2str(rs->key_id));
+    rs->key_contextid = string_strdupz(buffer);
 
-    snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", st->context, rs->key_name);
-    rs->key_contextname = strdupz(buffer);
+    snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_context(st), string2str(rs->key_name));
+    rs->key_contextname = string_strdupz(buffer);
 
-    snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", st->name, rs->key_id);
-    rs->key_fullnameid = strdupz(buffer);
+    snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_name(st), string2str(rs->key_id));
+    rs->key_fullnameid = string_strdupz(buffer);
 
-    snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", st->name, rs->key_name);
-    rs->key_fullnamename = strdupz(buffer);
+    snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_name(st), string2str(rs->key_name));
+    rs->key_fullnamename = string_strdupz(buffer);
 
     // CHART VARIABLES FOR THIS DIMENSION
     // -----------------------------------
@@ -119,8 +119,8 @@ static inline void rrddimvar_create_variables(RRDDIMVAR *rs) {
     // - $id
     // - $name
 
-    rs->var_local_id           = rrdvar_create_and_index("local", &st->rrdvar_root_index, rs->key_id, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
-    rs->var_local_name         = rrdvar_create_and_index("local", &st->rrdvar_root_index, rs->key_name, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
+    rs->var_local_id           = rrdvar_create_and_index("local", st->rrdvar_root_index, rs->key_id, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
+    rs->var_local_name         = rrdvar_create_and_index("local", st->rrdvar_root_index, rs->key_name, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
 
     // FAMILY VARIABLES FOR THIS DIMENSION
     // -----------------------------------
@@ -131,10 +131,10 @@ static inline void rrddimvar_create_variables(RRDDIMVAR *rs) {
     // - $chart-context.id
     // - $chart-context.name
 
-    rs->var_family_id          = rrdvar_create_and_index("family", &st->rrdfamily->rrdvar_root_index, rs->key_id, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
-    rs->var_family_name        = rrdvar_create_and_index("family", &st->rrdfamily->rrdvar_root_index, rs->key_name, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
-    rs->var_family_contextid   = rrdvar_create_and_index("family", &st->rrdfamily->rrdvar_root_index, rs->key_contextid, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
-    rs->var_family_contextname = rrdvar_create_and_index("family", &st->rrdfamily->rrdvar_root_index, rs->key_contextname, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
+    rs->var_family_id          = rrdvar_create_and_index("family", st->rrdfamily->rrdvar_root_index, rs->key_id, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
+    rs->var_family_name        = rrdvar_create_and_index("family", st->rrdfamily->rrdvar_root_index, rs->key_name, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
+    rs->var_family_contextid   = rrdvar_create_and_index("family", st->rrdfamily->rrdvar_root_index, rs->key_contextid, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
+    rs->var_family_contextname = rrdvar_create_and_index("family", st->rrdfamily->rrdvar_root_index, rs->key_contextname, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
 
     // HOST VARIABLES FOR THIS DIMENSION
     // -----------------------------------
@@ -145,33 +145,32 @@ static inline void rrddimvar_create_variables(RRDDIMVAR *rs) {
     // - $chart-name.id
     // - $chart-name.name
 
-    rs->var_host_chartidid      = rrdvar_create_and_index("host", &host->rrdvar_root_index, rs->key_fullidid, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
-    rs->var_host_chartidname    = rrdvar_create_and_index("host", &host->rrdvar_root_index, rs->key_fullidname, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
-    rs->var_host_chartnameid    = rrdvar_create_and_index("host", &host->rrdvar_root_index, rs->key_fullnameid, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
-    rs->var_host_chartnamename  = rrdvar_create_and_index("host", &host->rrdvar_root_index, rs->key_fullnamename, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
+    rs->var_host_chartidid      = rrdvar_create_and_index("host", host->rrdvar_root_index, rs->key_fullidid, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
+    rs->var_host_chartidname    = rrdvar_create_and_index("host", host->rrdvar_root_index, rs->key_fullidname, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
+    rs->var_host_chartnameid    = rrdvar_create_and_index("host", host->rrdvar_root_index, rs->key_fullnameid, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
+    rs->var_host_chartnamename  = rrdvar_create_and_index("host", host->rrdvar_root_index, rs->key_fullnamename, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
 }
 
 RRDDIMVAR *rrddimvar_create(RRDDIM *rd, RRDVAR_TYPE type, const char *prefix, const char *suffix, void *value, RRDVAR_OPTIONS options) {
     RRDSET *st = rd->rrdset;
     (void)st;
 
-    debug(D_VARIABLES, "RRDDIMSET create for chart id '%s' name '%s', dimension id '%s', name '%s%s%s'", st->id, st->name, rd->id, (prefix)?prefix:"", rd->name, (suffix)?suffix:"");
+    debug(D_VARIABLES, "RRDDIMSET create for chart id '%s' name '%s', dimension id '%s', name '%s%s%s'", rrdset_id(st), rrdset_name(st), rrddim_id(rd), (prefix)?prefix:"", rrddim_name(rd), (suffix)?suffix:"");
 
     if(!prefix) prefix = "";
     if(!suffix) suffix = "";
 
     RRDDIMVAR *rs = (RRDDIMVAR *)callocz(1, sizeof(RRDDIMVAR));
 
-    rs->prefix = strdupz(prefix);
-    rs->suffix = strdupz(suffix);
+    rs->prefix = string_strdupz(prefix);
+    rs->suffix = string_strdupz(suffix);
 
     rs->type = type;
     rs->value = value;
     rs->options = options;
     rs->rrddim = rd;
 
-    rs->next = rd->variables;
-    rd->variables = rs;
+    DOUBLE_LINKED_LIST_PREPEND_UNSAFE(rd->variables, rs, prev, next);
 
     rrddimvar_create_variables(rs);
 
@@ -182,7 +181,7 @@ void rrddimvar_rename_all(RRDDIM *rd) {
     RRDSET *st = rd->rrdset;
     (void)st;
 
-    debug(D_VARIABLES, "RRDDIMSET rename for chart id '%s' name '%s', dimension id '%s', name '%s'", st->id, st->name, rd->id, rd->name);
+    debug(D_VARIABLES, "RRDDIMSET rename for chart id '%s' name '%s', dimension id '%s', name '%s'", rrdset_id(st), rrdset_name(st), rrddim_id(rd), rrddim_name(rd));
 
     RRDDIMVAR *rs, *next = rd->variables;
     while((rs = next)) {
@@ -193,25 +192,13 @@ void rrddimvar_rename_all(RRDDIM *rd) {
 
 void rrddimvar_free(RRDDIMVAR *rs) {
     RRDDIM *rd = rs->rrddim;
-    RRDSET *st = rd->rrdset;
-    debug(D_VARIABLES, "RRDDIMSET free for chart id '%s' name '%s', dimension id '%s', name '%s', prefix='%s', suffix='%s'", st->id, st->name, rd->id, rd->name, rs->prefix, rs->suffix);
+    debug(D_VARIABLES, "RRDDIMSET free for chart id '%s' name '%s', dimension id '%s', name '%s', prefix='%s', suffix='%s'", rrdset_id(rd->rrdset), rrdset_name(rd->rrdset), rrddim_id(rd), rrddim_name(rd), string2str(rs->prefix), string2str(rs->suffix));
 
     rrddimvar_free_variables(rs);
 
-    if(rd->variables == rs) {
-        debug(D_VARIABLES, "RRDDIMSET removing first entry for chart id '%s' name '%s', dimension id '%s', name '%s'", st->id, st->name, rd->id, rd->name);
-        rd->variables = rs->next;
-    }
-    else {
-        debug(D_VARIABLES, "RRDDIMSET removing non-first entry for chart id '%s' name '%s', dimension id '%s', name '%s'", st->id, st->name, rd->id, rd->name);
-        RRDDIMVAR *t;
-        for (t = rd->variables; t && t->next != rs; t = t->next) ;
-        if(!t) error("RRDDIMVAR '%s' not found in dimension '%s/%s' variables linked list", rs->key_name, st->id, rd->id);
-        else t->next = rs->next;
-    }
+    DOUBLE_LINKED_LIST_REMOVE_UNSAFE(rd->variables, rs, prev, next);
 
-    freez(rs->prefix);
-    freez(rs->suffix);
+    string_freez(rs->prefix);
+    string_freez(rs->suffix);
     freez(rs);
 }
-