diff options
author | Costa Tsaousis <costa@netdata.cloud> | 2022-09-19 23:46:13 +0300 |
---|---|---|
committer | GitHub <noreply@github.com> | 2022-09-19 23:46:13 +0300 |
commit | cb7af25c09d8775d1967cb0553268075cda868d4 (patch) | |
tree | 9e86bc359bb2b1ec72d3a1382236703dc633ad63 /database/rrdlabels.c | |
parent | 62246029160025a8d6503d9fbb617c7b029b9126 (diff) |
RRD structures managed by dictionaries (#13646)
* rrdset - in progress
* rrdset optimal constructor; rrdset conflict
* rrdset final touches
* re-organization of rrdset object members
* prevent use-after-free
* dictionary dfe supports also counting of iterations
* rrddim managed by dictionary
* rrd.h cleanup
* DICTIONARY_ITEM now is referencing actual dictionary items in the code
* removed rrdset linked list
* Revert "removed rrdset linked list"
This reverts commit 690d6a588b4b99619c2c5e10f84e8f868ae6def5.
* removed rrdset linked list
* added comments
* Switch chart uuid to static allocation in rrdset
Remove unused functions
* rrdset_archive() and friends...
* always create rrdfamily
* enable ml_free_dimension
* rrddim_foreach done with dfe
* most custom rrddim loops replaced with rrddim_foreach
* removed accesses to rrddim->dimensions
* removed locks that are no longer needed
* rrdsetvar is now managed by the dictionary
* set rrdset is rrdsetvar, fixes https://github.com/netdata/netdata/pull/13646#issuecomment-1242574853
* conflict callback of rrdsetvar now properly checks if it has to reset the variable
* dictionary registered callbacks accept as first parameter the DICTIONARY_ITEM
* dictionary dfe now uses internal counter to report; avoided excess variables defined with dfe
* dictionary walkthrough callbacks get dictionary acquired items
* dictionary reference counters that can be dupped from zero
* added advanced functions for get and del
* rrdvar managed by dictionaries
* thread safety for rrdsetvar
* faster rrdvar initialization
* rrdvar string lengths should match in all add, del, get functions
* rrdvar internals hidden from the rest of the world
* rrdvar is now acquired throughout netdata
* hide the internal structures of rrdsetvar
* rrdsetvar is now acquired through out netdata
* rrddimvar managed by dictionary; rrddimvar linked list removed; rrddimvar structures hidden from the rest of netdata
* better error handling
* dont create variables if not initialized for health
* dont create variables if not initialized for health again
* rrdfamily is now managed by dictionaries; references of it are acquired dictionary items
* type checking on acquired objects
* rrdcalc renaming of functions
* type checking for rrdfamily_acquired
* rrdcalc managed by dictionaries
* rrdcalc double free fix
* host rrdvars is always needed
* attempt to fix deadlock 1
* attempt to fix deadlock 2
* Remove unused variable
* attempt to fix deadlock 3
* snprintfz
* rrdcalc index in rrdset fix
* Stop storing active charts and computing chart hashes
* Remove store active chart function
* Remove compute chart hash function
* Remove sql_store_chart_hash function
* Remove store_active_dimension function
* dictionary delayed destruction
* formatting and cleanup
* zero dictionary base on rrdsetvar
* added internal error to log delayed destructions of dictionaries
* typo in rrddimvar
* added debugging info to dictionary
* debug info
* fix for rrdcalc keys being empty
* remove forgotten unlock
* remove deadlock
* Switch to metadata version 5 and drop
chart_hash
chart_hash_map
chart_active
dimension_active
v_chart_hash
* SQL cosmetic changes
* do not busy wait while destroying a referenced dictionary
* remove deadlock
* code cleanup; re-organization;
* fast cleanup and flushing of dictionaries
* number formatting fixes
* do not delete configured alerts when archiving a chart
* rrddim obsolete linked list management outside dictionaries
* removed duplicate contexts call
* fix crash when rrdfamily is not initialized
* dont keep rrddimvar referenced
* properly cleanup rrdvar
* removed some locks
* Do not attempt to cleanup chart_hash / chart_hash_map
* rrdcalctemplate managed by dictionary
* register callbacks on the right dictionary
* removed some more locks
* rrdcalc secondary index replaced with linked-list; rrdcalc labels updates are now executed by health thread
* when looking up for an alarm look using both chart id and chart name
* host initialization a bit more modular
* init rrdlabels on host update
* preparation for dictionary views
* improved comment
* unused variables without internal checks
* service threads isolation and worker info
* more worker info in service thread
* thread cancelability debugging with internal checks
* strings data races addressed; fixes https://github.com/netdata/netdata/issues/13647
* dictionary modularization
* Remove unused SQL statement definition
* unit-tested thread safety of dictionaries; removed data race conditions on dictionaries and strings; dictionaries now can detect if the caller is holds a write lock and automatically all the calls become their unsafe versions; all direct calls to unsafe version is eliminated
* remove worker_is_idle() from the exit of service functions, because we lose the lock time between loops
* rewritten dictionary to have 2 separate locks, one for indexing and another for traversal
* Update collectors/cgroups.plugin/sys_fs_cgroup.c
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
* Update collectors/cgroups.plugin/sys_fs_cgroup.c
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
* Update collectors/proc.plugin/proc_net_dev.c
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
* fix memory leak in rrdset cache_dir
* minor dictionary changes
* dont use index locks in single threaded
* obsolete dict option
* rrddim options and flags separation; rrdset_done() optimization to keep array of reference pointers to rrddim;
* fix jump on uninitialized value in dictionary; remove double free of cache_dir
* addressed codacy findings
* removed debugging code
* use the private refcount on dictionaries
* make dictionary item desctructors work on dictionary destruction; strictier control on dictionary API; proper cleanup sequence on rrddim;
* more dictionary statistics
* global statistics about dictionary operations, memory, items, callbacks
* dictionary support for views - missing the public API
* removed warning about unused parameter
* chart and context name for cloud
* chart and context name for cloud, again
* dictionary statistics fixed; first implementation of dictionary views - not currently used
* only the master can globally delete an item
* context needs netdata prefix
* fix context and chart it of spins
* fix for host variables when health is not enabled
* run garbage collector on item insert too
* Fix info message; remove extra "using"
* update dict unittest for new placement of garbage collector
* we need RRDHOST->rrdvars for maintaining custom host variables
* Health initialization needs the host->host_uuid
* split STRING to its own files; no code changes other than that
* initialize health unconditionally
* unit tests do not pollute the global scope with their variables
* Skip initialization when creating archived hosts on startup. When a child connects it will initialize properly
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
Diffstat (limited to 'database/rrdlabels.c')
-rw-r--r-- | database/rrdlabels.c | 97 |
1 files changed, 57 insertions, 40 deletions
diff --git a/database/rrdlabels.c b/database/rrdlabels.c index 6ab1ea66b4..6472743122 100644 --- a/database/rrdlabels.c +++ b/database/rrdlabels.c @@ -478,9 +478,7 @@ typedef struct rrdlabel { RRDLABEL_SRC label_source; } RRDLABEL; -static void rrdlabel_insert_callback(const char *name, void *value, void *data) { - (void)name; - DICTIONARY *dict = (DICTIONARY *)data; (void)dict; +static void rrdlabel_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *dict_ptr __maybe_unused) { RRDLABEL *lb = (RRDLABEL *)value; // label_value is already allocated by the STRING @@ -488,42 +486,43 @@ static void rrdlabel_insert_callback(const char *name, void *value, void *data) lb->label_source &= ~RRDLABEL_FLAG_OLD; } -static void rrdlabel_delete_callback(const char *name, void *value, void *data) { - (void)name; - DICTIONARY *dict = (DICTIONARY *)data; (void)dict; +static void rrdlabel_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *dict_ptr __maybe_unused) { RRDLABEL *lb = (RRDLABEL *)value; string_freez(lb->label_value); lb->label_value = NULL; } -static void rrdlabel_conflict_callback(const char *name, void *oldvalue, void *newvalue, void *data) { - (void)name; - DICTIONARY *dict = (DICTIONARY *)data; (void)dict; +static bool rrdlabel_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *oldvalue, void *newvalue, void *dict_ptr __maybe_unused) { RRDLABEL *lbold = (RRDLABEL *)oldvalue; RRDLABEL *lbnew = (RRDLABEL *)newvalue; if(lbold->label_value == lbnew->label_value) { // they are the same + lbold->label_source |= lbnew->label_source; lbold->label_source |= RRDLABEL_FLAG_OLD; lbold->label_source &= ~RRDLABEL_FLAG_NEW; // free the new one string_freez(lbnew->label_value); + + return false; } - else { - // they are different - string_freez(lbold->label_value); - lbold->label_value = lbnew->label_value; - lbold->label_source = lbnew->label_source; - lbold->label_source |= RRDLABEL_FLAG_NEW; - lbold->label_source &= ~RRDLABEL_FLAG_OLD; - } + + // they are different + + string_freez(lbold->label_value); + lbold->label_value = lbnew->label_value; + lbold->label_source = lbnew->label_source; + lbold->label_source |= RRDLABEL_FLAG_NEW; + lbold->label_source &= ~RRDLABEL_FLAG_OLD; + + return true; } DICTIONARY *rrdlabels_create(void) { - DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_DONT_OVERWRITE_VALUE); + DICTIONARY *dict = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE); dictionary_register_insert_callback(dict, rrdlabel_insert_callback, dict); dictionary_register_delete_callback(dict, rrdlabel_delete_callback, dict); dictionary_register_conflict_callback(dict, rrdlabel_conflict_callback, dict); @@ -623,7 +622,7 @@ void rrdlabels_add_pair(DICTIONARY *dict, const char *string, RRDLABEL_SRC ls) { // rrdlabels_get_value_to_buffer_or_null() void rrdlabels_get_value_to_buffer_or_null(DICTIONARY *labels, BUFFER *wb, const char *key, const char *quote, const char *null) { - DICTIONARY_ITEM *acquired_item = dictionary_get_and_acquire_item(labels, key); + const DICTIONARY_ITEM *acquired_item = dictionary_get_and_acquire_item(labels, key); RRDLABEL *lb = dictionary_acquired_item_value(acquired_item); if(lb && lb->label_value) @@ -638,7 +637,7 @@ void rrdlabels_get_value_to_buffer_or_null(DICTIONARY *labels, BUFFER *wb, const // rrdlabels_get_value_to_char_or_null() void rrdlabels_get_value_to_char_or_null(DICTIONARY *labels, char **value, const char *key) { - DICTIONARY_ITEM *acquired_item = dictionary_get_and_acquire_item(labels, key); + const DICTIONARY_ITEM *acquired_item = dictionary_get_and_acquire_item(labels, key); RRDLABEL *lb = dictionary_acquired_item_value(acquired_item); *value = (lb && lb->label_value) ? strdupz(string2str(lb->label_value)) : NULL; @@ -650,10 +649,7 @@ void rrdlabels_get_value_to_char_or_null(DICTIONARY *labels, char **value, const // rrdlabels_unmark_all() // remove labels RRDLABEL_FLAG_OLD and RRDLABEL_FLAG_NEW from all dictionary items -static int remove_flags_old_new(const char *name, void *value, void *data) { - (void)name; - (void)data; - +static int remove_flags_old_new(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data __maybe_unused) { RRDLABEL *lb = (RRDLABEL *)value; if(lb->label_source & RRDLABEL_FLAG_OLD) lb->label_source &= ~RRDLABEL_FLAG_OLD; @@ -671,12 +667,13 @@ void rrdlabels_unmark_all(DICTIONARY *labels) { // rrdlabels_remove_all_unmarked() // remove dictionary items that are neither old, nor new -static int remove_not_old_not_new_callback(const char *name, void *value, void *data) { +static int remove_not_old_not_new_callback(const DICTIONARY_ITEM *item, void *value, void *data) { + const char *name = dictionary_acquired_item_name(item); DICTIONARY *dict = (DICTIONARY *)data; RRDLABEL *lb = (RRDLABEL *)value; if(!(lb->label_source & (RRDLABEL_FLAG_OLD | RRDLABEL_FLAG_NEW | RRDLABEL_FLAG_PERMANENT))) { - dictionary_del_having_write_lock(dict, name); + dictionary_del(dict, name); return 1; } @@ -696,7 +693,8 @@ struct labels_walkthrough { void *data; }; -static int labels_walkthrough_callback(const char *name, void *value, void *data) { +static int labels_walkthrough_callback(const DICTIONARY_ITEM *item, void *value, void *data) { + const char *name = dictionary_acquired_item_name(item); struct labels_walkthrough *d = (struct labels_walkthrough *)data; RRDLABEL *lb = (RRDLABEL *)value; @@ -728,7 +726,8 @@ int rrdlabels_sorted_walkthrough_read(DICTIONARY *labels, int (*callback)(const // rrdlabels_migrate_to_these() // migrate an existing label list to a new list, INPLACE -static int copy_label_to_dictionary_callback(const char *name, void *value, void *data) { +static int copy_label_to_dictionary_callback(const DICTIONARY_ITEM *item, void *value, void *data) { + const char *name = dictionary_acquired_item_name(item); DICTIONARY *dst = (DICTIONARY *)data; RRDLABEL *lb = (RRDLABEL *)value; labels_add_already_sanitized(dst, name, string2str(lb->label_value), lb->label_source); @@ -765,7 +764,8 @@ struct simple_pattern_match_name_value { char equal; }; -static int simple_pattern_match_name_only_callback(const char *name, void *value, void *data) { +static int simple_pattern_match_name_only_callback(const DICTIONARY_ITEM *item, void *value, void *data) { + const char *name = dictionary_acquired_item_name(item); struct simple_pattern_match_name_value *t = (struct simple_pattern_match_name_value *)data; (void)value; @@ -775,7 +775,8 @@ static int simple_pattern_match_name_only_callback(const char *name, void *value return 0; } -static int simple_pattern_match_name_and_value_callback(const char *name, void *value, void *data) { +static int simple_pattern_match_name_and_value_callback(const DICTIONARY_ITEM *item, void *value, void *data) { + const char *name = dictionary_acquired_item_name(item); struct simple_pattern_match_name_value *t = (struct simple_pattern_match_name_value *)data; RRDLABEL *lb = (RRDLABEL *)value; @@ -841,7 +842,9 @@ bool rrdlabels_match_simple_pattern(DICTIONARY *labels, const char *simple_patte // ---------------------------------------------------------------------------- // Log all labels -static int rrdlabels_log_label_to_buffer_callback(const char *name, void *value, void *data) { +static int rrdlabels_log_label_to_buffer_callback(const DICTIONARY_ITEM *item, void *value, void *data) { + const char *name = dictionary_acquired_item_name(item); + BUFFER *wb = (BUFFER *)data; RRDLABEL *lb = (RRDLABEL *)value; @@ -891,7 +894,8 @@ struct labels_to_buffer { size_t count; }; -static int label_to_buffer_callback(const char *name, void *value, void *data) { +static int label_to_buffer_callback(const DICTIONARY_ITEM *item, void *value, void *data) { + const char *name = dictionary_acquired_item_name(item); struct labels_to_buffer *t = (struct labels_to_buffer *)data; RRDLABEL *lb = (RRDLABEL *)value; @@ -961,17 +965,30 @@ void rrdset_update_rrdlabels(RRDSET *st, DICTIONARY *new_rrdlabels) { if (new_rrdlabels) rrdlabels_migrate_to_these(st->rrdlabels, new_rrdlabels); - rrdcalc_update_rrdlabels(st); + rrdset_save_rrdlabels_to_sql(st); +} + +void rrdset_save_rrdlabels_to_sql(RRDSET *st) { + if(!st->rrdlabels) return; + + size_t old_version = st->rrdlabels_last_saved_version; + size_t new_version = dictionary_version(st->rrdlabels); + + if(new_version != old_version) { + // TODO - we should also cleanup sqlite from old new_rrdlabels that have been removed - // TODO - we should also cleanup sqlite from old new_rrdlabels that have been removed - BUFFER *sql_buf = buffer_create(1024); - struct label_str tmp = {.sql = sql_buf, .count = 0 }; - uuid_unparse_lower(*st->chart_uuid, tmp.uuid_str); - rrdlabels_walkthrough_read(st->rrdlabels, chart_label_store_to_sql_callback, &tmp); - db_execute(buffer_tostring(sql_buf)); - buffer_free(sql_buf); + BUFFER *sql_buf = buffer_create(1024); + struct label_str tmp = {.sql = sql_buf, .count = 0}; + uuid_unparse_lower(st->chart_uuid, tmp.uuid_str); + rrdlabels_walkthrough_read(st->rrdlabels, chart_label_store_to_sql_callback, &tmp); + db_execute(buffer_tostring(sql_buf)); + buffer_free(sql_buf); + + st->rrdlabels_last_saved_version = new_version; + } } + // ---------------------------------------------------------------------------- // rrdlabels unit test |