diff options
author | Costa Tsaousis <costa@netdata.cloud> | 2022-09-19 23:46:13 +0300 |
---|---|---|
committer | GitHub <noreply@github.com> | 2022-09-19 23:46:13 +0300 |
commit | cb7af25c09d8775d1967cb0553268075cda868d4 (patch) | |
tree | 9e86bc359bb2b1ec72d3a1382236703dc633ad63 /database/rrdcalctemplate.c | |
parent | 62246029160025a8d6503d9fbb617c7b029b9126 (diff) |
RRD structures managed by dictionaries (#13646)
* rrdset - in progress
* rrdset optimal constructor; rrdset conflict
* rrdset final touches
* re-organization of rrdset object members
* prevent use-after-free
* dictionary dfe supports also counting of iterations
* rrddim managed by dictionary
* rrd.h cleanup
* DICTIONARY_ITEM now is referencing actual dictionary items in the code
* removed rrdset linked list
* Revert "removed rrdset linked list"
This reverts commit 690d6a588b4b99619c2c5e10f84e8f868ae6def5.
* removed rrdset linked list
* added comments
* Switch chart uuid to static allocation in rrdset
Remove unused functions
* rrdset_archive() and friends...
* always create rrdfamily
* enable ml_free_dimension
* rrddim_foreach done with dfe
* most custom rrddim loops replaced with rrddim_foreach
* removed accesses to rrddim->dimensions
* removed locks that are no longer needed
* rrdsetvar is now managed by the dictionary
* set rrdset is rrdsetvar, fixes https://github.com/netdata/netdata/pull/13646#issuecomment-1242574853
* conflict callback of rrdsetvar now properly checks if it has to reset the variable
* dictionary registered callbacks accept as first parameter the DICTIONARY_ITEM
* dictionary dfe now uses internal counter to report; avoided excess variables defined with dfe
* dictionary walkthrough callbacks get dictionary acquired items
* dictionary reference counters that can be dupped from zero
* added advanced functions for get and del
* rrdvar managed by dictionaries
* thread safety for rrdsetvar
* faster rrdvar initialization
* rrdvar string lengths should match in all add, del, get functions
* rrdvar internals hidden from the rest of the world
* rrdvar is now acquired throughout netdata
* hide the internal structures of rrdsetvar
* rrdsetvar is now acquired through out netdata
* rrddimvar managed by dictionary; rrddimvar linked list removed; rrddimvar structures hidden from the rest of netdata
* better error handling
* dont create variables if not initialized for health
* dont create variables if not initialized for health again
* rrdfamily is now managed by dictionaries; references of it are acquired dictionary items
* type checking on acquired objects
* rrdcalc renaming of functions
* type checking for rrdfamily_acquired
* rrdcalc managed by dictionaries
* rrdcalc double free fix
* host rrdvars is always needed
* attempt to fix deadlock 1
* attempt to fix deadlock 2
* Remove unused variable
* attempt to fix deadlock 3
* snprintfz
* rrdcalc index in rrdset fix
* Stop storing active charts and computing chart hashes
* Remove store active chart function
* Remove compute chart hash function
* Remove sql_store_chart_hash function
* Remove store_active_dimension function
* dictionary delayed destruction
* formatting and cleanup
* zero dictionary base on rrdsetvar
* added internal error to log delayed destructions of dictionaries
* typo in rrddimvar
* added debugging info to dictionary
* debug info
* fix for rrdcalc keys being empty
* remove forgotten unlock
* remove deadlock
* Switch to metadata version 5 and drop
chart_hash
chart_hash_map
chart_active
dimension_active
v_chart_hash
* SQL cosmetic changes
* do not busy wait while destroying a referenced dictionary
* remove deadlock
* code cleanup; re-organization;
* fast cleanup and flushing of dictionaries
* number formatting fixes
* do not delete configured alerts when archiving a chart
* rrddim obsolete linked list management outside dictionaries
* removed duplicate contexts call
* fix crash when rrdfamily is not initialized
* dont keep rrddimvar referenced
* properly cleanup rrdvar
* removed some locks
* Do not attempt to cleanup chart_hash / chart_hash_map
* rrdcalctemplate managed by dictionary
* register callbacks on the right dictionary
* removed some more locks
* rrdcalc secondary index replaced with linked-list; rrdcalc labels updates are now executed by health thread
* when looking up for an alarm look using both chart id and chart name
* host initialization a bit more modular
* init rrdlabels on host update
* preparation for dictionary views
* improved comment
* unused variables without internal checks
* service threads isolation and worker info
* more worker info in service thread
* thread cancelability debugging with internal checks
* strings data races addressed; fixes https://github.com/netdata/netdata/issues/13647
* dictionary modularization
* Remove unused SQL statement definition
* unit-tested thread safety of dictionaries; removed data race conditions on dictionaries and strings; dictionaries now can detect if the caller is holds a write lock and automatically all the calls become their unsafe versions; all direct calls to unsafe version is eliminated
* remove worker_is_idle() from the exit of service functions, because we lose the lock time between loops
* rewritten dictionary to have 2 separate locks, one for indexing and another for traversal
* Update collectors/cgroups.plugin/sys_fs_cgroup.c
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
* Update collectors/cgroups.plugin/sys_fs_cgroup.c
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
* Update collectors/proc.plugin/proc_net_dev.c
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
* fix memory leak in rrdset cache_dir
* minor dictionary changes
* dont use index locks in single threaded
* obsolete dict option
* rrddim options and flags separation; rrdset_done() optimization to keep array of reference pointers to rrddim;
* fix jump on uninitialized value in dictionary; remove double free of cache_dir
* addressed codacy findings
* removed debugging code
* use the private refcount on dictionaries
* make dictionary item desctructors work on dictionary destruction; strictier control on dictionary API; proper cleanup sequence on rrddim;
* more dictionary statistics
* global statistics about dictionary operations, memory, items, callbacks
* dictionary support for views - missing the public API
* removed warning about unused parameter
* chart and context name for cloud
* chart and context name for cloud, again
* dictionary statistics fixed; first implementation of dictionary views - not currently used
* only the master can globally delete an item
* context needs netdata prefix
* fix context and chart it of spins
* fix for host variables when health is not enabled
* run garbage collector on item insert too
* Fix info message; remove extra "using"
* update dict unittest for new placement of garbage collector
* we need RRDHOST->rrdvars for maintaining custom host variables
* Health initialization needs the host->host_uuid
* split STRING to its own files; no code changes other than that
* initialize health unconditionally
* unit tests do not pollute the global scope with their variables
* Skip initialization when creating archived hosts on startup. When a child connects it will initialize properly
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
Co-authored-by: Vladimir Kobal <vlad@prokk.net>
Diffstat (limited to 'database/rrdcalctemplate.c')
-rw-r--r-- | database/rrdcalctemplate.c | 202 |
1 files changed, 174 insertions, 28 deletions
diff --git a/database/rrdcalctemplate.c b/database/rrdcalctemplate.c index 0db76d6c04..87e085c932 100644 --- a/database/rrdcalctemplate.c +++ b/database/rrdcalctemplate.c @@ -1,6 +1,5 @@ // SPDX-License-Identifier: GPL-3.0-or-later -#define NETDATA_HEALTH_INTERNALS #include "rrd.h" // ---------------------------------------------------------------------------- @@ -11,45 +10,84 @@ * @param rt is the template used to create the chart. * @param st is the chart where the alarm will be attached. */ -void rrdcalctemplate_check_conditions_and_link(RRDCALCTEMPLATE *rt, RRDSET *st, RRDHOST *host) { + +static char *rrdcalc_alert_name_with_dimension(const char *name, size_t namelen, const char *dim, size_t dimlen) { + char *newname,*move; + + newname = mallocz(namelen + dimlen + 2); + move = newname; + memcpy(move, name, namelen); + move += namelen; + + *move++ = '_'; + memcpy(move, dim, dimlen); + move += dimlen; + *move = '\0'; + + return newname; +} + +bool rrdcalctemplate_check_rrdset_conditions(RRDCALCTEMPLATE *rt, RRDSET *st, RRDHOST *host) { if(rt->context != st->context) - return; + return false; - if (rt->charts_pattern && !simple_pattern_matches(rt->charts_pattern, rrdset_name(st))) - return; + if(rt->foreach_dimension_pattern && !rrdset_number_of_dimensions(st)) + return false; + + if (rt->charts_pattern && !simple_pattern_matches(rt->charts_pattern, rrdset_name(st)) && !simple_pattern_matches(rt->charts_pattern, rrdset_id(st))) + return false; if (rt->family_pattern && !simple_pattern_matches(rt->family_pattern, rrdset_family(st))) - return; + return false; if (rt->module_pattern && !simple_pattern_matches(rt->module_pattern, rrdset_module_name(st))) - return; + return false; if (rt->plugin_pattern && !simple_pattern_matches(rt->plugin_pattern, rrdset_plugin_name(st))) - return; + return false; if(host->rrdlabels && rt->host_labels_pattern && !rrdlabels_match_simple_pattern_parsed(host->rrdlabels, rt->host_labels_pattern, '=')) + return false; + + return true; +} + +void rrdcalctemplate_check_rrddim_conditions_and_link(RRDCALCTEMPLATE *rt, RRDSET *st, RRDDIM *rd, RRDHOST *host) { + if (simple_pattern_matches(rt->foreach_dimension_pattern, rrddim_id(rd)) || simple_pattern_matches(rt->foreach_dimension_pattern, rrddim_name(rd))) { + char *overwrite_alert_name = rrdcalc_alert_name_with_dimension( + rrdcalctemplate_name(rt), string_strlen(rt->name), rrddim_name(rd), string_strlen(rd->name)); + rrdcalc_add_from_rrdcalctemplate(host, rt, st, overwrite_alert_name, rrddim_name(rd)); + freez(overwrite_alert_name); + } +} + +void rrdcalctemplate_check_conditions_and_link(RRDCALCTEMPLATE *rt, RRDSET *st, RRDHOST *host) { + if(!rrdcalctemplate_check_rrdset_conditions(rt, st, host)) + return; + + if(!rt->foreach_dimension_pattern) { + rrdcalc_add_from_rrdcalctemplate(host, rt, st, NULL, NULL); return; + } - RRDCALC *rc = rrdcalc_create_from_template(host, rt, rrdset_id(st)); - if (unlikely(!rc)) - info("Health tried to create alarm from template '%s' on chart '%s' of host '%s', but it failed", rrdcalctemplate_name(rt), rrdset_id(st), rrdhost_hostname(host)); -#ifdef NETDATA_INTERNAL_CHECKS - else if (rc->rrdset != st && !rc->foreachdim) //When we have a template with foreachdim, the child will be added to the index late - error("Health alarm '%s.%s' should be linked to chart '%s', but it is not", rrdcalc_chart_name(rc), rrdcalc_name(rc), rrdset_id(st)); -#endif + RRDDIM *rd; + rrddim_foreach_read(rd, st) { + rrdcalctemplate_check_rrddim_conditions_and_link(rt, st, rd, host); + } + rrddim_foreach_done(rd); } -void rrdcalctemplate_link_matching(RRDSET *st) { +void rrdcalctemplate_link_matching_templates_to_rrdset(RRDSET *st) { RRDHOST *host = st->rrdhost; - RRDCALCTEMPLATE *rt; - foreach_rrdcalctemplate_in_rrdhost(host, rt) + RRDCALCTEMPLATE *rt; + foreach_rrdcalctemplate_read(host, rt) { rrdcalctemplate_check_conditions_and_link(rt, st, host); + } + foreach_rrdcalctemplate_done(rt); } -inline void rrdcalctemplate_free(RRDCALCTEMPLATE *rt) { - if(unlikely(!rt)) return; - +static void rrdcalctemplate_free_internals(RRDCALCTEMPLATE *rt) { expression_free(rt->calculation); expression_free(rt->warning); expression_free(rt->critical); @@ -77,19 +115,127 @@ inline void rrdcalctemplate_free(RRDCALCTEMPLATE *rt) { string_freez(rt->units); string_freez(rt->info); string_freez(rt->dimensions); - string_freez(rt->foreachdim); + string_freez(rt->foreach_dimension); string_freez(rt->host_labels); - simple_pattern_free(rt->spdim); + simple_pattern_free(rt->foreach_dimension_pattern); simple_pattern_free(rt->host_labels_pattern); - freez(rt); } -inline void rrdcalctemplate_unlink_and_free(RRDHOST *host, RRDCALCTEMPLATE *rt) { +void rrdcalctemplate_free_unused_rrdcalctemplate_loaded_from_config(RRDCALCTEMPLATE *rt) { if(unlikely(!rt)) return; - debug(D_HEALTH, "Health removing template '%s' of host '%s'", rrdcalctemplate_name(rt), rrdhost_hostname(host)); + rrdcalctemplate_free_internals(rt); + freez(rt); +} +static void rrdcalctemplate_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdcalctemplate, void *added_bool) { + RRDCALCTEMPLATE *rt = rrdcalctemplate; (void)rt; + + bool *added = added_bool; + *added = true; + + debug(D_HEALTH, "Health configuration adding template '%s'" + ": context '%s'" + ", exec '%s'" + ", recipient '%s'" + ", green " NETDATA_DOUBLE_FORMAT_AUTO + ", red " NETDATA_DOUBLE_FORMAT_AUTO + ", lookup: group %d" + ", after %d" + ", before %d" + ", options %u" + ", dimensions '%s'" + ", for each dimension '%s'" + ", update every %d" + ", calculation '%s'" + ", warning '%s'" + ", critical '%s'" + ", source '%s'" + ", delay up %d" + ", delay down %d" + ", delay max %d" + ", delay_multiplier %f" + ", warn_repeat_every %u" + ", crit_repeat_every %u", + rrdcalctemplate_name(rt), + (rt->context)?string2str(rt->context):"NONE", + (rt->exec)?rrdcalctemplate_exec(rt):"DEFAULT", + (rt->recipient)?rrdcalctemplate_recipient(rt):"DEFAULT", + rt->green, + rt->red, + (int)rt->group, + rt->after, + rt->before, + rt->options, + (rt->dimensions)?rrdcalctemplate_dimensions(rt):"NONE", + (rt->foreach_dimension)?rrdcalctemplate_foreachdim(rt):"NONE", + rt->update_every, + (rt->calculation)?rt->calculation->parsed_as:"NONE", + (rt->warning)?rt->warning->parsed_as:"NONE", + (rt->critical)?rt->critical->parsed_as:"NONE", + rrdcalctemplate_source(rt), + rt->delay_up_duration, + rt->delay_down_duration, + rt->delay_max_duration, + rt->delay_multiplier, + rt->warn_repeat_every, + rt->crit_repeat_every + ); +} + +static void rrdcalctemplate_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdcalctemplate, void *rrdhost __maybe_unused) { + RRDCALCTEMPLATE *rt = rrdcalctemplate; + rrdcalctemplate_free_internals(rt); +} + +void rrdcalctemplate_index_init(RRDHOST *host) { + if(!host->rrdcalctemplate_root_index) { + host->rrdcalctemplate_root_index = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE); + + dictionary_register_insert_callback(host->rrdcalctemplate_root_index, rrdcalctemplate_insert_callback, NULL); + dictionary_register_delete_callback(host->rrdcalctemplate_root_index, rrdcalctemplate_delete_callback, host); + } +} + +void rrdcalctemplate_index_destroy(RRDHOST *host) { + dictionary_destroy(host->rrdcalctemplate_root_index); + host->rrdcalctemplate_root_index = NULL; +} + +inline void rrdcalctemplate_delete_all(RRDHOST *host) { + dictionary_flush(host->rrdcalctemplate_root_index); +} + +#define RRDCALCTEMPLATE_MAX_KEY_SIZE 1024 +static size_t rrdcalctemplate_key(char *dst, size_t dst_len, const char *name, const char *family_match) { + return snprintfz(dst, dst_len, "%s/%s", name, (family_match && *family_match)?family_match:"*"); +} + +void rrdcalctemplate_add_from_config(RRDHOST *host, RRDCALCTEMPLATE *rt) { + if(unlikely(!rt->context)) { + error("Health configuration for template '%s' does not have a context", rrdcalctemplate_name(rt)); + return; + } + + if(unlikely(!rt->update_every)) { + error("Health configuration for template '%s' has no frequency (parameter 'every'). Ignoring it.", rrdcalctemplate_name(rt)); + return; + } + + if(unlikely(!RRDCALCTEMPLATE_HAS_DB_LOOKUP(rt) && !rt->calculation && !rt->warning && !rt->critical)) { + error("Health configuration for template '%s' is useless (no calculation, no warning and no critical evaluation)", rrdcalctemplate_name(rt)); + return; + } + + char key[RRDCALCTEMPLATE_MAX_KEY_SIZE + 1]; + size_t key_len = rrdcalctemplate_key(key, RRDCALCTEMPLATE_MAX_KEY_SIZE, rrdcalctemplate_name(rt), rrdcalctemplate_family_match(rt)); - DOUBLE_LINKED_LIST_REMOVE_UNSAFE(host->alarms_templates, rt, prev, next); + bool added = false; + dictionary_set_advanced(host->rrdcalctemplate_root_index, key, (ssize_t)(key_len + 1), rt, sizeof(*rt), &added); - rrdcalctemplate_free(rt); + if(added) + freez(rt); + else { + info("Health configuration template '%s' already exists for host '%s'.", rrdcalctemplate_name(rt), rrdhost_hostname(host)); + rrdcalctemplate_free_unused_rrdcalctemplate_loaded_from_config(rt); + } } |