summaryrefslogtreecommitdiffstats
path: root/database
AgeCommit message (Collapse)Author
2022-05-03Configurable storage engine for Netdata agents: step 1 (#12776)Adrien Béraud
* rrd: move API structures out of rrddim_volatile In C, unlike C++, it's not possible to reference a nested structure from outside this structure. Since we later want to use rrddim_query_ops and rrddim_collect_ops separately from rrddim_volatile, move these nested structures out. * rrd: use opaque handle types for different memory modes
2022-05-03Check for chart obsoletion on children re-connections (#12707)Emmanuel Vasilakis
* check for chart obsoletion on children connections * use rrdset_is_obsolete
2022-05-03One way allocator to double the speed of parallel context queries (#12787)Costa Tsaousis
* one way allocator to speed up context queries * fixed a bug while expanding memory pages * reworked for clarity and finally fixed the bug of allocating memory beyond the page size * further optimize allocation step to minimize the number of allocations made * implement strdup with memcpy instead of strcpy * added documentation * prevent an uninitialized use of owa * added callocz() interface * integrate onewayalloc everywhere - apart sql queries * one way allocator is now used in context queries using archived charts in sql * align on the size of pointers * forgotten freez() * removed not needed memcpys * give unique names to global variables to avoid conflicts with system definitions
2022-05-02Reduce alert events sent to the cloud. (#12544)Emmanuel Vasilakis
* filter * update filter * queue removed directly * more * logging * cleanup * cleanup 2 * cleanup 3 * finalize instead of reset
2022-05-02Avoid clearing already unset flags. (#12727)vkalintiris
If memory mode is save, map or ram the set's flags are initialized to 0. Otherwise, the set is calloc'd which will make the set have 0 flags.
2022-05-02Make atomics a hard-dep. (#12730)vkalintiris
They are used extensively throughout our code base, and not having support for them does not generate a thread-safe agent.
2022-04-28feat(dbengine): make dbengine page cache undumpable and dedupuble (#12765)Ilya Mashchenko
* make netdata more awesome * reworked on-madvise and mmap to provide clarity
2022-04-26chore: reduce logging in rrdset (#12739)Ilya Mashchenko
2022-04-20Remove commented code. (#12726)vkalintiris
2022-04-20Store alert log entries even if alert is repeating. (#12226)Emmanuel Vasilakis
* store alarm log entries even if it is repeating * log times repeated for an alert
2022-04-19Add the ability to perform a data query using an offline node id (#12650)Stelios Fragkakis
* Add the ability to build a host structure by node id to execute queries for archived hosts * Add the ability to execute queries from the cloud for archived hosts by node id * Add free_temporary_host function
2022-04-19Fix Valgrind errors (#12619)Vladimir Kobal
2022-04-12Delete ML-related data of a host in the proper order. (#12672)vkalintiris
Initialization of ML-related structures and threads should happen when the underlying RRD objects have been fully initialized. Destruction should happen in the opposite way, ie. before deleting an RRD host/dimension.
2022-04-12Add a chart label filter parameter in context data queries (#12652)Stelios Fragkakis
* Add function to filter chart labels * Add new parameter to filter chart labels on context queries * Change swagger * Better formatting for swagger
2022-04-06feat: add support for cloud providers info to /api/v1/info (#12613)Ilya Mashchenko
2022-04-05Fix a compilation warning (#12608)Vladimir Kobal
2022-04-04Update ML-related charts (#12574)vkalintiris
* Move CPU usage stats under netdata charts Use the hostname in each chart's name, and the machine GUID in each chart's id. * Move anomaly_detection.* charts to child host instance. * Add option to enable/disable streaming of ML-related charts. * Update priority of prediction/training charts.
2022-04-04chore: disable chart exists message (#12594)Ilya Mashchenko
2022-04-01Fix issue with charts not properly synchronized with the cloud (#12451)Stelios Fragkakis
* Add function to check a specific chart * If a chart is not obsoleted, check if the liveness needs to be updated * Calculate liveness based on a (constant * update_every) for each dimension * Scan all dimensions when the retention message is constructed and update liveness if needed * If initial state, set to computed live * Set computed live state to dimension * Add a maximum dimension cleanup on startup to prevent message flood * Schedule chart updates if charts streaming is enabled * Adjust live state for dimension * The query executed will have a valid dimension uuid only if memory mode is dbengine
2022-04-01Fix memory leaks on Netdata exit (#12511)Vladimir Kobal
* Fix memory leaks in dimensions and charts * Initialize superblock memory regions * Clean up static threads * Fix memory leaks in compression * Fix memory leaks in rrdcaltemplate * Fix memory leaks in health config * Fix ACLK memory leaks
2022-03-31Respect dimension hidden option when executing a query and building the ↵Stelios Fragkakis
dimension list from the database (#12570)
2022-03-31Improve ACLK sync logging (#12534)Stelios Fragkakis
* Switch messages to ACLK RES, ACLK REQ, ACLK STA instead of OG, IN and just AC * Lookup hostname by node id * Record hostname when receiving an ACK for a chart sequence * Additional log_access info * Adjust log message when receing health log request * Remove redundant ACK log message * Remove duplicate log message * Remove duplicate sql statements * Rearrange variable definition for clarity * Make sure node is a valid UUID (check return code)
2022-03-31Initialize foreach alarms of dimensions in health thread. (#12452)vkalintiris
The previous approach required us to try wr-lock the host after locking a chart and sleeping on failure. Lock contention would lead to alarms not being created and the agent to become unresponsive.
2022-03-30Don't send alert events without wc->host (#12547)Emmanuel Vasilakis
* if wc->host is null dont send events * we will always have wc->host * free claim_id
2022-03-28Skip `foreach` alarms for dimensions of anomaly rate chart. (#12441)vkalintiris
Health is not enabled for the anomaly rates chart. This was missed in the original PR that added support for tracking anomaly rates with dbengine. The side-effect was that the agent would block when opening the dashboard before its initialization was done.
2022-03-24Dont send a snapshot with snapshot id 0 (#12469)Emmanuel Vasilakis
2022-03-16Fix os_name for older agent versions streaming to a parent (#12425)Stelios Fragkakis
2022-03-15Remove backends subsystem (#12146)Vladimir Kobal
2022-03-14Dont check host health enabled if host is null (#12392)Emmanuel Vasilakis
2022-03-11Only store alert hashes once per health config iteration (#12292)Emmanuel Vasilakis
* only store alert hashes when iterated from localhost * store hashes on start and health reload, at least for one pass of a host
2022-03-10Replace write with read locks (#12309)Emmanuel Vasilakis
2022-03-10Fix chart synchronization with the cloud #12356Stelios Fragkakis
2022-03-09Improve agent to cloud synchronization performance (#12348)Stelios Fragkakis
* Switch to prepare statement when storing active charts / dimensions * Switch to prepare statement when storing chart labels * Switch to prepare statement when doing a node id lookup * Switch to prepare statement when loading the node id for a host * Improve performance by avoiding db query * Use prepare statement when counting pending chart messages to send to the cloud * Delay locking while preparing commands * No need to use buffer, avoid memory allocation overhead * Switch to prepare statement when loading pending chart updates to send to the cloud
2022-03-09Adds more info to aclk-state API call (#12231)Timotej S
2022-03-08Adjust cloud dimension update frequency (#12284)Stelios Fragkakis
* Queue a chart immediately to the cloud * Do not inform the cloud immediately if a dimension stopped collecting use MAX(obsoletion time, 1.5 * update_every) * Notify cloud immediately on dimension deletion * Add debug messages * Do not schedule an update if we are shutting down
2022-02-28Setting a DB version (to make future schema changes / migration easier) (#12249)Stelios Fragkakis
2022-02-25Add a fix to correctly register child nodes to the cloud via a parent (#12241)Stelios Fragkakis
* Add a trigger to populate the node_instance table. This will allow older agent versions pre v1.31 to connect to the cloud via the parent * Minor fix : Make the trigger creation a separate statement
2022-02-24Fix node information send to the cloud for older agent versions (#12223)Stelios Fragkakis
* Find the correct host netdata version from streaming info if not localhost * Handle old netdata versions that do not supply information during the streaming connection * Send unknown agent version if child is not connected
2022-02-24Track anomaly rates with DBEngine. (#12083)vkalintiris
* Track anomaly rates with DBEngine. This commit adds support for tracking anomaly rates with DBEngine. We do so by creating a single chart with id "anomaly_detection.anomaly_rates" for each trainable/predictable host, which is responsible for tracking the anomaly rate of each dimension that we train/predict for that host. The rrdset->state->is_ar_chart boolean flag is set to true only for anomaly rates charts. We use this flag to: - Disable exposing the anomaly rates charts through the functionality in backends/, exporting/ and streaming/. - Skip generation of configuration options for the name, algorithm, multiplier, divisor of each dimension in an anomaly rates chart. - Skip the creation of health variables for anomaly rates dimensions. - Skip the chart/dim queue of ACLK. - Post-process the RRDR result of an anomaly rates chart, so that we can return a sorted, trimmed number of anomalous dimensions. In a child/parent configuration where both the child and the parent run ML for the child, we want to be able to stream the rest of the ML-related charts to the parent. To be able to do this without any chart name collisions, the charts are now created on localhost and their IDs and titles have the node's machine_guid and hostname as a suffix, respectively. * Fix exporting_engine tests. * Restore default ML configuration. The reverted changes where meant for local testing only. This commit restores the default values that we want to have when someone runs anomaly detection on their node. * Set context for anomaly_detection.* charts. * Check for anomaly rates chart only with a valid pointer. * Remove duplicate code. * Use a more descriptive name for id/title pair variable
2022-02-23Store dimension hidden option in the metadata db (#12196)Stelios Fragkakis
* Add a function to update dimension options in the metadata database * Update the option for dimension to be hidden/unhinden when rrdim_hide/rrdim_unhide is called * Store the hidden option for dimensions to the database
2022-02-23Improve cleaning up of orphan hosts (#12201)Stelios Fragkakis
* Move the rrdhost_cleanup_orphan_hosts_nolock to the service that processes obsolete charts * Add OPCODE to mark a host as orphan * Queue cmd to mark a host as orphan
2022-02-22Remove chart specific configuration from netdata.conf except enabled (#12209)Stelios Fragkakis
2022-02-22Skip info field in protobuf alerts messages if it doesn't exist. (#12210)Emmanuel Vasilakis
* dont assume info field exists * add info field to documentation
2022-02-21Add serial numbers to chart names (#12067)Vladimir Kobal
2022-02-21Fix compilation warnings on macOS (#12082)Vladimir Kobal
2022-02-18adds install method to /api/v1/info as label (#12040)Timotej S
* install_type to rrdhost_system_info and as labels
2022-02-17Docs: Removed Google Analytics tags (#12145)Tina Luedtke
2022-02-08Disable hashes for charts and alerts if openssl is not available or cloud is ↵Emmanuel Vasilakis
disabled (#12071) * disable hashes for charts and alerts if openssl is not available * create hashes if disable_cloud has not been defined and https has been defined
2022-01-19Handle re-claim while the agent is running in new architecture (#11924)Emmanuel Vasilakis
* re-connect when re-claiming * send the previous claim_id when disconnecting * use same block for aclk_kill_link * free prev_claimed_id
2022-01-19Fix queue removed alerts (#11996)Emmanuel Vasilakis
* delay queueing removed alerts * parenthesis * remove debug