summaryrefslogtreecommitdiffstats
path: root/database
AgeCommit message (Collapse)Author
2023-11-08Switch charts / chart to use buffer json functions (#16359)Stelios Fragkakis
Switch charts / chart to use buffer json
2023-11-07add rrddim_get_last_stored_value to simplify function code in internal ↵Ilya Mashchenko
collectors (#16348)
2023-11-07Improve agent to cloud status update process (#16342)Stelios Fragkakis
* Send node update info only if the host has finished replication * Log number of hosts replicating / pending to load context * Remove prefix (thread name is enough)
2023-11-07Better database corruption detention during runtime (#16343)Stelios Fragkakis
Detect database corruption doing query executing and schedule recovery on next restart
2023-11-06Improve unittests (#16329)Stelios Fragkakis
* Update flags during label copy Update unittests * Return proper value * Improve double check in unittests
2023-11-06give the streaming function to nightly users (#16346)Costa Tsaousis
2023-11-06Keep precompiled statements for alarm log queries to improve performance ↵Stelios Fragkakis
(#16321) * Keep precompiled statements for alarm log queries to improve performance * Check bind result
2023-11-02Systemd units function (#16318)Costa Tsaousis
* split systemd-journal.c * split fstat caching * split systemd-journal further * working systemd-units function * do not enable systemd-units when libsystemd does not provide the interface * move the header to the right place * mixed parantheses * update codacy exlcusions * update codacy exlcusions * update codacy exlcusions * added option to show show expanded filters by default * keep the original extension and decode descriptions too * updated systemd-units function to handle all known unit states * dont show the path by default * final touches * remove trailing spaces
2023-11-02Fix journal file index when collision is detected (#16319)Stelios Fragkakis
Add index properly if collision is detected
2023-11-01Optimize database before agent shutdown (#16317)Stelios Fragkakis
Optimize database before shutdown
2023-11-01Improve shutdown when collectors are active (#16315)Stelios Fragkakis
* Collectors should not be running at this point, but allow shutdown to continue after several retries (workaround) * Proceed with shutdown after 10 attempts
2023-10-31Improve dimension ML model load (#16262)Stelios Fragkakis
* Prepare metadata sync thread cleanup earlier in the shutdown process * Set flag for the dimensions that need ML MODEL load instead of queueing a message in the event loop * Process the dimension ML load during the normal dimension metadata save loop * Use spinlock for cmd queue / dequeue instead of mutex Cleanup queue structure * Remove old ML model load code * Rebase and cleanup
2023-10-29Fix label copy (#16297)Stelios Fragkakis
Fix label copy, more details in unittest
2023-10-29fix missing labels from parents (#16296)Costa Tsaousis
* maintain in /tmp/stream-receiver-X.txt a copy of metadata received * stream log metadata to /tmp/stream-sender-localhost.txt * log the stream of all senders * cleanup use of X_update_metadata() functions * fix for last commit * rrdlabel unmark/mark/delete unmarked restored
2023-10-28do not propagate upstream internal label sources (#16295)Costa Tsaousis
2023-10-28fix various issues identified by coverity (#16294)Costa Tsaousis
2023-10-28fix missing labels from parents (#16293)Costa Tsaousis
increate metadata version to resend chart upstream when labels and other metadata are updated
2023-10-28fix retention loading (#16290)Costa Tsaousis
2023-10-27Faster parents (#16127)Costa Tsaousis
* cache ctx in collection handle * cache rd together with rda * do not repeatedy call rrdcontexts - cached collection status; optimize pluginsd_acquire_dimension() * fix unit tests * do the absolutely minimum while updating timestamps, ensure validity during reading them * when the stream is INTERPOLATED, buffer outstanding data for up to 50ms if the buffer contains DATA only. * remove the spinlock from mrg * remove the metric flags that are not used any more * mrg writers can be different threads * update first time when latest clean is also updated * cleanup * set hot page with a simple atomic operation * sender sets chart slot for every chart * work on senders without SLOT * enable SLOT capability * send slot at BEGIN when SLOT is enabled * fix slot generation and parsing * send slot while re-streaming * use the sender capabilities, not the receiver * cleanup * add slots support to all chart and dimension related plugin commands * fix condition * fix calculation * check sender capabilties * assign slots in constructors * we need the dimension slot at the DIMENSION keyword * more debug info in case of dimension mismatch * ensure the RRDDIM EXPOSED flag is multi-threaded and set it after the sender buffer has been committed, so that replication will not send dimensions prematurely * fix renumbering on child restart * reset rda caching when receiving a chart definition * optimize pluginsd_end_v2() * do not do zero sized allocations * trust the chart slot id of the child * cleanup charts on pluginsd thread exit * better cleanup * find the chart and put it in the slot, if it not already there * move slots array to host * initialize pluginsd slots properly * add slots to replay begin; do not cleanup slots that dont belong to a chart * cleanup on obsolete * cleanup slots on obsoletions * cleanup and renames about obsoletion * rewrite obsolation service code to remove race conditions * better service obsoletion log * added debugging * more debug * exposed flag now compares versions * removed debugging messages * respolve conflicts * fix replication check for unsent dimensions
2023-10-27ZSTD and GZIP/DEFLATE streaming support (#16268)Costa Tsaousis
* move compression header to compression.h * prototype with zstd compression * updated capabilities * no need for resetting compression * left-over reset function * use ZSTD_compressStream() instead of ZSTD_compressStream2() for backwards compatibility * remove call to LZ4_decoderRingBufferSize() * debug signature failures * fix the buffers of lz4 * fix decoding of zstd * detect compression based on initialization; prefer ZSTD over LZ4 * allow both lz4 and zstd * initialize zstd streams * define missing ZSTD_CLEVEL_DEFAULT * log zero compressed size * debug log * flush compression buffer * add sender compression statistics * removed debugging messages * do not fail if zstd is not available * cleanup and buildinfo * fix max message size, use zstd level 1, add compressio ratio reporting * use compression level 1 * fix ratio title * better compression error logs * for backwards compatibility use buffers of COMPRESSION_MAX_CHUNK * switch to default compression level * additional streaming error conditions detection * do not expose compression stats when compression is not enabled * test for the right lz4 functions * moved lz4 and zstd to their own files * add gzip streaming compression * gzip error handling * added unittest for streaming compression * eliminate a copy of the uncompressed data during zstd compression * eliminate not needed zstd allocations * cleanup * decode gzip with Z_SYNC_FLUSH * set the decoding gzip algorithm * user configuration for compression levels and compression algorithms order * fix exclusion of not preferred compressions * remove now obsolete compression define, since gzip is always available * rename compression algorithms order in stream.conf * move common checks in compression.c * cleanup * backwards compatible error checking
2023-10-27Small optimization of alert queries (#16282)Emmanuel Vasilakis
2023-10-20Fix label copy to correctly handle duplicate keys (#16249)Stelios Fragkakis
Fix rrdlabel copy to correctly handle duplicate keys Enhance the corresponding unit tests
2023-10-20Drop an unused index from aclk_alert table (#16242)Stelios Fragkakis
* Drop unused aclk_alert index * Log messages only when compiled with NETDATA_INTERNAL_CHECKS
2023-10-18Improve context load on startup (#16203)Stelios Fragkakis
* Retrieve last connected timestamp from the database (host->last_connected) * Improve context load performance Check for agent shutdown while context load in progress Log information about host load start and finish * Remove check for slot as it will only reach this part when a slot is found
2023-10-18Reuse ML load prepared statement (#16240)Stelios Fragkakis
Reuse ML load prepared statement and release resources on each batch load Fix parameter to ML model load to be in seconds not usec
2023-10-18Improvements for labels handling (#16172)Stelios Fragkakis
* Add additional checks rrdlabels_find_label_with_key_unsafe finds a label with specified key (not only if it exists but with difefrent value) Quick check if label already exists (avoids JudyLIns) * Add migration unit test Add additional unit test to verify that adding a label with the same key will replace its value
2023-10-18Fix dimension HETEROGENEOUS check (#16234)Stelios Fragkakis
2023-10-18Fix statistics calculation in 32bit systems (#16222)Stelios Fragkakis
Use uint64_t for calculations
2023-10-17Fix meta unittest (#16221)Stelios Fragkakis
Fix log message (queue has no limit now) Fix unittest
2023-10-16dynamic meta queue size (#16218)Costa Tsaousis
* dynamic meta queue size * meta cleanup
2023-10-16Add summary to /alerts (#16213)Emmanuel Vasilakis
2023-10-16allow patterns in journal queries (#16210)Costa Tsaousis
2023-10-14journal timeout (#16195)Costa Tsaousis
* stop the query 250ms before the timeout, to allow sending back partial responses * on timeout return partial responses * give it 500ms * give some additional timeout to plugins.d garbage collection * define an extension to the timeout for all intermediate hops * hunting for the crash... * set value name and len to zero * remove unneeded memset()
2023-10-13Fix access of memory after free (#16185)Stelios Fragkakis
* Proper init to avoid use after free * CID 400083 Unchecked return value
2023-10-10Batch ML model load commands (#16155)Stelios Fragkakis
Batch ML model load
2023-10-10Don't queue removed when there is a newer alert (#16157)Emmanuel Vasilakis
* dont queue removed when there is a newer * proper concat * add host_id
2023-10-10Fix compilation warnings (#16158)Stelios Fragkakis
Drop warning when parent is not accepting job status updates
2023-10-06Code improvements (#16104)Stelios Fragkakis
* Remove unused functions * No need for prepare statement because the function is not used frequently * Remove db_meta check, already assumed valid * Remove D_ACLK_SYNC and D_METADATALOG, fix log message * Reuse prepared statements per run to avoid sql parsing all the time * Keep rowid in charts and dimensions * Host and chart labels keep rowids * Don't store internal flags * Remove commented out code * Formatting * Fix algorithm when updating dimension
2023-10-06Remove family from alerts (#16025)Emmanuel Vasilakis
* remove loading and storing families from alert configs * remove families from silencers * remove from alarm log * start remove from alarm-notify.sh.in * fix test alarm * rebase * remove from api/v1/alarm_log * remove from alert stream * remove from config stream * remove from more * remove from swagger for health api * revert md changes * remove from health cmd api test
2023-10-04journal: fix the 1 second latency in play mode (#16123)Costa Tsaousis
provide a relative_to_absolute function that does not touch the current realtime time
2023-10-03Skip database migration steps in new installation (#16071)Stelios Fragkakis
* For new installation skip database migration steps * Simplify logging * Count database tables to determine if database is empty * Report extended error message
2023-10-02Avoid duplicate keys in labels (#16014)Stelios Fragkakis
* Avoid duplicate keys in labels * Properly delete old label * Simplify memory statistics logging
2023-10-02Send alerts summary field to cloud (#16056)Emmanuel Vasilakis
* new aclk schema * transmit summary to cloud and expose in v2/alerts * missing assign
2023-09-29Dyncfg add streaming support (#15791)Timotej S
* dyncfg fncnames as constants * add helper macros to know parser streaming/plugin * plugins dictionary per RRDHOST * api_request_v2_config add support for /host/ * streamify pluginsd_register_plugin * streamify pluginsd_register_module * streamify report_job_status * streamify dyncfg get functions * module_type2str * add job type and flags * add DYNCFG_REGISTER_JOB * implement register job * push all to parent at startup * add helper function is_dyncfg_function * forward virtual functions trough streaming * separate job2json * add api/v2/job_statuses * do cleanup on streaming * streamify set functions * support FUNCTION_PAYLOAD trough streaming * WIP tests * dont attempt loading non-localhost configs * move cfg persistence to proper place * prevent race * properly update job state at runtime * cleanup 1 * job2json add missing reason * add tests * correct HTTP code * add test * streamify delete_job_cb * add DELETE_JOB keyword * job delete over streaming * add tests for create and delete job over parent * rrdpush common checks to macro * add missing forwarders * fix jobs according to test results * more tests * review comment 1 * codacy remove valid warning * codacy ruby fixes * fix wrong rc check * minimal test plugin for child * add test * dict walk insted of master lock * minor - english spelling fixes * thiago comments 1 * minor - rename folder to dynconf * enable only when built with -DNETDATA_TEST_DYNCFG * minor - compiler warning * create dir post daemonization * stricter URL check
2023-09-28Convert the ML database (#16046)Stelios Fragkakis
* Convert a db to WAL with auto vacuum * Use single sqlite configuration function * Remove UNUSED statements
2023-09-27Fix coverity 402975 (#16058)Stelios Fragkakis
Bind value as 64bit
2023-09-26Maintain node's last connected timestamp in the db (#15979)Stelios Fragkakis
* Maintain node's last connected timestamp in the db * Rebase -- switch to version database v14
2023-09-26Switch to uint64_t to avoid overflow in 32bit systems (#16048)Stelios Fragkakis
2023-09-26Fix summary field in table (#16050)Emmanuel Vasilakis
fix summary field in table
2023-09-19Add a summary field to alerts (#15886)Emmanuel Vasilakis
* add a summary field to alerts * add summary field to db * rebase * better migration * rebase * change email notification * revert to silent * use macro * add the summary field to some alerts * add more summary fields * change migration function * add to postgres alerts * add summary to vernemq * more summary fields * more summary fields * fixes * add doc
generated by cgit v1.2.3 (git 2.25.1) at 2024-09-07 17:42:05 +0000