| Age | Commit message (Collapse) | Author |
|---|
|
* use brackets around vars
* update doc
* change doc
* try fix codacy
* fix 2
|
|
* Add profile.plugin
Creates the specified number of charts/dimensions, and supports
backfilling with pseudo-historical data.
* Bump
* Remove wrongly merged line.
* Use the number of models specified from the config section.
* Add option to consult all ML models.
* Remove profiling option consuming all models.
* Add underscore after chart name prefix.
* prediction -> dimensions chart
* reorder funcs
* Split charts across types with correct priority
* Ignore training request when chart is under replication.
* Track global number of models consulted.
* Cleanup config.
* initial readme updates
* fix readme
* readme
* Fix function definition when ML is disabled.
* Add dummy ml_chart_update_{begin,end}
* Remove profile_plugin
* Define chart priorities under collectors/all.h
* s/curr_t/current_time/
* Use libnetdata's lock/thread wrappers.
* Fix autotools & cmake builds.
* Delete ML dimensions & charts.
* Let users of buffer preprocessing to handle memory.
* Add separate API calls to start/stop ML threads.
Co-authored-by: Andrew Maguire <andrewm4894@gmail.com>
|
|
(#14172)
|
|
* Add profile.plugin
Creates the specified number of charts/dimensions, and supports
backfilling with pseudo-historical data.
* Bump
* Remove wrongly merged line.
* Use the number of models specified from the config section.
* Add option to consult all ML models.
* Remove profiling option consuming all models.
* Add underscore after chart name prefix.
* prediction -> dimensions chart
* reorder funcs
* Split charts across types with correct priority
* Ignore training request when chart is under replication.
* Track global number of models consulted.
* Cleanup config.
* initial readme updates
* fix readme
* readme
* Fix function definition when ML is disabled.
* Add dummy ml_chart_update_{begin,end}
* Remove profile_plugin
* Define chart priorities under collectors/all.h
* s/curr_t/current_time/
Co-authored-by: Andrew Maguire <andrewm4894@gmail.com>
|
|
page alignment is a property of metric collection, so it has been moved to metric collection only
|
|
* proper use for atomic_compare_exchange()
* diskspace plugin is multi-threaded but it uses single threaded dictionaries
|
|
* Wait for pending read to complete before destroying the page
* fix page alignment crash
* Compare copy of descriptor
* prevent workers crashes by disabling cancellability on critical areas and separate sqlite3 statistics to its own worker job
* do not update sqlite3 stats when they are slow
* do not query sqlite3 statistics when they are slow
* flipped condition
* sqlite3 proper timeout calculation
Co-authored-by: Costa Tsaousis <costa@netdata.cloud>
|
|
* replication fixes 9
* no room metric is now absolute
* decrement senders full and flip the actions on sender full and sender available
* finer timings
* execute all requests, unconditionally, once they are in the replication queue; worker charts are now sorted
* remove left-over debug message
* log verification of replication completion only when there are no pending requests any more
* use up to 50% of the sender buffer for replication responses
|
|
* move global statistics workers to a separate thread; query statistics per query source; query statistics for ML, exporters, backfilling; reset replication point in time every 10 seconds, instead of every 1; fix compilation warnings; optimize the replication queries code; prevent long tail of replication requests (big sleeps); provide query statistics about replication ; optimize replication sender when most senders are full; optimize replication_request_get_first_available(); reset replication completion calculation;
* remove workers utilization from global statistics thread
|
|
use the faster monotonic clock in workers and replication; avoid unecessary statistics function on every request on replication - gather them all together once every second; check the chart flags on all mirrored hosts, not only the ones that have a sender; cleanup and unify replication logs; added child world time to REND; fix first BEGIN been transmitted when replication starts;
|
|
* pluginsd cleanup; replication logic cleanup; fix bug in replication begin
* log replication start/stop and change the keyword of NETDATA_LOG_REPLICATION_REQUESTS logs to REPLAY
* dont ask for data the child does not have; log fixes
* more pluginsd cleanup
* count sender dictionary entries
* fix dictionary_flush()
|
|
* cleanup and additional information about replication
* fix deadlock on sender mutex
* do not ignore start streaming empty requests; when there duplicate requests, merge them
* flipped the flag
* final touch
* added queued flag on the charts to prevent them from being obsoleted by the service thread
|
|
* Remove calls to rrdset_next().
* Rm checks plugin
* Update documentantion
* Call rrdset_next from within rrdset_done
This wraps up the removal of rrdset_next from internal collectors, which
removes a lot of unecessary code and the need for if/else clauses in
every place.
The pluginsd parser is the only component that calls rrdset_next*()
functions because it's not strictly speaking a collector but more of a
collector manager/proxy.
With the current changes it's possible to simplify the API we expose
from RRD significantly, but this will be follow-up work in the future.
* Remove stale reference to checks.plugin
* Fix RRD unit test
rrdset_next is not meant to be called from these tests.
* Fix db engine unit test.
* Schedule rrdset_next when we have completed at least one collection.
* Mark chart creation clauses as unlikely.
* Add missing brace to fix FreeBSD plugin.
|
|
(#14031)
* use 2 levels of judy arrays to speed up replication on very busy parents
* delete requests from judy when executed
* do not process requests when the sender is not connected; not all requests removed are executed, so count the executed accurately
* flush replication data on sender disconnect
* cache used buffer ratio in sender structure
* cleanup replication requests when they are not valid any more
* properly update inner and outer judy arrays on deletions
* detailed replication stats
* fix bug in dictionary where deletion and flushes lead to crashes
* replication should only report retention of tier 0
* replication now has 2 buffer limits 10 -> 30 %
* detailed statistics about resets and skipped requests
* register worker metrics
* added counter for waits
* make new requests discoverable
* make it continue on iterations properly
|
|
* streaming compression, query planner and replication fixes
* remove journal v2 stats from global statistics
* disable sql for checking past sql UUIDs
* single threaded replication
* final replication thread using dictionaries and JudyL for sorting the pending requests
* do not timeout the sending socket when there are pending replication requests
* streaming receiver using read() instead of fread()
* remove FILE * from streaming - now using posix read() and write()
* increase timeouts to 10 minutes
* apply sender timeout only when there are metrics that are supposed to be streamed
* error handling in replication
* remove retries on socket read timeout; better error messages
* take into account inbound traffic too to detect that a connection is stale
* remove race conditions from replication thread
* make sure deleted entries are marked as executed, so that even if deletion fails, they will not be executed
* 2 minutes timeout to retry streaming to a parent that already has this node
* remove unecessary condition check
* fix compilation warnings
* include judy in replication
* wrappers to handle retries for SSL_read and SSL_write
* compressed bytes read monitoring
* recursive locks on replication to make it faster during flush or cleanup
* replication completion chart at the receiver side
* simplified recursive mutex
* simplified recursive mutex again
|
|
|
|
Revert "New journal disk based indexing for agent memory reduction (#13885)"
This reverts commit 224b051a2b2bab39a4b536e531ab9ca590bf31bb.
|
|
Revert "enable dbengine tiering by default (#13914)"
This reverts commit 2de2e45656dc9875326d7d0e0f76de61d1377e2a.
|
|
enable tiering by default
|
|
* cleanup capas + add func capa
* make it const
* fixes
* freez
|
|
* Add read only option to netdata_mmap so files are accessed ousing PROT_READ
* Initial functions to write the new journal file and switch to the new indexing
* Cleanup code, add parameters to pg_cache_punch_hole to avoid updating page latets oldest times
pg_cache insert to have parameter if page index locked needs to be done
Page eviction functions will try to deallocate the descriptor as well (pg_cache_punch_hole without page_index time updates)
Cleanup messages during startup
* Cleanup messages during startup
* Disbale extent caching for now, add placeholder for journal indexing and activation while the agent is running
* Add main function to populate descriptors by checking the new journal indexing
* prevent crash
* fix for binary search crash
* Avoid Time-of-check time-of-use filesystem race condition
* always add a page
* populate fixes - it is still incomplete
* pg_cache_insert returns the descriptor that ends up in the page_index
* Add populate next (Fix 1)
* Fix compilation warnings, reactivate extent caching
* Add populate next (Fix 2)
* Add populate next (Fix 3) switch to the next entry or journal file when asking to populate descriptor with next
* Fix resource leak and wrong sizeof
* Rework page population (part 1)
* Additional checksums added / journal validation
* Cleanup (part 1)
* Locking added and Cleanup (part 2)
* Close journal file after new journal index activation
* Skip warning when compiling without NETDATA_INTERNAL_CHECKS
* Ignore empty index file (header and trailer and no metrics)
* Try to remove all evicted descriptors (may prevent slight memory increase)
* Evict pages also when we succesfully do try_reserve
* Precache pages and cleanup
* Add a separate cleanup thread to release unused descriptors
* Check existence of key correctly
* Fix total file size calculation
* Statistics for journal descriptors
* Track and release jourval v2 descriptors
* Do not try to allocate pages for locality if under pressure
* Do not track v2 descriptors when populating the page_index
* Track page descriptors as they are inserted in the page index (per journal file)
Scan journal files for pending items to cleanup
Cleanup v2 descriptors only if they are not populated
Check before adding to page cache to avoid memory allocation /free
* Close journal file that has been processed and migrated to the new index
Check for valid file before trying to truncate / close. This file has been closed during startup
* Better calculation for the number of prefetched data pages based on the query end time
Code cleanup and comments
Add v2 populated descriptor expiration based on journal access time
* Code cleanup
* Faster indexing
Better journal validation (more sanity checks)
Detect new datafile/ journal creation and trigger index generation
Switch to the new index / mark descriptors in memory as needed
Update journal access time when a descriptor is returned
Code cleanup (part 1)
* Re activate descriptor clean
Code cleanup
* Allow locality precaching
* Allow locality precaching for the same page alignment
* Descriptor cleanup internal changed
* Disable locality precaching
* Precache only if not under pressure / internal cleanup at 60 seconds
* Remove unused functions
* Migrate on startup always
Make sure the metric uuid is valid (we have a page_index)
Prevent crash if no datafile is available when logging an error
Remove unused functions
* New warn limit for precaching
Stress test v2 descriptor cleanup
- Every 1s cleanup if it doesnt exist in cache
- 60s cache eviction
* Arrayalloc internal checks on free activated with NETDATA_ARRAYALLOC_INTERNAL_CHECKS
Ability to add DESCRIPTOR_EXPIRATION_TIME and DESCRIPTOR_INTERVAL_CLEANUP during compile
Defaults DESCRIPTOR_INTERVAL_CLEANUP = 60 and DESCRIPTOR_EXPIRATION_TIME = 600
* Lookup page index correctly
* Calculate index time once
* Detect a duplicate page when doing cache insert and during flushing of pages
* Better logging
* Descriptor validation (extent vs page index) when building an index file while the agent is running
* Mark invalid entries in the journal v2 file
* Schedule an index rebuild if a descriptor is found without an extent in the timerange we are processing
Release descriptor lock to prevent random shutdown locks
* Proper unlock
* Skip descriptor cleanup when journal file v2 migration is running
* Fix page cache statistics
Remove multiple entries of the page_index from the page cache
Cleanup
* Adjust preload pages on pg_cache_next. Handle invalid descriptor properly
Unlock properly
* Better handling of invalid pages
Journal indexing during runtime will scan all files to find potential ones to index
* Reactivate migration on startup
Evict descriptors to cause migration
Don't count the entries in page index (calculate when processing the extent list)
Check for valid extent since we may set the extent to NULL on startup if it is invalid
Better structure init
Address valgrind issues
* Add don't fork/dump option
* Add separate lock to protect accessing a datafile's extent list
Comment out some unused code (for now)
Abort descriptor cleanup if we are force flushing pages (page cache under pressure)
* Check for index and schedule when data flush completes
Configure max datafile size during compilation
Keep a separate JudyL array for descriptors
Skip quota test if we are deleting descriptors or explicitly flushing pages under pressure
* Fix
* set function when waiters are waken up
* add the line number to trace the deadlock
* add thread id
* add wait list
* init to zero
* disable thread cancelability inside dbengine rrdeng_load_page_next()
* make sure the owner is the thread
* disable thread cancelability for replication as a whole
* Check and queue indexing after first page flush
* Queue indexing after a small delay to allow some time for page flushing
* tracing of waiters only when compiled with internal checks
* Mark descr with extent_entry
* Return page timeout
* Check if a journalfile is ready to be indexed
Migrate the descriptors or evict if possible
Compilation warning fix
* Use page index if indexing during startup
Mark if journalfile should be checked depending on whether we can migrate or delete a page during indexing
* require 3x max message size as sender buffer
* fix for the msg of the adaptive buffer size
* fix for the msg of the duplicate replication commands
* Disable descriptor deletion during migration
* Detect descriptor with same start page time
* sender sorts replication requests before fullfilling them; receiver does not send duplicate replication requests
* dbengine never allows past timestamps to be collected
* do not accept values same as last data point stored in dbengine
* replicate non-overlapping ranges
* a better replication logic to avoid sending overlapping data to parents
* Do not start journal migration in parallel
* Always update page index times
* Fix page index first / last times on load
* internal log when replication responses do not match the requests or when replication commands are sent while there are others inflight
* do not log out of bounds RBEGIN if it is the last replication command we sent
* better checking of past data collection points
* better checking of past data collection points - optimized
* fix corruption during decompression of streaming
* Add config to disable journal indexing
Add config parameter for detailed journal integrity check (Metric chain validation check during startup)
pg cache insert drop check for existing page
Fix crc calculation for metric headers
* children disable compression globally, only when the compression gives an error
* turn boolean member into RRDHOST OPTION
* Compilation warnings
* Remove unused code
* replication sender statistics
* replication sender statistics set to 100% when no replication requests are pending
* Fix casting warning
Co-authored-by: Costa Tsaousis <costa@netdata.cloud>
|
|
|
|
contexts are not set to collected until replication finishes for them; sender thread disables cancelability while replication queries are executed;
|
|
* remove health_thread_stop
* soft sleeps
|
|
(#13968)
break active-active loop from replicating non-existing child to each-other
|
|
|
|
* reduce alert events to the cloud
* proper column, set filtered when queing existing
* increase max removed period to a day
* add constraint, fix queries
|
|
* do not resend charts upstream when chart variables are being updated
* re-stream archived hosts that are now being collected
|
|
|
|
* prevent RRDDIM from vanishing while replay is working with it
* set chart last access time
* set chart last access time everytime someone finds it
* do not replay dimensions that are archived
* remove the obsolete flag from dimensions that replayed; do not process archived dimensions
* cleanup db_metric_handle refcount of hidden dimensions
* more information in page alignment fatal
* do not fatal() of page alignment reset when the caller is the only writer
|
|
|
|
* Revert "Use llvm's ar and ranlib when compiling with clang (#13854)"
This reverts commit a9135f47bbb36e9cb437b18a7109607569580db7.
* Profile plugin
* Fix macos static thread
* Add support for replication
- Add a new capability for replication, when not supported the agent
should behave as previously.
- When replication is supported, the text protocol supports the
following new commands:
- CHART_DEFINITION_END: send the first/last entry of the child
- REPLAY_RRDSET_BEGIN: sends the name of the chart we are
replicating
- REPLAY_RRDSET_HEADER: sends a line describing the columns of the
following command (ie. start-time, end-time, dim1-name, ...)
- REPLAY_RRDSET_DONE: sends values to push for a specific start/end
time
- REPLAY_RRDSET_END: send the (a) update every of the chart, (b)
first/last entries in DB, (c) whether the child's been told to
start streaming, (d) original after/before period to replicate.
- REPLAY_CHART: Sent from a parent to a child, specifying (a)
the chart name we want data for, (b) whether the child should
start streaming once it has fullfilled the request with the
aforementioned commands, (c) after/before of the data the parent
wants
- As a consequence of the new protocol, streaming is disabled for all
charts on a new connection. It's enabled once replication is finished.
- The configuration parameters are specified from within stream.conf:
- "enable replication = yes|no"
- "seconds to replicate = 3600"
- "replication step = 600" (ie. how many seconds to fill per
roundtrip request.
* Minor fixes
- quote set and dim ids
- start streaming after writing replicated data to the buffer
- write replicated data only when buffer is less than 50% full.
- use reentrant iteration for charts
* Do not send chart definitions on connection.
* Track replication status through rrdset flags.
* Add debug flag for noisy log messages.
* Add license notice.
* Iterate charts with reentrant loop
* Set replication finished flag when streaming is disabled.
* Revert "Profile plugin"
This reverts commit 468fc9386e5283e0865fae56e9989b8ec83de14d.
Used only for testing purposes.
* Revert "Revert "Use llvm's ar and ranlib when compiling with clang (#13854)""
This reverts commit 27c955c58d95aed6c44d42e8b675f0cf3ca45c6d.
Reapply commit that I had to revert in order to be able to build the
agent on MacOS.
* Build replication source files with CMake.
* Pass number of words in plugind functions.
* Use get_word instead of indexing words.
* Use size_t instead of int.
* Pay only what we use when splitting words.
* no need to redefine PLUGINSD_MAX_WORDS
* fix formatting warning
* all usages of pluginsd_split_words() should use the return value to ensure non-cached results reuse; no need to lock the host to find a chart
* keep a sender dictionary with all the replication commands received and remove replication commands from charts
* do not replicate future data
* use last_updated to find the end of the db
* uniformity of replication logs
* rewrite of the query logic
* replication.c in C; debug info in human readable dates
* update the chart on every replication row
* update all chart members so that rrdset_done() can continue
* update the protocol to push one dimension per line and transfer data collection state to parent
* fix formatting
* remove replication object from pluginsd
* shorter communication
* fix typo
* support for replication proxies
* proper use of flags
* set receiver replication finished flag on charts created after the sender has been connected
* clear RRDSET_FLAG_SYNC_CLOCK on replicated charts
* log storing of nulls
* log first store
* log update every switches
* test ignoring timestamps but sending a point just after replication end
* replication should work on end_time
* use replicated timestamps
* at the final replication step, replicate all the remaining points
* cleanup code from tests
* print timestamps as unsigned long long
* more formating changes; fix conflicting type of replicate_chart_response()
* updated stream.conf
* always respond to replication requests
* in non-dbengine db modes, do not replicate more than the database size
* advance the db pointer of legacy db modes
* should be multiplied by update_every
* fix buggy label parsing - identified by codacy
* dont log error on history mismatches for db mode dbengine
* allow SSL requests to streaming children
* dont use ssl variable
Co-authored-by: Costa Tsaousis <costa@netdata.cloud>
|
|
|
|
chart (#13911)
* timeframe matching should take into account the update frequency of the chart
* twice update frequency is needed
|
|
|
|
|
|
(#13886)
|
|
* Add functions to find chart uuid and dimension UUIDs from context
* Remove old functions that access the sqlite database directly
* Use new functions to fetch the UUIDs for chart and dimensions
* Remove unused function
|
|
* use print macros
* cast instead
|
|
dont create REMOVED after a REMOVED
|
|
|
|
* Use MMAP to read an extent from the datafile. Fallback to uv_fs_read if that fails
* Schedule the fallback using uv_fs_read in the MMAP callback
* Check the return value of mmap correctly
* Add unlikely
|
|
* initial implementation of QUERY_TARGET
* rrd2rrdr() interface
* rrddim_find_best_tier_for_timeframe() ported
* added dimension filtering
* added db object in query target
* rrd2rrdr() ported
* working on formatters
* working on jsonwrapper
* finally, it compiles...
* 1st run without crashes
* query planer working
* cleanup old code
* review changes
* fix also changing data collection frequency
* fix signess
* fix rrdlabels and dimension ordering
* fixes
* remove unused variable
* ml should accept NULL response from rrd2rrdr()
* number formatting fixes
* more number formatting fixes
* more number formatting fixes
* support mc parallel queries
* formatting and cleanup
* added rrd2rrdr_legacy() as a simplified interface to run a query
* make sure rrdset_find_natural_update_every_for_timeframe() returns a value
* make signed comparisons
* weights endpoint using rrdcontexts
* fix for legacy db modes and cleanup
* fix for chart_ids and remove AR chart from weights endpoint
* Ignore command if not initialized yet
* remove unused members
* properly initialize window
* code cleanup - rrddim linked list is gone; rrdset rwlock is gone too
* reviewed RRDR.internal members
* eliminate unnecessary members of QUERY_TARGET
* more complete query ids; more detailed information on aborted queries
* properly terminate option strings
* query id contains group_options which is controlled by users, so escaping is necessary
* tense in query id
* tense in query id - again
* added the remaining query options to the query id
* Expose hidden option to the dimension
* use the hidden flag when loading context dimensions
* Specify table alias for option
* dont update chart last access time, unless at least a dimension of the chart will be queried
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
|
|
* Rebased
* rebased
* health_execute_pending_updates -> health_execute_delayed_initializations
* fix labels for current host only
* missing bracket
* misc fixes, reload health for disconnected hosts
* remove volatile, add comment
|
|
* fix tiers update frequency
* dont change mqtt_websockets
|
|
* Remove old metalog text fle processing
* Add metadata event loop
* Move functions from sqlite_functions.c to sqlite_metadata.c
Queue updates to the metadata event loop
Migration to remove unused tables
Cleanup unused functions
* Queue chart labels to metadata
* Store chart labels to metadata
* During shutdown, run full speed
* Add shutdown prepare
Handle SHUTDOWN in the cmd queue function
Add worker thread to handle host/chart/dimension metadata doing dictionary traversals
* Remove unused RRDIM_FLAG_ACLK
Add flags to trigger host/chart/dimension metadata processing
* Incremental processing of chart metadata writes
* Store host labels
* Remove redundant return statements
* Change unit tests / cleanup
* Fix rescheduling
* Schedule chart labels update by setting the RRDSET_FLAG_METADATA_UPDATE flag
* Queue commands to update metadata for dimension and host labels
* Make sure we do a final scan to store metadata during shutdown (if needed)
* Remove unused structures
Adjust queue size since we do batch processing of updates without queueing individual messages
Remove pragma mmap for now
Fix memory leak during sqlite unittest (minor)
* Dont update if we are in archive mode
* Cleanup
* Build entire message payload and store
* Initialize worker completion properly
* Properly skip host check for pending metadata updates
* Report bind param failures
Add worker request inside the data payload
Initialize variables to silence warnings
Rebase on master
* Report the chart id (not the dimension) and the dimension id when storing a dimension
* Compilation warnings in 32bit
* Add DEFINE for the queries
* Remove commented out code
* * Remove items parameter from unitest
* Remove commented out code
* sqlite_metadata.h contains only public items
* Use sleep_usec instead of usleep
* Rename metadata_database_init_cmd_queue to metadata_init_cmd_queue
* Rename metadata_database_enq_cmd_noblock to metadata_enq_cmd_noblock
|
|
Add variants of functions allowing callers to specify the timeval to use.
These functions will be prove helfupl in writting tests that are both
(a) memory mode agnostic, and
(b) oblivious of rrd internals.
|
|
Increase page_index reference count on metric_get
Don't free the db_metric_handle
|
|
rebased
|
|
* dbengine free from RRDSET and RRDDIM
* fix for excess parameters to query ops
* add comment about ML
* update_every from int to uint32_t
* rrddim_mem storage engine working
* fixes for update_every_s
* working dbengine
* a lot of changes in dbengine regarding timestamps
* better logging of not sequential points
* rrdset_done() now gives aligned timestamps for higher tiers
* dont change the end_time of descriptors, because they cant be loaded back
* fixes for cmake
* fixes for db mode ram
* Global counters for dbengine loading errors.
Ensure dbengine store metrics always has aligned metrics or breaks the page when storing new data.
* update lgtm config
* fixes for 32-bit systems
* update unittests
* Don't try to find and create a host on the fly if not already in memory
* Remove unused functions
* print backtrace in case of fatal
* always set ctx to page_index
* detect ctx and metric uuid discrepancies
* use legacy uuid if multihost is not available
* fix for last commit
* prevent repeating log
* Do not try to access archived charts when executing a data query
* Remove unused function
* log inconsistent collections once every 10 mins
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
|
