| Age | Commit message (Collapse) | Author |
|---|
|
* replication cancels pending queries on exit
* log when waiting for inflight queries
* when there are collected and not-collected metrics, use the context priority from the collected only
* Write metadata with a faster pace
* Remove journal file size limit and sync mode to 0 / Drop wal checkpoint for now
* Wrap in a big transaction remaining metadata writes (test 1)
* fix higher tiers when tiering iterations = 2
* dbengine always returns db-aligned points; query engine expands the queries by 2 points in every direction to have enough data for interpolation
* Wrap in a big transaction metadata writes (test 2)
* replication cancelling fix
* do not first and last entry in replication when the db has no retention
* fix internal check condition
* Increase metadata write batch size
* always apply error limit to dbengine logs
* Remove code that processes the obsolete health.db files
* cleanup in query.c
* do not allow queries to go beyond db boundaries
* prevent internal log for +1 delta in timestamp
* detect gap pages in conflicts
* double protection for gap injection in main cache
* Add checkpoint to prevent large WAL while running
Remove unused and duplicate functions
* do not allocate chart cache dir if not needed
* add more info to unittests
* revert query expansion to satisfy unittests
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
|
|
|
|
Revert "Misc SSL improvements (#14317)"
This reverts commit f4d017ccf7385df3e152eeb5ea1259702ab01043.
|
|
* Remove old documentation check related config files.
These are no longer used by any of our CI as far as I can tell.
* Purge LGTM configuration and references.
It has been fully shut down, so none of this works anymore.
* Purge Travis CI config, scripts, and references.
We are no longer using Travis CI, so all of this is useless.
* Pureg config for other CI tools we are no longer using.
* Remove old packaging related test scripts.
These haven’t been used in years, and are potentially confusing for new
contributors.
* Restore MLC configuration.
It is, in fact, still in use.
* Fix bogus CI config in dist files.
* Fix botched merge in CODEOWNERS.
|
|
* Moving the cloud docs under /docs/cloud (previous location: netdata/learn/*)
* Added metadata on almost every document of the old learn site for the new ingest process of learn.
* Map old learn document to their best fit as topic related docs.
Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>
Co-authored-by: DShreve2 <david@netdata.cloud>
Co-authored-by: hugovalente-pm <hugo@netdata.cloud>
|
|
* set web client to poll when ssl error want read or write
* turn to function
|
|
* cache 100 pages for each size our tiers need
* smarter page caching
* account the caching structures
* dynamic max number of cached pages
* make variables const to ensure they are not changed
* make sure replication timestamps do not go to the future
* replication now sends chart and dimension states atomically; replication receivers ignores chart and dimension states when rbegin is also ignored
* make sure all pages are flushed on shutdown
* take into account empty points too
* when recalculating retention update first_time_s on metrics only when they are bigger
* Report the datafile number we use to recalculate retention
* Report the datafile number we use to recalculate retention
* rotate db at startup
* make query plans overlap
* Calculate properly first time s
* updated event labels
* negative page caching fix
* Atempt to create missing tables on query failure
* Atempt to create missing tables on query failure (part 2)
* negative page caching for all gaps, to eliminate jv2 scans
* Fix unittest
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
|
|
* add consul license alarm
* minor
|
|
* track memory footprint of Netdata
* track db modes alloc/ram/save/map
* track system info; track sender and receiver
* fixes
* more fixes
* track workers memory, onewayalloc memory; unify judyhs size estimation
* track replication structures and buffers
* Properly clear host RRDHOST_FLAG_METADATA_UPDATE flag
* flush the replication buffer every 1000 times the circular buffer is found empty
* dont take timestamp too frequently in sender loop
* sender buffers are not used by the same thread as the sender, so they were never recreated - fixed it
* free sender thread buffer on replication threads when replication is idle
* use the last sender flag as a timestamp of the last buffer recreation
* free cbuffer before reconnecting
* recreate cbuffer on every flush
* timings for journal v2 loading
* inlining of metric and cache functions
* aral likely/unlikely
* free left-over thread buffers
* fix NULL pointer dereference in replication
* free sender thread buffer on sender thread too
* mark ctx as used before flushing
* better logging on ctx datafiles closing
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
|
|
Remove undocumented archivedcharts endpoint. Use context endpoint instead
Remove unused functions to lookup chart and dimension UUIDs
Drop/Add new index for dimension and chart tables
|
|
* cleanup journal v2 mounts periodically
* fix for last commit
* re-enable loading page from disk when the arrangement of pages requires it
* Remove unused statistics
* Estimate diskspace when the current datafile is full and queue a rotate command (Currently it will not attempt to estimate end size for journals)
Queue a command to check quota on startup per tier
* apps.plugin now exposes RSS chart
* shorter thread names to make debugging easier, since thread names can only be 15 characters
* more thread names fixes
* allow an apps_groups.conf target to be pid 0 or 1
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
|
|
check session variable before resuming it
|
|
* query planer weight calculation using long long
* adjust replication query ahead pipeline for smaller systems
* do not generate huge replication messages
* add message to indicate replication message was interrupted
* improved message
* max replication size 25% of sender buffer
* fix for last commit
* use less cache and smaller page sizes and fewer threads on 32-bits
* fix reserved libuv workers for 32bits
* fix detection of 32/64 bit
|
|
* allow extents to be merged for as long as possible
* do not block the event loop while recalculating retention due to datafile rotation
* buffers are incrementally cleaned up, every second, by just 1 entry
* fix order of commands
* remove newline
* measure cancelled extent read requests
* count all cancelled extent requests
* do not double count failed pages
* fixed cancelled name
* Fix error and warnings when compiling with --disable-dbengine
* when the timeframe is outside retention and whole query should fail
* do not mark as failed pages that have been loaded but have been skipped
* added chart to show cache memory calculation variables
* LONG_MAX for 32-bit compatibility
* fix cache size calculation on 32-bit
* fix cache size calculation on 32-bit - use unsinged long long
* fix compilation warnings on 32-bits
* fix another compilation warning on 32-bits
* fix compilation warnings on older 32-bit compilers
* fix compilation warnings on older 32-bit compilers - more of them
* disable ML threads joining
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
|
|
* enable retries for SSL_ERROR_WANT_READ
* only when bytes is <= 0
* treat ERROR_WANT_READ/WRITE as 0 bytes
* dont close connection on zero bytes
* reuse ssl connection
* treat zero bytes
* ifdef for old openssl
* revert check
|
|
* allow running multiple evictors and flushers
* flipped aggressive and critical evictions
* dont run more than 1 evictor
* switch to batch evictions when the size of the cache is critical
* remove batching of evictions
* dedup extent load pending requests
* accounting for merged extents
* always use double linked list
* add extent merging to the overall cache hit ratio
* support requeuing merged extents to higher priorities
* fix function name
* query planner now prefers higher tiers even when they miss some data at the end, which it fills from lower tiers; adding the option "plan" to jsonwrap now renders the query plan
* update statistics after every dimension completes
* use the retention of all tiers to calculate coverage per tier
* use the original window of the query for the planner
* give 2.5% befenit for each higher tier
* update cmd->priority so that it be requeued multiple times
* merged extent pages is a cache hit
* fixed dbegnine cache hit stats
|
|
size (#14247)
* allow the cache to grow when huge queries are running that exceed the size of the cache
* queue preloading of queries
* finalize prepared queries on timeout
|
|
* count open cache pages refering to datafile
* eliminate waste flush attempts
* remove eliminated variable
* journal v2 scanning split functions
* avoid locking open cache for a long time while migrating to journal v2
* dont acquire datafile for the loop; disable thread cancelability while a query is running
* work on datafile acquiring
* work on datafile deletion
* work on datafile deletion again
* logs of dbengine should start with DBENGINE
* thread specific key for queries to check if a query finishes without a finalize
* page_uuid is not used anymore
* Cleanup judy traversal when building new v2
Remove not needed calls to metric registry
* metric is 8 bytes smaller; timestamps are protected with a spinlock; timestamps in metric are now always coherent
* disable checks for invalid time-ranges
* Remove type from page details
* report scanning time
* remove infinite loop from datafile acquire for deletion
* remove infinite loop from datafile acquire for deletion again
* trace query handles
* properly allocate array of dimensions in replication
* metrics cleanup
* metrics registry uses arrayalloc
* arrayalloc free should be protected by lock
* use array alloc in page cache
* journal v2 scanning fix
* datafile reference leaking hunding
* do not load metrics of future timestamps
* initialize reasons
* fix datafile reference leak
* do not load pages that are entirely overlapped by others
* expand metric retention atomically
* split replication logic in initialization and execution
* replication prepare ahead queries
* replication prepare ahead queries fixed
* fix replication workers accounting
* add router active queries chart
* restore accounting of pages metadata sources; cleanup replication
* dont count skipped pages as unroutable
* notes on services shutdown
* do not migrate to journal v2 too early, while it has pending dirty pages in the main cache for the specific journal file
* do not add pages we dont need to pdc
* time in range re-work to provide info about past and future matches
* finner control on the pages selected for processing; accounting of page related issues
* fix invalid reference to handle->page
* eliminate data collection handle of pg_lookup_next
* accounting for queries with gaps
* query preprocessing the same way the processing is done; cache now supports all operations on Judy
* dynamic libuv workers based on number of processors; minimum libuv workers 8; replication query init ahead uses libuv workers - reserved ones (3)
* get into pdc all matching pages from main cache and open cache; do not do v2 scan if main cache and open cache can satisfy the query
* finner gaps calculation; accounting of overlapping pages in queries
* fix gaps accounting
* move datafile deletion to worker thread
* tune libuv workers and thread stack size
* stop netdata threads gradually
* run indexing together with cache flush/evict
* more work on clean shutdown
* limit the number of pages to evict per run
* do not lock the clean queue for accesses if it is not possible at that time - the page will be moved to the back of the list during eviction
* economies on flags for smaller page footprint; cleanup and renames
* eviction moves referenced pages to the end of the queue
* use murmur hash for indexing partition
* murmur should be static
* use more indexing partitions
* revert number of partitions to number of cpus
* cancel threads first, then stop services
* revert default thread stack size
* dont execute replication requests of disconnected senders
* wait more time for services that are exiting gradually
* fixed last commit
* finer control on page selection algorithm
* default stacksize of 1MB
* fix formatting
* fix worker utilization going crazy when the number is rotating
* avoid buffer full due to replication preprocessing of requests
* support query priorities
* add count of spins in spinlock when compiled with netdata internal checks
* remove prioritization from dbengine queries; cache now uses mutexes for the queues
* hot pages are now in sections judy arrays, like dirty
* align replication queries to optimal page size
* during flushing add to clean and evict in batches
* Revert "during flushing add to clean and evict in batches"
This reverts commit 8fb2b69d068499eacea6de8291c336e5e9f197c7.
* dont lock clean while evicting pages during flushing
* Revert "dont lock clean while evicting pages during flushing"
This reverts commit d6c82b5f40aeba86fc7aead062fab1b819ba58b3.
* Revert "Revert "during flushing add to clean and evict in batches""
This reverts commit ca7a187537fb8f743992700427e13042561211ec.
* dont cross locks during flushing, for the fastest flushes possible
* low-priority queries load pages synchronously
* Revert "low-priority queries load pages synchronously"
This reverts commit 1ef2662ddcd20fe5842b856c716df134c42d1dc7.
* cache uses spinlock again
* during flushing, dont lock the clean queue at all; each item is added atomically
* do smaller eviction runs
* evict one page at a time to minimize lock contention on the clean queue
* fix eviction statistics
* fix last commit
* plain should be main cache
* event loop cleanup; evictions and flushes can now happen concurrently
* run flush and evictions from tier0 only
* remove not needed variables
* flushing open cache is not needed; flushing protection is irrelevant since flushing is global for all tiers; added protection to datafiles so that only one flusher can run per datafile at any given time
* added worker jobs in timer to find the slow part of it
* support fast eviction of pages when all_of_them is set
* revert default thread stack size
* bypass event loop for dispatching read extent commands to workers - send them directly
* Revert "bypass event loop for dispatching read extent commands to workers - send them directly"
This reverts commit 2c08bc5bab12881ae33bc73ce5dea03dfc4e1fce.
* cache work requests
* minimize memory operations during flushing; caching of extent_io_descriptors and page_descriptors
* publish flushed pages to open cache in the thread pool
* prevent eventloop requests from getting stacked in the event loop
* single threaded dbengine controller; support priorities for all queries; major cleanup and restructuring of rrdengine.c
* more rrdengine.c cleanup
* enable db rotation
* do not log when there is a filter
* do not run multiple migration to journal v2
* load all extents async
* fix wrong paste
* report opcodes waiting, works dispatched, works executing
* cleanup event loop memory every 10 minutes
* dont dispatch more work requests than the number of threads available
* use the dispatched counter instead of the executing counter to check if the worker thread pool is full
* remove UV_RUN_NOWAIT
* replication to fill the queues
* caching of extent buffers; code cleanup
* caching of pdc and pd; rework on journal v2 indexing, datafile creation, database rotation
* single transaction wal
* synchronous flushing
* first cancel the threads, then signal them to exit
* caching of rrdeng query handles; added priority to query target; health is now low prio
* add priority to the missing points; do not allow critical priority in queries
* offload query preparation and routing to libuv thread pool
* updated timing charts for the offloaded query preparation
* caching of WALs
* accounting for struct caches (buffers); do not load extents with invalid sizes
* protection against memory booming during replication due to the optimal alignment of pages; sender thread buffer is now also reset when the circular buffer is reset
* also check if the expanded before is not the chart later updated time
* also check if the expanded before is not after the wall clock time of when the query started
* Remove unused variable
* replication to queue less queries; cleanup of internal fatals
* Mark dimension to be updated async
* caching of extent_page_details_list (epdl) and datafile_extent_offset_list (deol)
* disable pgc stress test, under an ifdef
* disable mrg stress test under an ifdef
* Mark chart and host labels, host info for async check and store in the database
* dictionary items use arrayalloc
* cache section pages structure is allocated with arrayalloc
* Add function to wakeup the aclk query threads and check for exit
Register function to be called during shutdown after signaling the service to exit
* parallel preparation of all dimensions of queries
* be more sensitive to enable streaming after replication
* atomically finish chart replication
* fix last commit
* fix last commit again
* fix last commit again again
* fix last commit again again again
* unify the normalization of retention calculation for collected charts; do not enable streaming if more than 60 points are to be transferred; eliminate an allocation during replication
* do not cancel start streaming; use high priority queries when we have locked chart data collection
* prevent starvation on opcodes execution, by allowing 2% of the requests to be re-ordered
* opcode now uses 2 spinlocks one for the caching of allocations and one for the waiting queue
* Remove check locks and NETDATA_VERIFY_LOCKS as it is not needed anymore
* Fix bad memory allocation / cleanup
* Cleanup ACLK sync initialization (part 1)
* Don't update metric registry during shutdown (part 1)
* Prevent crash when dashboard is refreshed and host goes away
* Mark ctx that is shutting down.
Test not adding flushed pages to open cache as hot if we are shutting down
* make ML work
* Fix compile without NETDATA_INTERNAL_CHECKS
* shutdown each ctx independently
* fix completion of quiesce
* do not update shared ML charts
* Create ML charts on child hosts.
When a parent runs a ML for a child, the relevant-ML charts
should be created on the child host. These charts should use
the parent's hostname to differentiate multiple parents that might
run ML for a child.
The only exception to this rule is the training/prediction resource
usage charts. These are created on the localhost of the parent host,
because they provide information specific to said host.
* check new ml code
* first save the database, then free all memory
* dbengine prep exit before freeing all memory; fixed deadlock in cache hot to dirty; added missing check to query engine about metrics without any data in the db
* Cleanup metadata thread (part 2)
* increase refcount before dispatching prep command
* Do not try to stop anomaly detection threads twice.
A separate function call has been added to stop anomaly detection threads.
This commit removes the left over function calls that were made
internally when a host was being created/destroyed.
* Remove allocations when smoothing samples buffer
The number of dims per sample is always 1, ie. we are training and
predicting only individual dimensions.
* set the orphan flag when loading archived hosts
* track worker dispatch callbacks and threadpool worker init
* make ML threads joinable; mark ctx having flushing in progress as early as possible
* fix allocation counter
* Cleanup metadata thread (part 3)
* Cleanup metadata thread (part 4)
* Skip metadata host scan when running unittest
* unittest support during init
* dont use all the libuv threads for queries
* break an infinite loop when sleep_usec() is interrupted
* ml prediction is a collector for several charts
* sleep_usec() now makes sure it will never loop if it passes the time expected; sleep_usec() now uses nanosleep() because clock_nanosleep() misses signals on netdata exit
* worker_unregister() in netdata threads cleanup
* moved pdc/epdl/deol/extent_buffer related code to pdc.c and pdc.h
* fixed ML issues
* removed engine2 directory
* added dbengine2 files in CMakeLists.txt
* move query plan data to query target, so that they can be exposed by in jsonwrap
* uniform definition of query plan according to the other query target members
* event_loop should be in daemon, not libnetdata
* metric_retention_by_uuid() is now part of the storage engine abstraction
* unify time_t variables to have the suffix _s (meaning: seconds)
* old dbengine statistics become "dbengine io"
* do not enable ML resource usage charts by default
* unify ml chart families, plugins and modules
* cleanup query plans from query target
* cleanup all extent buffers
* added debug info for rrddim slot to time
* rrddim now does proper gap management
* full rewrite of the mem modes
* use library functions for madvise
* use CHECKSUM_SZ for the checksum size
* fix coverity warning about the impossible case of returning a page that is entirely in the past of the query
* fix dbengine shutdown
* keep the old datafile lock until a new datafile has been created, to avoid creating multiple datafiles concurrently
* fine tune cache evictions
* dont initialize health if the health service is not running - prevent crash on shutdown while children get connected
* rename AS threads to ACLK[hostname]
* prevent re-use of uninitialized memory in queries
* use JulyL instead of JudyL for PDC operations - to test it first
* add also JulyL files
* fix July memory accounting
* disable July for PDC (use Judy)
* use the function to remove datafiles from linked list
* fix july and event_loop
* add july to libnetdata subdirs
* rename time_t variables that end in _t to end in _s
* replicate when there is a gap at the beginning of the replication period
* reset postponing of sender connections when a receiver is connected
* Adjust update every properly
* fix replication infinite loop due to last change
* packed enums in rrd.h and cleanup of obsolete rrd structure members
* prevent deadlock in replication: replication_recalculate_buffer_used_ratio_unsafe() deadlocking with replication_sender_delete_pending_requests()
* void unused variable
* void unused variables
* fix indentation
* entries_by_time calculation in VD was wrong; restored internal checks for checking future timestamps
* macros to caclulate page entries by time and size
* prevent statsd cleanup crash on exit
* cleanup health thread related variables
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
Co-authored-by: vkalintiris <vasilis@netdata.cloud>
|
|
|
|
|
|
(#14073)
|
|
|
|
|
|
|
|
* move global statistics workers to a separate thread; query statistics per query source; query statistics for ML, exporters, backfilling; reset replication point in time every 10 seconds, instead of every 1; fix compilation warnings; optimize the replication queries code; prevent long tail of replication requests (big sleeps); provide query statistics about replication ; optimize replication sender when most senders are full; optimize replication_request_get_first_available(); reset replication completion calculation;
* remove workers utilization from global statistics thread
|
|
|
|
remove retries
|
|
* streaming compression, query planner and replication fixes
* remove journal v2 stats from global statistics
* disable sql for checking past sql UUIDs
* single threaded replication
* final replication thread using dictionaries and JudyL for sorting the pending requests
* do not timeout the sending socket when there are pending replication requests
* streaming receiver using read() instead of fread()
* remove FILE * from streaming - now using posix read() and write()
* increase timeouts to 10 minutes
* apply sender timeout only when there are metrics that are supposed to be streamed
* error handling in replication
* remove retries on socket read timeout; better error messages
* take into account inbound traffic too to detect that a connection is stale
* remove race conditions from replication thread
* make sure deleted entries are marked as executed, so that even if deletion fails, they will not be executed
* 2 minutes timeout to retry streaming to a parent that already has this node
* remove unecessary condition check
* fix compilation warnings
* include judy in replication
* wrappers to handle retries for SSL_read and SSL_write
* compressed bytes read monitoring
* recursive locks on replication to make it faster during flush or cleanup
* replication completion chart at the receiver side
* simplified recursive mutex
* simplified recursive mutex again
|
|
Revert "New journal disk based indexing for agent memory reduction (#13885)"
This reverts commit 224b051a2b2bab39a4b536e531ab9ca590bf31bb.
|
|
* Add read only option to netdata_mmap so files are accessed ousing PROT_READ
* Initial functions to write the new journal file and switch to the new indexing
* Cleanup code, add parameters to pg_cache_punch_hole to avoid updating page latets oldest times
pg_cache insert to have parameter if page index locked needs to be done
Page eviction functions will try to deallocate the descriptor as well (pg_cache_punch_hole without page_index time updates)
Cleanup messages during startup
* Cleanup messages during startup
* Disbale extent caching for now, add placeholder for journal indexing and activation while the agent is running
* Add main function to populate descriptors by checking the new journal indexing
* prevent crash
* fix for binary search crash
* Avoid Time-of-check time-of-use filesystem race condition
* always add a page
* populate fixes - it is still incomplete
* pg_cache_insert returns the descriptor that ends up in the page_index
* Add populate next (Fix 1)
* Fix compilation warnings, reactivate extent caching
* Add populate next (Fix 2)
* Add populate next (Fix 3) switch to the next entry or journal file when asking to populate descriptor with next
* Fix resource leak and wrong sizeof
* Rework page population (part 1)
* Additional checksums added / journal validation
* Cleanup (part 1)
* Locking added and Cleanup (part 2)
* Close journal file after new journal index activation
* Skip warning when compiling without NETDATA_INTERNAL_CHECKS
* Ignore empty index file (header and trailer and no metrics)
* Try to remove all evicted descriptors (may prevent slight memory increase)
* Evict pages also when we succesfully do try_reserve
* Precache pages and cleanup
* Add a separate cleanup thread to release unused descriptors
* Check existence of key correctly
* Fix total file size calculation
* Statistics for journal descriptors
* Track and release jourval v2 descriptors
* Do not try to allocate pages for locality if under pressure
* Do not track v2 descriptors when populating the page_index
* Track page descriptors as they are inserted in the page index (per journal file)
Scan journal files for pending items to cleanup
Cleanup v2 descriptors only if they are not populated
Check before adding to page cache to avoid memory allocation /free
* Close journal file that has been processed and migrated to the new index
Check for valid file before trying to truncate / close. This file has been closed during startup
* Better calculation for the number of prefetched data pages based on the query end time
Code cleanup and comments
Add v2 populated descriptor expiration based on journal access time
* Code cleanup
* Faster indexing
Better journal validation (more sanity checks)
Detect new datafile/ journal creation and trigger index generation
Switch to the new index / mark descriptors in memory as needed
Update journal access time when a descriptor is returned
Code cleanup (part 1)
* Re activate descriptor clean
Code cleanup
* Allow locality precaching
* Allow locality precaching for the same page alignment
* Descriptor cleanup internal changed
* Disable locality precaching
* Precache only if not under pressure / internal cleanup at 60 seconds
* Remove unused functions
* Migrate on startup always
Make sure the metric uuid is valid (we have a page_index)
Prevent crash if no datafile is available when logging an error
Remove unused functions
* New warn limit for precaching
Stress test v2 descriptor cleanup
- Every 1s cleanup if it doesnt exist in cache
- 60s cache eviction
* Arrayalloc internal checks on free activated with NETDATA_ARRAYALLOC_INTERNAL_CHECKS
Ability to add DESCRIPTOR_EXPIRATION_TIME and DESCRIPTOR_INTERVAL_CLEANUP during compile
Defaults DESCRIPTOR_INTERVAL_CLEANUP = 60 and DESCRIPTOR_EXPIRATION_TIME = 600
* Lookup page index correctly
* Calculate index time once
* Detect a duplicate page when doing cache insert and during flushing of pages
* Better logging
* Descriptor validation (extent vs page index) when building an index file while the agent is running
* Mark invalid entries in the journal v2 file
* Schedule an index rebuild if a descriptor is found without an extent in the timerange we are processing
Release descriptor lock to prevent random shutdown locks
* Proper unlock
* Skip descriptor cleanup when journal file v2 migration is running
* Fix page cache statistics
Remove multiple entries of the page_index from the page cache
Cleanup
* Adjust preload pages on pg_cache_next. Handle invalid descriptor properly
Unlock properly
* Better handling of invalid pages
Journal indexing during runtime will scan all files to find potential ones to index
* Reactivate migration on startup
Evict descriptors to cause migration
Don't count the entries in page index (calculate when processing the extent list)
Check for valid extent since we may set the extent to NULL on startup if it is invalid
Better structure init
Address valgrind issues
* Add don't fork/dump option
* Add separate lock to protect accessing a datafile's extent list
Comment out some unused code (for now)
Abort descriptor cleanup if we are force flushing pages (page cache under pressure)
* Check for index and schedule when data flush completes
Configure max datafile size during compilation
Keep a separate JudyL array for descriptors
Skip quota test if we are deleting descriptors or explicitly flushing pages under pressure
* Fix
* set function when waiters are waken up
* add the line number to trace the deadlock
* add thread id
* add wait list
* init to zero
* disable thread cancelability inside dbengine rrdeng_load_page_next()
* make sure the owner is the thread
* disable thread cancelability for replication as a whole
* Check and queue indexing after first page flush
* Queue indexing after a small delay to allow some time for page flushing
* tracing of waiters only when compiled with internal checks
* Mark descr with extent_entry
* Return page timeout
* Check if a journalfile is ready to be indexed
Migrate the descriptors or evict if possible
Compilation warning fix
* Use page index if indexing during startup
Mark if journalfile should be checked depending on whether we can migrate or delete a page during indexing
* require 3x max message size as sender buffer
* fix for the msg of the adaptive buffer size
* fix for the msg of the duplicate replication commands
* Disable descriptor deletion during migration
* Detect descriptor with same start page time
* sender sorts replication requests before fullfilling them; receiver does not send duplicate replication requests
* dbengine never allows past timestamps to be collected
* do not accept values same as last data point stored in dbengine
* replicate non-overlapping ranges
* a better replication logic to avoid sending overlapping data to parents
* Do not start journal migration in parallel
* Always update page index times
* Fix page index first / last times on load
* internal log when replication responses do not match the requests or when replication commands are sent while there are others inflight
* do not log out of bounds RBEGIN if it is the last replication command we sent
* better checking of past data collection points
* better checking of past data collection points - optimized
* fix corruption during decompression of streaming
* Add config to disable journal indexing
Add config parameter for detailed journal integrity check (Metric chain validation check during startup)
pg cache insert drop check for existing page
Fix crc calculation for metric headers
* children disable compression globally, only when the compression gives an error
* turn boolean member into RRDHOST OPTION
* Compilation warnings
* Remove unused code
* replication sender statistics
* replication sender statistics set to 100% when no replication requests are pending
* Fix casting warning
Co-authored-by: Costa Tsaousis <costa@netdata.cloud>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
fix https://github.com/netdata/netdata/issues/13918
|
|
Co-authored-by: Shyam Sreevalsan <shyam@netdata.cloud>
|
|
|
|
* use print macros
* cast instead
|
|
|
|
|
|
* initial implementation of QUERY_TARGET
* rrd2rrdr() interface
* rrddim_find_best_tier_for_timeframe() ported
* added dimension filtering
* added db object in query target
* rrd2rrdr() ported
* working on formatters
* working on jsonwrapper
* finally, it compiles...
* 1st run without crashes
* query planer working
* cleanup old code
* review changes
* fix also changing data collection frequency
* fix signess
* fix rrdlabels and dimension ordering
* fixes
* remove unused variable
* ml should accept NULL response from rrd2rrdr()
* number formatting fixes
* more number formatting fixes
* more number formatting fixes
* support mc parallel queries
* formatting and cleanup
* added rrd2rrdr_legacy() as a simplified interface to run a query
* make sure rrdset_find_natural_update_every_for_timeframe() returns a value
* make signed comparisons
* weights endpoint using rrdcontexts
* fix for legacy db modes and cleanup
* fix for chart_ids and remove AR chart from weights endpoint
* Ignore command if not initialized yet
* remove unused members
* properly initialize window
* code cleanup - rrddim linked list is gone; rrdset rwlock is gone too
* reviewed RRDR.internal members
* eliminate unnecessary members of QUERY_TARGET
* more complete query ids; more detailed information on aborted queries
* properly terminate option strings
* query id contains group_options which is controlled by users, so escaping is necessary
* tense in query id
* tense in query id - again
* added the remaining query options to the query id
* Expose hidden option to the dimension
* use the hidden flag when loading context dimensions
* Specify table alias for option
* dont update chart last access time, unless at least a dimension of the chart will be queried
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
|
|
add ACLK access to ml_info
|
|
Add pandas collector.
|
|
* dbengine free from RRDSET and RRDDIM
* fix for excess parameters to query ops
* add comment about ML
* update_every from int to uint32_t
* rrddim_mem storage engine working
* fixes for update_every_s
* working dbengine
* a lot of changes in dbengine regarding timestamps
* better logging of not sequential points
* rrdset_done() now gives aligned timestamps for higher tiers
* dont change the end_time of descriptors, because they cant be loaded back
* fixes for cmake
* fixes for db mode ram
* Global counters for dbengine loading errors.
Ensure dbengine store metrics always has aligned metrics or breaks the page when storing new data.
* update lgtm config
* fixes for 32-bit systems
* update unittests
* Don't try to find and create a host on the fly if not already in memory
* Remove unused functions
* print backtrace in case of fatal
* always set ctx to page_index
* detect ctx and metric uuid discrepancies
* use legacy uuid if multihost is not available
* fix for last commit
* prevent repeating log
* Do not try to access archived charts when executing a data query
* Remove unused function
* log inconsistent collections once every 10 mins
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
|
|
(#13810)
* overload libc memory allocators with custom ones to trace all allocations
* grab libc pointers for external c plugins
* use -ldl when necessary; fallback to work without dlsym when it is not available
* initialize global variable
* add optional dl libs
* dynamically link every library function when needed for the first time
* prevent crashes on musl libc
* another attempt
* dont dereference function
* attempt no 3
* attempt no 4
* cleanup - all attempts failed
* dont enable tracing of allocations
* missing parenthesis
|
|
* full memory tracking and profiling of Netdata Agent
* initialize dbengine only when it is needed
* handling of dbengine compiled but not available
* restore unittest
* restore unittest again
* more improvements about ifdef dbengine
* fix compilation when dbengine is not enabled
* check if dbengine is enabled on exit
* call freez() not free()
* aral unittest
* internal checks activate trace allocations; dev mode activates internal checks
|
|
By default functions are declared as extern in C/C++ headers. The goal
of this PR is to reduce the wall of text that many headers have and,
more importantly, to make the declaration of extern'd variables - of
which we have many dispersed in various places - easily and quickly
identifiable.
Automatically generated with:
$ git grep -l '^extern.*(' '**.h' | \
grep -v libjudy | \
grep -v 'sqlite3.h' | \
xargs sed -i -e 's/extern \(.*(.*$\)/\1/'
This is a NFC.
|
|
* fix bad merge
* fix for ssl_security_cert
|
