| Age | Commit message (Collapse) | Author |
|---|
|
* additional check for consistency of rrdr
* do not query the host if the contexts are not initialized for it
|
|
* query timestamps are now pre-determined and alignment on timestamps is guarranteed
* turn internal_fatal() to internal_error() to investigate the issue
* handle query when no data exist in the db
* check for non NULL dict when running dictionary garbage collect
* support API v2 requests via ACLK
* add nodes detailed information to /api/v2/nodes
* fixed keys and added dummy nodes for completeness
* added nodes_hard_hash, alerts_hard_hash, alerts_soft_hash; started building a nodes status object to reflect the current status of a node
* make sure replication does not double count charts that are already being replicated
* expose min and max in sts structures
* added view_minimum_value and view_maximum_value; percentage calculation is now an additional pass on the data, removed from formatters; absolute value calculation is now done at the query level, removed from formatters
* respect trimming in percentage calculation; updated swagger
* api/v2/weights preparative work to support multi-node queries - still single node though
* multi-node /api/v2/weights endpoint, supporting all the filtering parameters of /api/v2/data
* when passing the raw option, the query exposes the hidden dimensions
* fix compilation issues on older systems
* the query engine now calculates per dimension min, max, sum, count, anomaly count
* use the macro to calculate storage point anomaly rate
* weights endpoint exposing version hashes
* weights method=value shows min, max, average, sum, count, anomaly count, anomaly rate
* query: expose RESET flag; do not add the same point multiple times to the aggregated point
* weights: more compact output
* weights requests can be interrupted
* all /api/v2 requests can be interrupted and timeout
* allow relative timestamps in weights
* fix macos compilation warnings
* Revert "fix macos compilation warnings"
This reverts commit 8a1d24e41e9b58de566ac59f0c4b1c465bcc0592.
* /api/v2/data group-by now works on dimension names, not ids
* /api/v2/weights does not query metrics without retention and new output format
* /api/v2/weights value and anomaly queries do context queries when contexts are filtered; query timeout is now always in ms
|
|
Co-authored-by: netdatabot <netdatabot@users.noreply.github.com>
|
|
* Update README.md
* Add info on the cloud API
* Update web/api/README.md
Co-authored-by: Tasos Katsoulas <12612986+tkatsoulas@users.noreply.github.com>
---------
Co-authored-by: Tasos Katsoulas <12612986+tkatsoulas@users.noreply.github.com>
|
|
accept all=true for alarms api v1
|
|
Co-authored-by: netdatabot <netdatabot@users.noreply.github.com>
|
|
* expose the order of group by
* key renames in json wrapper v2
* added group by context and group by units
* added view_average_values
* fix for view_average_values when percentage is specified
* option group-by-labels is enabling the exposure of all the labels that are used for each of the final grouped dimensions
* when executing group by queries, allocate one dimension data at a time - not all of them
* respect hidden dimensions
* cancel running data query on socket error
* use poll to detect socket errors
* use POLLRDHUP to detect half closed connections
* make sure POLLRDHUP is available
* do not destroy aral-by-size arals
* completed documentation of /api/v2/data.
* moved min, max back to view; updated swagger yaml and json
* default format for /api/v2/data is json2
|
|
* max web request size to 64KB
* fix the request too big message
* increase max request reading tries to 100
* support for bigger web requests
* add "avg" as a shortcut for "average" to both group by aggregation and time aggregation; discard the last partial points of a query in play mode, up to max update every; group by hidden dimensions too
* better implementation for partial data trimming
* added group_by=selected to return only one dimension for all selected metrics
* fix acceptance of group_by=selected
* passing option "raw" disables partial data trimming
* remove obsolete option "plan"; use "debug"
* fix view.min and view.max calculation - there were 2 bugs: a) min and max were reset for every row and b) min and max were corrupted by GBC and AR printing
* per row annotations
* added time column to point annotations
* disable caching for /api/v2/contexts responses
* added api format json2 that returns an array for each points, having all the point values and annotations in them
* work on swagger about /api/v2
* prevent infinite loop
* cleanup and swagger work
* allow negative simple pattern expressions to work as expected
* do not lookup in the dictionary empty names
* garbage collect dictionaries
* make query_target allocate less aggressively; queries fill the remaining points with nulls
* reusable query ops to save memory on huge queries
* move parts of query plans into query ops to save query target memory
* remove storage engine from query metric tiers, to save memory, and recalculate it when it is needed
|
|
* Refactor ML code.
This commit introduces only non-functional changes.
Originally, the C++ code exposed C functions to be called
from the rest of the agent. When we migrated from C++ to C,
we did not eliminate these wrapper functions to make the
PR easier to understand and keep the total LOC low.
This commit removes the wrapper functions and "reclaims" the
`ml_` prefix that we used for the public API of the old
implementation.
Also, the nlohmann Json library has been removed and its
functionality was replaced with the equivalent Json functionality
that we added in libnetdata's BUFFERs.
* Remove missing headers from build systems.
* Fix CMake build.
* rrddim_free is outside of rrd "internals" now.
|
|
|
|
* preparation for /api/v2/contexts
* working /api/v2/contexts
* add anomaly rate information in all statistics; when sum-count is requested, return sums and counts instead of averages
* minor fix
* query targegt now accurately counts hosts, contexts, instances, dimensions, metrics
* cleanup /api/v2/contexts
* full text search with /api/v2/contexts
* simple patterns now support the option to search ignoring case
* full text search API with /api/v2/q
* simple pattern execution optimization
* do not show q when not given
* full text search accounting
* separated /api/v2/nodes from /api/v2/contexts
* fix ssv queries for group_by
* count query instances queried and failed per context and host
* split rrdcontext.c to multiple files
* add query totals
* fix anomaly rate calculation; provide "ni" for indexing hosts
* do not generate zero valued members
* faster calculation of anomaly rate; by just summing integers for each db points and doing math once for every generated point
* fix typo when printing dimensions totals
* added option minify to remove spaces and newlines fron JSON output
* send instance ids and names when they differ
* do not add in query target dimensions, instances, contexts and hosts for which there is no retention in the current timeframe
* fix for the previous + renames and code cleanup
* when a dimension is filtered, include in the response all the other dimensions that are selectable
* do not add nodes that do not have retention in the current window
* move selection of dimensions to query_dimension_add(), instead of query_metric_add()
* increase the pre-processing capacity of queries
* generate instance fqdn ids and names only when they are needed
* provide detailed statistics about tiers retention, queries, points, update_every
* late allocation of query dimensions
* cleanup
* more cleanup
* support for annotations per displayed point, RESET and PARTIAL
* new type annotations
* if a chart is not linked to contexts and it is collected, link it when it is collected
* make ML run reentrant
* make ML rrdr query synchronous
* optimize replication memory allocation of replication_sort_entry
* change units to percentage, when requesting a coefficinet of variation, or a percentage query
* initialize replication before starting main threads
* properly decrement no room requests counter
* propagate the non-zero flag to group-by
* the same by avoiding the extra loop
* respect non-zero in all dimension arrays
* remove dictionary garbage collection from dictionary_entries() and dictionary_version()
* be more verbose when jv2 indexing is postponed
* prevent infinite loop
* use hidden dimensions even when dimensions pattern is unset
* traverse hosts using dictionaries
* fix dictionary unittests
|
|
* go to normal only if its in stream mode
* use ssl specific flags for wait receive and send
* remove brackets
|
|
* make the title metadta the H1
* Update collectors/python.d.plugin/zscores/README.md
* Update libnetdata/ebpf/README.md
* Update ml/README.md
* Update libnetdata/string/README.md
---------
Co-authored-by: Chris Akritidis <43294513+cakrit@users.noreply.github.com>
|
|
* reorg batch 1
* remove duplicate cloud custom dashboard and agent dashboard
* Simplify the root web/README
* Merge streaming references
* Make enable streaming the overall intro and the README the reference
* Remove reference-streaming document
* Update overview pages
|
|
|
|
|
|
* fundamentals for having /api/v2/ working
* use an atomic to prevent writing to internal pipe too much
* first attempt of multi-node, multi-context, multi-chart, multi-dimension queries
* v2 jsonwrap
* first attempt for group by
* cleaned up RRDR and fixed group by
* improvements to /api/v2/api
* query instance may be realloced, so pointers to it get invalid; solved memory leaks
* count of quried metrics in summary information
* provide detailed information about selected, excluded, queried and failed metrics for each entity
* select instances by fqdn too
* add timing information to json output
* link charts to rrdcontexts, if a query comes in and it is found unlinked
* calculate min, max, sum, average, volume, count per metric
* api v2 parameters naming
* renders alerts and units
* render machine_guid and node_id in all sections it is relevant
* unified keys
* group by now takes into account units and when there are multiple units involved, it creates a dimension per unit
* request and detailed are hidden behind an option
* summary includes only a flattened list of alerts
* alert counts per host and instance
* count of grouped metrics per dimension
* added contexts to summary
* added chart title
* added dimension priorities and chart type
* support for multiple group by at the same time
* minor fixes
* labels are now a tree
* keys uniformity
* filtering by alerts, both having a specific alert and having a specific alert in a specific status
* added scope of hosts and contexts
* count of instances on contexts and hosts
* make the api return valid responses even when the response contains no data
* calculate average and contribution % for every item in the summary
* fix compilation warnings
* fix compilation warnings - again
|
|
* set to wait receive/send when ssl returns wait read/write
* compare the bytes
* set to normal to prevent going into stream mode with incomplete request
* disable wait send
|
|
* Fix coverity 383236: Resource leak
* Fix coverity 382915 : Logically dead code
* Fix coverity 379133 : Division or modulo by float zero
* Fix coverity 382783 : Copy into fixed size buffer
* Fix coverity 381151 : Missing unlock
* Fix coverity 381903 : Dereference after null check
|
|
|
|
|
|
* sort per reachable in v0/info. re-index per hostname
* remove whitespace
* keep the new host if the old is archived or orphan
|
|
optimization (#14493)
* first work on standardizing json formatting
* renamed old grouping to time_grouping and added group_by
* add dummy functions to enable compilation
* buffer json api work
* jsonwrap opening with buffer_json_X() functions
* cleanup
* storage for quotes
* optimize buffer printing for both numbers and strings
* removed ; from define
* contexts json generation using the new json functions
* fix buffer overflow at unit test
* weights endpoint using new json api
* fixes to weights endpoint
* check buffer overflow on all buffer functions
* do synchronous queries for weights
* buffer_flush() now resets json state too
* content type typedef
* print double values that are above the max 64-bit value
* str2ndd() can now parse values above UINT64_MAX
* faster number parsing by avoiding double calculations as much as possible
* faster number parsing
* faster hex parsing
* accurate printing and parsing of double values, even for very large numbers that cannot fit in 64bit integers
* full printing and parsing without using library functions - and related unit tests
* added IEEE754 streaming capability to enable streaming of double values in hex
* streaming and replication to transfer all values in hex
* use our own str2ndd for set2
* remove subnormal check from ieee
* base64 encoding for numbers, instead of hex
* when increasing double precision, also make sure the fractional number printed is aligned to the wanted precision
* str2ndd_encoded() parses all encoding formats, including integers
* prevent uninitialized use
* /api/v1/info using the new json API
* Fix error when compiling with --disable-ml
* Remove redundant 'buffer_unittest' declaration
* Fix formatting
* Fix formatting
* Fix formatting
* fix buffer unit test
* apps.plugin using the new JSON API
* make sure the metrics registry does not accept negative timestamps
* do not allow pages with negative timestamps to be loaded from db files; do not accept pages with negative timestamps in the cache
* Fix more formatting
---------
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
|
|
Update README.md
|
|
* Reorge exporter integrations
* Reorg alert notifications.
change mdx to md for alert related files
* Move all .mdx to .md, including links.
|
|
* fix broken links in claim/README.md
* delete broken link in docs/guidelines.md
* fix broken links
* fix broken link
* fix broken links
* fix broken links
* fix broken links
* fix broken links
* fix broken links
* remove broken link
* fix broken link
* fix broken links
* fix broken links
* fix broken links
* fix broken link
* fix linking phrasing
* fix broken links batch
* fix broken links second batch
* fix broken links
* fix broken links
* fix broken links
* Update COLLECTORS.md
* fix broken links
* fix broken links
|
|
|
|
|
|
* add metadata for learn
* first batch of adding metadata to md files
* second batch of adding metadata to md files
* third batch of adding metadata to md files
* test one sidebar_label
* add missing sidebar_labels
* add missing sidebar_labels to files left behind
* test, ansible doc is stubborn
* fix
* fix
* fix
* don't use questionmarks in the sidebar label
* don't use exclamation marks and symbols in the sidebar label
* fix style guide
* fixes
* fixes
|
|
* do not report dimensions that failed to be queried
* renamed SELECTED to QUERIED to have clarity on what it means
* fix wrong placement of continue
|
|
|
|
Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>
|
|
* parallel initialization of tiers
* do not spawn multiple dbengine event loops
* user configurable dbengine parallel initialization
* size netdata based on the real cpu cores available on the system netdata runs, not on the system monitored
* user configurable system cpus
* move cpuset parsing to os.c/.h
* fix replication of misaligned chart dimensions
* give a different path to each tier thread
* statically allocate the path into the initialization structure
* use aral for reusing dbengine pages
* dictionaries uses ARAL for fixed sized values
* fix compilation without internal checks
* journal v2 index uses aral
* test to see judy allocations
* judy allocations using aral
* Add config option to select if dbengine will use direct I/O (default is yes)
* V1 journafiles will use uv_fs_read instead of mmap (respect the direct I/O setting)
* Remove sqlite3IsMemdb as it is unused
* Fix compilation error when --disable-dbengine is used
* use aral for dbengine work_cmds
* changed aral API to support new features
* pgc and mrg aral overheads
* rrdeng opcodes using aral
* better structuring and naming
* dbegnine query handles using aral
* page descriptors using aral
* remove obsolete linking
* extent io descriptors using aral
* aral keeps one last page alive
* add missing return value
* added judy aral overhead
* pdc now uses aral
* page_details now use aral
* epdl and deol using aral - make sure ARALs are initialized before spawning the event loop
* remove unused linking
* pgc now uses one aral per partition
* aral measure maximum allocation queue
* aral to allocate pages in parallel
* aral parallel pages allocation when needed
* aral cleanup
* track page allocation and page population separately
---------
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
|
|
remove equality
|
|
* acquiring / releasing interface for metrics
* metrics registry statistics
* cleanup metrics registry by deleting metrics when they dont have retention anymore; do not double copy the data of pages to be flushed
* print the tier in retention summary
* Open files with buffered instead of direct I/O (test)
* added more metrics stats and fixed unittest
* rename writer functions to avoid confusion with refcounting
* do not release a metric that is not acquired
* Revert to use direct I/O on write -- use direct I/O on read as well
* keep track of ARAL overhead and add it to the memory chart
* aral full check via api
* Cleanup
* give names to ARALs and PGCs
* aral improvements
* restore query expansion to the future
* prefer higher resolution tier when switching plans
* added extent read statistics
* smoother joining of tiers at query engine
* fine tune aral max allocation size
* aral restructuring to hide its internals from the rest of netdata
* aral restructuring; addtion of defrag option to aral to keep the linked list sorted - enabled by default to test it
* fully async aral
* some statistics and cleanup
* fix infinite loop while calculating retention
* aral docs and defragmenting disabled by default
* fix bug and add optimization when defragmenter is not enabled
* aral stress test
* aral speed report and documentation
* added internal checks that all pages are full
* improve internal log about metrics deletion
* metrics registry uses one aral per partition
* metrics registry aral max size to 512 elements per page
* remove data_structures/README.md dependency
---------
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
|
|
* replication cancels pending queries on exit
* log when waiting for inflight queries
* when there are collected and not-collected metrics, use the context priority from the collected only
* Write metadata with a faster pace
* Remove journal file size limit and sync mode to 0 / Drop wal checkpoint for now
* Wrap in a big transaction remaining metadata writes (test 1)
* fix higher tiers when tiering iterations = 2
* dbengine always returns db-aligned points; query engine expands the queries by 2 points in every direction to have enough data for interpolation
* Wrap in a big transaction metadata writes (test 2)
* replication cancelling fix
* do not first and last entry in replication when the db has no retention
* fix internal check condition
* Increase metadata write batch size
* always apply error limit to dbengine logs
* Remove code that processes the obsolete health.db files
* cleanup in query.c
* do not allow queries to go beyond db boundaries
* prevent internal log for +1 delta in timestamp
* detect gap pages in conflicts
* double protection for gap injection in main cache
* Add checkpoint to prevent large WAL while running
Remove unused and duplicate functions
* do not allocate chart cache dir if not needed
* add more info to unittests
* revert query expansion to satisfy unittests
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
|
|
|
|
Revert "Misc SSL improvements (#14317)"
This reverts commit f4d017ccf7385df3e152eeb5ea1259702ab01043.
|
|
* Remove old documentation check related config files.
These are no longer used by any of our CI as far as I can tell.
* Purge LGTM configuration and references.
It has been fully shut down, so none of this works anymore.
* Purge Travis CI config, scripts, and references.
We are no longer using Travis CI, so all of this is useless.
* Pureg config for other CI tools we are no longer using.
* Remove old packaging related test scripts.
These haven’t been used in years, and are potentially confusing for new
contributors.
* Restore MLC configuration.
It is, in fact, still in use.
* Fix bogus CI config in dist files.
* Fix botched merge in CODEOWNERS.
|
|
* Moving the cloud docs under /docs/cloud (previous location: netdata/learn/*)
* Added metadata on almost every document of the old learn site for the new ingest process of learn.
* Map old learn document to their best fit as topic related docs.
Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>
Co-authored-by: DShreve2 <david@netdata.cloud>
Co-authored-by: hugovalente-pm <hugo@netdata.cloud>
|
|
* set web client to poll when ssl error want read or write
* turn to function
|
|
* cache 100 pages for each size our tiers need
* smarter page caching
* account the caching structures
* dynamic max number of cached pages
* make variables const to ensure they are not changed
* make sure replication timestamps do not go to the future
* replication now sends chart and dimension states atomically; replication receivers ignores chart and dimension states when rbegin is also ignored
* make sure all pages are flushed on shutdown
* take into account empty points too
* when recalculating retention update first_time_s on metrics only when they are bigger
* Report the datafile number we use to recalculate retention
* Report the datafile number we use to recalculate retention
* rotate db at startup
* make query plans overlap
* Calculate properly first time s
* updated event labels
* negative page caching fix
* Atempt to create missing tables on query failure
* Atempt to create missing tables on query failure (part 2)
* negative page caching for all gaps, to eliminate jv2 scans
* Fix unittest
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
|
|
* add consul license alarm
* minor
|
|
* track memory footprint of Netdata
* track db modes alloc/ram/save/map
* track system info; track sender and receiver
* fixes
* more fixes
* track workers memory, onewayalloc memory; unify judyhs size estimation
* track replication structures and buffers
* Properly clear host RRDHOST_FLAG_METADATA_UPDATE flag
* flush the replication buffer every 1000 times the circular buffer is found empty
* dont take timestamp too frequently in sender loop
* sender buffers are not used by the same thread as the sender, so they were never recreated - fixed it
* free sender thread buffer on replication threads when replication is idle
* use the last sender flag as a timestamp of the last buffer recreation
* free cbuffer before reconnecting
* recreate cbuffer on every flush
* timings for journal v2 loading
* inlining of metric and cache functions
* aral likely/unlikely
* free left-over thread buffers
* fix NULL pointer dereference in replication
* free sender thread buffer on sender thread too
* mark ctx as used before flushing
* better logging on ctx datafiles closing
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
|
|
Remove undocumented archivedcharts endpoint. Use context endpoint instead
Remove unused functions to lookup chart and dimension UUIDs
Drop/Add new index for dimension and chart tables
|
|
* cleanup journal v2 mounts periodically
* fix for last commit
* re-enable loading page from disk when the arrangement of pages requires it
* Remove unused statistics
* Estimate diskspace when the current datafile is full and queue a rotate command (Currently it will not attempt to estimate end size for journals)
Queue a command to check quota on startup per tier
* apps.plugin now exposes RSS chart
* shorter thread names to make debugging easier, since thread names can only be 15 characters
* more thread names fixes
* allow an apps_groups.conf target to be pid 0 or 1
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
|
|
check session variable before resuming it
|
|
* query planer weight calculation using long long
* adjust replication query ahead pipeline for smaller systems
* do not generate huge replication messages
* add message to indicate replication message was interrupted
* improved message
* max replication size 25% of sender buffer
* fix for last commit
* use less cache and smaller page sizes and fewer threads on 32-bits
* fix reserved libuv workers for 32bits
* fix detection of 32/64 bit
|
|
* allow extents to be merged for as long as possible
* do not block the event loop while recalculating retention due to datafile rotation
* buffers are incrementally cleaned up, every second, by just 1 entry
* fix order of commands
* remove newline
* measure cancelled extent read requests
* count all cancelled extent requests
* do not double count failed pages
* fixed cancelled name
* Fix error and warnings when compiling with --disable-dbengine
* when the timeframe is outside retention and whole query should fail
* do not mark as failed pages that have been loaded but have been skipped
* added chart to show cache memory calculation variables
* LONG_MAX for 32-bit compatibility
* fix cache size calculation on 32-bit
* fix cache size calculation on 32-bit - use unsinged long long
* fix compilation warnings on 32-bits
* fix another compilation warning on 32-bits
* fix compilation warnings on older 32-bit compilers
* fix compilation warnings on older 32-bit compilers - more of them
* disable ML threads joining
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
|
|
* enable retries for SSL_ERROR_WANT_READ
* only when bytes is <= 0
* treat ERROR_WANT_READ/WRITE as 0 bytes
* dont close connection on zero bytes
* reuse ssl connection
* treat zero bytes
* ifdef for old openssl
* revert check
|
