summaryrefslogtreecommitdiffstats
path: root/streaming
AgeCommit message (Collapse)Author
2020-07-28Implemented multihost database (#9556)Stelios Fragkakis
* Hard code a node for non-legacy multidb test Skip dbengine initialization for new incoming children Add code to switch to multidb ctx when accessing the dbengine * When a non-legacy streaming connection is detected, use the multidb metadata log context * Clear the superblock memory to avoid random data written in the metadata log * Activate the host detection during compaction Activate the host detection during metadata log chart updates Keep the host in the user object during replay of the HOST command * Add defaults for health / rrdpush on HOST metadata replay Check for legacy status on host creation by checking is_archived and if not conclusive, call is_legacy_child() Use defaults from the stream.conf * Count hosts only if not archived When host switches from archived to active update rrd_hosts_available Remove archived hosts from charts and info * Change parameter from "multidb disk space" to "dbengine multihost disk space" Remove unused variables Fix compilation error when dbengine is disabled Fix condition for machine_guid directory creation under cache_dir * Enable multidb disk space file creation. * Stop deleting dimensions when rotating archived metrics if the dimension is active in a different database engine. * Fix old bug in the code that confused obsolete hosts with orphan hosts. * Do not delete multi-host DB host files. * Discard dbengine state when a legacy memory mode instantiates to avoid inconsistencies. * Identify metadata that collide with non-dbengine memory mode hosts and ignore them. * Handle non-dbengine localhost with dbengine archived charts in localhost and streaming. * Ignore archived hosts in streaming. * Add documentation before merging to master. Co-authored-by: Markos Fountoulakis <markos.fountoulakis.senior@gmail.com>
2020-07-02Fixed typo in the streaming readme (#9378)Jonathan Barda
Changed `ndoes` to `nodes`.
2020-07-02Added missing caps letters (#9379)Jonathan Barda
It appeared me that few words were missing a first letter in caps.
2020-06-29Fix children version on stream (#9438)thiagoftsm
Store Netdata child version instead parent version inside `HOST` structure.
2020-06-22Fixed concurrency bug in health sending HOST VARIABLE to receiver (#9396)thiagoftsm
The recent changes to the streaming component wipe the send buffer when the mutex is acquired to prevent stale data and expose concurrency bugs. The callback from health was called from another thread to write into the stream buffer without any protection. This protects the access to the buffer.
2020-06-19Fixes the race-hazard in streaming during the shutdown sequence (#9370)Andrew Moss
The streaming component detects when a receiver stream has closed, and stops an attached sender on the same host. This is to support proxy configurations where the stream is passed through. During the shutdown sequence, once netdata_exit has been set no thread should touch any RRDHOST structure as the non-static threads are not joined before the database shuts down. The destruction of the thread state has been separated from the cleanup and can be called from two points. If the thread can detach itself from the host (i.e. it is not during the shutdown sequence) then it does so and destroys the state. During shutdown the thread leaves the state intact so that it can be destroyed during the host destruction, and the host destruction now cancels the thread to ensure a consistent sequence of events.
2020-06-12Change streaming terminology to parent/child in docs (#9312)Joel Hans
* Intial pass through docs * Dash instead of slash * To parent/child * Child nodes * Change diagrams * Allowlist * Fixes for Andrew * Remove from build_external * Change in proc
2020-06-12Change streaming terminology to parent-child in the code (#9323)Andrew Moss
2020-06-12Add support for persistent metadata (#9324)Stelios Fragkakis
* Implemented collector metadata logging * Added persistent GUIDs for charts and dimensions * Added metadata log replay and automatic compaction * Added detection of charts with no active collector (archived) * Added new endpoint to report archived charts via `/api/v1/archivedcharts` * Added support for collector metadata update Co-authored-by: Markos Fountoulakis <44345837+mfundul@users.noreply.github.com>
2020-06-08Fix crash in #9291 (#9297)Andrew Moss
Did not account for the path where rrdhost_find_or_create fails and terminates the thread before setting rpt->host to a valid pointer.
2020-06-04Fix Coverity defects 359164, 359165 and 358989. (#9268)Andrew Moss
Removed uses of the host lock that could deadlock senders and replaced with the new fine-grained mutex.
2020-06-03Fix bugs in streaming and enable support for gap filling (#9214)Andrew Moss
This PR adds (inactive) support that we will use to fill the gaps on chart when a receiving agent goes offline and the sender reconnects. The streaming component has been reworked to make the connection bi-directional and fix several outstanding bugs in the area. * Fixed an incorrect case of version negotiation. Removed fatal() on exhaustion of fds. * Fixed cases that fell through to polling the socket after closing. * Fixed locking of data related to sender and receiver in the host structure. * Added fine-grained locks to reduce contention. * Added circular buffer to sender to prevent starvation in high-latency conditions. * Fixed case where agent is a proxy and negotiated different streaming versions with sender and receiver. * Changed interface to new parser to put the buffering code in streaming. * Fixed the bug that stopped senders from reconnecting after their socket times out - this was part of the scaling fixes that provide an early shortcut path for rejecting connections without lock contention. * Uses fine-grained locking and a different approach to thread shutdown instead. * Added liveness detection to connections to allow selection of the best connection.
2020-05-25Implement new incremental parser (#9074)Stelios Fragkakis
Implemented a new parser for the pluginsd language
2020-05-11Docs: Fix internal links and remove obsolete admonitions (#8946)Joel Hans
* Fixed a few more links * Remove old syntax * Abs-relative links to files in docs folder * Trying to fix nother doc learn link * Fix a few more links * Add testing doc * Tracking down mysteries * Cleanup * Update broken external links * Remove index.html that appeared from testing * Fix remainder of links
2020-04-21capitalize title (#8712)Zack Shoylev
2020-04-14Docs: Standardize links between documentation (#8638)Joel Hans
* Trying out some absolute-ish links * Try one out on installer * Testing logic * Trying out some more links * Fixing links * Fix links in python collectors * Changed a bunch more links * Fix build errors * Another push of links * Fix build error and add more links * Complete first pass * Fix final broken links * Fix links to files * Fix for Netlify * Two more fixes
2020-04-03cove355076: Config per section (#8588)thiagoftsm
This commit brings the possibility to search an option directly when we already have a section
2020-03-31Extend TLS Support (#8505)thiagoftsm
* tls13: This commit brings TLS 1.3 to Netdata * tls13: Update variables on slave side * tls13: Fix compilation error for old libraries * tls13: Fix compilation error for old libraries 2 * tls13 remove ciphers * tls13: TLS versions This commit brings the missing tls versions accpeted for Netdata and it also brings documentation update related to these versions * tls13: Remove dupplication This commit removes wrong dupplication of code * tls13: Documentation This commit brings fix for the documentation * tls13: Remove magic number This commit removes the magic number to allow the code to be readable * tls13: TLS version Small adjust with TLS version * tls13: Security Init This commit removes array from the function and overwrite the magic number with a string * tls13: Remove new variable name from stream * tls13: OpenSSL versions and old key name This commit removes the new key names and also update the names used to define openssl version
2020-03-16Fix streaming scaling (#8375)Markos Fountoulakis
* Disallow multiple streaming connections to the same master agent * Reject multiple streaming connections quickly without blocking * Increase timeout for systemd service shutdown to give time to flush the db. * Optimize page correlation ID to use atomic counter instead of locks * Reduce contention in global configuration mutex * Optimize complexity of inserting configuration sections from O(N) to O(1) * Reduce overhead of clockgettime() by utilizing CLOCK_MONOTONIC_COARSE when applicable. * Fix unit test compile errors
2020-03-12Enconde slave fields (#8216)thiagoftsm
* stream_encode: Enconde slave This commit encodes the messages before to send them from master to slave * stream_encode: Remove comma This commit changes the comma to semi-colon to bring a pattern to code
2020-03-10Bulk add frontmatter to all documentation (#8354)Joel Hans
* Bulk add frontmatter * A few extra edge cases
2020-02-10Fix missing variables on stream (#8011)thiagoftsm
* stream_container: Bring the missing container variables to stream * stream_container: Missing variables This commit brings 4 new variables that were missed to stream
2020-02-05Stream with version (#7851)thiagoftsm
* stream_forward: Fix protocol This commit brings the necessary fixes to the protocol * stream_forward: Fix old slave support This commit fixes the communication with old versions of Netdata * stream_forward: Remove declaration There was a wrong declaration inside a block, so I am removing it * stream_forward: USe version This commit brings the use of version instead flags to stream * stream_forward: Remove variable This commit removes useless variable from hand shake * stream_forward: Change message Change the message setting the protocol version on it * stream_forward: Fix version number * stream_forward: readable definition The definition and the variables were using the same data type, but with different declaration, this commit fixes this. * stream_forward: Set master version inside message This commit updates the message used that there was a successfull connection with master * stream_forward: FIx wrong version This commit fixes the multiple set for stream version * stream_forward: Reorganize code This commit reorganizes code to speed up the processing * stream_forward: Adjust code This commit removes an unecessary else * stream_forward: Brings old structure This commits returns a previous necessary to the code * stream_forward: fix error report This commit fixes the error report that was happening when the stream version does not match * stream_forward: Fixes msg and remove unecessary call
2020-02-01Parse host tags (#7702)Vladimir Kobal
* Fix memory leaks * Check for configuration options * Parse simple tags * Parse JSON tags * Remove an unnecessary check * Parse a JSON object * Parse a JSON array * Update the documentation * Fix host locks
2020-01-27Update stream documentation bringing explanation of some errors (#6995)thiagoftsm
* stream_doc: Update documentation Update documentation for stream to clarify the users some errors * stream_doc: Update documentation 2 Explain more errors that can happen on master * Fix errors reported This commit fixes errors reported by @cosmix * Fix and more errors This commit brings more fixes for the text and description for more errors * stream_doc: Gramatical fixes This commit brings all the fixes suggested by Joel * stream_doc: Gramatical fixes This commit brings the last fix suggested by Joel * stream_doc: Message format I missed a message format in the previous sprint * Fix doc: This commit fix the errors reported by Joel * Fix test This commit brings part of the fixes requested from Christopher * New explanation: The previous explanation for these two features were not good enough, so I rewrite it to avoid confusion * stream_doc: Fix text After to receive the text from Joel, I am bringing all the fix for the new section on documentation
2020-01-21Issue 7488 docker labels (#7770)Andrew Moss
Improve the metadata detection for containers. The system_info structure has been updated to hold separate copies of OS_NAME, OS_ID, OS_ID_LIKE, OS_VERSION, OS_VERSION_ID and OS_DETECTION for both the container environment and the host. This new information is communicated through the /api/v1/info endpoint. For the streaming interface a partial copy of the info is carried until the stream protocol is upgraded. The anonymous_statistics script has been updated to carry the new data to Google Analytics. Some minor improvements have been made to OS-X / FreeBSD detection, and the detection of virtualization. The docs have been updated to explain how to pass the host environment to the docker container running Netdata.
2020-01-14Stream with labels (#7549)thiagoftsm
This commit enables streaming host labels
2019-12-16Labels issues (#7515)Andrew Moss
Initial work on host labels from the dedicated branch. Includes work for issues #7096, #7400, #7411, #7369, #7410, #7458, #7459, #7412 and #7408 by @vlvkobal, @thiagoftsm, @cakrit and @amoss.
2019-11-14stream_ssl_slave: Fix socket overwrite (#7306)thiagoftsm
When a slave had SSL activate for stream and local access it was overwritten the addresses, this PR fixes this problem that was not allowed the stream works 100%
2019-11-11Makefile.am files indentation (#7252)Konstantinos Natsakis
* Use 4 spaces for indentation of non-recipe lines in Makefile.am files * Be more consistent in the use of space before = in Makefile.am files
2019-08-15Fix Markdown Lint warnings (#6664)Promise Akpan
* make remark access all directories * detailed fix after autofix by remark lint * cross check autofix for this set of files * crosscheck more files * crosschecking and small fixes * crosscheck autofixed md files
2019-08-13 Change "netdata" to "Netdata" in all docs (#6621)Joel Hans
* First pass of changing netdata to Netdata * Second pass of netdata -> Netdata * Starting work on netdata with no whitespace after * Pass for netdata with no whitespace at the end * Pass for netdata with no whitespace at the front
2019-07-29Add configurable default locations for trusted CA certificates (#6549)thiagoftsm
* sslcertificate: Trust certificate The netdata could not allow invalid certificate or certificate with invalid chain this commit fixes this! * sslcertificate: Changing name We are binging the same names used by the OpenSSL library to simplify the understand of the parameters * sslcertificate: Name changes and explicity directory This commit fix the problem with Streams and rename correctly the files in the option, it also uses stat to define the existence of a file * sslcertificate: Documentation Fix grammar for the newest section in the documentation * sslcertificate: Rename variables The old variables did not represent well what they are doing, so it was renamed
2019-07-27netdata/streaming: fix bug for non-ssl installations - we used a definition ↵Pavlos Emm. Katsoulakis
that only exists on ssl-enabled installs
2019-07-25Fix parsing SSL ACL along with others (#6468)thiagoftsm
* sslstream: ACL parser It was noticed in the issue 6457 that the some ACLs were not parsing correctly when they were along SSL acl, this commit fixes this' * sslstream: remove comments This commit removes the comments that were present while I was testing the code * sslstream: Tests This commit adds ACL tests to check the Netdata response to them * sslstream: Tests Fix the extension to upload the files * sslstream: more tests In this commit I am bringing more tests, including the ssl tests' * sslstream: leading space Remove leading space from variable that was creating problem with shellcheck * sslstream: glob Remove special character from script * sslstream: Makefile The Makefile diretives were pointed to wrong files * sslstream: Missing stream encrypt This commit solves the problem of the stream not be encrypted, but it is not the final solution, because the parser made is incomplete. * sslstream: Finish encrypt channel This commit brings the step that I was missing, the complete encryptation in the communication between Master and Slave * sslstream: Fix argument in script After the latest tests, it was verified that two arguments given to a function inside the script were not correct, with this PR I am fixing this! * sslstream: Fix argument in info Instead to call a function to deliver an integer I was passing a size_t value. Only cmake showed this, but not in my clion! :/ * sslstream: Fix redirect When we were having different SSL configuration, the system were not applying the option for all * sslstream: Update documentation Our documentation was not clear about the rules according our code so I am updating the text to explain for the users * sslstream: Adjust script With this last commit, I am adjusting the tests to avoid false positive * sslstream: Missing elif The previous commit had a missing elif in the shell script * sslstream: Split ports Before this commit Netdata was having SSL as a global option, now it has as a real ACL. * sslstream: reduce context The stream variable will not be affected in the master side, it is only necessary on the slave side, so I am reducing the context of it * sslstream: Force SSL When the user has certificate and he does not set any SSL flag, it is necessary to append the SSL=force flag * sslstream: Default flag It is necessary to have a default flag when the SSL flags are not SET * sslstream: remove comments Remove comments from the scrip * sslstream: moving flag It is better the flag to be set inside socket instead everytime there is a new connection * sslstream: documentation Fix a sentence in the web/server/README.md
2019-07-10SSL vs. TLS consistency and clarification in documentation (#6414)Joel Hans
* Changed references of SSL to TLS where appropriate * A few more tweaks on SSL->TLS * Fixed whitespace issue from @vlvkobal * Fixing sentence on OpenTSDB for @thiagoftsm
2019-06-27Backend and SSL! (#6220)thiagoftsm
* SSL_backend Begin of the encryptation of backend! * SSL_backend changing opentsdb! * SSL_backend fix HTTP message with JSON! * SSL_backend HTTP API done! * SSL_fix_format preparing to connect with proxy! * SSL_backend wip SSL send/receive ! * SSL_backend working with proxy * SSL_backend removing comments! * SSL_backend docummentation! * SSL_backend review]! * SSL_backend organizing! * Alarm_backend remove comments! * SSL_backend! * SSL_backend typedef! * SSL_backend bring switch! * SSL_backend commiting format changes! * SSL_backend fix github parser! * SSL_Backend fix format! * SSL_backend switch everything! * SSL_backend reviewing! * SSL_backend comments! * SSL_backend indentation! * SSL_backend indentation 3! * SSL_backend documentation! * SSL_backend hidden pointer! * SSL_backend missing space * SSL_backend change documentation! * SSL_backend change documentation 2!
2019-06-24SSL_better_name change name! (#6336)thiagoftsm
2019-06-19Chart name streaming (#6304)Vladimir Kobal
* Update name when it is changed * Add info about custom chart names on slaves
2019-06-14Update streaming documentation and config file with more info on how health ↵Chris Akritidis
enabled by default = auto works. (#6281)
2019-06-03netdata/daemon: SSL fix - broken compilation case when ssl library not ↵thiagoftsm
present! (#6201) * SSL_fix fix the compilation case the library is not present!
2019-05-31SSL implementation for Netdata (#5956)thiagoftsm
* SSL implementation for Netdata * Upload of fixes asked by @paulkatsoulakis and @cakrit * Fix local computer * Adding openssl to webserver * fixing.. * HTTPS almost there * Codacity * HTTPS day 3 * HTTPS without Bio step 1 * HTTPS without Bio step 2 * HTTPS without Bio step 3 * HTTPS without Bio step 4 * HTTPS without Bio step 5 * HTTPS without Bio step 6 * HTTPS without Bio step 7 * HTTPS without Bio step 8 * HTTPS without Bio step 9 * HTTPS without Bio step 10 * SSL on streaming 1 * Daily pull * HTTPS without Bio step 11 * HTTPS without Bio step 12 * HTTPS without Bio step 13 * HTTPS without Bio step 14 * SSL_Interception change documentation * HTTPS without Bio step 15 * HTTPS without Bio step 16 * SSL_Interception fix codacity * SSL_Interception fix doc * SSL_Interception comments * SSL_Interception fixing problems! * SSL_Interception killing bugs * SSL_Interception changing parameter * SSL_Implementation documentation and script * SSL_Implementation multiple fixes * SSL_Implementation installer and cipher * SSL_Implementation Redirect 301 * SSL_Implementation webserver doc and install-or-update.sh * SSL_Implementation error 00000001:lib(0):func(0):reason(1) * SSL_Implementation web server doc * SSL_Implementation SEGFAULT on Fedora * SSL_Implementation fix ^SSL=force|optional * SSL_Implementation Redirect and Ciphers * SSL_Implementation race condition 1 * SSL_Implementation Fix Location * SSL_Implementation Fix Location 2 * SSL_Implementation Fix stream * SSL_Implementation Fix stream 2 * SSL_Implementation Fix stream 3 * SSL_Implementation last problems! * SSL_Implementation adjusts to commit! * SSL_Implementation documentation permission! * SSL_Implementation documentation permission 2! * SSL_Implementation documentation permission 3!
2019-05-28Documentation fixes (#6144)Chris Akritidis
* Correct my-netdata menu to node menu and mention Netdata cloud in the registry * rebase and fix replace of main readme link * remove comma
2019-05-27Remove unecessary 2nd call of get_system_info at startup (#6128)Chris Akritidis
2019-05-24Remove system_info copying (#6113)Vladimir Kobal
* Remove system_info copying * Return res flag
2019-05-15Database engine (#5282)Markos Fountoulakis
* Database engine prototype version 0 * Database engine initial integration with netdata POC * Scalable database engine with file and memory management. * Database engine integration with netdata * Added MIN MAX definitions to fix alpine build of travis CI * Bugfix for backends and new DB engine, remove useless rrdset_time2slot() calls and erroneous checks * DB engine disk protocol correction * Moved DB engine storage file location to /var/cache/netdata/{host}/dbengine * Fix configure to require openSSL for DB engine * Fix netdata daemon health not holding read lock when iterating chart dimensions * Optimized query API for new DB engine and old netdata DB fallback code-path * netdata database internal query API improvements and cleanup * Bugfix for DB engine queries returning empty values * Added netdata internal check for data queries for old and new DB * Added statistics to DB engine and fixed memory corruption bug * Added preliminary charts for DB engine statistics * Changed DB engine ratio statistics to incremental * Added netdata statistics charts for DB engine internal statistics * Fix for netdata not compiling successfully when missing dbengine dependencies * Added DB engine functional test to netdata unittest command parameter * Implemented DB engine dataset generator based on example.random chart * Fix build error in CI * Support older versions of libuv1 * Fixes segmentation fault when using multiple DB engine instances concurrently * Fix memory corruption bug * Fixed createdataset advanced option not exiting * Fix for DB engine not working on FreeBSD * Support FreeBSD library paths of new dependencies * Workaround for unsupported O_DIRECT in OS X * Fix unittest crashing during cleanup * Disable DB engine FS caching in Apple OS X since O_DIRECT is not available * Fix segfault when unittest and DB engine dataset generator don't have permissions to create temporary host * Modified DB engine dataset generator to create multiple files * Toned down overzealous page cache prefetcher * Reduce internal memory fragmentation for page-cache data pages * Added documentation describing the DB engine * Documentation bugfixes * Fixed unit tests compilation errors since last rebase * Added note to back-up the DB engine files in documentation * Added codacy fix. * Support old gcc versions for atomic counters in DB engine
2019-05-14Fix segmentation fault (#6011)Vladimir Kobal
* Fix segmentation fault * Make system info printing safe * Fix quotes for OS name
2019-05-14Fix Coverity defects (#6008)Vladimir Kobal
2019-05-13Add system info streaming (#5996)Vladimir Kobal
* Add system info streaming * Fix segmentation fault in unit testing
2019-04-17Fix segmentation fault in streaming (#5882)Vladimir Kobal
generated by cgit v1.2.3 (git 2.25.1) at 2024-09-08 23:57:03 +0000