summaryrefslogtreecommitdiffstats
path: root/streaming
AgeCommit message (Collapse)Author
2020-11-10streaming: fix a typo in the README.md (#10225)Ilya Mashchenko
2020-10-22Disregard host tags configuration pointer and keep duplicate string from ↵Markos Fountoulakis
streaming handshake. (#10121)
2020-10-20Replace memcpy() with memmove() for overlapping memory ranges. (#10097)Markos Fountoulakis
2020-09-29Add docsv2 project to master branch (#10000)Joel Hans
* Add overview docs to docsv2 project * Add quickstart docs to docsv2 project (#9865) * Init quickstart docs * Begin work on quickstart guides * Finish quickstart drafts * Tweaks to both quickstarts * Add titles * Copyedit pass to both docs * Fixes for Amy and Jen * Add Get doc to docsv2 project (#9854) * Init get file * Add some links * Change h2 to h1 * Rephrase * Add configure docs to docsv2 project (#9878) * Add overview docs to docsv2 project (#9849) * Init files * Add drafts of what and why * Finish initial drafts * Fix edit URL * Copyedit pass * Finish initial drafts of configure docs * Copyedit all docs * Fixes for Amy * Fixes for Jen * Add collect docs to the docsv2 project (#9932) * Init files * Finish first two collect docs * Finish drafts of collect docs * Copyedit pass * Fixes for Amy * Fix for Jen * Add visualize docs to the docsv2 project (#9943) * Add visualize docs * Copyedits and cleanup * New images and features * Copyedit pass and cleanup * Missing word * Fixes for Jen * Add monitor docs to docsv2 project (#9949) * Finish drafts of monitor docs * Copyedit pass * Remove whitespace * Fixes for Jen * Add store docs to docsv2 project (#9969) * Finalize store documents * Fix import path * Finishing edit section * Copyedit pass * Add export docs to docsv2 project (#9986) * Add install and claim videos to Get doc * Finish drafts of exporting docs plus other tweaks * Init new exporting READMEs * Copyedit pass and new links * Fixes for Amy, Vlad, Jen * Fix links in docsv2 project (#9993) * Fix links * Fix a bunch of links ahead of export merge * Fix additional links * Fix links, nuke what-is-netdata * Fixing a few last links * Improve product images in overview * Remove extra paren * Quick tweaks for Jen * Fixes for Jen * Access fix * Remove extra word
2020-09-07Fix lock order reversal. (#9888)Markos Fountoulakis
2020-09-04Fix race condition with orphan hosts (#9862)Markos Fountoulakis
* Fix race condition between orphan host cleanup and new streaming connections. * Remove health enabling from log replay, it will be handled at streaming connection time.
2020-08-27fixes proxy forwarding claim_id to old parent (#9828)Timotej S
when v3 child reloads claim state while connected to v3 proxy the proxy forwarded it to parent despite it being v2
2020-08-26Adds claimed_id streaming (#9804)Timotej S
* streams claimed_id of child nodes to parents * adds this information into /api/v1/info
2020-08-19Fix long stats.d chart names (thanks to @vince-lessbits) (#9783)Andrew Moss
The read buffer in the streaming receiver was too small to allow the longest lines in the plugins.d protocol to be reassembled.
2020-08-11Remove broken optimization in the sender thread (#9703)Andrew Moss
The sender thread avoided locking the circular buffer to check if there was outstanding data on the connection. The condition it needs (unsent data) grows monotonically w.r.t. other threads as the collectors can add data but only this thread can remove it. However, it cached the pointer into the buffer as a side-effect and then reused it later during the transmission. This fails if the buffer is resized by a collector thread. Peeking at the buffer sizes without locking could fail in the same situation. The optimization is removed and the sender thread now locks the mutex before checking the buffer, throws away the data buffer pointer and releases the mutex over the poll() operation. It then reacquires the mutex and checks the buffer size and data pointer again when it performs the send.
2020-07-28Implemented multihost database (#9556)Stelios Fragkakis
* Hard code a node for non-legacy multidb test Skip dbengine initialization for new incoming children Add code to switch to multidb ctx when accessing the dbengine * When a non-legacy streaming connection is detected, use the multidb metadata log context * Clear the superblock memory to avoid random data written in the metadata log * Activate the host detection during compaction Activate the host detection during metadata log chart updates Keep the host in the user object during replay of the HOST command * Add defaults for health / rrdpush on HOST metadata replay Check for legacy status on host creation by checking is_archived and if not conclusive, call is_legacy_child() Use defaults from the stream.conf * Count hosts only if not archived When host switches from archived to active update rrd_hosts_available Remove archived hosts from charts and info * Change parameter from "multidb disk space" to "dbengine multihost disk space" Remove unused variables Fix compilation error when dbengine is disabled Fix condition for machine_guid directory creation under cache_dir * Enable multidb disk space file creation. * Stop deleting dimensions when rotating archived metrics if the dimension is active in a different database engine. * Fix old bug in the code that confused obsolete hosts with orphan hosts. * Do not delete multi-host DB host files. * Discard dbengine state when a legacy memory mode instantiates to avoid inconsistencies. * Identify metadata that collide with non-dbengine memory mode hosts and ignore them. * Handle non-dbengine localhost with dbengine archived charts in localhost and streaming. * Ignore archived hosts in streaming. * Add documentation before merging to master. Co-authored-by: Markos Fountoulakis <markos.fountoulakis.senior@gmail.com>
2020-07-02Fixed typo in the streaming readme (#9378)Jonathan Barda
Changed `ndoes` to `nodes`.
2020-07-02Added missing caps letters (#9379)Jonathan Barda
It appeared me that few words were missing a first letter in caps.
2020-06-29Fix children version on stream (#9438)thiagoftsm
Store Netdata child version instead parent version inside `HOST` structure.
2020-06-22Fixed concurrency bug in health sending HOST VARIABLE to receiver (#9396)thiagoftsm
The recent changes to the streaming component wipe the send buffer when the mutex is acquired to prevent stale data and expose concurrency bugs. The callback from health was called from another thread to write into the stream buffer without any protection. This protects the access to the buffer.
2020-06-19Fixes the race-hazard in streaming during the shutdown sequence (#9370)Andrew Moss
The streaming component detects when a receiver stream has closed, and stops an attached sender on the same host. This is to support proxy configurations where the stream is passed through. During the shutdown sequence, once netdata_exit has been set no thread should touch any RRDHOST structure as the non-static threads are not joined before the database shuts down. The destruction of the thread state has been separated from the cleanup and can be called from two points. If the thread can detach itself from the host (i.e. it is not during the shutdown sequence) then it does so and destroys the state. During shutdown the thread leaves the state intact so that it can be destroyed during the host destruction, and the host destruction now cancels the thread to ensure a consistent sequence of events.
2020-06-12Change streaming terminology to parent/child in docs (#9312)Joel Hans
* Intial pass through docs * Dash instead of slash * To parent/child * Child nodes * Change diagrams * Allowlist * Fixes for Andrew * Remove from build_external * Change in proc
2020-06-12Change streaming terminology to parent-child in the code (#9323)Andrew Moss
2020-06-12Add support for persistent metadata (#9324)Stelios Fragkakis
* Implemented collector metadata logging * Added persistent GUIDs for charts and dimensions * Added metadata log replay and automatic compaction * Added detection of charts with no active collector (archived) * Added new endpoint to report archived charts via `/api/v1/archivedcharts` * Added support for collector metadata update Co-authored-by: Markos Fountoulakis <44345837+mfundul@users.noreply.github.com>
2020-06-08Fix crash in #9291 (#9297)Andrew Moss
Did not account for the path where rrdhost_find_or_create fails and terminates the thread before setting rpt->host to a valid pointer.
2020-06-04Fix Coverity defects 359164, 359165 and 358989. (#9268)Andrew Moss
Removed uses of the host lock that could deadlock senders and replaced with the new fine-grained mutex.
2020-06-03Fix bugs in streaming and enable support for gap filling (#9214)Andrew Moss
This PR adds (inactive) support that we will use to fill the gaps on chart when a receiving agent goes offline and the sender reconnects. The streaming component has been reworked to make the connection bi-directional and fix several outstanding bugs in the area. * Fixed an incorrect case of version negotiation. Removed fatal() on exhaustion of fds. * Fixed cases that fell through to polling the socket after closing. * Fixed locking of data related to sender and receiver in the host structure. * Added fine-grained locks to reduce contention. * Added circular buffer to sender to prevent starvation in high-latency conditions. * Fixed case where agent is a proxy and negotiated different streaming versions with sender and receiver. * Changed interface to new parser to put the buffering code in streaming. * Fixed the bug that stopped senders from reconnecting after their socket times out - this was part of the scaling fixes that provide an early shortcut path for rejecting connections without lock contention. * Uses fine-grained locking and a different approach to thread shutdown instead. * Added liveness detection to connections to allow selection of the best connection.
2020-05-25Implement new incremental parser (#9074)Stelios Fragkakis
Implemented a new parser for the pluginsd language
2020-05-11Docs: Fix internal links and remove obsolete admonitions (#8946)Joel Hans
* Fixed a few more links * Remove old syntax * Abs-relative links to files in docs folder * Trying to fix nother doc learn link * Fix a few more links * Add testing doc * Tracking down mysteries * Cleanup * Update broken external links * Remove index.html that appeared from testing * Fix remainder of links
2020-04-21capitalize title (#8712)Zack Shoylev
2020-04-14Docs: Standardize links between documentation (#8638)Joel Hans
* Trying out some absolute-ish links * Try one out on installer * Testing logic * Trying out some more links * Fixing links * Fix links in python collectors * Changed a bunch more links * Fix build errors * Another push of links * Fix build error and add more links * Complete first pass * Fix final broken links * Fix links to files * Fix for Netlify * Two more fixes
2020-04-03cove355076: Config per section (#8588)thiagoftsm
This commit brings the possibility to search an option directly when we already have a section
2020-03-31Extend TLS Support (#8505)thiagoftsm
* tls13: This commit brings TLS 1.3 to Netdata * tls13: Update variables on slave side * tls13: Fix compilation error for old libraries * tls13: Fix compilation error for old libraries 2 * tls13 remove ciphers * tls13: TLS versions This commit brings the missing tls versions accpeted for Netdata and it also brings documentation update related to these versions * tls13: Remove dupplication This commit removes wrong dupplication of code * tls13: Documentation This commit brings fix for the documentation * tls13: Remove magic number This commit removes the magic number to allow the code to be readable * tls13: TLS version Small adjust with TLS version * tls13: Security Init This commit removes array from the function and overwrite the magic number with a string * tls13: Remove new variable name from stream * tls13: OpenSSL versions and old key name This commit removes the new key names and also update the names used to define openssl version
2020-03-16Fix streaming scaling (#8375)Markos Fountoulakis
* Disallow multiple streaming connections to the same master agent * Reject multiple streaming connections quickly without blocking * Increase timeout for systemd service shutdown to give time to flush the db. * Optimize page correlation ID to use atomic counter instead of locks * Reduce contention in global configuration mutex * Optimize complexity of inserting configuration sections from O(N) to O(1) * Reduce overhead of clockgettime() by utilizing CLOCK_MONOTONIC_COARSE when applicable. * Fix unit test compile errors
2020-03-12Enconde slave fields (#8216)thiagoftsm
* stream_encode: Enconde slave This commit encodes the messages before to send them from master to slave * stream_encode: Remove comma This commit changes the comma to semi-colon to bring a pattern to code
2020-03-10Bulk add frontmatter to all documentation (#8354)Joel Hans
* Bulk add frontmatter * A few extra edge cases
2020-02-10Fix missing variables on stream (#8011)thiagoftsm
* stream_container: Bring the missing container variables to stream * stream_container: Missing variables This commit brings 4 new variables that were missed to stream
2020-02-05Stream with version (#7851)thiagoftsm
* stream_forward: Fix protocol This commit brings the necessary fixes to the protocol * stream_forward: Fix old slave support This commit fixes the communication with old versions of Netdata * stream_forward: Remove declaration There was a wrong declaration inside a block, so I am removing it * stream_forward: USe version This commit brings the use of version instead flags to stream * stream_forward: Remove variable This commit removes useless variable from hand shake * stream_forward: Change message Change the message setting the protocol version on it * stream_forward: Fix version number * stream_forward: readable definition The definition and the variables were using the same data type, but with different declaration, this commit fixes this. * stream_forward: Set master version inside message This commit updates the message used that there was a successfull connection with master * stream_forward: FIx wrong version This commit fixes the multiple set for stream version * stream_forward: Reorganize code This commit reorganizes code to speed up the processing * stream_forward: Adjust code This commit removes an unecessary else * stream_forward: Brings old structure This commits returns a previous necessary to the code * stream_forward: fix error report This commit fixes the error report that was happening when the stream version does not match * stream_forward: Fixes msg and remove unecessary call
2020-02-01Parse host tags (#7702)Vladimir Kobal
* Fix memory leaks * Check for configuration options * Parse simple tags * Parse JSON tags * Remove an unnecessary check * Parse a JSON object * Parse a JSON array * Update the documentation * Fix host locks
2020-01-27Update stream documentation bringing explanation of some errors (#6995)thiagoftsm
* stream_doc: Update documentation Update documentation for stream to clarify the users some errors * stream_doc: Update documentation 2 Explain more errors that can happen on master * Fix errors reported This commit fixes errors reported by @cosmix * Fix and more errors This commit brings more fixes for the text and description for more errors * stream_doc: Gramatical fixes This commit brings all the fixes suggested by Joel * stream_doc: Gramatical fixes This commit brings the last fix suggested by Joel * stream_doc: Message format I missed a message format in the previous sprint * Fix doc: This commit fix the errors reported by Joel * Fix test This commit brings part of the fixes requested from Christopher * New explanation: The previous explanation for these two features were not good enough, so I rewrite it to avoid confusion * stream_doc: Fix text After to receive the text from Joel, I am bringing all the fix for the new section on documentation
2020-01-21Issue 7488 docker labels (#7770)Andrew Moss
Improve the metadata detection for containers. The system_info structure has been updated to hold separate copies of OS_NAME, OS_ID, OS_ID_LIKE, OS_VERSION, OS_VERSION_ID and OS_DETECTION for both the container environment and the host. This new information is communicated through the /api/v1/info endpoint. For the streaming interface a partial copy of the info is carried until the stream protocol is upgraded. The anonymous_statistics script has been updated to carry the new data to Google Analytics. Some minor improvements have been made to OS-X / FreeBSD detection, and the detection of virtualization. The docs have been updated to explain how to pass the host environment to the docker container running Netdata.
2020-01-14Stream with labels (#7549)thiagoftsm
This commit enables streaming host labels
2019-12-16Labels issues (#7515)Andrew Moss
Initial work on host labels from the dedicated branch. Includes work for issues #7096, #7400, #7411, #7369, #7410, #7458, #7459, #7412 and #7408 by @vlvkobal, @thiagoftsm, @cakrit and @amoss.
2019-11-14stream_ssl_slave: Fix socket overwrite (#7306)thiagoftsm
When a slave had SSL activate for stream and local access it was overwritten the addresses, this PR fixes this problem that was not allowed the stream works 100%
2019-11-11Makefile.am files indentation (#7252)Konstantinos Natsakis
* Use 4 spaces for indentation of non-recipe lines in Makefile.am files * Be more consistent in the use of space before = in Makefile.am files
2019-08-15Fix Markdown Lint warnings (#6664)Promise Akpan
* make remark access all directories * detailed fix after autofix by remark lint * cross check autofix for this set of files * crosscheck more files * crosschecking and small fixes * crosscheck autofixed md files
2019-08-13 Change "netdata" to "Netdata" in all docs (#6621)Joel Hans
* First pass of changing netdata to Netdata * Second pass of netdata -> Netdata * Starting work on netdata with no whitespace after * Pass for netdata with no whitespace at the end * Pass for netdata with no whitespace at the front
2019-07-29Add configurable default locations for trusted CA certificates (#6549)thiagoftsm
* sslcertificate: Trust certificate The netdata could not allow invalid certificate or certificate with invalid chain this commit fixes this! * sslcertificate: Changing name We are binging the same names used by the OpenSSL library to simplify the understand of the parameters * sslcertificate: Name changes and explicity directory This commit fix the problem with Streams and rename correctly the files in the option, it also uses stat to define the existence of a file * sslcertificate: Documentation Fix grammar for the newest section in the documentation * sslcertificate: Rename variables The old variables did not represent well what they are doing, so it was renamed
2019-07-27netdata/streaming: fix bug for non-ssl installations - we used a definition ↵Pavlos Emm. Katsoulakis
that only exists on ssl-enabled installs
2019-07-25Fix parsing SSL ACL along with others (#6468)thiagoftsm
* sslstream: ACL parser It was noticed in the issue 6457 that the some ACLs were not parsing correctly when they were along SSL acl, this commit fixes this' * sslstream: remove comments This commit removes the comments that were present while I was testing the code * sslstream: Tests This commit adds ACL tests to check the Netdata response to them * sslstream: Tests Fix the extension to upload the files * sslstream: more tests In this commit I am bringing more tests, including the ssl tests' * sslstream: leading space Remove leading space from variable that was creating problem with shellcheck * sslstream: glob Remove special character from script * sslstream: Makefile The Makefile diretives were pointed to wrong files * sslstream: Missing stream encrypt This commit solves the problem of the stream not be encrypted, but it is not the final solution, because the parser made is incomplete. * sslstream: Finish encrypt channel This commit brings the step that I was missing, the complete encryptation in the communication between Master and Slave * sslstream: Fix argument in script After the latest tests, it was verified that two arguments given to a function inside the script were not correct, with this PR I am fixing this! * sslstream: Fix argument in info Instead to call a function to deliver an integer I was passing a size_t value. Only cmake showed this, but not in my clion! :/ * sslstream: Fix redirect When we were having different SSL configuration, the system were not applying the option for all * sslstream: Update documentation Our documentation was not clear about the rules according our code so I am updating the text to explain for the users * sslstream: Adjust script With this last commit, I am adjusting the tests to avoid false positive * sslstream: Missing elif The previous commit had a missing elif in the shell script * sslstream: Split ports Before this commit Netdata was having SSL as a global option, now it has as a real ACL. * sslstream: reduce context The stream variable will not be affected in the master side, it is only necessary on the slave side, so I am reducing the context of it * sslstream: Force SSL When the user has certificate and he does not set any SSL flag, it is necessary to append the SSL=force flag * sslstream: Default flag It is necessary to have a default flag when the SSL flags are not SET * sslstream: remove comments Remove comments from the scrip * sslstream: moving flag It is better the flag to be set inside socket instead everytime there is a new connection * sslstream: documentation Fix a sentence in the web/server/README.md
2019-07-10SSL vs. TLS consistency and clarification in documentation (#6414)Joel Hans
* Changed references of SSL to TLS where appropriate * A few more tweaks on SSL->TLS * Fixed whitespace issue from @vlvkobal * Fixing sentence on OpenTSDB for @thiagoftsm
2019-06-27Backend and SSL! (#6220)thiagoftsm
* SSL_backend Begin of the encryptation of backend! * SSL_backend changing opentsdb! * SSL_backend fix HTTP message with JSON! * SSL_backend HTTP API done! * SSL_fix_format preparing to connect with proxy! * SSL_backend wip SSL send/receive ! * SSL_backend working with proxy * SSL_backend removing comments! * SSL_backend docummentation! * SSL_backend review]! * SSL_backend organizing! * Alarm_backend remove comments! * SSL_backend! * SSL_backend typedef! * SSL_backend bring switch! * SSL_backend commiting format changes! * SSL_backend fix github parser! * SSL_Backend fix format! * SSL_backend switch everything! * SSL_backend reviewing! * SSL_backend comments! * SSL_backend indentation! * SSL_backend indentation 3! * SSL_backend documentation! * SSL_backend hidden pointer! * SSL_backend missing space * SSL_backend change documentation! * SSL_backend change documentation 2!
2019-06-24SSL_better_name change name! (#6336)thiagoftsm
2019-06-19Chart name streaming (#6304)Vladimir Kobal
* Update name when it is changed * Add info about custom chart names on slaves
2019-06-14Update streaming documentation and config file with more info on how health ↵Chris Akritidis
enabled by default = auto works. (#6281)
generated by cgit v1.2.3 (git 2.25.1) at 2024-08-05 08:17:57 +0000