Add code for LZ4 streaming data compression (#11821)

* Add code for LZ4 streaming data compression

* Fix LGTM alert

* Add lz4 library for link when compression enabled

* Add LZ4_resetStream_fast presence detection

* Disable compression for older LZ4 libraries

* Correct LZ4 API check

* [Testing Stream Compression] Debug msgs and report.md

* Add LZ4 library version using LZ4_initStream

* Fixed bug in SSL mode

* [Testing compression] - Add compression info messages

* Set compression enabled by default, update doc

* Update streaming/README.md

Co-authored-by: DShreve2 <david@netdata.cloud>

* [Agent Negotiation] Compression as separate capability

* [Agent Negotiation] Compression as separate capability - default compression variable always active

* Add code to negotiate compression

* [Agent Negotiation] Based on stream version

* [Agent Negotiation] Version based - fix compilation error

* [Agent Negotiation] Fix glob var default_compression_enbaled=0 affects all the connections - Handle compression - stream version based

* [Agent Negotiation - Compression] - Add control flag in 1. sender/receiver state & 2. stream.conf per child

* [Agent Negotiation - Compression] Fix stream.conf key, mguid control

* [Agent Negotiate Compression] Fine control on stream.conf per key,mguid for each child

* [Agent Negotiation Compression] Stop destroying compressor for runtime configuration + Update Readme.md

* [Agent Negotiation Compression] Use stream_version 4 if compression is disabled

* Correct child's compression check

* [Agent Negotiation Compression] Create streaming compression section in docs.

* [Agent Negotiation Compresion] Remove redundant debug msgs

* [Stream Compression] - integrate compression build info & config info in api/v1/info endpoint.

* [Agent Negotiation] Finalize README.md

* [Agent Stream Compression] Fix buildinfo json, Finalize readme.md

* [Agent Stream Compression] Negotiate compression based on stream version

* [Agent Stream Compression] Stream compression control per child in stream.conf |  per AP_KEY, MACHINE_GUID

* [Agent Stream Compression] Avoid destroying compressor enabling runtime configuration + Update Readme.md

* [Agent Stream Compression] - Provide stream compression build info & config info in api/v1/info endpoint + Update Readme.md

* [Agent Stream Compression] Fix rebase conflicts

* [Agent Stream Compression] Fix more rebase conflicts

* [Agent Stream Compression] 1. Stream version based negotiation 2. per child stream.conf control 3. finalize docs 4. stream compression build info in web api

* [Agent Stream Compression] 1. Stream version based negotiation 2. per child stream.conf control 3. finalize docs 4. stream compression build info in web api

* [Agent Stream Compression] Change unsuccessful buffer check to error

* [Agent Stream Compression] Readme.md proof-read corrections, downgrade to stream_version_clabels, add shields for supported versions, EOF lint

* [Agent Stream Compression] Fix missed lz4 library on Alpine Linux

* Phrasal review

Co-authored-by: odynik <odynik.ee@gmail.com>
Co-authored-by: DShreve2 <david@netdata.cloud>
Co-authored-by: Tina Lüdtke <tina@kickoke.com>

1 files changed, 19 insertions, 0 deletions

diff --git a/streaming/stream.conf b/streaming/stream.conf
index b5142632db..3c363fad69 100644
--- a/streaming/stream.conf
+++ b/streaming/stream.conf
@@ -60,6 +60,12 @@
     # The API_KEY to use (as the sender)
     api key =
 
+    # Stream Compresssion
+    #
+    # The netdata child is configurated to enable stream compression by default. 
+    # You can control stream compression in this agent with options: yes | no
+    #enable compression = yes
+
     # The timeout to connect and send metrics
     timeout seconds = 60
 
@@ -156,6 +162,12 @@
     #default proxy api key = API_KEY
     #default proxy send charts matching = *
 
+    # Stream Compresssion
+    #
+    # The stream with the child can be configurated to enable stream compression. 
+    # You can control stream compression in this parent agent stream with options: yes | no
+    #enable compression = yes
+
 
 # -----------------------------------------------------------------------------
 # 3. PER SENDING HOST SETTINGS, ON PARENT NETDATA
@@ -203,3 +215,10 @@
     #proxy destination = IP:PORT IP:PORT ...
     #proxy api key = API_KEY
     #proxy send charts matching = *
+
+    # Stream Compresssion
+    #
+    # The stream with the child can be configurated to enable stream compression. 
+    # You can control stream compression in this parent agent stream with options: yes | no
+    #enable compression = yes
+  
\ No newline at end of file