Streaming interpolated values (#14431)

* first commit - untested

* fix wrong begin command

* added set v2 too

* debug to log stream buffer

* debug to log stream buffer

* faster streaming printing

* mark charts and dimensions as collected

* use stream points even if sender is not enabled

* comment out stream debug log

* parse null as nan

* custom begin v2

* custom set v2; replication now copies the anomalous flag too

* custom end v2

* enabled stream log test

* renamed to BEGIN2, SET2, END2

* dont mix up replay and v2 members in user object

* fix typo

* cleanup

* support to v2 to v1 proxying

* mark updated dimensions as such

* do not log unknown flags

* comment out stream debug log

* send also the chart id on BEGIN2, v2 to v2

* update the data collections counter

* v2 values are transferred in hex

* faster hex parsing

* a little more generic hex and dec printing and parsing

* fix hex parsing

* minor optimization in dbengine api

* turn debugging into info message

* generalized the timings tracking, so that it can be used in more places

* commented out debug info

* renamed conflicting variable with macro

* remove wrong edits

* integrated ML and added cleanup in case parsing is interrupted

* disable data collection locking during v2

* cleanup stale ML locks; send updated chart variables during v2; add info to find stale locks

* inject an END2 between repeated BEGIN2 from rrdset_done()

* test: remove lockless single-threaded logic from dictionary and aral and apply the right acquire/release memory order to reference counters

* more fine grained dictionary atomics

* remove unecessary return values

* pointer validation under NETDATA_DICTIONARY_VALIDATE_POINTERS

* Revert "pointer validation under NETDATA_DICTIONARY_VALIDATE_POINTERS"

This reverts commit 846cdf2713e2a7ee2ff797f38db11714228800e9.

* Revert "remove unecessary return values"

This reverts commit 8c87d30f4d86f0f5d6b4562cf74fe7447138bbff.

* Revert "more fine grained dictionary atomics"

This reverts commit 984aec4234a340d197d45239ff9a10fd479fcf3c.

* Revert "test: remove lockless single-threaded logic from dictionary and aral and apply the right acquire/release memory order to reference counters"

This reverts commit c460b3d0ad497d2641bd0ea1d63cec7c052e74e4.

* Apply again "pointer validation under NETDATA_DICTIONARY_VALIDATE_POINTERS" while keeping the improved atomic operations.

This reverts commit f158d009

* fix last commit

* fix last commit again

* optimizations in dbengine

* do not send anomaly bit on non-supporting agents (send it when the INTERPOLATED capability is available)

* break long empty-points-loops in rrdset_done()

* decide page alignment on new page allocation, not on every point collected

* create max size pages but no smaller than 1/3

* Fix compilation when --disable-ml is specified

* Return false

* fixes for NETDATA_LOG_REPLICATION_REQUESTS

* added compile option NETDATA_WITHOUT_WORKERS_LATENCY

* put timings in BEGIN2, SET2, END2

* isolate begin2 ml

* revert repositioning data collection lock

* fixed multi-threading of statistics

* do not lookup dimensions all the time if they come in the same order

* update used on iteration, not on every points; also do better error handling

---------

Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>

1 files changed, 54 insertions, 0 deletions

diff --git a/libnetdata/libnetdata.h b/libnetdata/libnetdata.h
index c504bd4bdb..8d50a55c57 100644
--- a/libnetdata/libnetdata.h
+++ b/libnetdata/libnetdata.h
@@ -661,6 +661,60 @@ static inline size_t indexing_partition(Word_t ptr, Word_t modulo) {
     }
 }
 
+typedef enum {
+    TIMING_STEP_INTERNAL = 0,
+
+    TIMING_STEP_BEGIN2_PREPARE,
+    TIMING_STEP_BEGIN2_FIND_CHART,
+    TIMING_STEP_BEGIN2_PARSE,
+    TIMING_STEP_BEGIN2_ML,
+    TIMING_STEP_BEGIN2_PROPAGATE,
+    TIMING_STEP_BEGIN2_STORE,
+
+    TIMING_STEP_SET2_PREPARE,
+    TIMING_STEP_SET2_LOOKUP_DIMENSION,
+    TIMING_STEP_SET2_PARSE,
+    TIMING_STEP_SET2_ML,
+    TIMING_STEP_SET2_PROPAGATE,
+    TIMING_STEP_RRDSET_STORE_METRIC,
+    TIMING_STEP_DBENGINE_FIRST_CHECK,
+    TIMING_STEP_DBENGINE_CHECK_DATA,
+    TIMING_STEP_DBENGINE_PACK,
+    TIMING_STEP_DBENGINE_PAGE_FIN,
+    TIMING_STEP_DBENGINE_MRG_UPDATE,
+    TIMING_STEP_DBENGINE_PAGE_ALLOC,
+    TIMING_STEP_DBENGINE_CREATE_NEW_PAGE,
+    TIMING_STEP_DBENGINE_FLUSH_PAGE,
+    TIMING_STEP_SET2_STORE,
+
+    TIMING_STEP_END2_PREPARE,
+    TIMING_STEP_END2_PUSH_V1,
+    TIMING_STEP_END2_ML,
+    TIMING_STEP_END2_RRDSET,
+    TIMING_STEP_END2_PROPAGATE,
+    TIMING_STEP_END2_STORE,
+
+    // terminator
+    TIMING_STEP_MAX,
+} TIMING_STEP;
+
+typedef enum {
+    TIMING_ACTION_INIT,
+    TIMING_ACTION_STEP,
+    TIMING_ACTION_FINISH,
+} TIMING_ACTION;
+
+#ifdef NETDATA_TIMING_REPORT
+#define timing_init() timing_action(TIMING_ACTION_INIT, TIMING_STEP_INTERNAL)
+#define timing_step(step) timing_action(TIMING_ACTION_STEP, step)
+#define timing_report() timing_action(TIMING_ACTION_FINISH, TIMING_STEP_INTERNAL)
+#else
+#define timing_init() debug_dummy()
+#define timing_step(step) debug_dummy()
+#define timing_report() debug_dummy()
+#endif
+void timing_action(TIMING_ACTION action, TIMING_STEP step);
+
 # ifdef __cplusplus
 }
 # endif