DBENGINE v2 - improvements part 9 (#14326)

* on shutdown stop data collection for all hosts instead of freeing their memory

* print number of sql statements per metadata host scan

* print timings with metadata checking

* use dbengine API to figure out of a database is legacy

* Recalculate retention after a datafile deletion

* validate child timestamps during replication

* main cache uses a lockless aral per partition, protected by the partition index lock

* prevent ML crash

* Revert "main cache uses a lockless aral per partition, protected by the partition index lock"

This reverts commit 6afc01527dc5c66548b4bc8a1d63c026c3149358.

* Log direct index and binary searches

* distribute metrics more evenly across time

* statistics about retention recalculation

* fix crash

* Reverse the binary search to calculate retention

* more optimization on retention calculation

* removed commented old code

Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>

1 files changed, 20 insertions, 4 deletions

diff --git a/database/rrdhost.c b/database/rrdhost.c
index 516d733daa..b25fc72d21 100644
--- a/database/rrdhost.c
+++ b/database/rrdhost.c
@@ -33,10 +33,8 @@ time_t rrdhost_free_orphan_time_s = 3600;
 
 bool is_storage_engine_shared(STORAGE_INSTANCE *engine __maybe_unused) {
 #ifdef ENABLE_DBENGINE
-    for(size_t tier = 0; tier < storage_tiers ;tier++) {
-        if (engine == (STORAGE_INSTANCE *)multidb_ctx[tier])
-            return true;
-    }
+    if(!rrdeng_is_legacy(engine))
+        return true;
 #endif
 
     return false;
@@ -1223,6 +1221,15 @@ void rrdhost_free_all(void) {
     rrd_unlock();
 }
 
+void rrd_finalize_collection_for_all_hosts(void) {
+    RRDHOST *host;
+    rrd_wrlock();
+    rrdhost_foreach_read(host) {
+        rrdhost_finalize_collection(host);
+    }
+    rrd_unlock();
+}
+
 // ----------------------------------------------------------------------------
 // RRDHOST - save host files
 
@@ -1391,6 +1398,15 @@ void reload_host_labels(void) {
     rrdpush_send_host_labels(localhost);
 }
 
+void rrdhost_finalize_collection(RRDHOST *host) {
+    info("Stopping data collection for host '%s'...", rrdhost_hostname(host));
+
+    RRDSET *st;
+    rrdset_foreach_write(st, host)
+        rrdset_finalize_collection(st, true);
+    rrdset_foreach_done(st);
+}
+
 // ----------------------------------------------------------------------------
 // RRDHOST - delete host files