Implement the main flow for the Exporting Engine (#7149)

* Add top level tests

* Add a skeleton for preparing buffers

* Initialize graphite instance

* Prepare buffers for all instances

* Add Grafite collected value formatter

* Add support for exporting.conf read and parsing

* - Use new exporting_config instead of netdata_config

* Implement Grafite worker

* Disable exporting engine compilation if libuv is not available

* Add mutex locks

- Configure connectors as connector_<type> in sections of exporting.conf

- Change exporting_select_type to check for connector_ fields

* - Override exporting_config structure if there no exporting.conf so that
  look ups don't fail and we maintain backwards compatibility

* Separate fixtures in unit tests

* Test exporting_discard_responce

* Test response receiving

* Test buffer sending

* Test simple connector worker

- Instance section has the format connector:instance_name
  e.g graphite:my_graphite_instance

- Connectors with : in their name e.g graphite:plaintext are reserved
  So graphite:plaintext is not accepted because it would activate an
  instance with name "plaintext"
  It should be graphite:plaintext:instance_name

* - Enable the add_connector_instance to cleanup the internal structure
  by passing NULL,not NULL arguments

* Implement configurable update interval

- Add additional check to verify instance uniqueness across connectors

* Add host and chart filters

* Add the value calculation over a database series

* Add the calculated over stored data graphite connector

* Add tests for graphite connector

* Add JSON connector

* Add tests for JSON formatting functions

* Add OpenTSDB connector

* Add tests for the OpenTSDB connector

* Add temporaty notes to the documentation

1 files changed, 60 insertions, 0 deletions

diff --git a/exporting/exporting_engine.c b/exporting/exporting_engine.c
new file mode 100644
index 0000000000..24d8756dc2
--- /dev/null
+++ b/exporting/exporting_engine.c
@@ -0,0 +1,60 @@
+// SPDX-License-Identifier: GPL-3.0-or-later
+
+#include "exporting_engine.h"
+
+/**
+ * Exporting engine main
+ *
+ * The main thread used to control the exporting engine.
+ *
+ * @param ptr a pointer to netdata_static_structure.
+ *
+ * @return It always returns NULL.
+ */
+void *exporting_main(void *ptr)
+{
+    (void)ptr;
+
+    struct engine *engine = read_exporting_config();
+    if (!engine) {
+        info("EXPORTING: no exporting connectors configured");
+        return NULL;
+    }
+
+    if (init_connectors(engine) != 0) {
+        error("EXPORTING: cannot initialize exporting connectors");
+        return NULL;
+    }
+
+    usec_t step_ut = localhost->rrd_update_every * USEC_PER_SEC;
+    heartbeat_t hb;
+    heartbeat_init(&hb);
+
+    while (!netdata_exit) {
+        heartbeat_next(&hb, step_ut);
+        engine->now = now_realtime_sec();
+
+        if (mark_scheduled_instances(engine)) {
+            if (prepare_buffers(engine) != 0) {
+                error("EXPORTING: cannot prepare data to send");
+                return NULL;
+            }
+        }
+
+        if (notify_workers(engine) != 0) {
+            error("EXPORTING: cannot communicate with exporting connector instance working threads");
+            return NULL;
+        }
+
+        if (send_internal_metrics(engine) != 0) {
+            error("EXPORTING: cannot send metrics for the operation of exporting engine");
+            return NULL;
+        }
+
+#ifdef UNIT_TESTING
+        break;
+#endif
+    }
+
+    return NULL;
+}