DBENGINE v2 - improvements part 11 (#14337)

* acquiring / releasing interface for metrics

* metrics registry statistics

* cleanup metrics registry by deleting metrics when they dont have retention anymore; do not double copy the data of pages to be flushed

* print the tier in retention summary

* Open files with buffered instead of direct I/O (test)

* added more metrics stats and fixed unittest

* rename writer functions to avoid confusion with refcounting

* do not release a metric that is not acquired

* Revert to use direct I/O on write -- use direct I/O on read as well

* keep track of ARAL overhead and add it to the memory chart

* aral full check via api

* Cleanup

* give names to ARALs and PGCs

* aral improvements

* restore query expansion to the future

* prefer higher resolution tier when switching plans

* added extent read statistics

* smoother joining of tiers at query engine

* fine tune aral max allocation size

* aral restructuring to hide its internals from the rest of netdata

* aral restructuring; addtion of defrag option to aral to keep the linked list sorted - enabled by default to test it

* fully async aral

* some statistics and cleanup

* fix infinite loop while calculating retention

* aral docs and defragmenting disabled by default

* fix bug and add optimization when defragmenter is not enabled

* aral stress test

* aral speed report and documentation

* added internal checks that all pages are full

* improve internal log about metrics deletion

* metrics registry uses one aral per partition

* metrics registry aral max size to 512 elements per page

* remove data_structures/README.md dependency

---------

Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>

1 files changed, 0 insertions, 55 deletions

diff --git a/libnetdata/arrayalloc/arrayalloc.h b/libnetdata/arrayalloc/arrayalloc.h
deleted file mode 100644
index de6bd60d74..0000000000
--- a/libnetdata/arrayalloc/arrayalloc.h
+++ /dev/null
@@ -1,55 +0,0 @@
-
-#ifndef ARRAYALLOC_H
-#define ARRAYALLOC_H 1
-
-#include "../libnetdata.h"
-
-typedef struct arrayalloc {
-    size_t requested_element_size;
-    size_t initial_elements;
-    const char *filename;
-    char **cache_dir;
-    bool use_mmap;
-
-    // private members - do not touch
-    struct {
-        bool mmap;
-        bool lockless;
-        bool initialized;
-        size_t element_size;
-        size_t page_ptr_offset;
-        size_t file_number;
-        size_t natural_page_size;
-        size_t allocation_multiplier;
-        size_t max_alloc_size;
-        SPINLOCK spinlock;
-        struct arrayalloc_page *pages;
-    } internal;
-} ARAL;
-
-ARAL *arrayalloc_create(size_t element_size, size_t elements, const char *filename, char **cache_dir, bool mmap, bool lockless);
-int aral_unittest(size_t elements);
-
-#ifdef NETDATA_TRACE_ALLOCATIONS
-
-#define arrayalloc_mallocz(ar) arrayalloc_mallocz_internal(ar, __FILE__, __FUNCTION__, __LINE__)
-#define arrayalloc_freez(ar, ptr) arrayalloc_freez_internal(ar, ptr, __FILE__, __FUNCTION__, __LINE__)
-#define arrayalloc_destroy(ar) arrayalloc_destroy_internal(ar, __FILE__, __FUNCTION__, __LINE__)
-
-void *arrayalloc_mallocz_internal(ARAL *ar, const char *file, const char *function, size_t line);
-void arrayalloc_freez_internal(ARAL *ar, void *ptr, const char *file, const char *function, size_t line);
-void arrayalloc_destroy_internal(ARAL *ar, const char *file, const char *function, size_t line);
-
-#else // NETDATA_TRACE_ALLOCATIONS
-
-#define arrayalloc_mallocz(ar) arrayalloc_mallocz_internal(ar)
-#define arrayalloc_freez(ar, ptr) arrayalloc_freez_internal(ar, ptr)
-#define arrayalloc_destroy(ar) arrayalloc_destroy_internal(ar)
-
-void *arrayalloc_mallocz_internal(ARAL *ar);
-void arrayalloc_freez_internal(ARAL *ar, void *ptr);
-void arrayalloc_destroy_internal(ARAL *ar);
-
-#endif // NETDATA_TRACE_ALLOCATIONS
-
-#endif // ARRAYALLOC_H