New logging layer (#16357)

* cleanup of logging - wip

* first working iteration

* add errno annotator

* replace old logging functions with netdata_logger()

* cleanup

* update error_limit

* fix remanining error_limit references

* work on fatal()

* started working on structured logs

* full cleanup

* default logging to files; fix all plugins initialization

* fix formatting of numbers

* cleanup and reorg

* fix coverity issues

* cleanup obsolete code

* fix formatting of numbers

* fix log rotation

* fix for older systems

* add detection of systemd journal via stderr

* finished on access.log

* remove left-over transport

* do not add empty fields to the logs

* journal get compact uuids; X-Transaction-ID header is added in web responses

* allow compiling on systems without memfd sealing

* added libnetdata/uuid directory

* move datetime formatters to libnetdata

* add missing files

* link the makefiles in libnetdata

* added uuid_parse_flexi() to parse UUIDs with and without hyphens; the web server now read X-Transaction-ID and uses it for functions and web responses

* added stream receiver, sender, proc plugin and pluginsd log stack

* iso8601 advanced usage; line_splitter module in libnetdata; code cleanup

* add message ids to streaming inbound and outbound connections

* cleanup line_splitter between lines to avoid logging garbage; when killing children, kill them with SIGABRT if internal checks is enabled

* send SIGABRT to external plugins only if we are not shutting down

* fix cross cleanup in pluginsd parser

* fatal when there is a stack error in logs

* compile netdata with -fexceptions

* do not kill external plugins with SIGABRT

* metasync info logs to debug level

* added severity to logs

* added json output; added options per log output; added documentation; fixed issues mentioned

* allow memfd only on linux

* moved journal low level functions to journal.c/h

* move health logs to daemon.log with proper priorities

* fixed a couple of bugs; health log in journal

* updated docs

* systemd-cat-native command to push structured logs to journal from the command line

* fix makefiles

* restored NETDATA_LOG_SEVERITY_LEVEL

* fix makefiles

* systemd-cat-native can also work as the logger of Netdata scripts

* do not require a socket to systemd-journal to log-as-netdata

* alarm notify logs in native format

* properly compare log ids

* fatals log alerts; alarm-notify.sh working

* fix overflow warning

* alarm-notify.sh now logs the request (command line)

* anotate external plugins logs with the function cmd they run

* added context, component and type to alarm-notify.sh; shell sanitization removes control character and characters that may be expanded by bash

* reformatted alarm-notify logs

* unify cgroup-network-helper.sh

* added quotes around params

* charts.d.plugin switched logging to journal native

* quotes for logfmt

* unify the status codes of streaming receivers and senders

* alarm-notify: dont log anything, if there is nothing to do

* all external plugins log to stderr when running outside netdata; alarm-notify now shows an error when notifications menthod are needed but are not available

* migrate cgroup-name.sh to new logging

* systemd-cat-native now supports messages with newlines

* socket.c logs use priority

* cleanup log field types

* inherit the systemd set INVOCATION_ID if found

* allow systemd-cat-native to send messages to a systemd-journal-remote URL

* log2journal command that can convert structured logs to journal export format

* various fixes and documentation of log2journal

* updated log2journal docs

* updated log2journal docs

* updated documentation of fields

* allow compiling without libcurl

* do not use socket as format string

* added version information to newly added tools

* updated documentation and help messages

* fix the namespace socket path

* print errno with error

* do not timeout

* updated docs

* updated docs

* updated docs

* log2journal updated docs and params

* when talking to a remote journal, systemd-cat-native batches the messages

* enable lz4 compression for systemd-cat-native when sending messages to a systemd-journal-remote

* Revert "enable lz4 compression for systemd-cat-native when sending messages to a systemd-journal-remote"

This reverts commit b079d53c11f6687cd64d804fdd7b24c0492bf245.

* note about uncompressed traffic

* log2journal: code reorg and cleanup to make modular

* finished rewriting log2journal

* more comments

* rewriting rules support

* increased limits

* updated docs

* updated docs

* fix old log call

* use journal only when stderr is connected to journal

* update netdata.spec for libcurl, libpcre2 and log2journal

* pcre2-devel

* do not require pcre2 in centos < 8, amazonlinux < 2023, open suse

* log2journal only on systems pcre2 is available

* ignore log2journal in .gitignore

* avoid log2journal on centos 7, amazonlinux 2 and opensuse

* add pcre2-8 to static build

* undo last commit

* Bundle to static

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

* Add build deps for deb packages

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

* Add dependencies; build from source

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

* Test build for amazon linux and centos expect to fail for suse

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

* fix minor oversight

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

* Reorg code

* Add the install from source (deps) as a TODO
* Not enable the build on suse ecosystem

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

---------

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>
Co-authored-by: Tasos Katsoulas <tasos@netdata.cloud>

1 files changed, 179 insertions, 0 deletions

diff --git a/libnetdata/uuid/uuid.c b/libnetdata/uuid/uuid.c
new file mode 100644
index 0000000000..55b66db9b3
--- /dev/null
+++ b/libnetdata/uuid/uuid.c
@@ -0,0 +1,179 @@
+// SPDX-License-Identifier: GPL-3.0-or-later
+
+#include "../libnetdata.h"
+
+void uuid_unparse_lower_compact(const uuid_t uuid, char *out) {
+    static const char *hex_chars = "0123456789abcdef";
+    for (int i = 0; i < 16; i++) {
+        out[i * 2] = hex_chars[(uuid[i] >> 4) & 0x0F];
+        out[i * 2 + 1] = hex_chars[uuid[i] & 0x0F];
+    }
+    out[32] = '\0'; // Null-terminate the string
+}
+
+inline int uuid_parse_compact(const char *in, uuid_t uuid) {
+    if (strlen(in) != 32)
+        return -1; // Invalid input length
+
+    for (int i = 0; i < 16; i++) {
+        int high = hex_char_to_int(in[i * 2]);
+        int low = hex_char_to_int(in[i * 2 + 1]);
+
+        if (high < 0 || low < 0)
+            return -1; // Invalid hexadecimal character
+
+        uuid[i] = (high << 4) | low;
+    }
+
+    return 0; // Success
+}
+
+int uuid_parse_flexi(const char *in, uuid_t uu) {
+    if(!in || !*in)
+        return -1;
+
+    size_t hexCharCount = 0;
+    size_t hyphenCount = 0;
+    const char *s = in;
+    int byteIndex = 0;
+    uuid_t uuid; // work on a temporary place, to not corrupt the previous value of uu if we fail
+
+    while (*s && byteIndex < 16) {
+        if (*s == '-') {
+            s++;
+            hyphenCount++;
+
+            if (unlikely(hyphenCount > 4))
+                // Too many hyphens
+                return -2;
+        }
+
+        if (likely(isxdigit(*s))) {
+            int high = hex_char_to_int(*s++);
+            hexCharCount++;
+
+            if (likely(isxdigit(*s))) {
+                int low = hex_char_to_int(*s++);
+                hexCharCount++;
+
+                uuid[byteIndex++] = (high << 4) | low;
+            }
+            else
+                // Not a valid UUID (expected a pair of hex digits)
+                return -3;
+        }
+        else
+            // Not a valid UUID
+            return -4;
+    }
+
+    if (unlikely(byteIndex < 16))
+        // Not enough data to form a UUID
+        return -5;
+
+    if (unlikely(hexCharCount != 32))
+        // wrong number of hex digits
+        return -6;
+
+    if(unlikely(hyphenCount != 0 && hyphenCount != 4))
+        // wrong number of hyphens
+        return -7;
+
+    // copy the final value
+    memcpy(uu, uuid, sizeof(uuid_t));
+
+    return 0;
+}
+
+
+// ----------------------------------------------------------------------------
+// unit test
+
+static inline void remove_hyphens(const char *uuid_with_hyphens, char *uuid_without_hyphens) {
+    while (*uuid_with_hyphens) {
+        if (*uuid_with_hyphens != '-') {
+            *uuid_without_hyphens++ = *uuid_with_hyphens;
+        }
+        uuid_with_hyphens++;
+    }
+    *uuid_without_hyphens = '\0';
+}
+
+int uuid_unittest(void) {
+    const int num_tests = 100000;
+    int failed_tests = 0;
+
+    int i;
+    for (i = 0; i < num_tests; i++) {
+        uuid_t original_uuid, parsed_uuid;
+        char uuid_str_with_hyphens[UUID_STR_LEN], uuid_str_without_hyphens[UUID_COMPACT_STR_LEN];
+
+        // Generate a random UUID
+        switch(i % 2) {
+            case 0:
+                uuid_generate(original_uuid);
+                break;
+
+            case 1:
+                uuid_generate_random(original_uuid);
+                break;
+        }
+
+        // Unparse it with hyphens
+        bool lower = false;
+        switch(i % 3) {
+            case 0:
+                uuid_unparse_lower(original_uuid, uuid_str_with_hyphens);
+                lower = true;
+                break;
+
+            case 1:
+                uuid_unparse(original_uuid, uuid_str_with_hyphens);
+                break;
+
+            case 2:
+                uuid_unparse_upper(original_uuid, uuid_str_with_hyphens);
+                break;
+        }
+
+        // Remove the hyphens
+        remove_hyphens(uuid_str_with_hyphens, uuid_str_without_hyphens);
+
+        if(lower) {
+            char test[UUID_COMPACT_STR_LEN];
+            uuid_unparse_lower_compact(original_uuid, test);
+            if(strcmp(test, uuid_str_without_hyphens) != 0) {
+                printf("uuid_unparse_lower_compact() failed, expected '%s', got '%s'\n",
+                       uuid_str_without_hyphens, test);
+                failed_tests++;
+            }
+        }
+
+        // Parse the UUID string with hyphens
+        int parse_result = uuid_parse_flexi(uuid_str_with_hyphens, parsed_uuid);
+        if (parse_result != 0) {
+            printf("uuid_parse_flexi() returned -1 (parsing error) for UUID with hyphens: %s\n", uuid_str_with_hyphens);
+            failed_tests++;
+        } else if (uuid_compare(original_uuid, parsed_uuid) != 0) {
+            printf("uuid_parse_flexi() parsed value mismatch for UUID with hyphens: %s\n", uuid_str_with_hyphens);
+            failed_tests++;
+        }
+
+        // Parse the UUID string without hyphens
+        parse_result = uuid_parse_flexi(uuid_str_without_hyphens, parsed_uuid);
+        if (parse_result != 0) {
+            printf("uuid_parse_flexi() returned -1 (parsing error) for UUID without hyphens: %s\n", uuid_str_without_hyphens);
+            failed_tests++;
+        }
+        else if(uuid_compare(original_uuid, parsed_uuid) != 0) {
+            printf("uuid_parse_flexi() parsed value mismatch for UUID without hyphens: %s\n", uuid_str_without_hyphens);
+            failed_tests++;
+        }
+
+        if(failed_tests)
+            break;
+    }
+
+    printf("UUID: failed %d out of %d tests.\n", failed_tests, i);
+    return failed_tests;
+}