New logging layer (#16357)

* cleanup of logging - wip

* first working iteration

* add errno annotator

* replace old logging functions with netdata_logger()

* cleanup

* update error_limit

* fix remanining error_limit references

* work on fatal()

* started working on structured logs

* full cleanup

* default logging to files; fix all plugins initialization

* fix formatting of numbers

* cleanup and reorg

* fix coverity issues

* cleanup obsolete code

* fix formatting of numbers

* fix log rotation

* fix for older systems

* add detection of systemd journal via stderr

* finished on access.log

* remove left-over transport

* do not add empty fields to the logs

* journal get compact uuids; X-Transaction-ID header is added in web responses

* allow compiling on systems without memfd sealing

* added libnetdata/uuid directory

* move datetime formatters to libnetdata

* add missing files

* link the makefiles in libnetdata

* added uuid_parse_flexi() to parse UUIDs with and without hyphens; the web server now read X-Transaction-ID and uses it for functions and web responses

* added stream receiver, sender, proc plugin and pluginsd log stack

* iso8601 advanced usage; line_splitter module in libnetdata; code cleanup

* add message ids to streaming inbound and outbound connections

* cleanup line_splitter between lines to avoid logging garbage; when killing children, kill them with SIGABRT if internal checks is enabled

* send SIGABRT to external plugins only if we are not shutting down

* fix cross cleanup in pluginsd parser

* fatal when there is a stack error in logs

* compile netdata with -fexceptions

* do not kill external plugins with SIGABRT

* metasync info logs to debug level

* added severity to logs

* added json output; added options per log output; added documentation; fixed issues mentioned

* allow memfd only on linux

* moved journal low level functions to journal.c/h

* move health logs to daemon.log with proper priorities

* fixed a couple of bugs; health log in journal

* updated docs

* systemd-cat-native command to push structured logs to journal from the command line

* fix makefiles

* restored NETDATA_LOG_SEVERITY_LEVEL

* fix makefiles

* systemd-cat-native can also work as the logger of Netdata scripts

* do not require a socket to systemd-journal to log-as-netdata

* alarm notify logs in native format

* properly compare log ids

* fatals log alerts; alarm-notify.sh working

* fix overflow warning

* alarm-notify.sh now logs the request (command line)

* anotate external plugins logs with the function cmd they run

* added context, component and type to alarm-notify.sh; shell sanitization removes control character and characters that may be expanded by bash

* reformatted alarm-notify logs

* unify cgroup-network-helper.sh

* added quotes around params

* charts.d.plugin switched logging to journal native

* quotes for logfmt

* unify the status codes of streaming receivers and senders

* alarm-notify: dont log anything, if there is nothing to do

* all external plugins log to stderr when running outside netdata; alarm-notify now shows an error when notifications menthod are needed but are not available

* migrate cgroup-name.sh to new logging

* systemd-cat-native now supports messages with newlines

* socket.c logs use priority

* cleanup log field types

* inherit the systemd set INVOCATION_ID if found

* allow systemd-cat-native to send messages to a systemd-journal-remote URL

* log2journal command that can convert structured logs to journal export format

* various fixes and documentation of log2journal

* updated log2journal docs

* updated log2journal docs

* updated documentation of fields

* allow compiling without libcurl

* do not use socket as format string

* added version information to newly added tools

* updated documentation and help messages

* fix the namespace socket path

* print errno with error

* do not timeout

* updated docs

* updated docs

* updated docs

* log2journal updated docs and params

* when talking to a remote journal, systemd-cat-native batches the messages

* enable lz4 compression for systemd-cat-native when sending messages to a systemd-journal-remote

* Revert "enable lz4 compression for systemd-cat-native when sending messages to a systemd-journal-remote"

This reverts commit b079d53c11f6687cd64d804fdd7b24c0492bf245.

* note about uncompressed traffic

* log2journal: code reorg and cleanup to make modular

* finished rewriting log2journal

* more comments

* rewriting rules support

* increased limits

* updated docs

* updated docs

* fix old log call

* use journal only when stderr is connected to journal

* update netdata.spec for libcurl, libpcre2 and log2journal

* pcre2-devel

* do not require pcre2 in centos < 8, amazonlinux < 2023, open suse

* log2journal only on systems pcre2 is available

* ignore log2journal in .gitignore

* avoid log2journal on centos 7, amazonlinux 2 and opensuse

* add pcre2-8 to static build

* undo last commit

* Bundle to static

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

* Add build deps for deb packages

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

* Add dependencies; build from source

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

* Test build for amazon linux and centos expect to fail for suse

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

* fix minor oversight

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

* Reorg code

* Add the install from source (deps) as a TODO
* Not enable the build on suse ecosystem

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

---------

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>
Co-authored-by: Tasos Katsoulas <tasos@netdata.cloud>

1 files changed, 2096 insertions, 910 deletions

diff --git a/libnetdata/log/log.c b/libnetdata/log/log.c
index 02bb776c5b..b771cf96ea 100644
--- a/libnetdata/log/log.c
+++ b/libnetdata/log/log.c
@@ -1,472 +1,301 @@
 // SPDX-License-Identifier: GPL-3.0-or-later
 
-#include <daemon/main.h>
+#define SD_JOURNAL_SUPPRESS_LOCATION
+
 #include "../libnetdata.h"
+#include <daemon/main.h>
+
+#ifdef __FreeBSD__
+#include <sys/endian.h>
+#endif
+
+#ifdef __APPLE__
+#include <machine/endian.h>
+#endif
 
 #ifdef HAVE_BACKTRACE
 #include <execinfo.h>
 #endif
 
-int web_server_is_multithreaded = 1;
+#ifdef HAVE_SYSTEMD
+#include <systemd/sd-journal.h>
+#endif
+
+#include <syslog.h>
 
 const char *program_name = "";
+
 uint64_t debug_flags = 0;
 
-int access_log_syslog = 0;
-int error_log_syslog = 0;
-int collector_log_syslog = 0;
-int output_log_syslog = 0;  // debug log
-int health_log_syslog = 0;
+#ifdef ENABLE_ACLK
+int aclklog_enabled = 0;
+#endif
+
+// ----------------------------------------------------------------------------
+
+struct nd_log_source;
+static bool nd_log_limit_reached(struct nd_log_source *source);
 
-int stdaccess_fd = -1;
-FILE *stdaccess = NULL;
+// ----------------------------------------------------------------------------
+// logging method
+
+typedef enum  __attribute__((__packed__)) {
+    NDLO_DISABLED = 0,
+    NDLO_DEVNULL,
+    NDLO_DEFAULT,
+    NDLO_JOURNAL,
+    NDLO_SYSLOG,
+    NDLO_STDOUT,
+    NDLO_STDERR,
+    NDLO_FILE,
+} ND_LOG_OUTPUT;
+
+
+static struct {
+    ND_LOG_OUTPUT output;
+    const char *name;
+} nd_log_outputs[] = {
+        { .output = NDLO_DISABLED, .name = "none" },
+        { .output = NDLO_DEVNULL, .name = "/dev/null" },
+        { .output = NDLO_DEFAULT, .name = "default" },
+        { .output = NDLO_JOURNAL, .name = "journal" },
+        { .output = NDLO_SYSLOG, .name = "syslog" },
+        { .output = NDLO_STDOUT, .name = "stdout" },
+        { .output = NDLO_STDERR, .name = "stderr" },
+        { .output = NDLO_FILE, .name = "file" },
+};
+
+static ND_LOG_OUTPUT nd_log_output2id(const char *output) {
+    if(!output || !*output)
+        return NDLO_DEFAULT;
+
+    size_t entries = sizeof(nd_log_outputs) / sizeof(nd_log_outputs[0]);
+    for(size_t i = 0; i < entries ;i++) {
+        if(strcmp(nd_log_outputs[i].name, output) == 0)
+            return nd_log_outputs[i].output;
+    }
 
-int stdhealth_fd = -1;
-FILE *stdhealth = NULL;
+    return NDLO_FILE;
+}
 
-int stdcollector_fd = -1;
-FILE *stderror = NULL;
+static const char *nd_log_id2output(ND_LOG_OUTPUT output) {
+    size_t entries = sizeof(nd_log_outputs) / sizeof(nd_log_outputs[0]);
+    for(size_t i = 0; i < entries ;i++) {
+        if(output == nd_log_outputs[i].output)
+            return nd_log_outputs[i].name;
+    }
 
-const char *stdaccess_filename = NULL;
-const char *stderr_filename = NULL;
-const char *stdout_filename = NULL;
-const char *facility_log = NULL;
-const char *stdhealth_filename = NULL;
-const char *stdcollector_filename = NULL;
+    return "unknown";
+}
 
-netdata_log_level_t global_log_severity_level = NETDATA_LOG_LEVEL_INFO;
+// ----------------------------------------------------------------------------
+// workaround strerror_r()
 
-#ifdef ENABLE_ACLK
-const char *aclklog_filename = NULL;
-int aclklog_fd = -1;
-FILE *aclklog = NULL;
-int aclklog_syslog = 1;
-int aclklog_enabled = 0;
+#if defined(STRERROR_R_CHAR_P)
+// GLIBC version of strerror_r
+static const char *strerror_result(const char *a, const char *b) { (void)b; return a; }
+#elif defined(HAVE_STRERROR_R)
+// POSIX version of strerror_r
+static const char *strerror_result(int a, const char *b) { (void)a; return b; }
+#elif defined(HAVE_C__GENERIC)
+
+// what a trick!
+// http://stackoverflow.com/questions/479207/function-overloading-in-c
+static const char *strerror_result_int(int a, const char *b) { (void)a; return b; }
+static const char *strerror_result_string(const char *a, const char *b) { (void)b; return a; }
+
+#define strerror_result(a, b) _Generic((a), \
+    int: strerror_result_int, \
+    char *: strerror_result_string \
+    )(a, b)
+
+#else
+#error "cannot detect the format of function strerror_r()"
 #endif
 
+static const char *errno2str(int errnum, char *buf, size_t size) {
+    return strerror_result(strerror_r(errnum, buf, size), buf);
+}
+
 // ----------------------------------------------------------------------------
-// Log facility(https://tools.ietf.org/html/rfc5424)
+// facilities
 //
-// The facilities accepted in the Netdata are in according with the following
-// header files for their respective operating system:
-// sys/syslog.h (Linux )
+// sys/syslog.h (Linux)
 // sys/sys/syslog.h (FreeBSD)
 // bsd/sys/syslog.h (darwin-xnu)
 
-#define LOG_AUTH_KEY "auth"
-#define LOG_AUTHPRIV_KEY "authpriv"
-#ifdef __FreeBSD__
-# define LOG_CONSOLE_KEY "console"
-#endif
-#define LOG_CRON_KEY "cron"
-#define LOG_DAEMON_KEY "daemon"
-#define LOG_FTP_KEY "ftp"
-#ifdef __APPLE__
-# define LOG_INSTALL_KEY "install"
-#endif
-#define LOG_KERN_KEY "kern"
-#define LOG_LPR_KEY "lpr"
-#define LOG_MAIL_KEY "mail"
-//#define LOG_INTERNAL_MARK_KEY "mark"
-#ifdef __APPLE__
-# define LOG_NETINFO_KEY "netinfo"
-# define LOG_RAS_KEY "ras"
-# define LOG_REMOTEAUTH_KEY "remoteauth"
-#endif
-#define LOG_NEWS_KEY "news"
-#ifdef __FreeBSD__
-# define LOG_NTP_KEY "ntp"
-#endif
-#define LOG_SECURITY_KEY "security"
-#define LOG_SYSLOG_KEY "syslog"
-#define LOG_USER_KEY "user"
-#define LOG_UUCP_KEY "uucp"
-#ifdef __APPLE__
-# define LOG_LAUNCHD_KEY "launchd"
-#endif
-#define LOG_LOCAL0_KEY "local0"
-#define LOG_LOCAL1_KEY "local1"
-#define LOG_LOCAL2_KEY "local2"
-#define LOG_LOCAL3_KEY "local3"
-#define LOG_LOCAL4_KEY "local4"
-#define LOG_LOCAL5_KEY "local5"
-#define LOG_LOCAL6_KEY "local6"
-#define LOG_LOCAL7_KEY "local7"
-
-static int log_facility_id(const char *facility_name)
-{
-    static int
-        hash_auth = 0,
-        hash_authpriv = 0,
-#ifdef __FreeBSD__
-        hash_console = 0,
-#endif
-        hash_cron = 0,
-        hash_daemon = 0,
-        hash_ftp = 0,
-#ifdef __APPLE__
-        hash_install = 0,
-#endif
-        hash_kern = 0,
-        hash_lpr = 0,
-        hash_mail = 0,
-//      hash_mark = 0,
-#ifdef __APPLE__
-        hash_netinfo = 0,
-        hash_ras = 0,
-        hash_remoteauth = 0,
-#endif
-        hash_news = 0,
-#ifdef __FreeBSD__
-        hash_ntp = 0,
-#endif
-        hash_security = 0,
-        hash_syslog = 0,
-        hash_user = 0,
-        hash_uucp = 0,
-#ifdef __APPLE__
-        hash_launchd = 0,
-#endif
-        hash_local0 = 0,
-        hash_local1 = 0,
-        hash_local2 = 0,
-        hash_local3 = 0,
-        hash_local4 = 0,
-        hash_local5 = 0,
-        hash_local6 = 0,
-        hash_local7 = 0;
-
-    if(unlikely(!hash_auth))
-    {
-        hash_auth = simple_hash(LOG_AUTH_KEY);
-        hash_authpriv = simple_hash(LOG_AUTHPRIV_KEY);
-#ifdef __FreeBSD__
-        hash_console = simple_hash(LOG_CONSOLE_KEY);
-#endif
-        hash_cron = simple_hash(LOG_CRON_KEY);
-        hash_daemon = simple_hash(LOG_DAEMON_KEY);
-        hash_ftp = simple_hash(LOG_FTP_KEY);
-#ifdef __APPLE__
-        hash_install = simple_hash(LOG_INSTALL_KEY);
-#endif
-        hash_kern = simple_hash(LOG_KERN_KEY);
-        hash_lpr = simple_hash(LOG_LPR_KEY);
-        hash_mail = simple_hash(LOG_MAIL_KEY);
-//        hash_mark = simple_uhash();
-#ifdef __APPLE__
-        hash_netinfo = simple_hash(LOG_NETINFO_KEY);
-        hash_ras = simple_hash(LOG_RAS_KEY);
-        hash_remoteauth = simple_hash(LOG_REMOTEAUTH_KEY);
-#endif
-        hash_news = simple_hash(LOG_NEWS_KEY);
-#ifdef __FreeBSD__
-        hash_ntp = simple_hash(LOG_NTP_KEY);
-#endif
-        hash_security = simple_hash(LOG_SECURITY_KEY);
-        hash_syslog = simple_hash(LOG_SYSLOG_KEY);
-        hash_user = simple_hash(LOG_USER_KEY);
-        hash_uucp = simple_hash(LOG_UUCP_KEY);
-#ifdef __APPLE__
-        hash_launchd = simple_hash(LOG_LAUNCHD_KEY);
-#endif
-        hash_local0 = simple_hash(LOG_LOCAL0_KEY);
-        hash_local1 = simple_hash(LOG_LOCAL1_KEY);
-        hash_local2 = simple_hash(LOG_LOCAL2_KEY);
-        hash_local3 = simple_hash(LOG_LOCAL3_KEY);
-        hash_local4 = simple_hash(LOG_LOCAL4_KEY);
-        hash_local5 = simple_hash(LOG_LOCAL5_KEY);
-        hash_local6 = simple_hash(LOG_LOCAL6_KEY);
-        hash_local7 = simple_hash(LOG_LOCAL7_KEY);
-    }
+static struct {
+    int facility;
+    const char *name;
+} nd_log_facilities[] = {
+        { LOG_AUTH, "auth" },
+        { LOG_AUTHPRIV, "authpriv" },
+        { LOG_CRON, "cron" },
+        { LOG_DAEMON, "daemon" },
+        { LOG_FTP, "ftp" },
+        { LOG_KERN, "kern" },
+        { LOG_LPR, "lpr" },
+        { LOG_MAIL, "mail" },
+        { LOG_NEWS, "news" },
+        { LOG_SYSLOG, "syslog" },
+        { LOG_USER, "user" },
+        { LOG_UUCP, "uucp" },
+        { LOG_LOCAL0, "local0" },
+        { LOG_LOCAL1, "local1" },
+        { LOG_LOCAL2, "local2" },
+        { LOG_LOCAL3, "local3" },
+        { LOG_LOCAL4, "local4" },
+        { LOG_LOCAL5, "local5" },
+        { LOG_LOCAL6, "local6" },
+        { LOG_LOCAL7, "local7" },
 
-    int hash = simple_hash(facility_name);
-    if ( hash == hash_auth )
-    {
-        return LOG_AUTH;
-    }
-    else if  ( hash == hash_authpriv )
-    {
-        return LOG_AUTHPRIV;
-    }
 #ifdef __FreeBSD__
-    else if ( hash == hash_console )
-    {
-        return LOG_CONSOLE;
-    }
-#endif
-    else if ( hash == hash_cron )
-    {
-        return LOG_CRON;
-    }
-    else if ( hash == hash_daemon )
-    {
-        return LOG_DAEMON;
-    }
-    else if ( hash == hash_ftp )
-    {
-        return LOG_FTP;
-    }
-#ifdef __APPLE__
-    else if ( hash == hash_install )
-    {
-        return LOG_INSTALL;
-    }
+        { LOG_CONSOLE, "console" },
+        { LOG_NTP, "ntp" },
+
+        // FreeBSD does not consider 'security' as deprecated.
+        { LOG_SECURITY, "security" },
+#else
+        // For all other O/S 'security' is mapped to 'auth'.
+        { LOG_AUTH, "security" },
 #endif
-    else if ( hash == hash_kern )
-    {
-        return LOG_KERN;
-    }
-    else if ( hash == hash_lpr )
-    {
-        return LOG_LPR;
-    }
-    else if ( hash == hash_mail )
-    {
-        return LOG_MAIL;
-    }
-    /*
-    else if ( hash == hash_mark )
-    {
-        //this is internal for all OS
-        return INTERNAL_MARK;
-    }
-    */
+
 #ifdef __APPLE__
-    else if ( hash == hash_netinfo )
-    {
-        return LOG_NETINFO;
-    }
-    else if ( hash == hash_ras )
-    {
-        return LOG_RAS;
-    }
-    else if ( hash == hash_remoteauth )
-    {
-        return LOG_REMOTEAUTH;
-    }
-#endif
-    else if ( hash == hash_news )
-    {
-        return LOG_NEWS;
-    }
-#ifdef __FreeBSD__
-    else if ( hash == hash_ntp )
-    {
-        return LOG_NTP;
-    }
-#endif
-    else if ( hash == hash_security )
-    {
-        //FreeBSD is the unique that does not consider
-        //this facility deprecated. We are keeping
-        //it for other OS while they are kept in their headers.
-#ifdef __FreeBSD__
-        return LOG_SECURITY;
-#else
-        return LOG_AUTH;
+        { LOG_INSTALL, "install" },
+        { LOG_NETINFO, "netinfo" },
+        { LOG_RAS,     "ras" },
+        { LOG_REMOTEAUTH, "remoteauth" },
+        { LOG_LAUNCHD, "launchd" },
+
 #endif
+};
+
+static int nd_log_facility2id(const char *facility) {
+    size_t entries = sizeof(nd_log_facilities) / sizeof(nd_log_facilities[0]);
+    for(size_t i = 0; i < entries ;i++) {
+        if(strcmp(nd_log_facilities[i].name, facility) == 0)
+            return nd_log_facilities[i].facility;
     }
-    else if ( hash == hash_syslog )
-    {
-        return LOG_SYSLOG;
-    }
-    else if ( hash == hash_user )
-    {
-        return LOG_USER;
-    }
-    else if ( hash == hash_uucp )
-    {
-        return LOG_UUCP;
-    }
-    else if ( hash == hash_local0 )
-    {
-        return LOG_LOCAL0;
-    }
-    else if ( hash == hash_local1 )
-    {
-        return LOG_LOCAL1;
-    }
-    else if ( hash == hash_local2 )
-    {
-        return LOG_LOCAL2;
-    }
-    else if ( hash == hash_local3 )
-    {
-        return LOG_LOCAL3;
-    }
-    else if ( hash == hash_local4 )
-    {
-        return LOG_LOCAL4;
-    }
-    else if ( hash == hash_local5 )
-    {
-        return LOG_LOCAL5;
-    }
-    else if ( hash == hash_local6 )
-    {
-        return LOG_LOCAL6;
-    }
-    else if ( hash == hash_local7 )
-    {
-        return LOG_LOCAL7;
+
+    return LOG_DAEMON;
+}
+
+static const char *nd_log_id2facility(int facility) {
+    size_t entries = sizeof(nd_log_facilities) / sizeof(nd_log_facilities[0]);
+    for(size_t i = 0; i < entries ;i++) {
+        if(nd_log_facilities[i].facility == facility)
+            return nd_log_facilities[i].name;
     }
-#ifdef __APPLE__
-    else if ( hash == hash_launchd )
-    {
-        return LOG_LAUNCHD;
+
+    return "daemon";
+}
+
+// ----------------------------------------------------------------------------
+// priorities
+
+static struct {
+    ND_LOG_FIELD_PRIORITY priority;
+    const char *name;
+} nd_log_priorities[] = {
+        { .priority = NDLP_EMERG, .name = "emergency" },
+        { .priority = NDLP_EMERG, .name = "emerg" },
+        { .priority = NDLP_ALERT, .name = "alert" },
+        { .priority = NDLP_CRIT, .name = "critical" },
+        { .priority = NDLP_CRIT, .name = "crit" },
+        { .priority = NDLP_ERR, .name = "error" },
+        { .priority = NDLP_ERR, .name = "err" },
+        { .priority = NDLP_WARNING, .name = "warning" },
+        { .priority = NDLP_WARNING, .name = "warn" },
+        { .priority = NDLP_NOTICE, .name = "notice" },
+        { .priority = NDLP_INFO, .name = NDLP_INFO_STR },
+        { .priority = NDLP_DEBUG, .name = "debug" },
+};
+
+int nd_log_priority2id(const char *priority) {
+    size_t entries = sizeof(nd_log_priorities) / sizeof(nd_log_priorities[0]);
+    for(size_t i = 0; i < entries ;i++) {
+        if(strcmp(nd_log_priorities[i].name, priority) == 0)
+            return nd_log_priorities[i].priority;
     }
-#endif
 
-    return LOG_DAEMON;
+    return NDLP_INFO;
 }
 
-//we do not need to use this now, but I already created this function to be
-//used case necessary.
-/*
-char *log_facility_name(int code)
-{
-    char *defvalue = { "daemon" };
-    switch(code)
-    {
-        case LOG_AUTH:
-            {
-                return "auth";
-            }
-        case LOG_AUTHPRIV:
-            {
-                return "authpriv";
-            }
-#ifdef __FreeBSD__
-        case LOG_CONSOLE:
-            {
-                return "console";
-            }
-#endif
-        case LOG_CRON:
-            {
-                return "cron";
-            }
-        case LOG_DAEMON:
-            {
-                return defvalue;
-            }
-        case LOG_FTP:
-            {
-                return "ftp";
-            }
-#ifdef __APPLE__
-        case LOG_INSTALL:
-            {
-                return "install";
-            }
-#endif
-        case LOG_KERN:
-            {
-                return "kern";
-            }
-        case LOG_LPR:
-            {
-                return "lpr";
-            }
-        case LOG_MAIL:
-            {
-                return "mail";
-            }
-#ifdef __APPLE__
-        case LOG_NETINFO:
-            {
-                return "netinfo" ;
-            }
-        case LOG_RAS:
-            {
-                return "ras";
-            }
-        case LOG_REMOTEAUTH:
-            {
-                return "remoteauth";
-            }
-#endif
-        case LOG_NEWS:
-            {
-                return "news";
-            }
-#ifdef __FreeBSD__
-        case LOG_NTP:
-            {
-                return "ntp" ;
-            }
-        case LOG_SECURITY:
-            {
-                return "security";
-            }
-#endif
-        case LOG_SYSLOG:
-            {
-                return "syslog";
-            }
-        case LOG_USER:
-            {
-                return "user";
-            }
-        case LOG_UUCP:
-            {
-                return "uucp";
-            }
-        case LOG_LOCAL0:
-            {
-                return "local0";
-            }
-        case LOG_LOCAL1:
-            {
-                return "local1";
-            }
-        case LOG_LOCAL2:
-            {
-                return "local2";
-            }
-        case LOG_LOCAL3:
-            {
-                return "local3";
-            }
-        case LOG_LOCAL4:
-            {
-                return "local4" ;
-            }
-        case LOG_LOCAL5:
-            {
-                return "local5";
-            }
-        case LOG_LOCAL6:
-            {
-                return "local6";
-            }
-        case LOG_LOCAL7:
-            {
-                return "local7" ;
-            }
-#ifdef __APPLE__
-        case LOG_LAUNCHD:
-            {
-                return "launchd";
-            }
-#endif
+static const char *nd_log_id2priority(ND_LOG_FIELD_PRIORITY priority) {
+    size_t entries = sizeof(nd_log_priorities) / sizeof(nd_log_priorities[0]);
+    for(size_t i = 0; i < entries ;i++) {
+        if(priority == nd_log_priorities[i].priority)
+            return nd_log_priorities[i].name;
     }
 
-    return defvalue;
+    return NDLP_INFO_STR;
+}
+
+// ----------------------------------------------------------------------------
+// log sources
+
+const char *log_sources_str[] = {
+        [NDLS_UNSET] = "UNSET",
+        [NDLS_ACCESS] = "access",
+        [NDLS_ACLK] = "aclk",
+        [NDLS_COLLECTORS] = "collector",
+        [NDLS_DAEMON] = "daemon",
+        [NDLS_HEALTH] = "health",
+        [NDLS_DEBUG] = "debug",
+};
+
+static const char *nd_log_source2str(ND_LOG_SOURCES source) {
+    size_t entries = sizeof(log_sources_str) / sizeof(log_sources_str[0]);
+    if(source < entries)
+        return log_sources_str[source];
+
+    return "UNKNOWN";
 }
-*/
 
 // ----------------------------------------------------------------------------
+// log output formats
+
+typedef enum __attribute__((__packed__)) {
+    NDLOF_JOURNAL,
+    NDLOF_LOGFMT,
+    NDLOF_JSON,
+} ND_LOG_OUTPUT_FORMAT;
+
+static struct {
+    ND_LOG_OUTPUT_FORMAT format;
+    const char *name;
+} nd_log_formats[] = {
+        { .format = NDLOF_JOURNAL, .name = "journal" },
+        { .format = NDLOF_LOGFMT, .name = "logfmt" },
+        { .format = NDLOF_JSON, .name = "json" },
+};
+
+static ND_LOG_OUTPUT_FORMAT nd_log_format2id(const char *format) {
+    if(!format || !*format)
+        return NDLOF_LOGFMT;
+
+    size_t entries = sizeof(nd_log_formats) / sizeof(nd_log_formats[0]);
+    for(size_t i = 0; i < entries ;i++) {
+        if(strcmp(nd_log_formats[i].name, format) == 0)
+            return nd_log_formats[i].format;
+    }
 
-void syslog_init() {
-    static int i = 0;
+    return NDLOF_LOGFMT;
+}
 
-    if(!i) {
-        openlog(program_name, LOG_PID,log_facility_id(facility_log));
-        i = 1;
+static const char *nd_log_id2format(ND_LOG_OUTPUT_FORMAT format) {
+    size_t entries = sizeof(nd_log_formats) / sizeof(nd_log_formats[0]);
+    for(size_t i = 0; i < entries ;i++) {
+        if(format == nd_log_formats[i].format)
+            return nd_log_formats[i].name;
     }
+
+    return "logfmt";
 }
 
+// ----------------------------------------------------------------------------
+// format dates
+
 void log_date(char *buffer, size_t len, time_t now) {
     if(unlikely(!buffer || !len))
         return;
@@ -476,7 +305,7 @@ void log_date(char *buffer, size_t len, time_t now) {
 
     tmp = localtime_r(&t, &tmbuf);
 
-    if (tmp == NULL) {
+    if (unlikely(!tmp)) {
         buffer[0] = '\0';
         return;
     }
@@ -487,552 +316,1918 @@ void log_date(char *buffer, size_t len, time_t now) {
     buffer[len - 1] = '\0';
 }
 
-static netdata_mutex_t log_mutex = NETDATA_MUTEX_INITIALIZER;
-static inline void log_lock() {
-    netdata_mutex_lock(&log_mutex);
-}
-static inline void log_unlock() {
-    netdata_mutex_unlock(&log_mutex);
-}
+// ----------------------------------------------------------------------------
 
-static FILE *open_log_file(int fd, FILE *fp, const char *filename, int *enabled_syslog, int is_stdaccess, int *fd_ptr) {
-    int f, devnull = 0;
+struct nd_log_limit {
+    usec_t started_monotonic_ut;
+    uint32_t counter;
+    uint32_t prevented;
 
-    if(!filename || !*filename || !strcmp(filename, "none") ||  !strcmp(filename, "/dev/null")) {
-        filename = "/dev/null";
-        devnull = 1;
-    }
+    uint32_t throttle_period;
+    uint32_t logs_per_period;
+    uint32_t logs_per_period_backup;
+};
 
-    if(!strcmp(filename, "syslog")) {
-        filename = "/dev/null";
-        devnull = 1;
+#define ND_LOG_LIMITS_DEFAULT (struct nd_log_limit){ .logs_per_period = ND_LOG_DEFAULT_THROTTLE_LOGS, .logs_per_period_backup = ND_LOG_DEFAULT_THROTTLE_LOGS, .throttle_period = ND_LOG_DEFAULT_THROTTLE_PERIOD, }
+#define ND_LOG_LIMITS_UNLIMITED (struct nd_log_limit){  .logs_per_period = 0, .logs_per_period_backup = 0, .throttle_period = 0, }
 
-    syslog_init();
-        if(enabled_syslog) *enabled_syslog = 1;
-    }
-    else if(enabled_syslog) *enabled_syslog = 0;
-
-    // don't do anything if the user is willing
-    // to have the standard one
-    if(!strcmp(filename, "system")) {
-        if(fd != -1 && !is_stdaccess) {
-            if(fd_ptr) *fd_ptr = fd;
-            return fp;
-        }
+struct nd_log_source {
+    SPINLOCK spinlock;
+    ND_LOG_OUTPUT method;
+    ND_LOG_OUTPUT_FORMAT format;
+    const char *filename;
+    int fd;
+    FILE *fp;
 
-        filename = "stderr";
-    }
+    ND_LOG_FIELD_PRIORITY min_priority;
+    const char *pending_msg;
+    struct nd_log_limit limits;
+};
 
-    if(!strcmp(filename, "stdout"))
-        f = STDOUT_FILENO;
+static __thread ND_LOG_SOURCES overwrite_thread_source = 0;
 
-    else if(!strcmp(filename, "stderr"))
-        f = STDERR_FILENO;
+void nd_log_set_thread_source(ND_LOG_SOURCES source) {
+    overwrite_thread_source = source;
+}
 
-    else {
-        f = open(filename, O_WRONLY | O_APPEND | O_CREAT, 0664);
-        if(f == -1) {
-            netdata_log_error("Cannot open file '%s'. Leaving %d to its default.", filename, fd);
-            if(fd_ptr) *fd_ptr = fd;
-            return fp;
+static struct {
+    uuid_t invocation_id;
+
+    ND_LOG_SOURCES overwrite_process_source;
+
+    struct nd_log_source sources[_NDLS_MAX];
+
+    struct {
+        bool initialized;
+    } journal;
+
+    struct {
+        bool initialized;
+        int fd;
+    } journal_direct;
+
+    struct {
+        bool initialized;
+        int facility;
+    } syslog;
+
+    struct {
+        SPINLOCK spinlock;
+        bool initialized;
+    } std_output;
+
+    struct {
+        SPINLOCK spinlock;
+        bool initialized;
+    } std_error;
+
+} nd_log = {
+        .overwrite_process_source = 0,
+        .journal = {
+                .initialized = false,
+        },
+        .journal_direct = {
+                .initialized = false,
+                .fd = -1,
+        },
+        .syslog = {
+                .initialized = false,
+                .facility = LOG_DAEMON,
+        },
+        .std_output = {
+                .spinlock = NETDATA_SPINLOCK_INITIALIZER,
+                .initialized = false,
+        },
+        .std_error = {
+                .spinlock = NETDATA_SPINLOCK_INITIALIZER,
+                .initialized = false,
+        },
+        .sources = {
+                [NDLS_UNSET] = {
+                        .spinlock = NETDATA_SPINLOCK_INITIALIZER,
+                        .method = NDLO_DISABLED,
+                        .format = NDLOF_JOURNAL,
+                        .filename = NULL,
+                        .fd = -1,
+                        .fp = NULL,
+                        .min_priority = NDLP_EMERG,
+                        .limits = ND_LOG_LIMITS_UNLIMITED,
+                },
+                [NDLS_ACCESS] = {
+                        .spinlock = NETDATA_SPINLOCK_INITIALIZER,
+                        .method = NDLO_DEFAULT,
+                        .format = NDLOF_LOGFMT,
+                        .filename = LOG_DIR "/access.log",
+                        .fd = -1,
+                        .fp = NULL,
+                        .min_priority = NDLP_DEBUG,
+                        .limits = ND_LOG_LIMITS_UNLIMITED,
+                },
+                [NDLS_ACLK] = {
+                        .spinlock = NETDATA_SPINLOCK_INITIALIZER,
+                        .method = NDLO_FILE,
+                        .format = NDLOF_LOGFMT,
+                        .filename = LOG_DIR "/aclk.log",
+                        .fd = -1,
+                        .fp = NULL,
+                        .min_priority = NDLP_DEBUG,
+                        .limits = ND_LOG_LIMITS_UNLIMITED,
+                },
+                [NDLS_COLLECTORS] = {
+                        .spinlock = NETDATA_SPINLOCK_INITIALIZER,
+                        .method = NDLO_DEFAULT,
+                        .format = NDLOF_LOGFMT,
+                        .filename = LOG_DIR "/collectors.log",
+                        .fd = STDERR_FILENO,
+                        .fp = NULL,
+                        .min_priority = NDLP_INFO,
+                        .limits = ND_LOG_LIMITS_DEFAULT,
+                },
+                [NDLS_DEBUG] = {
+                        .spinlock = NETDATA_SPINLOCK_INITIALIZER,
+                        .method = NDLO_DISABLED,
+                        .format = NDLOF_LOGFMT,
+                        .filename = LOG_DIR "/debug.log",
+                        .fd = STDOUT_FILENO,
+                        .fp = NULL,
+                        .min_priority = NDLP_DEBUG,
+                        .limits = ND_LOG_LIMITS_UNLIMITED,
+                },
+                [NDLS_DAEMON] = {
+                        .spinlock = NETDATA_SPINLOCK_INITIALIZER,
+                        .method = NDLO_DEFAULT,
+                        .filename = LOG_DIR "/daemon.log",
+                        .format = NDLOF_LOGFMT,
+                        .fd = -1,
+                        .fp = NULL,
+                        .min_priority = NDLP_INFO,
+                        .limits = ND_LOG_LIMITS_DEFAULT,
+                },
+                [NDLS_HEALTH] = {
+                        .spinlock = NETDATA_SPINLOCK_INITIALIZER,
+                        .method = NDLO_DEFAULT,
+                        .format = NDLOF_LOGFMT,
+                        .filename = LOG_DIR "/health.log",
+                        .fd = -1,
+                        .fp = NULL,
+                        .min_priority = NDLP_DEBUG,
+                        .limits = ND_LOG_LIMITS_UNLIMITED,
+                },
+        },
+};
+
+__attribute__((constructor)) void initialize_invocation_id(void) {
+    // check for a NETDATA_INVOCATION_ID
+    if(uuid_parse_flexi(getenv("NETDATA_INVOCATION_ID"), nd_log.invocation_id) != 0) {
+        // not found, check for systemd set INVOCATION_ID
+        if(uuid_parse_flexi(getenv("INVOCATION_ID"), nd_log.invocation_id) != 0) {
+            // not found, generate a new one
+            uuid_generate_random(nd_log.invocation_id);
         }
     }
 
-    // if there is a level-2 file pointer
-    // flush it before switching the level-1 fds
-    if(fp)
-        fflush(fp);
+    char uuid[UUID_COMPACT_STR_LEN];
+    uuid_unparse_lower_compact(nd_log.invocation_id, uuid);
+    setenv("NETDATA_INVOCATION_ID", uuid, 1);
+}
 
-    if(devnull && is_stdaccess) {
-        fd = -1;
-        fp = NULL;
-    }
+void nd_log_set_user_settings(ND_LOG_SOURCES source, const char *se