New logging layer (#16357)

* cleanup of logging - wip

* first working iteration

* add errno annotator

* replace old logging functions with netdata_logger()

* cleanup

* update error_limit

* fix remanining error_limit references

* work on fatal()

* started working on structured logs

* full cleanup

* default logging to files; fix all plugins initialization

* fix formatting of numbers

* cleanup and reorg

* fix coverity issues

* cleanup obsolete code

* fix formatting of numbers

* fix log rotation

* fix for older systems

* add detection of systemd journal via stderr

* finished on access.log

* remove left-over transport

* do not add empty fields to the logs

* journal get compact uuids; X-Transaction-ID header is added in web responses

* allow compiling on systems without memfd sealing

* added libnetdata/uuid directory

* move datetime formatters to libnetdata

* add missing files

* link the makefiles in libnetdata

* added uuid_parse_flexi() to parse UUIDs with and without hyphens; the web server now read X-Transaction-ID and uses it for functions and web responses

* added stream receiver, sender, proc plugin and pluginsd log stack

* iso8601 advanced usage; line_splitter module in libnetdata; code cleanup

* add message ids to streaming inbound and outbound connections

* cleanup line_splitter between lines to avoid logging garbage; when killing children, kill them with SIGABRT if internal checks is enabled

* send SIGABRT to external plugins only if we are not shutting down

* fix cross cleanup in pluginsd parser

* fatal when there is a stack error in logs

* compile netdata with -fexceptions

* do not kill external plugins with SIGABRT

* metasync info logs to debug level

* added severity to logs

* added json output; added options per log output; added documentation; fixed issues mentioned

* allow memfd only on linux

* moved journal low level functions to journal.c/h

* move health logs to daemon.log with proper priorities

* fixed a couple of bugs; health log in journal

* updated docs

* systemd-cat-native command to push structured logs to journal from the command line

* fix makefiles

* restored NETDATA_LOG_SEVERITY_LEVEL

* fix makefiles

* systemd-cat-native can also work as the logger of Netdata scripts

* do not require a socket to systemd-journal to log-as-netdata

* alarm notify logs in native format

* properly compare log ids

* fatals log alerts; alarm-notify.sh working

* fix overflow warning

* alarm-notify.sh now logs the request (command line)

* anotate external plugins logs with the function cmd they run

* added context, component and type to alarm-notify.sh; shell sanitization removes control character and characters that may be expanded by bash

* reformatted alarm-notify logs

* unify cgroup-network-helper.sh

* added quotes around params

* charts.d.plugin switched logging to journal native

* quotes for logfmt

* unify the status codes of streaming receivers and senders

* alarm-notify: dont log anything, if there is nothing to do

* all external plugins log to stderr when running outside netdata; alarm-notify now shows an error when notifications menthod are needed but are not available

* migrate cgroup-name.sh to new logging

* systemd-cat-native now supports messages with newlines

* socket.c logs use priority

* cleanup log field types

* inherit the systemd set INVOCATION_ID if found

* allow systemd-cat-native to send messages to a systemd-journal-remote URL

* log2journal command that can convert structured logs to journal export format

* various fixes and documentation of log2journal

* updated log2journal docs

* updated log2journal docs

* updated documentation of fields

* allow compiling without libcurl

* do not use socket as format string

* added version information to newly added tools

* updated documentation and help messages

* fix the namespace socket path

* print errno with error

* do not timeout

* updated docs

* updated docs

* updated docs

* log2journal updated docs and params

* when talking to a remote journal, systemd-cat-native batches the messages

* enable lz4 compression for systemd-cat-native when sending messages to a systemd-journal-remote

* Revert "enable lz4 compression for systemd-cat-native when sending messages to a systemd-journal-remote"

This reverts commit b079d53c11f6687cd64d804fdd7b24c0492bf245.

* note about uncompressed traffic

* log2journal: code reorg and cleanup to make modular

* finished rewriting log2journal

* more comments

* rewriting rules support

* increased limits

* updated docs

* updated docs

* fix old log call

* use journal only when stderr is connected to journal

* update netdata.spec for libcurl, libpcre2 and log2journal

* pcre2-devel

* do not require pcre2 in centos < 8, amazonlinux < 2023, open suse

* log2journal only on systems pcre2 is available

* ignore log2journal in .gitignore

* avoid log2journal on centos 7, amazonlinux 2 and opensuse

* add pcre2-8 to static build

* undo last commit

* Bundle to static

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

* Add build deps for deb packages

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

* Add dependencies; build from source

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

* Test build for amazon linux and centos expect to fail for suse

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

* fix minor oversight

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

* Reorg code

* Add the install from source (deps) as a TODO
* Not enable the build on suse ecosystem

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>

---------

Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud>
Co-authored-by: Tasos Katsoulas <tasos@netdata.cloud>

1 files changed, 103 insertions, 46 deletions

diff --git a/collectors/charts.d.plugin/charts.d.plugin.in b/collectors/charts.d.plugin/charts.d.plugin.in
index 34a5a656e3..e7906b5c75 100755
--- a/collectors/charts.d.plugin/charts.d.plugin.in
+++ b/collectors/charts.d.plugin/charts.d.plugin.in
@@ -16,24 +16,111 @@
 export PATH="${PATH}:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin"
 
 PROGRAM_FILE="$0"
-PROGRAM_NAME="$(basename $0)"
-PROGRAM_NAME="${PROGRAM_NAME/.plugin/}"
 MODULE_NAME="main"
 
-LOG_LEVEL_ERR=1
-LOG_LEVEL_WARN=2
-LOG_LEVEL_INFO=3
-LOG_LEVEL="$LOG_LEVEL_INFO"
-
-set_log_severity_level() {
-  case ${NETDATA_LOG_SEVERITY_LEVEL,,} in
-    "info") LOG_LEVEL="$LOG_LEVEL_INFO";;
-    "warn" | "warning") LOG_LEVEL="$LOG_LEVEL_WARN";;
-    "err" | "error") LOG_LEVEL="$LOG_LEVEL_ERR";;
+# -----------------------------------------------------------------------------
+# logging
+
+PROGRAM_NAME="$(basename "${0}")"
+
+# these should be the same with syslog() priorities
+NDLP_EMERG=0   # system is unusable
+NDLP_ALERT=1   # action must be taken immediately
+NDLP_CRIT=2    # critical conditions
+NDLP_ERR=3     # error conditions
+NDLP_WARN=4    # warning conditions
+NDLP_NOTICE=5  # normal but significant condition
+NDLP_INFO=6    # informational
+NDLP_DEBUG=7   # debug-level messages
+
+# the max (numerically) log level we will log
+LOG_LEVEL=$NDLP_INFO
+
+set_log_min_priority() {
+  case "${NETDATA_LOG_PRIORITY_LEVEL,,}" in
+    "emerg" | "emergency")
+      LOG_LEVEL=$NDLP_EMERG
+      ;;
+
+    "alert")
+      LOG_LEVEL=$NDLP_ALERT
+      ;;
+
+    "crit" | "critical")
+      LOG_LEVEL=$NDLP_CRIT
+      ;;
+
+    "err" | "error")
+      LOG_LEVEL=$NDLP_ERR
+      ;;
+
+    "warn" | "warning")
+      LOG_LEVEL=$NDLP_WARN
+      ;;
+
+    "notice")
+      LOG_LEVEL=$NDLP_NOTICE
+      ;;
+
+    "info")
+      LOG_LEVEL=$NDLP_INFO
+      ;;
+
+    "debug")
+      LOG_LEVEL=$NDLP_DEBUG
+      ;;
   esac
 }
 
-set_log_severity_level
+set_log_min_priority
+
+log() {
+  local level="${1}"
+  shift 1
+
+  [[ -n "$level" && -n "$LOG_LEVEL" && "$level" -gt "$LOG_LEVEL" ]] && return
+
+  systemd-cat-native --log-as-netdata <<EOFLOG
+INVOCATION_ID=${NETDATA_INVOCATION_ID}
+SYSLOG_IDENTIFIER=${PROGRAM_NAME}
+PRIORITY=${level}
+THREAD_TAG="charts.d.plugin"
+ND_LOG_SOURCE=collector
+MESSAGE=${MODULE_NAME}: ${*//[$'\r\n']}
+
+EOFLOG
+  # AN EMPTY LINE IS NEEDED ABOVE
+}
+
+info() {
+  log "$NDLP_INFO" "${@}"
+}
+
+warning() {
+  log "$NDLP_WARN" "${@}"
+}
+
+error() {
+  log "$NDLP_ERR" "${@}"
+}
+
+fatal() {
+  log "$NDLP_ALERT" "${@}"
+	echo "DISABLE"
+  exit 1
+}
+
+debug() {
+  [ "$debug" = "1" ] && log "$NDLP_DEBUG" "${@}"
+}
+
+# -----------------------------------------------------------------------------
+# check for BASH v4+ (required for associative arrays)
+
+if [ ${BASH_VERSINFO[0]} -lt 4 ]; then
+  echo >&2 "BASH version 4 or later is required (this is ${BASH_VERSION})."
+  exit 1
+fi
 
 # -----------------------------------------------------------------------------
 # create temp dir
@@ -62,39 +149,6 @@ logdate() {
 	date "+%Y-%m-%d %H:%M:%S"
 }
 
-log() {
-	local status="${1}"
-	shift
-
-	echo >&2 "$(logdate): ${PROGRAM_NAME}: ${status}: ${MODULE_NAME}: ${*}"
-
-}
-
-info() {
-  [[ -n "$LOG_LEVEL" && "$LOG_LEVEL_INFO" -gt "$LOG_LEVEL" ]] && return
-  log INFO "${@}"
-}
-
-warning() {
-  [[ -n "$LOG_LEVEL" && "$LOG_LEVEL_WARN" -gt "$LOG_LEVEL" ]] && return
-  log WARNING "${@}"
-}
-
-error() {
-  [[ -n "$LOG_LEVEL" && "$LOG_LEVEL_ERR" -gt "$LOG_LEVEL" ]] && return
-  log ERROR "${@}"
-}
-
-fatal() {
-	log FATAL "${@}"
-	echo "DISABLE"
-	exit 1
-}
-
-debug() {
-	[ $debug -eq 1 ] && log DEBUG "${@}"
-}
-
 # -----------------------------------------------------------------------------
 # check a few commands
 
@@ -194,12 +248,14 @@ while [ ! -z "$1" ]; do
 
 	if [ "$1" = "debug" -o "$1" = "all" ]; then
 		debug=1
+		LOG_LEVEL=$NDLP_DEBUG
 		shift
 		continue
 	fi
 
 	if [ -f "$chartsd/$1.chart.sh" ]; then
 		debug=1
+		LOG_LEVEL=$NDLP_DEBUG
 		chart_only="$(echo $1.chart.sh | sed "s/\.chart\.sh$//g")"
 		shift
 		continue
@@ -207,6 +263,7 @@ while [ ! -z "$1" ]; do
 
 	if [ -f "$chartsd/$1" ]; then
 		debug=1
+		LOG_LEVEL=$NDLP_DEBUG
 		chart_only="$(echo $1 | sed "s/\.chart\.sh$//g")"
 		shift
 		continue