path: root/collectors/log2journal/README.md

# log2journal

`log2journal` and `systemd-cat-native` can be used to convert a structured log file, such as the ones generated by web servers, into `systemd-journal` entries.

By combining these tools you can create advanced log processing pipelines sending any kind of structured text logs to systemd-journald. This is a simple, but powerful and efficient way to handle log processing.

The process involves the usual piping of shell commands, to get and process the log files in realtime.

The overall process looks like this:

```bash
tail -F /var/log/nginx/*.log       |\  # outputs log lines
  log2journal 'PATTERN'            |\  # outputs Journal Export Format
  systemd-cat-native                   # send to local/remote journald
```

Let's see the steps:

1. `tail -F /var/log/nginx/*.log`<br/>this command will tail all `*.log` files in `/var/log/nginx/`. We use `-F` instead of `-f` to ensure that files will still be tailed after log rotation.
2. `log2joural` is a Netdata program. It reads log entries and extracts fields, according to the PCRE2 pattern it accepts. It can also apply some basic operations on the fields, like injecting new fields or duplicating existing ones or rewriting their values. The output of `log2journal` is in Systemd Journal Export Format, and it looks like this:
    ```bash
   KEY1=VALUE1 # << start of the first log line
   KEY2=VALUE2
               # << log lines separator
   KEY1=VALUE1 # << start of the second log line
   KEY2=VALUE2
    ```
3. `systemd-cat-native` is a Netdata program. I can send the logs to a local `systemd-journald` (journal namespaces supported), or to a remote `systemd-journal-remote`.


## YAML configuration



## Real-life example

We have an nginx server logging in this format:

```bash
        log_format access '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '$request_length $request_time '
                    '"$http_referer" "$http_user_agent"';
```

First, let's find the right pattern for `log2journal`. We ask ChatGPT:

```
My nginx log uses this log format:

log_format access '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '$request_length $request_time '
                    '"$http_referer" "$http_user_agent"';

I want to use `log2joural` to convert this log for systemd-journal.
`log2journal` accepts a PCRE2 regular expression, using the named groups
in the pattern as the journal fields to extract from the logs.

Prefix all PCRE2 group names with `NGINX_` and use capital characters only. 

For the $request, use the field `MESSAGE` (without NGINX_ prefix), so that
it will appear in systemd journals as the message of the log.

Please give me the PCRE2 pattern.
```

ChatGPT replies with this:

```regexp
^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>[^"]+)" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"
```

Let's test it with a sample line (instead of `tail`):

```bash
# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 104 0.001 "-" "Go-http-client/1.1"' | log2journal '^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>[^"]+)" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"'
MESSAGE=GET /index.html HTTP/1.1
NGINX_BODY_BYTES_SENT=4172
NGINX_HTTP_REFERER=-
NGINX_HTTP_USER_AGENT=Go-http-client/1.1
NGINX_REMOTE_ADDR=1.2.3.4
NGINX_REMOTE_USER=-
NGINX_REQUEST_LENGTH=104
NGINX_REQUEST_TIME=0.001
NGINX_STATUS=200
NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000

```

As you can see, it extracted all the fields.

The `MESSAGE` however, has 3 fields by itself: the method, the URL and the procotol version. Let's ask ChatGPT to extract these too:

```
I see that the MESSAGE has 3 key items in it. The request method (GET, POST,
etc), the URL and HTTP protocol version.

I want to keep the MESSAGE as it is, with all the information in it, but also
extract the 3 items from it as separate fields.

Can this be done?
```

ChatGPT responded with this:

```regexp
^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>(?<NGINX_METHOD>[A-Z]+) (?<NGINX_URL>[^ ]+) HTTP/(?<NGINX_HTTP_VERSION>[^"]+))" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"
```

Let's test this too:

```bash
# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 104 0.001 "-" "Go-http-client/1.1"' | log2journal '^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>(?<NGINX_METHOD>[A-Z]+) (?<NGINX_URL>[^ ]+) HTTP/(?<NGINX_HTTP_VERSION>[^"]+))" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"'
MESSAGE=GET /index.html HTTP/1.1              # <<<<<<<<< MESSAGE
NGINX_BODY_BYTES_SENT=4172
NGINX_HTTP_REFERER=-
NGINX_HTTP_USER_AGENT=Go-http-client/1.1
NGINX_HTTP_VERSION=1.1                        # <<<<<<<<< VERSION
NGINX_METHOD=GET                              # <<<<<<<<< METHOD
NGINX_REMOTE_ADDR=1.2.3.4
NGINX_REMOTE_USER=-
NGINX_REQUEST_LENGTH=104
NGINX_REQUEST_TIME=0.001
NGINX_STATUS=200
NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000
NGINX_URL=/index.html                         # <<<<<<<<< URL

```

Ideally, we would want the 5xx errors to be red in our `journalctl` output. To achieve that we need to add a PRIORITY field to set the log level. Log priorities are numeric and follow the `syslog` priorities. Checking `/usr/include/sys/syslog.h` we can see these:

```c
#define LOG_EMERG       0       /* system is unusable */
#define LOG_ALERT       1       /* action must be taken immediately */
#define LOG_CRIT        2       /* critical conditions */
#define LOG_ERR         3       /* error conditions */
#define LOG_WARNING     4       /* warning conditions */
#define LOG_NOTICE      5       /* normal but significant condition */
#define LOG_INFO        6       /* informational */
#define LOG_DEBUG       7       /* debug-level messages */
```

Avoid setting priority to 0 (`LOG_EMERG`), because these will be on your terminal (the journal uses `wall` to let you know of such events). A good priority for errors is 3 (red in `journalctl`), or 4 (yellow in `journalctl`).

To set the PRIORITY field in the output, we can use `NGINX_STATUS` fields. We need a copy of it, which we will alter later.

We can instruct `log2journal` to duplicate `NGINX_STATUS`, like this: `log2journal --inject 'PRIORITY=${NGINX_STATUS}'`. Let's try it:

```bash
# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 104 0.001 "-" "Go-http-client/1.1"' | log2journal '^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>(?<NGINX_METHOD>[A-Z]+) (?<NGINX_URL>[^ ]+) HTTP/(?<NGINX_HTTP_VERSION>[^"]+))" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"' --inject 'PRIORITY=${NGINX_STATUS}'
MESSAGE=GET /index.html HTTP/1.1
NGINX_BODY_BYTES_SENT=4172
NGINX_HTTP_REFERER=-
NGINX_HTTP_USER_AGENT=Go-http-client/1.1
NGINX_HTTP_VERSION=1.1
NGINX_METHOD=GET
NGINX_REMOTE_ADDR=1.2.3.4
NGINX_REMOTE_USER=-
NGINX_REQUEST_LENGTH=104
NGINX_REQUEST_TIME=0.001
NGINX_STATUS=200
PRIORITY=200                                 # <<<<<<<<< PRIORITY IS HERE
NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000
NGINX_URL=/index.html

```

Now that we have the `PRIORITY` field equal to the `NGINX_STATUS`, we can use instruct `log2journal` to change it to a valid priority, by appending: `--rewrite 'PRIORITY=/^5/3' --rewrite 'PRIORITY=/.*/6'`. These rewrite commands say to match everything that starts with `5` and replace it with priority `3` (error) and everything else with priority `6` (info). Let's see it:

```bash
# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 104 0.001 "-" "Go-http-client/1.1"' | log2journal '^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>(?<NGINX_METHOD>[A-Z]+) (?<NGINX_URL>[^ ]+) HTTP/(?<NGINX_HTTP_VERSION>[^"]+))" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"' --inject 'PRIORITY=${NGINX_STATUS}' --rewrite 'PRIORITY=/^5/3' --rewrite 'PRIORITY=/.*/6'
MESSAGE=GET /index.html HTTP/1.1
NGINX_BODY_BYTES_SENT=4172
NGINX_HTTP_REFERER=-
NGINX_HTTP_USER_AGENT=Go-http-client/1.1
NGINX_HTTP_VERSION=1.1
NGINX_METHOD=GET
NGINX_REMOTE_ADDR=1.2.3.4
NGINX_REMOTE_USER=-
NGINX_REQUEST_LENGTH=104
NGINX_REQUEST_TIME=0.001
NGINX_STATUS=200
PRIORITY=6                                   # <<<<<<<<<< PRIORITY changed to 6
NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000
NGINX_URL=/index.html

```

Similarly, we could duplicate `${NGINX_URL}` to `NGINX_ENDPOINT` and then process it to remove any query string, or replace IDs in the URL path with constant names, thus giving us uniform endpoints independently of the parameters.

To complete the example, we can also inject a `SYSLOG_IDENTIFIER` with `log2journal`, using `--inject SYSLOG_IDENTIFIER=nginx-log`, like this:

```bash
# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 104 0.001 "-" "Go-http-client/1.1"' | log2journal '^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>(?<NGINX_METHOD>[A-Z]+) (?<NGINX_URL>[^ ]+) HTTP/(?<NGINX_HTTP_VERSION>[^"]+))" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"' --inject 'PRIORITY=${NGINX_STATUS}' --inject 'SYSLOG_IDENTIFIER=nginx' -rewrite 'PRIORITY=/^5/3' --rewrite 'PRIORITY=/.*/6'
MESSAGE=GET /index.html HTTP/1.1
NGINX_BODY_BYTES_SENT=4172
NGINX_HTTP_REFERER=-
NGINX_HTTP_USER_AGENT=Go-http-client/1.1
NGINX_HTTP_VERSION=1.1
NGINX_METHOD=GET
NGINX_REMOTE_ADDR=1.2.3.4
NGINX_REMOTE_USER=-
NGINX_REQUEST_LENGTH=104
NGINX_REQUEST_TIME=0.001
NGINX_STATUS=200
PRIORITY=6
NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000
NGINX_URL=/index.html
SYSLOG_IDENTIFIER=nginx-log               # <<<<<<<<< THIS HAS BEEN ADDED

```

Now the message is ready to be sent to a systemd-journal. For this we use `systemd-cat-native`. This command can send such messages to a journal running on the localhost, a local journal namespace, or a `systemd-journal-remote` running on another server. By just appending `| systemd-cat-native` to the command, the message will be sent to the local journal.


```bash
# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 104 0.001 "-" "Go-http-client/1.1"' | log2journal '^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>(?<NGINX_METHOD>[A-Z]+) (?<NGINX_URL>[^ ]+) HTTP/(?<NGINX_HTTP_VERSION>[^"]+))" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"' --inject 'PRIORITY=${NGINX_STATUS}' --inject 'SYSLOG_IDENTIFIER=nginx' -rewrite 'PRIORITY=/^5/3' --rewrite 'PRIORITY=/.*/6' | systemd-cat-native
# no output

# let's find the message
# journalctl -o verbose SYSLOG_IDENTIFIER=nginx
Sun 2023-11-19 04:34:06.583912 EET [s=1eb59e7934984104ab3b61f5d9648057;i=115b6d4;b=7282d89d2e6e4299969a6030302ff3e4;m=69b419673;t=60a783417ac72;x=2cec5dde8bf01ee7]
    PRIORITY=6
    _UID=0
    _GID=0
    _BOOT_ID=7282d89d2e6e4299969a6030302ff3e4
    _MACHINE_ID=6b72c55db4f9411dbbb80b70537bf3a8
    _HOSTNAME=costa-xps9500
    _RUNTIME_SCOPE=system
    _TRANSPORT=journal
    _CAP_EFFECTIVE=1ffffffffff
    _AUDIT_LOGINUID=1000
    _AUDIT_SESSION=1
    _SYSTEMD_CGROUP=/user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-59780d3d-a3ff-4a82-a6fe-8d17d2261106.scope
    _SYSTEMD_OWNER_UID=1000
    _SYSTEMD_UNIT=user@1000.service
    _SYSTEMD_USER_UNIT=vte-spawn-59780d3d-a3ff-4a82-a6fe-8d17d2261106.scope
    _SYSTEMD_SLICE=user-1000.slice
    _SYSTEMD_USER_SLICE=app-org.gnome.Terminal.slice
    _SYSTEMD_INVOCATION_ID=6195d8c4c6654481ac9a30e9a8622ba1
    _COMM=systemd-cat-nat
    MESSAGE=GET /index.html HTTP/1.1              # <<<<<<<<< CHECK
    NGINX_BODY_BYTES_SENT=4172                    # <<<<<<<<< CHECK
    NGINX_HTTP_REFERER=-                          # <<<<<<<<< CHECK
    NGINX_HTTP_USER_AGENT=Go-http-client/1.1      # <<<<<<<<< CHECK
    NGINX_HTTP_VERSION=1.1                        # <<<<<<<<< CHECK
    NGINX_METHOD=GET                              # <<<<<<<<< CHECK
    NGINX_REMOTE_ADDR=1.2.3.4                     # <<<<<<<<< CHECK
    NGINX_REMOTE_USER=-                           # <<<<<<<<< CHECK
    NGINX_REQUEST_LENGTH=104                      # <<<<<<<<< CHECK
    NGINX_REQUEST_TIME=0.001                      # <<<<<<<<< CHECK
    NGINX_STATUS=200                              # <<<<<<<<< CHECK
    NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000   # <<<<<<<<< CHECK
    NGINX_URL=/index.html                         # <<<<<<<<< CHECK
    SYSLOG_IDENTIFIER=nginx-log                   # <<<<<<<<< CHECK
    _PID=354312
    _SOURCE_REALTIME_TIMESTAMP=1700361246583912

```

So, the log line, with all its fields parsed, ended up in systemd-journal.

The complete example, would look like the following script.
Running this script with parameter `test` will produce output on the terminal for you to inspect.
Unmatched log entries are added to the journal with PRIORITY=1 (`ERR_ALERT`), so that you can spot them.

We also used the `--filename-key` of `log2journal`, which parses the filename when `tail` switches output
between files, and adds the field `NGINX_LOG_FILE` with the filename each log line comes from.

Finally, the script also adds the field `NGINX_STATUS_FAMILY` taking values `2xx`, `3xx`, etc, so that
it is easy to find all the logs of a specific status family.

```bash
#!/usr/bin/env bash

test=0
last=0
send_or_show='./systemd-cat-native'
[ "${1}" = "test" ] && test=1 && last=100 && send_or_show=cat

pattern='(?x)                          # Enable PCRE2 extended mode
^
(?<NGINX_REMOTE_ADDR>[^ ]+) \s - \s    # NGINX_REMOTE_ADDR
(?<NGINX_REMOTE_USER>[^ ]+) \s         # NGINX_REMOTE_USER
\[
  (?<NGINX_TIME_LOCAL>[^\]]+)          # NGINX_TIME_LOCAL
\]
\s+ "
(?<MESSAGE>                            # MESSAGE
  (?<NGINX_METHOD>[A-Z]+) \s+          # NGINX_METHOD
  (?<NGINX_URL>[^ ]+) \s+              # NGINX_URL
  HTTP/(?<NGINX_HTTP_VERSION>[^"]+)    # NGINX_HTTP_VERSION
)
" \s+
(?<NGINX_STATUS>\d+) \s+               # NGINX_STATUS
(?<NGINX_BODY_BYTES_SENT>\d+) \s+      # NGINX_BODY_BYTES_SENT
"(?<NGINX_HTTP_REFERER>[^"]*)" \s+     # NGINX_HTTP_REFERER
"(?<NGINX_HTTP_USER_AGENT>[^"]*)"      # NGINX_HTTP_USER_AGENT
'

tail -n $last -F /var/log/nginx/*access.log \
	| log2journal "${