Overhaul build CI. (#11699)agent-ng-latest

* Overhaul build CI.

This makes a number of changes that make our build CI easier to maintain
and much safer with respect to the possibility of accidentally pushing
broken nightlies.

* The release-build.yml workflow has been renamed to just build.yml
* The updater.yml and build-and-install.yml workflows have been
  consolidated into the build.yml workflow.
* The updater checks now use the dist tarball produced by the dist
  tarball build instead of building one for each matrix job.
* The updater and build checks now use a set of jobs for generating
  their test environments.
* The updater and build checks and their shared test environment builds
  now use a unified build matrix, handled in an external data file
  (.github/data/build-matrix.json).
* Nightly artifact uploads and release creation are now dependent on the
  updater checks passing.

* Properly handle generated build matrix.

* Minor fixes.

* Fix shellcheck complaints.

* Fixed stub script handling.

* Mangle the build matrix JSON properly.

* Properly handle stub scripts.

* Fix distro name handling.

* Fix typos.

* Fix test environment image handling.

* Fix updater checks.

* Skip CentOS 7 JSON-C bundling check.

* Fix handling of artifacts for updater check.

* Fix yamllint errors.

* Fix dist tarball handling.

* Fix updater check script permissions.

* Restructure build test Dockerfile for better caching.

* Remove parallelization limits for test environment builds.

These complete fast enough that it is perfectly reasonable to run them
all in parallel simultaneously. In the best case scenario, this will
save roughly 5-10 minutes of time during CI runs.

* Add more in-line documentation and tidy up a few unneeded parameters.

* Add checks to verify consolidated artifacts are good.

* Fix distfile name handling.

* Fix artifact checking.

* Better delineate phases of updater check in logs.

* Simplify updater checks.

* Print checksums of artifacts.

To simplify debugging.

* Fix handling of updater checks.

8 files changed, 662 insertions, 479 deletions

diff --git a/.github/data/build-matrix.json b/.github/data/build-matrix.json
new file mode 100644
index 0000000000..427819b250
--- /dev/null
+++ b/.github/data/build-matrix.json
@@ -0,0 +1,109 @@
+{
+  "include": [
+    {
+      "distro": "alpine:edge",
+      "artifact_key": "alpine-edge",
+      "pre": "apk add -U bash",
+      "rmjsonc": "apk del json-c-dev"
+    },
+    {
+      "distro": "alpine:3.14",
+      "artifact_key": "alpine-3.14",
+      "pre": "apk add -U bash",
+      "rmjsonc": "apk del json-c-dev"
+    },
+    {
+      "distro": "alpine:3.13",
+      "artifact_key": "alpine-3.13",
+      "pre": "apk add -U bash",
+      "rmjsonc": "apk del json-c-dev"
+    },
+    {
+      "distro": "alpine:3.12",
+      "artifact_key": "alpine-3.12",
+      "pre": "apk add -U bash",
+      "rmjsonc": "apk del json-c-dev"
+    },
+    {
+      "distro": "archlinux:latest",
+      "artifact_key": "archlinux",
+      "pre": "pacman --noconfirm -Syu && pacman --noconfirm -Sy grep libffi"
+    },
+    {
+      "distro": "centos:7",
+      "artifact_key": "centos7"
+    },
+    {
+      "distro": "centos:8",
+      "artifact_key": "centos8",
+      "rmjsonc": "dnf remove -y json-c-devel"
+    },
+    {
+      "distro": "debian:11",
+      "artifact_key": "debian11",
+      "pre": "apt-get update",
+      "rmjsonc": "apt-get remove -y libjson-c-dev"
+    },
+    {
+      "distro": "debian:10",
+      "artifact_key": "debian10",
+      "pre": "apt-get update",
+      "rmjsonc": "apt-get remove -y libjson-c-dev"
+    },
+    {
+      "distro": "debian:9",
+      "artifact_key": "debian9",
+      "pre": "apt-get update",
+      "rmjsonc": "apt-get remove -y libjson-c-dev"
+    },
+    {
+      "distro": "fedora:35",
+      "artifact_key": "fedora35",
+      "rmjsonc": "dnf remove -y json-c-devel"
+    },
+    {
+      "distro": "fedora:34",
+      "artifact_key": "fedora34",
+      "rmjsonc": "dnf remove -y json-c-devel"
+    },
+    {
+      "distro": "fedora:33",
+      "artifact_key": "fedora33",
+      "rmjsonc": "dnf remove -y json-c-devel"
+    },
+    {
+      "distro": "opensuse/leap:15.3",
+      "artifact_key": "opensuse15.3",
+      "rmjsonc": "zypper rm -y libjson-c-devel"
+    },
+    {
+      "distro": "opensuse/leap:15.2",
+      "artifact_key": "opensuse15.2",
+      "rmjsonc": "zypper rm -y libjson-c-devel"
+    },
+    {
+      "distro": "ubuntu:21.10",
+      "artifact_key": "ubuntu21.10",
+      "pre": "rm -f /etc/apt/apt.conf.d/docker && apt-get update",
+      "rmjsonc": "apt-get remove -y libjson-c-dev"
+    },
+    {
+      "distro": "ubuntu:21.04",
+      "artifact_key": "ubuntu21.04",
+      "pre": "apt-get update",
+      "rmjsonc": "apt-get remove -y libjson-c-dev"
+    },
+    {
+      "distro": "ubuntu:20.04",
+      "artifact_key": "ubuntu20.04",
+      "pre": "apt-get update",
+      "rmjsonc": "apt-get remove -y libjson-c-dev"
+    },
+    {
+      "distro": "ubuntu:18.04",
+      "artifact_key": "ubuntu18.04",
+      "pre": "apt-get update",
+      "rmjsonc": "apt-get remove -y libjson-c-dev"
+    }
+  ]
+}
diff --git a/.github/dockerfiles/Dockerfile.build_test b/.github/dockerfiles/Dockerfile.build_test
index 718230dfb9..aca89c09b6 100644
--- a/.github/dockerfiles/Dockerfile.build_test
+++ b/.github/dockerfiles/Dockerfile.build_test
@@ -4,10 +4,15 @@ FROM ${BASE}
 
 ARG PRE
 ENV PRE=${PRE}
+ARG RMJSONC
+ENV RMJSONC=${RMJSONC}
 ENV DO_NOT_TRACK=1
 ENV GITHUB_ACTIONS=true
 
+RUN echo "${PRE}" > /prep-cmd.sh && \
+    echo "${RMJSONC}" > /rmjsonc.sh && chmod +x /rmjsonc.sh && \
+    /bin/sh /prep-cmd.sh
+
 COPY . /netdata
 
-RUN /bin/sh /netdata/prep-cmd.sh
 RUN /netdata/packaging/installer/install-required-packages.sh --dont-wait --non-interactive netdata-all
diff --git a/.github/scripts/run-updater-check.sh b/.github/scripts/run-updater-check.sh
new file mode 100755
index 0000000000..e6969a2d13
--- /dev/null
+++ b/.github/scripts/run-updater-check.sh
@@ -0,0 +1,9 @@
+#!/bin/sh
+
+echo ">>> Installing Netdata..."
+/netdata/packaging/installer/kickstart.sh --dont-wait --disable-telemetry || exit 1
+echo ">>> Updating Netdata..."
+export NETDATA_NIGHTLIES_BASEURL="http://localhost:8080/artifacts/" # Pull the tarball from the local web server.
+/netdata/packaging/installer/netdata-updater.sh --not-running-from-cron --no-updater-self-update || exit 1
+echo ">>> Checking if update was successful..."
+/netdata/.github/scripts/check-updater.sh || exit 1
diff --git a/.github/workflows/build-and-install.yml b/.github/workflows/build-and-install.yml
deleted file mode 100644
index b3feb7356a..0000000000
--- a/.github/workflows/build-and-install.yml
+++ /dev/null
@@ -1,125 +0,0 @@
----
-name: Builder
-on:
-  push:
-    branches:
-      - master
-  pull_request: null
-env:
-  DO_NOT_TRACK: 1
-concurrency:
-  group: builder-${{ github.ref }}
-  cancel-in-progress: true
-jobs:
-  source-build:
-    name: Build & Install
-    strategy:
-      fail-fast: false
-      max-parallel: 8
-      matrix:
-        distro:
-          - 'alpine:edge'
-          - 'alpine:3.14'
-          - 'alpine:3.13'
-          - 'alpine:3.12'
-          - 'archlinux:latest'
-          - 'centos:8'
-          - 'centos:7'
-          - 'debian:bullseye' # 11
-          - 'debian:10'
-          - 'debian:9'
-          - 'fedora:35'
-          - 'fedora:34'
-          - 'fedora:33'
-          - 'opensuse/leap:15.2'
-          - 'opensuse/leap:15.3'
-          - 'ubuntu:21.10'
-          - 'ubuntu:21.04'
-          - 'ubuntu:20.04'
-          - 'ubuntu:18.04'
-        include:
-          - distro: 'alpine:edge'
-            pre: 'apk add -U bash'
-            rmjsonc: 'apk del json-c-dev'
-          - distro: 'alpine:3.14'
-            pre: 'apk add -U bash'
-            rmjsonc: 'apk del json-c-dev'
-          - distro: 'alpine:3.13'
-            pre: 'apk add -U bash'
-            rmjsonc: 'apk del json-c-dev'
-          - distro: 'alpine:3.12'
-            pre: 'apk add -U bash'
-            rmjsonc: 'apk del json-c-dev'
-
-          - distro: 'archlinux:latest'
-            pre: 'pacman --noconfirm -Syu && pacman --noconfirm -Sy grep libffi'
-
-          - distro: 'centos:8'
-            rmjsonc: 'dnf remove -y json-c-devel'
-
-          - distro: 'debian:bullseye' # 11
-            pre: 'apt-get update'
-            rmjsonc: 'apt-get remove -y libjson-c-dev'
-          - distro: 'debian:10'
-            pre: 'apt-get update'
-            rmjsonc: 'apt-get remove -y libjson-c-dev'
-          - distro: 'debian:9'
-            pre: 'apt-get update'
-            rmjsonc: 'apt-get remove -y libjson-c-dev'
-
-          - distro: 'fedora:35'
-            rmjsonc: 'dnf remove -y json-c-devel'
-          - distro: 'fedora:34'
-            rmjsonc: 'dnf remove -y json-c-devel'
-          - distro: 'fedora:33'
-            rmjsonc: 'dnf remove -y json-c-devel'
-
-          - distro: 'opensuse/leap:15.2'
-            rmjsonc: 'zypper rm -y libjson-c-devel'
-          - distro: 'opensuse/leap:15.3'
-            rmjsonc: 'zypper rm -y libjson-c-devel'
-
-          - distro: 'ubuntu:21.10'
-            pre: 'rm -f /etc/apt/apt.conf.d/docker* && apt-get update'
-            rmjsonc: 'apt-get remove -y libjson-c-dev'
-          - distro: 'ubuntu:21.04'
-            pre: 'apt-get update'
-            rmjsonc: 'apt-get remove -y libjson-c-dev'
-          - distro: 'ubuntu:20.04'
-            pre: 'apt-get update'
-            rmjsonc: 'apt-get remove -y libjson-c-dev'
-          - distro: 'ubuntu:18.04'
-            pre: 'apt-get update'
-            rmjsonc: 'apt-get remove -y libjson-c-dev'
-    runs-on: ubuntu-latest
-    steps:
-      - name: Git clone repository
-        uses: actions/checkout@v2
-        with:
-          submodules: recursive
-      - name: install-required-packages.sh on ${{ matrix.distro }}
-        env:
-          PRE: ${{ matrix.pre }}
-          RMJSONC: ${{ matrix.rmjsonc }}
-        run: |
-          echo $PRE > ./prep-cmd.sh
-          echo $RMJSONC > ./rmjsonc.sh && chmod +x ./rmjsonc.sh
-          docker build . -f .github/dockerfiles/Dockerfile.build_test -t test --build-arg BASE=${{ matrix.distro }}
-      - name: Regular build on ${{ matrix.distro }}
-        run: |
-          docker run --security-opt seccomp=unconfined -w /netdata test /bin/sh -c 'autoreconf -ivf && ./configure && make -j2'
-      - name: netdata-installer on ${{ matrix.distro }}, disable cloud
-        run: |
-          docker run --security-opt seccomp=unconfined -w /netdata test /bin/sh -c './netdata-installer.sh --dont-wait --dont-start-it --disable-cloud'
-      - name: netdata-installer on ${{ matrix.distro }}, require cloud
-        run: |
-          docker run --security-opt seccomp=unconfined -w /netdata test /bin/sh -c './netdata-installer.sh --dont-wait --dont-start-it --require-cloud'
-      - name: netdata-installer on ${{ matrix.distro }}, require cloud, require ACLK-NG
-        run: |
-          docker run --security-opt seccomp=unconfined -w /netdata -e NETDATA_CONFIGURE_OPTIONS='--with-aclk-ng' test \
-              /bin/sh -c './netdata-installer.sh --dont-wait --dont-start-it --require-cloud'
-      - name: netdata-installer on ${{ matrix.distro }}, require cloud, no JSON-C
-        if: matrix.rmjsonc != ''
-        run: |
-          docker run --security-opt seccomp=unconfined -w /netdata test \
-              /bin/sh -c '/netdata/rmjsonc.sh && ./netdata-installer.sh --dont-wait --dont-start-it --require-cloud'
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
new file mode 100644
index 0000000000..cbc291f8fa
--- /dev/null
+++ b/.github/workflows/build.yml
@@ -0,0 +1,537 @@
+---
+# Ci code for building release artifacts.
+name: Build
+on:
+  push: # Master branch checks only validate the build and generate artifacts for testing.
+    branches:
+      - master
+  pull_request: null # PR checks only validate the build and generate artifacts for testing.
+  workflow_dispatch: # Dispatch runs build and validate, then push to the appropriate storage location.
+    inputs:
+      type:
+        description: Build Type
+        default: nightly
+        required: true
+      version:
+        description: Version Tag
+        default: nightly
+        required: true
+concurrency: # This keeps multiple instances of the job from running concurrently for the same ref and event type.
+  group: release-${{ github.ref }}-${{ github.event_name }}
+  cancel-in-progress: true
+jobs:
+  build-dist: # Build the distribution tarball and store it as an artifact.
+    name: Build Distribution Tarball
+    runs-on: ubuntu-latest
+    outputs:
+      distfile: ${{ steps.build.outputs.distfile }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+          submodules: recursive
+      - name: Mark Stable
+        if: github.event_name == 'workflow_dispatch' && github.event.inputs.type != 'nightly'
+        run: |
+          sed -i 's/^RELEASE_CHANNEL="nightly" *#/RELEASE_CHANNEL="stable" #/' netdata-installer.sh
+      - name: Build
+        id: build
+        run: |
+          mkdir -p artifacts
+          ./packaging/installer/install-required-packages.sh --dont-wait --non-interactive netdata
+          autoreconf -ivf
+          ./configure --prefix=/usr \
+                      --sysconfdir=/etc \
+                      --localstatedir=/var \
+                      --libexecdir=/usr/libexec \
+                      --with-zlib \
+                      --with-math \
+                      --with-user=netdata
+          make dist
+          echo "::set-output name=distfile::$(find . -name 'netdata-*.tar.gz')"
+          cp netdata-*.tar.gz artifacts/
+      - name: Store
+        uses: actions/upload-artifact@v2
+        with:
+          name: dist-tarball
+          path: artifacts/*.tar.gz
+          retention-days: 30
+      - name: Failure Notification
+        uses: rtCamp/action-slack-notify@v2
+        env:
+          SLACK_COLOR: 'danger'
+          SLACK_FOOTER: ''
+          SLACK_ICON_EMOJI: ':github-actions:'
+          SLACK_TITLE: 'Distribution tarball build failed:'
+          SLACK_USERNAME: 'GitHub Actions'
+          SLACK_MESSAGE: "Distribution tarball build failed."
+          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK_URL }}
+        if: >-
+          ${{
+            failure()
+            && startsWith(github.ref, 'refs/heads/master')
+            && github.event_name != 'pull_request'
+          }}
+
+  build-static: # Build the static binary archives, and store them as artifacts.
+    name: Build Static
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        arch:
+          - 'x86_64'
+          - 'armv7l'
+          - 'aarch64'
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+          submodules: recursive
+      - name: Mark Stable
+        if: github.event_name == 'workflow_dispatch' && github.event.inputs.type != 'nightly'
+        run: |
+          sed -i 's/^RELEASE_CHANNEL="nightly" *#/RELEASE_CHANNEL="stable" #/' netdata-installer.sh packaging/makeself/install-or-update.sh
+      - name: Build
+        run: .github/scripts/build-static.sh ${{ matrix.arch }}
+      - name: Store
+        uses: actions/upload-artifact@v2
+        with:
+          name: static-archive
+          path: artifacts/*.gz.run
+          retention-days: 30
+      - name: Failure Notification
+        uses: rtCamp/action-slack-notify@v2
+        env:
+          SLACK_COLOR: 'danger'
+          SLACK_FOOTER: ''
+          SLACK_ICON_EMOJI: ':github-actions:'
+          SLACK_TITLE: 'Static build for ${{ matrix.arch }} failed:'
+          SLACK_USERNAME: 'GitHub Actions'
+          SLACK_MESSAGE: "Static build for ${{ matrix.arch }} failed."
+          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK_URL }}
+        if: >-
+          ${{
+            failure()
+            && startsWith(github.ref, 'refs/heads/master')
+            && github.event_name != 'pull_request'
+          }}
+
+  matrix: # Generate the shared build matrix for our build tests.
+    name: Prepare Build Matrix
+    runs-on: ubuntu-latest
+    outputs:
+      matrix: ${{ steps.set-matrix.outputs.matrix }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+      - name: Prepare tools
+        run: |
+          sudo apt-get update && sudo apt-get install -y jq
+      - name: Read build matrix
+        id: set-matrix
+        run: |
+          TASKS="$(jq -c . .github/data/build-matrix.json)"
+          echo "Generated Matrix: $TASKS"
+          echo "::set-output name=matrix::$TASKS"
+
+  prepare-test-images: # Prepare the test environments for our build checks. This also checks dependency handling code for each tested environment.
+    name: Prepare Test Environments
+    runs-on: ubuntu-latest
+    needs:
+      - matrix
+    strategy:
+      # Unlike the actal build tests, this completes _very_ fast (average of about 3 minutes for each job), so we
+      # just run everything in parallel instead lof limiting job concurrency.
+      fail-fast: false
+      matrix: ${{ fromJson(needs.matrix.outputs.matrix) }}
+    steps:
+      - name: Git clone repository
+        uses: actions/checkout@v2
+      - name: Setup Buildx
+        uses: docker/setup-buildx-action@v1
+      - name: Build test environment
+        uses: docker/build-push-action@v2
+        with:
+          push: false
+          load: false
+          file: .github/dockerfiles/Dockerfile.build_test
+          build-args: |
+            BASE=${{ matrix.distro }}
+            PRE=${{ matrix.pre }}
+            RMJSONC=${{ matrix.rmjsonc }}
+          outputs: type=oci,dest=/tmp/image.tar
+          tags: test:${{ matrix.artifact_key }}
+      - name: Upload image artifact
+        uses: actions/upload-artifact@v2
+        with:
+          name: ${{ matrix.artifact_key }}-test-env
+          path: /tmp/image.tar
+          retention-days: 30
+      - name: Failure Notification
+        uses: rtCamp/action-slack-notify@v2
+        env:
+          SLACK_COLOR: 'danger'
+          SLACK_FOOTER: ''
+          SLACK_ICON_EMOJI: ':github-actions:'
+          SLACK_TITLE: 'Test environment preparation for ${{ matrix.distro }} failed:'
+          SLACK_USERNAME: 'GitHub Actions'
+          SLACK_MESSAGE: "Test environment preparation for ${{ matrix.distro }} failed."
+          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK_URL }}
+        if: >-
+          ${{
+            failure()
+            && startsWith(github.ref, 'refs/heads/master')
+            && github.event_name != 'pull_request'
+          }}
+
+  source-build: # Test various source build arrangements.
+    name: Test Source Build
+    runs-on: ubuntu-latest
+    needs:
+      - matrix
+      - prepare-test-images
+    strategy:
+      fail-fast: false
+      max-parallel: 8
+      matrix: ${{ fromJson(needs.matrix.outputs.matrix) }}
+    steps:
+      - name: Git clone repository
+        uses: actions/checkout@v2
+        with:
+          submodules: recursive
+      - name: Fetch test environment
+        uses: actions/download-artifact@v2
+        with:
+          name: ${{ matrix.artifact_key }}-test-env
+      - name: Load test environment
+        id: load
+        run: |
+          docker load --input image.tar | tee image-info.txt
+          echo "::set-output name=image::$(cut -d ':' -f 3 image-info.txt)"
+      - name: Regular build on ${{ matrix.distro }}
+        run: |
+          docker run --security-opt seccomp=unconfined -w /netdata sha256:${{ steps.load.outputs.image }} \
+              /bin/sh -c 'autoreconf -ivf && ./configure && make -j2'
+      - name: netdata-installer on ${{ matrix.distro }}, disable cloud
+        run: |
+          docker run --security-opt seccomp=unconfined -w /netdata sha256:${{ steps.load.outputs.image }} \
+              /bin/sh -c './netdata-installer.sh --dont-wait --dont-start-it --disable-cloud'
+      - name: netdata-installer on ${{ matrix.distro }}, require cloud
+        run: |
+          docker run --security-opt seccomp=unconfined -w /netdata sha256:${{ steps.load.outputs.image }} \
+              /bin/sh -c './netdata-installer.sh --dont-wait --dont-start-it --require-cloud'
+      - name: netdata-installer on ${{ matrix.distro }}, require cloud, require ACLK-NG
+        run: |
+          docker run --security-opt seccomp=unconfined -w /netdata -e NETDATA_CONFIGURE_OPTIONS='--with-aclk-ng' \
+              sha256:${{ steps.load.outputs.image }} /bin/sh -c './netdata-installer.sh --dont-wait --dont-start-it --require-cloud'
+      - name: netdata-installer on ${{ matrix.distro }}, require cloud, no JSON-C
+        if: matrix.rmjsonc != ''
+        run: |
+          docker run --security-opt seccomp=unconfined -w /netdata sha256:${{ steps.load.outputs.image }} \
+              /bin/sh -c '/rmjsonc.sh && ./netdata-installer.sh --dont-wait --dont-start-it --require-cloud'
+      - name: Failure Notification
+        uses: rtCamp/action-slack-notify@v2
+        env:
+          SLACK_COLOR: 'danger'
+          SLACK_FOOTER: ''
+          SLACK_ICON_EMOJI: ':github-actions:'
+          SLACK_TITLE: 'Build tests for ${{ matrix.distro }} failed:'
+          SLACK_USERNAME: 'GitHub Actions'
+          SLACK_MESSAGE: "Build tests for ${{ matrix.distro }} failed."
+          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK_URL }}
+        if: >-
+          ${{
+            failure()
+            && startsWith(github.ref, 'refs/heads/master')
+            && github.event_name != 'pull_request'
+          }}
+
+  updater-check: # Test the generated dist archive using the updater code.
+    name: Test Generated Distfile and Updater Code
+    runs-on: ubuntu-latest
+    needs:
+      - build-dist
+      - matrix
+      - prepare-test-images
+    strategy:
+      fail-fast: false
+      max-parallel: 8
+      matrix: ${{ fromJson(needs.matrix.outputs.matrix) }}
+    services:
+      apache: # This gets used to serve the dist tarball for the updater script.
+        image: httpd:2.4
+        ports:
+          - 8080:80
+        volumes:
+          - ${{ github.workspace }}:/usr/local/apache2/htdocs/
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+      - name: Fetch dist tarball artifacts
+        uses: actions/download-artifact@v2
+        with:
+          name: dist-tarball
+          path: dist-tarball
+      - name: Prepare artifact directory
+        run: |
+          mkdir -p artifacts || exit 1
+          echo "9999.0.0-0" > artifacts/latest-version.txt || exit 1
+          cp dist-tarball/* artifacts || exit 1
+          cd artifacts || exit 1
+          ln -s ${{ needs.build-dist.outputs.distfile }} netdata-latest.tar.gz || exit 1
+          sha256sum -b ./* > "sha256sums.txt" || exit 1
+          cat sha256sums.txt
+      - name: Fetch test environment
+        uses: actions/download-artifact@v2
+        with:
+          name: ${{ matrix.artifact_key }}-test-env
+      - name: Load test environment
+        id: load
+        run: |
+          docker load --input image.tar | tee image-info.txt
+          echo "::set-output name=image::$(cut -d ':' -f 3 image-info.txt)"
+      - name: Install netdata and run the updater on ${{ matrix.distro }}
+        run: |
+          docker run --security-opt seccomp=unconfined -e DO_NOT_TRACK=1 --network host -w /netdata sha256:${{ steps.load.outputs.image }} \
+                /netdata/.github/scripts/run-updater-check.sh
+      - name: Failure Notification
+        uses: rtCamp/action-slack-notify@v2
+        env:
+          SLACK_COLOR: 'danger'
+          SLACK_FOOTER: ''
+          SLACK_ICON_EMOJI: ':github-actions:'
+          SLACK_TITLE: 'Updater checks for ${{ matrix.distro }} failed:'
+          SLACK_USERNAME: 'GitHub Actions'
+          SLACK_MESSAGE: "Updater checks for ${{ matrix.distro }} failed."
+          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK_URL }}
+        if: >-
+          ${{
+            failure()
+            && startsWith(github.ref, 'r