netdata/packaging/ci: Make Travis CI more strict on nightlies run (#5708)

* netdata/packaging/ci: Make nightlies fail hard on all errors except changelog (#5580)

1) Add the necessary comments for documentation
2) Reinstate set -e to fail hard on errors
3) When changelog generation fails, just report a log entry and continue so that we dont break the release

These changes are temporary, we are following up with more aggressive changes that re-distribute processing

* netdata/packaging/nightlies: Clean up nightlies, round one of many

1) Move ALL changelog generation related code, to generate changelog script
2) Remove that || logic from everywherem, except the part we execute changelog generation.
   We have said its optional, so do not fail nightlies if this script fails, just report the problem
3) change the gitignore change and use the more stable logic that monitors folder path utilizing git
4) Add a bit more verbosity everywhere so we can tell what is going on
5) Adjust generate changelog script, based on the newly migrated code.
   a) Variable declarations to the top, so that if something wrong with the generated ones we break early and avoid the mess
   b) put all that github handling, right after markmandel's changelog generation
   c) Gently handle github commands, add enough verbosity to identify where we broke and also check in the end what happened and clean-up github before you bail out
6) A couple of renames for variables here and there, for clarity and readability

* netdata/packaging/ci: nits and fixes
1) Make sure nightlies still dont run when no changes are seen since last nightly
2) Artifacts creation should happen even if docker breaks. Ensure that and on the next iteration we will restructure the pipeline to a more productive layout
3) settle a couple of shellcheck warnings

* netdata/packaging/docker: build and publish are using a BASH v4 capability. Make sure you break early with a message if we ever run on a different bash by accident

1 files changed, 38 insertions, 36 deletions

diff --git a/.travis/nightlies.sh b/.travis/nightlies.sh
index 740e569126..3791b21fce 100755
--- a/.travis/nightlies.sh
+++ b/.travis/nightlies.sh
@@ -1,45 +1,47 @@
 #!/bin/bash
-
-BAD_THING_HAPPENED=0
-
-if [ ! -f .gitignore ]; then
-	echo "Run as ./travis/$(basename "$0") from top level directory of git repository"
-	exit 1
+#
+# This is the nightlies orchastration script
+# It runs the following activities in order:
+# 1) Generate changelog
+# 2) Build docker images
+# 3) Publish docker images
+# 4) Generate the rest of the artifacts (Source code .tar.gz file and makeself binary generation)
+#
+# Copyright: SPDX-License-Identifier: GPL-3.0-or-later
+#
+# Author  : Pawel Krupa (paulfantom)
+# Author  : Pavlos Emm. Katsoulakis (paul@netdata.cloud)
+set -e
+
+FAIL=0
+
+# If we are not in netdata git repo, at the top level directory, fail
+TOP_LEVEL=$(basename "$(git rev-parse --show-toplevel)")
+CWD=$(git rev-parse --show-cdup || echo "")
+if [ -n "$CWD" ] || [ ! "${TOP_LEVEL}" == "netdata" ]; then
+    echo "Run as .travis/$(basename "$0") from top level directory of netdata git repository"
+    echo "Changelog generation process aborted"
+    exit 1
 fi
 
-export GIT_MAIL="bot@netdata.cloud"
-export GIT_USER="netdatabot"
-echo "--- Initialize git configuration ---"
-git config user.email "${GIT_MAIL}"
-git config user.name "${GIT_USER}"
-
-echo "--- UPDATE VERSION FILE ---"
 LAST_TAG=$(git describe --abbrev=0 --tags)
-NO_COMMITS=$(git rev-list "$LAST_TAG"..HEAD --count)
-if [ "$NO_COMMITS" == "$(rev <packaging/version | cut -d- -f 2 | rev)" ]; then
-	echo "Nothing changed since last nightly build"
+COMMITS_SINCE_RELEASE=$(git rev-list "$LAST_TAG"..HEAD --count)
+PREVIOUS_NIGHTLY_COUNT="$(rev <packaging/version | cut -d- -f 2 | rev)"
+
+# If no commits since release, just stop
+if [ "$COMMITS_SINCE_RELEASE" == "${PREVIOUS_NIGHTLY_COUNT}" ]; then
+	echo "No changes since last nighthly release"
 	exit 0
 fi
-echo "$LAST_TAG-$((NO_COMMITS + 1))-nightly" >packaging/version
-git add packaging/version || exit 1
-
-echo "--- GENERATE CHANGELOG ---"
-if .travis/generate_changelog.sh; then
-	git add CHANGELOG.md
-
-	echo "--- UPLOAD FILE CHANGES ---"
-	git commit -m '[ci skip] create nightly packages and update changelog'
-	git push "https://${GITHUB_TOKEN}:@$(git config --get remote.origin.url | sed -e 's/^https:\/\///')"
-else
-	git clean -xfd
-	BAD_THING_HAPPENED=1
-fi
 
-echo "--- BUILD & PUBLISH DOCKER IMAGES ---"
-packaging/docker/build.sh || BAD_THING_HAPPENED=1
-packaging/docker/publish.sh || BAD_THING_HAPPENED=1
+echo "--- Running Changelog generation ---"
+.travis/generate_changelog.sh "$LAST_TAG" "$COMMITS_SINCE_RELEASE" || echo "Changelog generation has failed, this is a soft error, process continues"
+
+echo "--- Build && publish docker images ---"
+# Do not fail artifacts creation if docker fails. We will be restructuring this on a follow up PR
+packaging/docker/build.sh && packaging/docker/publish.sh || echo "Failed to build and publish docker images"
 
-echo "--- BUILD ARTIFACTS ---"
-.travis/create_artifacts.sh || BAD_THING_HAPPENED=1
+echo "--- Build artifacts ---"
+.travis/create_artifacts.sh
 
-exit "${BAD_THING_HAPPENED}"
+exit "${FAIL}"