netdata/ci: nits and fixes around package release workflow (#6914)

* netdata/packaging: bring el/6 first, because this appears on all cases. Otherwise travis craps out the steps

* netdata/ci: Attempt to sanitise workflow for master branch commits

1) Rename Packaging for release, now named: Support activities on main branch
2) Adjust instruction messages to reflect more realistically the activities
3) Put labeler first on the stage, since this will always run
4) Make changelog and tag for release conditionally run, so that it does not confuse people

* netdata/ci: Sanitise workflow for release more (explained below)

We need to clean up tagging logic more and in order to do this we need to bring tagger logic to travis.yml
Then we have the complete control of tagging and triggering for changelog generation in travis.
To mitigate this, we need to:
1) make tagger script a lirbary. Keep the tagging logic in a method and remove the things needed when it was an executable (checks, executions etc)
2) make travis handle GIT_TAG checking and setting at the beginning. We need this outside of any conditionals so that we have GIT_TAG result available inside the steps
3) COMMIT_TAG_POINTS_AT can now become GIT_TAG, since this is what we actually wanted. We just named differently before to avoid any weird conflicts
4) make changelog generation for release to NOT source tagger any more. Also make it fail in case GIT_TAG is not there, we wont run the script if git tag not in place so that should be error if it happens
5) Rename .travis/generate_changelog_for_release.sh -> .travis/create_changelog_for_release.sh
6) Rename .travis/generate_changelog_and_tag_release.sh -> .travis/generate_changelog_for_release.sh

* netdata/ci: reinstate filename here. we basically do the git tag operation here still

* netdata/ci: simplify more the changelog generation process

1) Rename create_changelog_for_release.sh -> create_changelog.sh
2) Add comments and the standard path checks in create_changelog.sh
3) Do some minor changes to help diff with nightlies changelog (no functional change though, primarily messages and move flags lower)
4) Inside generate_changelog_for_nightlies.sh, use create_changelog.sh and remove code related to create changelog process
5) in nightlies.sh, use standard shebang adn fix some messages

Note: might follow up with more changes, this is just a first batch

* netdata/packaging: reinstate accidentally removed OPTS from other iterationns (different engineer)

5 files changed, 57 insertions, 88 deletions

diff --git a/.travis/generate_changelog_for_release.sh b/.travis/create_changelog.sh
index 3f68a6925d..8d4c12ee1f 100755
--- a/.travis/generate_changelog_for_release.sh
+++ b/.travis/create_changelog.sh
@@ -1,16 +1,24 @@
-#!/bin/bash
-
+#!/usr/bin/env bash
+#
+# Copyright: SPDX-License-Identifier: GPL-3.0-or-later
+#
+# Author  : Pavlos Emm. Katsoulakis (paul@netdata.cloud)
 set -e
 
-if [ ! -f .gitignore ]; then
-	echo "Run as ./travis/$(basename "$0") from top level directory of git repository"
-	exit 1
+# If we are not in netdata git repo, at the top level directory, fail
+TOP_LEVEL=$(basename "$(git rev-parse --show-toplevel)")
+CWD=$(git rev-parse --show-cdup || echo "")
+if [ -n "$CWD" ] || [ ! "${TOP_LEVEL}" == "netdata" ]; then
+    echo "Run as .travis/$(basename "$0") from top level directory of netdata git repository"
+    echo "Changelog creation aborted"
+    exit 1
 fi
 
 ORGANIZATION=$(echo "$TRAVIS_REPO_SLUG" | awk -F '/' '{print $1}')
 PROJECT=$(echo "$TRAVIS_REPO_SLUG" | awk -F '/' '{print $2}')
 GIT_MAIL=${GIT_MAIL:-"bot@netdata.cloud"}
 GIT_USER=${GIT_USER:-"netdatabot"}
+
 if [ -z ${GIT_TAG+x} ]; then
 	OPTS=""
 else
@@ -18,7 +26,7 @@ else
 fi
 
 if [ ! "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then
-	echo "Beta mode on ${TRAVIS_REPO_SLUG}, nothing else to do"
+	echo "Beta mode on ${TRAVIS_REPO_SLUG}, nothing else to do here"
 	exit 0
 fi
 
@@ -31,8 +39,8 @@ docker run -it -v "$(pwd)":/project markmandel/github-changelog-generator:latest
 	--project "${PROJECT}" \
 	--token "${GITHUB_TOKEN}" \
 	--since-tag "v1.10.0" \
-	--no-issues \
 	--unreleased-label "**Next release**" \
+	--no-issues \
 	--exclude-labels "stale,duplicate,question,invalid,wontfix,discussion,no changelog" \
-        --max-issues 500 \
-        --bug-labels IGNOREBUGS
+	--max-issues 500 \
+	--bug-labels IGNOREBUGS ${OPTS}
diff --git a/.travis/generate_changelog_and_tag_release.sh b/.travis/generate_changelog_and_tag_release.sh
index fb155b2645..bf5555b4cd 100755
--- a/.travis/generate_changelog_and_tag_release.sh
+++ b/.travis/generate_changelog_and_tag_release.sh
@@ -30,14 +30,11 @@ if [ ! -f .gitignore ]; then
 	exit 1
 fi
 
-echo "--- Executing Tagging facility to determine TAG ---"
-source .travis/tagger.sh
-
-echo "--- Changelog generator and tagger script starting ---"
-# If tagger script hasn't produced a TAG, there is nothing to do so bail out happy
+echo "--- Changelog generator script starting ---"
+# If we dont have a produced TAG there is nothing to do, so bail out happy
 if [ -z "${GIT_TAG}" ]; then
-	echo "GIT_TAG is empty, nothing to do for now (Value: $GIT_TAG)"
-	exit 0
+	echo "GIT_TAG is empty, that is not suppose to happen (Value: $GIT_TAG)"
+	exit 1
 fi
 
 if [ ! "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then
@@ -55,8 +52,8 @@ echo "---- UPDATE VERSION FILE ----"
 echo "$GIT_TAG" >packaging/version
 git add packaging/version
 
-echo "---- GENERATE CHANGELOG -----"
-./.travis/generate_changelog_for_release.sh
+echo "---- Create CHANGELOG -----"
+./.travis/create_changelog.sh
 git add CHANGELOG.md
 
 echo "---- COMMIT AND PUSH CHANGES ----"
diff --git a/.travis/generate_changelog_for_nightlies.sh b/.travis/generate_changelog_for_nightlies.sh
index 2e8da17f59..59173af3f6 100755
--- a/.travis/generate_changelog_for_nightlies.sh
+++ b/.travis/generate_changelog_for_nightlies.sh
@@ -1,6 +1,6 @@
-#!/bin/bash
+#!/usr/bin/env bash
 #
-# Changelog generation scriptlet.
+# Changelog generation scriptlet, for nightlies
 #
 # Copyright: SPDX-License-Identifier: GPL-3.0-or-later
 #
@@ -20,39 +20,18 @@ fi
 LAST_TAG="$1"
 COMMITS_SINCE_RELEASE="$2"
 NEW_VERSION="${LAST_TAG}-$((COMMITS_SINCE_RELEASE + 1))-nightly"
-ORG=$(echo "$TRAVIS_REPO_SLUG" | cut -d '/' -f1)
-PROJECT=$(echo "$TRAVIS_REPO_SLUG" | cut -d '/' -f 2)
 GIT_MAIL=${GIT_MAIL:-"bot@netdata.cloud"}
 GIT_USER=${GIT_USER:-"netdatabot"}
 PUSH_URL=$(git config --get remote.origin.url | sed -e 's/^https:\/\///')
 FAIL=0
-if [ -z ${GIT_TAG+x} ]; then
-	OPTS=""
-else
-	OPTS="--future-release ${GIT_TAG}"
-fi
-echo "We got $COMMITS_SINCE_RELEASE changes since $LAST_TAG, re-generating changelog"
 
 if [ ! "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then
 	echo "Beta mode on ${TRAVIS_REPO_SLUG}, nothing else to do here"
 	exit 0
 fi
 
-git checkout master
-git pull
-
-echo "Running project markmandel for github changelog generation"
-#docker run -it --rm -v "$(pwd)":/usr/local/src/your-app ferrarimarco/github-changelog-generator:1.14.3 \
-docker run -it -v "$(pwd)":/project markmandel/github-changelog-generator:latest \
-	--user "${ORG}" \
-	--project "${PROJECT}" \
-	--token "${GITHUB_TOKEN}" \
-	--since-tag "v1.10.0" \
-	--unreleased-label "**Next release**" \
-	--no-issues \
-	--exclude-labels "stale,duplicate,question,invalid,wontfix,discussion,no changelog" \
-	--max-issues 500 \
-	--bug-labels IGNOREBUGS
+echo "Running changelog creation mechanism"
+.travis/create_changelog.sh
 
 echo "Changelog created! Adding packaging/version(${NEW_VERSION}) and CHANGELOG.md to the repository"
 echo "${NEW_VERSION}" > packaging/version
diff --git a/.travis/nightlies.sh b/.travis/nightlies.sh
index 13a813d75d..e60ee27976 100755
--- a/.travis/nightlies.sh
+++ b/.travis/nightlies.sh
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 #
 # This is the nightly changelog generation script
 # It is responsible for two major activities:
@@ -29,7 +29,7 @@ PREVIOUS_NIGHTLY_COUNT="$(rev <packaging/version | cut -d- -f 2 | rev)"
 
 # If no commits since release, just stop
 if [ "${COMMITS_SINCE_RELEASE}" == "${PREVIOUS_NIGHTLY_COUNT}" ]; then
-	echo "No changes since last nighthly release"
+	echo "No changes since last nighthly release, nothing else to do"
 	exit 0
 fi
 
@@ -39,6 +39,7 @@ if [ ! "${TRAVIS_REPO_SLUG}" == "netdata/netdata" ]; then
 fi
 
 echo "--- Running Changelog generation ---"
+echo "We got $COMMITS_SINCE_RELEASE changes since $LAST_TAG, re-generating changelog"
 NIGHTLIES_CHANGELOG_FAILED=0
 .travis/generate_changelog_for_nightlies.sh "${LAST_TAG}" "${COMMITS_SINCE_RELEASE}" || NIGHTLIES_CHANGELOG_FAILED=1
 
diff --git a/.travis/tagger.sh b/.travis/tagger.sh
index adac35b48d..a775a826f4 100755
--- a/.travis/tagger.sh
+++ b/.travis/tagger.sh
@@ -1,6 +1,4 @@
-#!/bin/bash
-#
-# Original script is available at https://github.com/paulfantom/travis-helper/blob/master/releasing/releaser.sh
+# #BASH library
 #
 # Tags are generated by searching for a keyword in last commit message. Keywords are:
 #  - [patch] or [fix] to bump patch number
@@ -12,8 +10,6 @@
 #   - GITHUB_TOKEN variable set with GitHub token. Access level: repo.public_repo
 #   - git-semver python package (pip install git-semver)
 #
-# Note: Exported variables needed by .travis/draft_release.sh
-#
 # Original script is available at https://github.com/paulfantom/travis-helper/blob/master/releasing/releaser.sh
 #
 # Copyright: SPDX-License-Identifier: GPL-3.0-or-later
@@ -21,24 +17,11 @@
 # Author  : Pawel Krupa (paulfantom)
 # Author  : Pavlos Emm. Katsoulakis (paul@netdata.cloud)
 
-set -e
-
-
-# If we are not in netdata git repo, at the top level directory, fail
-TOP_LEVEL=$(basename "$(git rev-parse --show-toplevel)")
-CWD=$(git rev-parse --show-cdup || echo "")
-if [ -n "${CWD}" ] || [ ! "${TOP_LEVEL}" == "netdata" ]; then
-    echo "Run as .travis/$(basename "$0") from top level directory of netdata git repository"
-    echo "Changelog generation process aborted"
-    exit 1
-fi
-
-
 # Figure out what will be new release candidate tag based only on previous ones.
 # This assumes that RELEASES are in format of "v0.1.2" and prereleases (RCs) are using "v0.1.2-rc0"
 function set_tag_release_candidate() {
 	LAST_TAG=$(git semver)
-	echo "Last tag found is: ${LAST_TAG}"
+	echo "${0}: Last tag found is: ${LAST_TAG}"
 
 	if [[ $LAST_TAG =~ -rc* ]]; then
 		VERSION=$(echo "$LAST_TAG" | cut -d'-' -f 1)
@@ -47,31 +30,32 @@ function set_tag_release_candidate() {
 	else
 		VERSION="$(git semver --next-minor)"
 		RC=0
-		echo "Warning: Will set version to ${VERSION} (Last tag: ${LAST_TAG}) while tagged for release candidate generation"
+		echo "${0}: Warning: Will set version to ${VERSION} (Last tag: ${LAST_TAG}) while tagged for release candidate generation"
 	fi
+
 	GIT_TAG="v${VERSION}-rc${RC}"
+	echo "${0}: Generated a new tag, set to: (${GIT_TAG})"
 }
 
-echo "Determining TAG"
-
-# Check if current commit is tagged or not
-GIT_TAG=$(git tag --points-at)
-
-if [ -z "${GIT_TAG}" ]; then
-	git semver
-	# Figure out next tag based on commit message
-	echo "Last commit message: ${TRAVIS_COMMIT_MESSAGE}"
-	case "${TRAVIS_COMMIT_MESSAGE}" in
-	*"[netdata patch release]"*) GIT_TAG="v$(git semver --next-patch)" ;;
-	*"[netdata minor release]"*) GIT_TAG="v$(git semver --next-minor)" ;;
-	*"[netdata major release]"*) GIT_TAG="v$(git semver --next-major)" ;;
-	*"[netdata release candidate]"*) set_tag_release_candidate ;;
-	*)
-		echo "Keyword not detected. Exiting..."
-		exit 0
-		;;
-	esac
-fi
-
-echo "Setting up GIT_TAG to ${GIT_TAG}"
-export GIT_TAG
+function set_tag_for_release() {
+	echo "${0}: Checking for tag existence"
+	if [ -z "${GIT_TAG}" ]; then
+		echo "${0}: No tag was found, generating a new tag"
+		git semver
+
+		echo "${0}: Last commit message: ${TRAVIS_COMMIT_MESSAGE}"
+
+		# Figure out next tag based on commit message
+		case "${TRAVIS_COMMIT_MESSAGE}" in
+		*"[netdata patch release]"*) GIT_TAG="v$(git semver --next-patch)" ;;
+		*"[netdata minor release]"*) GIT_TAG="v$(git semver --next-minor)" ;;
+		*"[netdata major release]"*) GIT_TAG="v$(git semver --next-major)" ;;
+		*"[netdata release candidate]"*) set_tag_release_candidate ;;
+		*)
+			echo "${0}: Keyword not detected. Nothing to set for GIT_TAG"
+			;;
+		esac
+	else
+		echo "${0}: We seem to already have a GIT_TAG set to (${GIT_TAG})"
+	fi
+}