netdata/packaging: Binary distributions - clean up .DEB package generation process (#6465)

* netdata/packaging: [ci skip] reinstate dev branch to help testing

* netdata/packaging: [ci skip] Yes, make the right package choice on json-c for debian based distros..

* netdata/packaging: [ci skip] two more packages that differentiate on debian/ubuntu...............

* netdata/packaging: [ci skip] Add more package dependencies around building

* netdata/packaging: [ci skip] Introduce changelog generation technique for .DEB

1) Remove contrib/debian/changelog from gitignore, we want it in
2) Add variables to monitor latest release version and latest release date from git
3) Add a templated contrib/debian/changelog, that will be the starting point for our changelog generation.
   Either on stable release or nightly releases, we will be generating the logs since the previous release, to manage the content.
   Note: We might eventually take the effort to produce one large changelog and append each time the extra stuff
4) Add the steps to prepare the source and the changelog for building and execute the build

At this stage things should still be broken, but we should have come closer to a ready-to-build-the-package environment per distro

* netdata/packaging: [ci skip] Extract to specified directory, otherwise we dump it on a privileged folder using an unprivileged user (builder)

Also, a fix on sed command, missed the s keyword on the string
replacement

* netdata/packaging: [ci skip] Should be in-place replacement, use -i instead of -e

* netdata/packaging: [ci skip] wrong parameter usage

* netdata/packaging: [ci skip] factor out the two build commands to run as a single bundle. refactor trigger deb build script respectively

* netdata/packaging: [ci skip] Be sure to copy over the build script to the build path

* netdata/packaging: [ci skip] OCD - unnecessary s added

* netdata/packaging: [ci skip] Changelog generation should take place on the repo side, then copy it over to the destination (If that doesnt work well, will just provide a static link for changelog

* netdata/packaging: [ci skip] missed the ignore branch option
	Also, a nit on the gbp command

* netdata/packaging: [ci skip] link to the original path, then create symlink on build folder
                             Also, fix --since option

* netdata/packaging: [ci skip] Run replacement of changelog fields on the host, not on the container

* netdata/packaging: [ci skip] fix contrib path, you are no longer reading from container path

* netdata/packaging: [ci skip] fix version

* netdata/packaging: [ci skip] add dep

* netdata/packaging: [ci skip] fix changelog template

* netdata/packaging: [ci skip] Fix debian package build, we were looping through forever. Exclude contrib all together

* netdata/packaging: [ci skip] Add logic to gather all package related content from the container folder and push it for publishing

* netdata/packaging: [ci skip] fixes and nits
1) use the branch for the beta deployment
2) change the cp pattern

* netdata/packaging:[ci skip] add more debug info

* netdata/packaging: [ci skip] Adjust LXC directory permissions, so that the rest of the workflow can pull through

* netdata/packaging: [ci skip] Using expressions within double quotes doesnt work, use quotes only on the variable to avoid syntax warnings from shellchecking

* netdata/packaging: [ci skip] Update build deps

* netdata/packaging: Add runtime deps (hopefully)

* netdata/packaging: [ci skip] remove old distro

* netdata/packaging: [ci skip] dont let that space there, breaks the syntax

* netdata/packaging: [ci skip] missed one

* netdata/packaging: [ci skip] yank yank yank

* netdata/packaging: [ci skip] You missed www over there

* netdata/packaging: [ci skip] Remove broken distros (no container images). We will revisit the way to support multiple distros in a different way, lxc failed us

* netdata/packaging: Pull in go.d plugin, also fix chmod command to just pass the folder since with the -R doesnt require anything else

* netdata/packaging: [ci skip] Get cups, freeipmi and nfacct on

* netdata/packaging: [ci skip] pull different package in

* netdata/packaging: [ci skip] Install dependencies on the build machine

* netdata/packaging: [ci skip] dont do post-clean, not on all versions

* netdata/packaging: [ci skip] move go install seperately

* netdata/packaging: [ci skip] Towards a more appropriate go.d plugin install

* netdata/packaging: [ci skip] fix permissions setting

* netdata/packaging: [ci skip] fix freeipmi package install

* netdata/packaging: [ci skip] Fix build arch setting on the build command, also remove pre-clean, it breaks on debian/jessie

* netdata/packaging: [ci skip] fix postinst actions, we use libexec

* netdata/packaging: adjust www and go.d permissions

* netdata/packaging: [ci skip] Handle the different build tool version cases, also take care some dependency issues on older version jessie

* netdata/packaging: [ci skip] adjust the control file based on the distro

* netdata/packaging: [ci skip] Move control file preparation prior to packaging the source

* netdata/packaging: [ci skip] Adjust versioning scheme for the nightlies

* netdata/packaging: [ci skip] Let RPM be built too for testing

* netdata/packaging: [ci skip] Factor out version calculation

* netdata/packaging: [ci skip] Attempt to modify version properly

* netdata/packaging: [ci skip] Update documentationg regarding .DEB files

* netdata/packaging: [ci skip] autogen not available everywhere, also fix json-c deps

* netdata/packaging: Remove development settings, also remove beta deployment as its not needed

6 files changed, 197 insertions, 10 deletions

diff --git a/contrib/debian/changelog b/contrib/debian/changelog
new file mode 100644
index 0000000000..b635606ab9
--- /dev/null
+++ b/contrib/debian/changelog
@@ -0,0 +1,5 @@
+netdata (PREVIOUS_PACKAGE_VERSION) unstable; urgency=medium
+
+  * Initial Release
+
+-- Netdata Builder <bot@netdata.cloud> PREVIOUS_PACKAGE_DATE
diff --git a/contrib/debian/control b/contrib/debian/control
index 0f4f1bc059..36e5d6bc61 100644
--- a/contrib/debian/control
+++ b/contrib/debian/control
@@ -4,7 +4,22 @@ Build-Depends: debhelper (>= 9),
                dh-systemd (>= 1.5),
                dpkg-dev (>= 1.13.19),
                zlib1g-dev,
-               uuid-dev
+               uuid-dev,
+               libuv1-dev,
+               liblz4-dev,
+               libjudy-dev,
+               libssl-dev,
+               libmnl-dev,
+               libjson-c-dev,
+               libcups2-dev,
+               libipmimonitoring-dev,
+               libnetfilter-acct-dev,
+               autogen,
+               autoconf,
+               automake,
+               pkg-config,
+               curl,
+               gcc
 Section: net
 Priority: optional
 Maintainer: Costa Tsaousis <costa@tsaousis.gr>
@@ -16,6 +31,17 @@ Architecture: any
 Depends: adduser,
          libcap2-bin (>= 1:2.0),
          lsb-base (>= 3.1-23.2),
+         zlib1g,
+         libuuid1,
+         libuv1,
+         liblz4-1,
+         libjudydebian1,
+         openssl,
+         libmnl0,
+         libjson-c3,
+         cups,
+         freeipmi,
+         libnetfilter-acct1,
          ${misc:Depends},
          ${shlibs:Depends}
 Description: real-time charts for system monitoring
diff --git a/contrib/debian/control.jessie b/contrib/debian/control.jessie
new file mode 100644
index 0000000000..7478d2e3cc
--- /dev/null
+++ b/contrib/debian/control.jessie
@@ -0,0 +1,49 @@
+Source: netdata
+Build-Depends: debhelper (>= 9),
+               dh-autoreconf,
+               dh-systemd (>= 1.5),
+               dpkg-dev (>= 1.13.19),
+               zlib1g-dev,
+               uuid-dev,
+               liblz4-dev,
+               libjudy-dev,
+               libssl-dev,
+               libmnl-dev,
+               libjson-c-dev,
+               libcups2-dev,
+               libipmimonitoring-dev,
+               libnetfilter-acct-dev,
+               autogen,
+               autoconf,
+               automake,
+               pkg-config,
+               curl,
+               gcc
+Section: net
+Priority: optional
+Maintainer: Costa Tsaousis <costa@tsaousis.gr>
+Standards-Version: 3.9.6
+Homepage: https://github.com/netdata/netdata/wiki
+
+Package: netdata
+Architecture: any
+Depends: adduser,
+         libcap2-bin (>= 1:2.0),
+         lsb-base (>= 3.1-23.2),
+         zlib1g,
+         libuuid1,
+         liblz4-1,
+         libjudydebian1,
+         openssl,
+         libmnl0,
+         libjson-c3,
+         cups,
+         freeipmi,
+         libnetfilter-acct1,
+         ${misc:Depends},
+         ${shlibs:Depends}
+Description: real-time charts for system monitoring
+ Netdata is a daemon that collects data in realtime (per second)
+ and presents a web site to view and analyze them. The presentation
+ is also real-time and full of interactive charts that precisely
+ render all collected values.
diff --git a/contrib/debian/install_go.sh b/contrib/debian/install_go.sh
new file mode 100755
index 0000000000..17a3b409e9
--- /dev/null
+++ b/contrib/debian/install_go.sh
@@ -0,0 +1,96 @@
+#!/usr/bin/env bash
+
+LIB_DIR="$1"
+LIBEXEC_DIR="$2"
+
+# ############################################################
+# Package Go within netdata (TBD: Package it separately)
+safe_sha256sum() {
+	# Within the contexct of the installer, we only use -c option that is common between the two commands
+	# We will have to reconsider if we start non-common options
+	if command -v sha256sum >/dev/null 2>&1; then
+		sha256sum $@
+	elif command -v shasum >/dev/null 2>&1; then
+		shasum -a 256 $@
+	else
+		fatal "I could not find a suitable checksum binary to use"
+	fi
+}
+
+download_go() {
+	url="${1}"
+	dest="${2}"
+
+	if command -v curl >/dev/null 2>&1; then
+		curl -sSL --connect-timeout 10 --retry 3 "${url}" > "${dest}"
+	elif command -v wget >/dev/null 2>&1; then
+		wget -T 15 -O - "${url}" > "${dest}"
+	else
+		echo >&2
+		echo >&2 "Downloading go.d plugin from '${url}' failed because of missing mandatory packages."
+		echo >&2 "Either add packages or disable it by issuing '--disable-go' in the installer"
+		echo >&2
+		exit 1
+	fi
+}
+
+install_go() {
+	# When updating this value, ensure correct checksums in packaging/go.d.checksums
+	GO_PACKAGE_VERSION="v0.7.0"
+	ARCH_MAP=(
+		'i386::386'
+		'i686::386'
+		'x86_64::amd64'
+		'aarch64::arm64'
+		'armv64::arm64'
+		'armv6l::arm'
+		'armv7l::arm'
+		'armv5tel::arm'
+	)
+
+	if [ -z "${NETDATA_DISABLE_GO+x}" ]; then
+		echo >&2 "Install go.d.plugin"
+		ARCH=$(uname -m)
+		OS=$(uname -s | tr '[:upper:]' '[:lower:]')
+
+		for index in "${ARCH_MAP[@]}" ; do
+			KEY="${index%%::*}"
+			VALUE="${index##*::}"
+			if [ "$KEY" = "$ARCH" ]; then
+				ARCH="${VALUE}"
+				break
+			fi
+		done
+		tmp=$(mktemp -d /tmp/netdata-go-XXXXXX)
+		GO_PACKAGE_BASENAME="go.d.plugin-${GO_PACKAGE_VERSION}.${OS}-${ARCH}"
+		download_go "https://github.com/netdata/go.d.plugin/releases/download/${GO_PACKAGE_VERSION}/${GO_PACKAGE_BASENAME}" "${tmp}/${GO_PACKAGE_BASENAME}"
+		download_go "https://github.com/netdata/go.d.plugin/releases/download/${GO_PACKAGE_VERSION}/config.tar.gz" "${tmp}/config.tar.gz"
+
+		if [ ! -f "${tmp}/${GO_PACKAGE_BASENAME}" ] || [ ! -f "${tmp}/config.tar.gz" ] || [ ! -s "${tmp}/config.tar.gz" ] || [ ! -s "${tmp}/${GO_PACKAGE_BASENAME}" ]; then
+			echo >&2 "Either check the error or consider disabling it by issuing '--disable-go' in the installer"
+			echo >&2
+			return 1
+		fi
+
+		grep "${GO_PACKAGE_BASENAME}\$" "packaging/go.d.checksums" > "${tmp}/sha256sums.txt" 2>/dev/null
+		grep "config.tar.gz" "packaging/go.d.checksums" >> "${tmp}/sha256sums.txt" 2>/dev/null
+
+		# Checksum validation
+		if ! (cd "${tmp}" && safe_sha256sum -c "sha256sums.txt"); then
+
+			echo >&2 "go.d plugin checksum validation failure."
+			echo >&2 "Either check the error or consider disabling it by issuing '--disable-go' in the installer"
+			echo >&2
+
+			echo "go.d.plugin package files checksum validation failed."
+			exit 1
+		fi
+
+		# Install files
+		tar -xf "${tmp}/config.tar.gz" -C "${LIB_DIR}/conf.d/"
+		mv "${tmp}/$GO_PACKAGE_BASENAME" "${LIBEXEC_DIR}/plugins.d/go.d.plugin"
+	fi
+	return 0
+}
+
+install_go
diff --git a/contrib/debian/netdata.postinst.in b/contrib/debian/netdata.postinst.in
index 29615f5413..44b53ccfbd 100644
--- a/contrib/debian/netdata.postinst.in
+++ b/contrib/debian/netdata.postinst.in
@@ -28,9 +28,10 @@ case "$1" in
         fi
 
         dpkg-statoverride --update --add --force root netdata 0775 /var/lib/netdata/registry
-        chown -R root:netdata /usr/share/netdata/*
-        chown -R root:netdata /usr/lib/@DEB_HOST_MULTIARCH@/netdata/plugins.d
-        setcap cap_dac_read_search,cap_sys_ptrace+ep /usr/lib/@DEB_HOST_MULTIARCH@/netdata/plugins.d/apps.plugin
+        chown -R root:netdata /usr/share/netdata
+        chown -R root:netdata /usr/libexec/netdata/plugins.d
+	chown -R root:netdata /var/lib/netdata/www
+        setcap cap_dac_read_search,cap_sys_ptrace+ep /usr/libexec/netdata/plugins.d/apps.plugin
 
 #PERMS#
         ;;
diff --git a/contrib/debian/rules b/contrib/debian/rules
index c19323960a..88a8ab36a6 100755
--- a/contrib/debian/rules
+++ b/contrib/debian/rules
@@ -17,7 +17,9 @@ TOP = $(CURDIR)/debian/netdata
 	#dh $@ --with autoreconf
 
 override_dh_auto_configure:
-	dh_auto_configure -- --with-math --with-webdir=/var/lib/netdata/www
+	autoreconf -ivf
+	dh_auto_configure -- --prefix=/usr --sysconfdir=/etc --localstatedir=/var \
+	--libexecdir=/usr/libexec --with-user=netdata --with-math --with-webdir=/var/lib/netdata/www
 
 debian/%.postinst: debian/%.postinst.in
 	sed 's/@DEB_HOST_MULTIARCH@/$(DEB_HOST_MULTIARCH)/g' $< > $@
@@ -34,7 +36,7 @@ override_dh_install: debian/netdata.postinst
 	mkdir -p "$(TOP)/usr/share/netdata"
 	for D in $$(find "$(TOP)/var/lib/netdata/www/" -maxdepth 1 -type d -printf '%f '); do \
 		echo Relocating $$D; \
-		mv "$(TOP)/var/lib/netdata/www/$$D" "$(TOP)/usr/share/netdata/$$D"; \
+		mv "$(TOP)/var/lib/netdata/www/$$D" "$(TOP)/usr/share/netdata/www/$$D"; \
 		ln -s "/usr/share/netdata/$$D" "$(TOP)/var/lib/netdata/www/$$D"; \
 	done
 
@@ -51,6 +53,10 @@ override_dh_install: debian/netdata.postinst
 	done
 	sed -i "/^#PERMS#/d" $(CURDIR)/debian/netdata.postinst
 
+	# Install go
+	#
+	debian/install_go.sh $(TOP)/usr/lib/$(DEB_HOST_MULTIARCH)/netdata/ $(TOP)/usr/libexec/netdata
+
 override_dh_installdocs:
 	dh_installdocs
 
@@ -58,10 +64,11 @@ override_dh_installdocs:
 		-name README.md \
 		-not -path './.travis/*' \
 		-not -path './debian/*' \
+		-not -path './contrib/*' \
 		-exec cp \
-						--parents \
-						--target $(TOP)/usr/share/doc/netdata/ \
-						{} \;
+		--parents \
+		--target $(TOP)/usr/share/doc/netdata/ \
+		{} \;
 
 override_dh_fixperms:
 	dh_fixperms
@@ -69,7 +76,10 @@ override_dh_fixperms:
 	# apps.plugin should only be runnable by the netdata user. It will be
 	# given extra capabilities in the postinst script.
 	#
-	chmod 0754 $(TOP)/usr/lib/$(DEB_HOST_MULTIARCH)/netdata/plugins.d/apps.plugin
+	chmod 0754 $(TOP)/usr/libexec/netdata/plugins.d/apps.plugin
+	chmod 0754 $(TOP)/usr/libexec/netdata/plugins.d/freeipmi.plugin
+	chmod 0754 $(TOP)/usr/libexec/netdata/plugins.d/perf.plugin
+	chmod 0750 $(TOP)/usr/libexec/netdata/plugins.d/go.d.plugin
 
 override_dh_installlogrotate:
 	cp system/netdata.logrotate debian/netdata.logrotate