netdata/packaging: Introducing automatic binary packages generation and delivery for RPM types (Phase 1) (#6223)

* netdata/packaging: Introducing automatic binary packages generation and delivery for RPM types (Phase 1)

 With this commit we introduce our packaging toolkits and workflows to automate the delivery for RPM packages to packagecloud.io,
 our packaging content delivery infrastructure. At this stage we have only prepared the required flows to publish our latest stable release.
 Missing items for now:
 1) Packaging versioning - we won't be providing increasing version numbers for the same upstream netdata version
 2) Nightlies - our nightlies won't be provided by this channel, yet

 With this changeset, we also introduce the basic artifacts for DEB packaging generation, with the same misses mentioned above.
 Debian is likely not to be functional within this sprint run though.

 Here's a more detailed list of the changes, as they were generated from the original branch:

89baa00b netdata/packging: Prior to the PR, place the master bracnh as the place to execute packaging
acfd2804 export variables at before_script
b95b112b remove that README.md, we will cover details from the distributions document in packaging folder
e52ab82e Update README.md
8df1f8e9 [ci skip] There is no need for package all option. There is absolutely no uniformity on available images or distro per arch to do our job for all at the same time, so remove it for now
6fc1600c remove a few of the unavailable architectures from distros
7e2741de missed arch, we actually have i686
64c72b4e Add arm64 architecture as a build option (to be refined on follow up commit for the distros not having arm64 or amd64 not to trigger build)
ead672b4 silence shellchecking, also fix a shameful recommendation from shellcheck so that array parsing can work
986d0e00 you disappoint me shellcheck, shame on you
a5c11a75 remove debug ls command
4b5fade6 nit - too obvious to spot
7edb9215 nits and fixes
06cf10b9 fix the obvious miss - you need to the full path appended before the container name, otherwise no folder..
d777f8ae Give me a break travis, whats wrong with you -- define packages directory at the yaml config level
f9756642 nits
ee253493 missed some old parts
1506105c we dont need container root variable here, we just need to go through the base LXC directory
03dc231c restructure packaging preparation process
529cdedb simplify, unify logic around build arch handling between creation and building
17624c14 pull packages directory variable in the code. We calculate base on arch
c47ea326 remove redudant package type var, also hard exit when not found the right stuff
8890041d Pull more arch dependent variables out of yaml config and into the code
832fbf71 remove unused variable
b54d6ccb Move check lower
0b1a3a7e handle build arch requirement, plus some shellcheck fixes
07bb01fe missed this, should pass container name instead
1ab8a3a4 we dont use build arch within this scope, ditch the check
6d3df1b3 Add i386 packaging keywords
1b403680 Remove arch from the descriptions and the environment variables We wont be building sections per arch in travis, to minimize config file size. for each distro we will always build both ARCHS unless the commit message instructs us to build specific architecture
8f38caa5 Drop support of 42.3
8e394906 its word counting here, focus
d3f6009c fix yanking command
5b4d7b9a dont call gpg flag on all distros, focus
7e0f4d64 better listing when yanking packages
bdb2d3aa skipping gpg checks, for now
75841a67 use newer distro instead
a439a9f9 downgrade base dist
eba0fba4 Merge branch 'master' into package-cloud-deployer
241c0e10 nits and fixeS
aad8d8bf fix naming, add update process in debian preparation
872b15af Separately list rpm and srpm contents for yanking
b3ef3a6f el/8 and fedora/31 not ready yet
d468cad7 Null commit
22135a2f fix stages for packaging
1464cd6b Null commit
5d44d4f9 implement yanking - when same version is uploaded, remove to re-add
0a4e2e46 sync the repo first of all
9f71583c Generalize repo tool definition on yaml config, so that we adjust per distro as needed
3988eb74 adjustments on the distro list, also remove coreutils -- not important
51571968 check if its just permission issue
38fa8b2b adjust syntax
e08ebd2b Due to the peculiarity of the way we do things, script out the packages preparation so that we can sudo them
0a3ce2c1 Now that we solved all other issues, make sure you add rpm building dependencies to LXC
aa0b370e a couple of nits and fixes 1) we need sudo within the container 2) just use wget everywhere, dont mix up too many packages for the same thing 3) do not try to add packages again, our dependency script should take care that. Single point of responsibility
a3f42e8d You need to request home directory to be created (need to review under the different distros for the right syntax though)
d7a3d0d0 dont go and add manually the required packages, call our dependency scriptlet
5c0888b7 skip GPG checking for now
9aa64bbe dont create from the python lib
19f8cc6b Add more keywords for better grouping
0d5b89d7 nits
e260a57d python libs not in good shape, try running create from command line instead
7fa58921 netdata/packaging/ci: Bingo - add container name on the RPM jobs
b99ff89e some nits and fixes, lets see
636057b2 bring back sudo, move to /var/lib/lxc, pass -E to sudo to preserve vars
1e487517 i wonder..
97583506 Update SuSE list
2e92649b fix doc - wrong distro details
6f53a4c2 Initial distro document -- WIP
d2ef5e4f Add some README stuff, also introduce yanking step in pre-deploy (empty script at this point)
465f293f Change the token
36431ab5 Add a package cloud wrapper (required for yanking RPMs)
2142a024 Auto-detect username for the beta deployment, so that others can use it. Secret key will remain as is for now, will do revisit this also, so that others can beta test the implementation too. Also include provide for the production deployment, when running on master
ab5a85ad Add more special conditions, also fix spacing and style a little bit more:
ce893eae build only on special conditions (branch name for now)
82340536 Attempt to find how to ref template in stages
5375cd93 one more thing - use ubuntu instead of download template, lets see
4ff26df4 Attempt to fetch lxc-templates (not installed by default
52715ba2 Bring back original implementation - tests gave some light to the problem
4607e7b8 lets see, try different template
4a453425 add some more debug stuff - will remove them once i figure it out
5ee2e03d Try to use the binaries for the creation, see what changes
95a22ef7 another re-arrangement
82855f91 Attempt to install a different version of libraries, as per another issue
fbda0d3a Attempt to debug weird container failure
d828de8c debugging
07bad55d reduce the noise, when not possible to create/modify the required directories dont do anything
6a2bd46b rename
bba22329 netdata/health: shellchecking - SC2236
103a7df9 Adjust naming
ea70d482 Run on Xenial
c2b3ad77 You should sudo create the folder
a2f00c26 Do not stop the build if one of the RPM fails, we want all of them to attempt to build
c367ab6c Cleanup, bring also RPM template in and lets see what happens
ba7b441b more nits -- add the right replacements
2dfbb3f5 move templated parts outside
3211bc11 another approach, missed something
2a47dd17 A more educated attempt based on other resources
670c35b7 first time in anchors, most likely is wrong but got to see what travis will do
6c00628e Binary release flows and other side fixes
1d89e519 Just in case, dont clean up if the env is messed up
06b8de6f start building up deb structure and flow, do some more re-arrangements also
05164ba6 longshot - bring the distro vars on the top to attempt to build a matrix. Also a fix: create the package folder prior populating it
c9334c24 Move RPM code to separate folder, make room for the deb implementations
8474c823 do not copy over everything, only the RPM directories
13d8ac0a netdata/packages: attempt to copy over the folder to a different location
3cb0edf6 remove temporarily to check deploy process
79d3b69c Permissions
05c3d4bd Add listing of container fs contents to see whats the result
eeee2249 fix container path, missed to adjust this after changing container name format
90a70b9c Add dependencies
34c3e1fa Dont forget to create the rpmbuild structure required
9bfb6026 reverse the order, so that we first add all required packages
663af0e5 Attempt to add all as user rather than w00t
0fa68530 more fixes 1) install wget within the container, so that we can then fetch our source 2) factor out the command execution, for cleaner code
23cc4517 better logging
da58043b skip formatting, do the traditional way for now
818508c7 more fixes 1) escape underscore too. Actually previously the complain was about underscore, i wasnt paying attention 2) Parse all command results and make sure you break the script if something goes wrong, otherwise travis wont know the failures
78b9be3a Add more debug messages and also fix the non-escaped character % in string
2037f2c5 fixes and next steps 1) Use sudo, attempt to make privileged containers this way in case there is something messed up with homedirs in travis (long shot) 2) Implement the build process for the next step (not tested, we 'll see how it works) 3) Change the container name, to something more specific to the build we are preparing to help us identify the container more accurately (and avoid possible conflicts)
9dac37bf revert attempt, obviously it didnt provide any value
2d975a7d Attempt to catch any stacktrace, if any is raised
93826afe Reinstate skip_cleanup variable, as its needed for deploy anyway
54fd126a That is weird, but seems like lxc is there on subsequent runs, make sure you clear before you start
f876e6a8 adjustments 1) add version string statically for now, for testing 2) bring more vars in 3) create the container from python all the way
67df8903 Flesh out the workflow for building up the builder environment on one distro
0906b41e change distro, dont forget
17e78ee1 fix distro keyword
26ca17d7 Create an experimental stage and bring onboard an encrypted token
4b0d8ce3 Introduce an experimental package cloud deployment stage

* netdata/packaging: Remove hardcoded tmp director as per Codacy feedback

* netdata/packaging: Update distributions document

Update DISTRIBUTIONS.md based on Codacy warnings (round 1)

* netdata/packaging: lets try to attend a few errors

* Removing DISTRIBUTIONS.md from this PR

As this is part of a separate task, it will be added on a separate branch and linked to the respective task, to unblock RPM generation

1 files changed, 46 insertions, 0 deletions

diff --git a/.travis/README.md b/.travis/README.md
index 03ac2edd62..b7b61ecb4f 100644
--- a/.travis/README.md
+++ b/.travis/README.md
@@ -95,3 +95,49 @@ During packaging we are preparing the release changelog information and run the
 ## Publish for release
 The publishing stage is the most complex part in publishing. This is the stage were we generate and publish docker images,
 prepare the release artifacts and get ready with the release draft.
+
+### Package Management workflows
+As part of our goal to provide the best support to our customers, we have created a set of CI workflows to automatically produce
+DEB and RPM for multiple distributions. These workflows are implemented under the templated stages '_DEB_TEMPLATE' and '_RPM_TEMPLATE'.
+We currently plan to actively support the following Operating Systems, with a plan to further expand this list following our users needs.
+
+### Operating systems supported
+The following distributions are supported
+- Debian versions
+  - Buster (TBD - not released yet, check [debian releases](https://www.debian.org/releases/) for details)
+  - Stretch
+  - Jessie
+  - Wheezy
+
+- Ubuntu versions
+  - Disco
+  - Cosmic
+  - Bionic
+  - artful
+
+- Enterprise Linux versions (Covers Redhat, CentOS, and Amazon Linux with version 6)
+  - Version 8 (TBD)
+  - Version 7
+  - Version 6
+
+- Fedora versions
+  - Version 31 (TBD)
+  - Version 30
+  - Version 29
+  - Version 28
+
+- OpenSuSE versions
+  - 15.1
+  - 15.0
+
+- Gentoo distributions
+  - TBD
+
+### Architectures supported
+We plan to support amd64, x86 and arm64 architectures. As of June 2019 only amd64 and x86 will become available, as we are still working on solving issues with the architecture.
+
+The Package deployment can be triggered manually by executing an empty commit with the following message pattern: `[Package PACKAGE_TYPE PACKAGE_ARCH] DESCRIBE_THE_REASONING_HERE`.
+Travis Yaml configuration allows the user to combine package type and architecture as necessary to regenerate the current stable release (For example tag v1.15.0 as of 4th of May 2019)
+Sample patterns to trigger building of packages for all AMD64 supported architecture:
+- '[Package AMD64 RPM]': Build & publish all amd64 available RPM packages
+- '[Package AMD64 DEB]': Build & publish all amd64 available DEB packages