netdata - Mirror of https://github.com/netdata/netdata

Age	Commit message (Collapse)	Author
2019-09-23	netdata/ci: nits and fixes around package release workflow (#6914)	Paul Emm. Katsoulakis
	* netdata/packaging: bring el/6 first, because this appears on all cases. Otherwise travis craps out the steps * netdata/ci: Attempt to sanitise workflow for master branch commits 1) Rename Packaging for release, now named: Support activities on main branch 2) Adjust instruction messages to reflect more realistically the activities 3) Put labeler first on the stage, since this will always run 4) Make changelog and tag for release conditionally run, so that it does not confuse people * netdata/ci: Sanitise workflow for release more (explained below) We need to clean up tagging logic more and in order to do this we need to bring tagger logic to travis.yml Then we have the complete control of tagging and triggering for changelog generation in travis. To mitigate this, we need to: 1) make tagger script a lirbary. Keep the tagging logic in a method and remove the things needed when it was an executable (checks, executions etc) 2) make travis handle GIT_TAG checking and setting at the beginning. We need this outside of any conditionals so that we have GIT_TAG result available inside the steps 3) COMMIT_TAG_POINTS_AT can now become GIT_TAG, since this is what we actually wanted. We just named differently before to avoid any weird conflicts 4) make changelog generation for release to NOT source tagger any more. Also make it fail in case GIT_TAG is not there, we wont run the script if git tag not in place so that should be error if it happens 5) Rename .travis/generate_changelog_for_release.sh -> .travis/create_changelog_for_release.sh 6) Rename .travis/generate_changelog_and_tag_release.sh -> .travis/generate_changelog_for_release.sh * netdata/ci: reinstate filename here. we basically do the git tag operation here still * netdata/ci: simplify more the changelog generation process 1) Rename create_changelog_for_release.sh -> create_changelog.sh 2) Add comments and the standard path checks in create_changelog.sh 3) Do some minor changes to help diff with nightlies changelog (no functional change though, primarily messages and move flags lower) 4) Inside generate_changelog_for_nightlies.sh, use create_changelog.sh and remove code related to create changelog process 5) in nightlies.sh, use standard shebang adn fix some messages Note: might follow up with more changes, this is just a first batch * netdata/packaging: reinstate accidentally removed OPTS from other iterationns (different engineer)
2019-09-20	Improve changelog generation and add it back to the pipeline (#6900)	Chris Akritidis
	#### Summary Preparation for #6899 Make changelog generation faster and ensure it won't crash. Put it back to the pipeline. ##### Component Name CI/CD ##### Additional Information - Added correct max-issues specification. Note that the number of issues affects the number of displayed PRs. There's no way to ensure that all the PRs in a tag will be displayed, but with a limit of 500 we know that the most recent releases will be complete. - Let the CHANGELOG show links to diffs - Put an invalid Bug tag, to ensure that fixed bugs are not listed (the result with the defaults was incorrect) - Enable running the changelog generation again
2019-07-26	netdata/ci: [ci skip] Do not include issues for now	Pavlos Emm. Katsoulakis

2019-05-03	netdata/packaging/ci: Create manual nightly deployment tool (#5899)	Paul Emm. Katsoulakis
	* netdata/packaging/ci: Create manual nightly deployment tool 1. Alter docker tools to shebang with /usr/bin/env, safer to detect the right BASH (>4.0) 2. Generate script packaging/manual_nightly_deployment.sh, that runs only from TLD of netdata GIT and provides the means to deploy to docker and GCS the latest nightly build of netdata. Only option passed to the script is whether you want to deploy in docker, gcs or both. Note: for GCS deployment the user has to define which bucket to update. Note2: this tool has a hard dependency on gsutil to be properly setup on the development environment * netdata/packaging/ci: Do not change global config for mail and user Since we are now starting to use these scripts externally, we need to make sure we dont mess developer config. It seems that we are touching global config, in an attempt to modify a single commit message. Alter this and instead use the --author option for the single commit that we care about instead of altering the universe Cheers
2019-04-18	Pipeline redesign master (#5885)	Paul Emm. Katsoulakis
	* netdata/packaging/ci: [WIP] Redesign packaging release * netdata/packaging/ci: Add explicitly os definition, also remove sourcing of slack, its already done and rename before_install -> install * netdata/packaging/ci: fix stray stage name * netdata/packaging/ci: style * netdata/packaging/ci: fix syntax for yaml * netdata/packaging/ci: Try removing this, doesnt seem to make much sense to be there (for now) * Revert "netdata/packaging/ci: Try removing this, doesnt seem to make much sense to be there (for now)" This reverts commit ad5556a3ccf6c11ebc9026942af8013d4d0ea666. wow travis - this is weird * netdata/packaging/ci: a few more changes * netdata/packaging/ci: This is totally experimental, will adjust after i test it * netdata/packaging/ci: Reinstate original design - the problem i am trying to solve is more travis-related and fundamental on the way travis works * netdata/packaging/ci: i wonder.. * Revert "netdata/packaging/ci: i wonder.." This reverts commit a160df4fcac866375e8319bf200f47263b2bcaa3. * netdata/packaging/ci: Fix broken items * netdata/packaging/ci: docker images are built and published independently. As long as tagging has taken place earlier, during packaging * netdata/packaging/ci: Following hints from https://github.com/travis-ci/travis-ci/issues/4681 and the mayhem triggered from this change So we adding the compiler, to see if it will get travis to build the right matrix. then linter broke on addons for some weird reason, so remove addons->apt->update too * netdata/packaging/ci: Now do the rest of the changes, since with addition of compiler, things got worse * netdata/packaging/ci: Ok, as expected the workaround from them did not work. Lets get creative though and assume they exclude what they added, so lets do that for os * netdata/packaging/ci: test slack * netdata/packaging/ci: Bring the modified slack script on this branch * netdata/packaging/ci: Add a few more slack messages * Fix empty stage occurrence * netdata/packaging/ci: Another set of restructure changes, explained below 1) Publish release -> Publish for release (minor rename for clarity) 2) Add some more work on the make dist testing (temp folder for root and for now just validate the existence of the tarball) 3) Add some more slack notifications (that is still beta, will have to properly refine once we are done with changes of the flow) 4) Restructure the release process, finally a) relase.sh becomes draft_release.sh and changelog_and_tag_release.sh, basically splitting up the steps into meaningful independent blocks b) Adjust the pipeline, now we run 'changelog_and_tag_release.sh' to get the latest changelog and tag the release, then we run artifacts submission and draft release generation on github still more things to re-check on this step, it will become clearer once i get it up to speed to run on my local repos end to end * netdata/packaging/ci: Fix wrong path usage * netdata/packaging/ci: Update webhook URL * netdata/packaging/ci: Update stage config as proposed by the fix from travis team * netdata/packaging/ci: Another round of restructuring, after 1-1 comparison with old version Briefly: 1) Publish for release stage will only run when there is a TAG defined (To be tested though, but thats the idea). The code previously had the releaser.sh script that did exactly that, if tagger script failed, it would stop process. Now we have distributed the processing in clear chunks and we pick up the process based on shared info that travis or the environment provides 2) Rename <<Create release (only on special commit msg)>> to a more appropriate name, we are not really creating release there its more complicated 3) Move the labeler execution to the packaging step, since publishing now not happening all the time. Previously we hacked this by exit 0 with the tagger and allow the labeler (which was the second part of the execution) to run. 4) before_install in the publishing stage is wrong. We only need this as before_deploy before deploy execution. That is where we use it, really. 5) Rename changelog_and_tag_release.sh to generate_changelog_and_tag_release.sh, just to be clearer. No problem with slightly longer file names 6) Do not let generate_changelog_and_tag_release.sh execute if GIT_TAG is not defined, just exit quietly with a message and a success code 7) Remove the tagger from the generate_changelog_and_tag_release.sh, its confusing to shove it in there. Pull it out and let it run on a clear step from the pipeline but within the same machine to make sure it shares environment with generate_changelog_and_tag_release.sh (So it has to be within the same job) This is it mostly, keep in mind that the nightly and release parts are not tested yet. After this commit, there will be more code that will allow me to execute all the flows on my pipeline and fully test and adjust the workflows as needed * netdata/packaging/ci: Add more notifications, adjust the existing ones too * netdata/packaging/ci: Add safeguards for beta, do a rename too 1) rename generate_changelog.sh -> generate_changelog_for_nightlies.sh, just to provide clarity on the usage. Note: we will simplify this later, this was a workaround that raised a bug on release after refactoring nightlies. We will have to clean this up properly once we merge the new pipeline flow 2) Add checks based on TRAVIS_REPO_SLUG to identify when we running on personal repo and abort mission critical actions like deploying or generating changelogs Note: The deployment step for now will simply fail, due to the lack of the necessary encryption keys on the environment. At a later iteration, we should have beta buckets on GCS and also beta repositories on Docker to properly test our publishing capability on our beta stack * netdata/packaging/ci: Refactor README.md -- still half way there, will resume when i am done with testing * netdata/packaging/ci: At this stage make the scripts harder to run, add checks to bypass tested execution. * netdata/packaging/ci: Seems that adding the install tag within the stage, kind of overriding the outer install pull out install items from packaging to the main install, no point to isolate two packages, just install all required all the time * netdata/packaging/ci: tag obviously cant work at that step, moving it down to the jobs and i need to recheck if the syntax is applicable for travis too * netdata/packaging/ci: So the jobs in a list dont share the environment, lets see if we do it otherwise. also remove the tag check, not work as expected * netdata/packaging/ci: Add another way to check the RC condition * netdata/packaging/ci: experimental - check conditions on message * netdata/packaging/ci: Introduce an alternate master branch for the workflow testing 1. I created pipeline-redesign-master that will be the master branch for my use cases 2. Modified travis to trigger all production workflows on that branch, so that i can experiment freely with test commits to trigger process and verify execution flow * remove spaces * netdata/packaging/ci: comment out * netdata/packaging/ci: wording nit * netdata/packaging/ci: adjust the condition properly * netdata/packaging/ci: test condition for triggering on specific keywords * netdata/packaging/ci: try different syntax * netdata/packaging/ci: remove depth, causes problems as we dont get all required objects from repo (adds --depth 50 in clone) * Missing equal sign in regex? * netdata/packaging/ci: Update tagger script 1) More verbose messages 2) Update comments copyrights etc 3) Rename release_candidate to set_tag_release_candidate 4) Add curlies on the variables, also change the way we check for path * netdata/packaging/ci: reinstate git depth * netdata/packaging/ci: Remove git depth from packaging too * netdata/packaging/ci: RTFM - was using wrong syntax for slack @here approach * netdata/packaging/ci: Using install tag at the stage layer, overrides the global one for some reason. fix that * netdata/packaging/ci: Dont use the pattern matching from travis, do it outside with a script * netdata/packaging/ci: Do it the other way, script execution doesnt work there * netdata/packaging/ci: Attempt to fix conditional on release, remove the faulty assumptions as provided by travis team * netdata/packaging/ci: reintroduce the tag checking * netdata/packaging/ci: you need to bring tag checking at the stage, as it is calculated during packaging step * netdata/packaging/ci: you cant escape that one for the moment, merge them and we will rework it later again * netdata/packaging/ci: dont forget git depth * netdata/packaging/ci: [WIP] Redesign packaging release * netdata/packaging/ci: Add explicitly os definition, also remove sourcing of slack, its already done and rename before_install -> install * netdata/packaging/ci: fix stray stage name * netdata/packaging/ci: Try removing this, doesnt seem to make much sense to be there (for now) * Revert "netdata/packaging/ci: Try removing this, doesnt seem to make much sense to be there (for now)" * netdata/packaging/ci: Reinstate original design - the problem i am trying to solve is more travis-related and fundamental on the way travis works * netdata/packaging/ci: Fix broken items * netdata/packaging/ci: docker images are built and published independently. As long as tagging has taken place earlier, during packaging * netdata/packaging/ci: Following hints from https://github.com/travis-ci/travis-ci/issues/4681 and the mayhem triggered from this change So we adding the compiler, to see if it will get travis to build the right matrix. then linter broke on addons for some weird reason, so remove addons->apt->update too * netdata/packaging/ci: Now do the rest of the changes, since with addition of compiler, things got worse * netdata/packaging/ci: Ok, as expected the workaround from them did not work. Lets get creative though and assume they exclude what they added, so lets do that for os * netdata/packaging/ci: test slack * netdata/packaging/ci: Bring the modified slack script on this branch * netdata/packaging/ci: Add a few more slack messages * Fix empty stage occurrence * netdata/packaging/ci: Another set of restructure changes, explained below 1) Publish release -> Publish for release (minor rename for clarity) 2) Add some more work on the make dist testing (temp folder for root and for now just validate the existence of the tarball) 3) Add some more slack notifications (that is still beta, will have to properly refine once we are done with changes of the flow) 4) Restructure the release process, finally a) relase.sh becomes draft_release.sh and changelog_and_tag_release.sh, basically splitting up the steps into meaningful independent blocks b) Adjust the pipeline, now we run 'changelog_and_tag_release.sh' to get the latest changelog and tag the release, then we run artifacts submission and draft release generation on github still more things to re-check on this step, it will become clearer once i get it up to speed to run on my local repos end to end * netdata/packaging/ci: Fix wrong path usage * netdata/packaging/ci: Update webhook URL * netdata/packaging/ci: Update stage config as proposed by the fix from travis team * netdata/packaging/ci: Another round of restructuring, after 1-1 comparison with old version Briefly: 1) Publish for release stage will only run when there is a TAG defined (To be tested though, but thats the idea). The code previously had the releaser.sh script that did exactly that, if tagger script failed, it would stop process. Now we have distributed the processing in clear chunks and we pick up the process based on shared info that travis or the environment provides 2) Rename <<Create release (only on special commit msg)>> to a more appropriate name, we are not really creating release there its more complicated 3) Move the labeler execution to the packaging step, since publishing now not happening all the time. Previously we hacked this by exit 0 with the tagger and allow the labeler (which was the second part of the execution) to run. 4) before_install in the publishing stage is wrong. We only need this as before_deploy before deploy execution. That is where we use it, really. 5) Rename changelog_and_tag_release.sh to generate_changelog_and_tag_release.sh, just to be clearer. No problem with slightly longer file names 6) Do not let generate_changelog_and_tag_release.sh execute if GIT_TAG is not defined, just exit quietly with a message and a success code 7) Remove the tagger from the generate_changelog_and_tag_release.sh, its confusing to shove it in there. Pull it out and let it run on a clear step from the pipeline but within the same machine to make sure it shares environment with generate_changelog_and_tag_release.sh (So it has to be within the same job) This is it mostly, keep in mind that the nightly and release parts are not tested yet. After this commit, there will be more code that will allow me to execute all the flows on my pipeline and fully test and adjust the workflows as needed * netdata/packaging/ci: Add more notifications, adjust the existing ones too * netdata/packaging/ci: Add safeguards for beta, do a rename too 1) rename generate_changelog.sh -> generate_changelog_for_nightlies.sh, just to provide clarity on the usage. Note: we will simplify this later, this was a workaround that raised a bug on release after refactoring nightlies. We will have to clean this up properly once we merge the new pipeline flow 2) Add checks based on TRAVIS_REPO_SLUG to identify when we running on personal repo and abort mission critical actions like deploying or generating changelogs Note: The deployment step for now will simply fail, due to the lack of the necessary encryption keys on the environment. At a later iteration, we should have beta buckets on GCS and also beta repositories on Docker to properly test our publishing capability on our beta stack * netdata/packaging/ci: Refactor README.md -- still half way there, will resume when i am done with testing * netdata/packaging/ci: At this stage make the scripts harder to run, add checks to bypass tested execution. * netdata/packaging/ci: Seems that adding the install tag within the stage, kind of overriding the outer install pull out install items from packaging to the main install, no point to isolate two packages, just install all required all the time * netdata/packaging/ci: tag obviously cant work at that step, moving it down to the jobs and i need to recheck if the syntax is applicable for travis too * netdata/packaging/ci: So the jobs in a list dont share the environment, lets see if we do it otherwise. also remove the tag check, not work as expected * netdata/packaging/ci: Add another way to check the RC condition * netdata/packaging/ci: experimental - check conditions on message * netdata/packaging/ci: Introduce an alternate master branch for the workflow testing 1. I created pipeline-redesign-master that will be the master branch for my use cases 2. Modified travis to trigger all production workflows on that branch, so that i can experiment freely with test commits to trigger process and verify execution flow * remove spaces * netdata/packaging/ci: comment out * netdata/packaging/ci: wording nit * netdata/packaging/ci: adjust the condition properly * netdata/packaging/ci: test condition for triggering on specific keywords * netdata/packaging/ci: try different syntax * netdata/packaging/ci: remove depth, causes problems as we dont get all required objects from repo (adds --depth 50 in clone) * Missing equal sign in regex? * netdata/packaging/ci: Update tagger script 1) More verbose messages 2) Update comments copyrights etc 3) Rename release_candidate to set_tag_release_candidate 4) Add curlies on the variables, also change the way we check for path * netdata/packaging/ci: reinstate git depth * netdata/packaging/ci: Remove git depth from packaging too * netdata/packaging/ci: RTFM - was using wrong syntax for slack @here approach * netdata/packaging/ci: Using install tag at the stage layer, overrides the global one for some reason. fix that * netdata/packaging/ci: Dont use the pattern matching from travis, do it outside with a script * netdata/packaging/ci: Do it the other way, script execution doesnt work there * netdata/packaging/ci: Attempt to fix conditional on release, remove the faulty assumptions as provided by travis team * netdata/packaging/ci: reintroduce the tag checking * netdata/packaging/ci: you need to bring tag checking at the stage, as it is calculated during packaging step * netdata/packaging/ci: you cant escape that one for the moment, merge them and we will rework it later again * netdata/packaging/ci: dont forget git depth * netdata/packaging/ci: bring back the dist checking * netdata/packaging/ci: fix syntax * netdata/packaging/ci: no depth * netdata/packaging/ci: Introduce make dist validation tasks: Install from dist file and verify service is starting up * netdata/packaging/ci: Add a couple more information for troubleshooting * netdata/packaging/ci: Add extra information prior to executing crucial operations * netdata/packaging/ci: Make rm error soft here * netdata/packaging/ci: This is not needed here * netdata/packaging/ci: Prepare for beta stage deployment process - introduce a beta provider - adjust key decryption process, different keys for beta, different for prod - Enter conditions to deploy beta only on beta process and production only on production process - Use travis variables, to allow multiple users have beta stages for deployment, if they want * netdata/packaging/ci: add one more message * netdata/packaging/ci: Bring back naming to the actual master branch * netdata/packaging/ci: rename beta to dev * netdata/packaging/ci: Make permanent restrictions for some of the stricts, so that we don't mess things up * netdata/packaging/ci: Update README that was missed and also make sure git depth false is at the places needed * netdata/packaging/ci: fix codacy complaints