summaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)Author
2021-03-07result-inspect: 0.1 -> 0.2Matthias Beyer
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-03-04Remove relative speed setting, select endpoint by utilization insteadMatthias Beyer
This patch removes the "speed" setting from the configuration, which was introduced to set a relative speed for each endpoint, with the idea that the scheduler then would select a faster node preferably. Instead, the utilization of an endpoint is now calculated (number of running jobs vs allowed maximum jobs on the endpoint), and the endpoint with lower utilization is selected. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-04Remove counting of package filesMatthias Beyer
This removes the counting of package files and using that value to set the length of the progress bar. This was just overhead and because of the duration of the bars to be visible (very short) simply not necessary. Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-03-04Do not pass progress bars by value, but by referenceMatthias Beyer
Because passing by value is simply not necessary here. Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-03-04Merge branch 'fix-env-filter-jobs'Matthias Beyer
2021-03-04Merge branch 'subcommand-versions'Matthias Beyer
2021-03-04Merge branch 'sched-improvements'Matthias Beyer
2021-03-04Add version for each subcommandMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-04Fix: Implementing jobs filtering only by envMatthias Beyer
Before this patch, one was only able to filter jobs by env if they were also filtered by submit. This patch fixes this by implementing env filtering for not-submit-filtered jobs. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-02Add comment what the "maxjobs" setting meansMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-02Implement scheduling with max jobs per endpointMatthias Beyer
This patch implements support for max jobs per endpoint. The number of running jobs on one endpoint are tracked with a wrapper around the Endpoint object, which increases the job counter on allocation and decreases it on deallocation. This way, the scheduler can know how many jobs are running on one endpoint and select the next endpoint accordingly. The loading/comparing is not perfect, so it might happen that more jobs run on one endpoint than configured, but this is the first step into the right direction. Also, the selection happens on a tokio job which runs in a loop{}. Because this almost blocks the whole executor thread, we use `tokio::task::yield_now()` as soon as there is no free endpoint anymore, to yield the execution to another future to free resources for doing actual work, not scheduling. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-01Remove RwLock around EndpointMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-01Merge branch 'remove-release-rm-artifact'Matthias Beyer
2021-03-01Merge branch 'release-update'Matthias Beyer
2021-03-01Merge branch 'git-info-to-container'Matthias Beyer
2021-03-01Add feature to pass git author and git commit information to containerMatthias Beyer
This patch implements the feature to be able to pass author and commit hash information from the repository to the container. This can be used to set packager or package description commit hash inside the build container, if desired. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-01Refactor: Open git repository in command::build()Matthias Beyer
This commit refactors the git2::Repository::open() call to be in the command::build() implementation, so we do not open the repository multiple times. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-01Refactor: Environment variable aggregationMatthias Beyer
This patch refactors the collecting of the environment variables in the `RunnableJob::build_from_job()` implementation as well as in the `RunnableJob::environemtn()` implementation. This results in fewer allocations, especially but not only because the `RunnableJob::environment()` function returns an iterator now and all clone() calls were removed. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-01Refactor: Collect environment resources before building resources for ↵Matthias Beyer
RunnableJob Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-01Fix: Remove idea to be able to remove an artifact by its filepathMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-01Add flag to be able to update without confirmation promptMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-01Add release update functionalityMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-01Fix: Ignore conflict on duplicated release store nameMatthias Beyer
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-03-01Implement "db releases" subcommandMatthias Beyer
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-03-01Implement "release rm" subcommandMatthias Beyer
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-02-25Merge branch 'multiple-release-stores'Matthias Beyer
2021-02-25Multiple release storesMatthias Beyer
This patch adds the ability to have more than one release store. With this patch, a user can (has to) configure release store names in the configuration file, and can then specify one of the configured names to release the artifacts to. This way, different release "channels" can be served, for example a stable channel and a rolling release channel (although "channel" is not in our wording). The code was adapted to be able to fetch releases from multiple release directories, in the crate::db::find_artifact implementation, so that re-using artifacts works across all release directories. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-25Add release stores in databaseMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-25Fix: Make sure multiprogress bars are hidden, if we hide progress barsMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-24Remove unused stuff from development shellMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-24Move constants to dedicated moduleMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-22Add required "release_stores" configurationMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-22Merge branch 'misc'Matthias Beyer
2021-02-22Merge branch 'remove-merged-store'Matthias Beyer
2021-02-22Merge branch 'staging-reuse-is-submit-reuse'Matthias Beyer
2021-02-22Fix: Debug output should match type nameMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-22Remove MergedStoresMatthias Beyer
The concept of the MergedStores type was okay in the beginning, but it got more and more complex to use properly and most of the time, we used the release/staging stores directly anyways. So this removes the MergedStores type, which is a preparation for the change to have multiple release stores. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-22Fix: Add missing license headerMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-22Fix: Add missing license headerMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-22Fix: Re-use submit UUID if staging directory is overwrittenMatthias Beyer
This patch changes the behaviour if the staging directory is re-used. If the staging directory is re-used, we take the submit UUID from there instead of generating a new one. This way, database entries are added to the already existing submit instead of added to a new one, but the artifacts are put into the staging store of the reused one. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-22Remove printlnMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-21Merge branch 'update-deps'Matthias Beyer
2021-02-21Update typed-builder: 0.8 -> 0.9Matthias Beyer
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-02-21Update shiplift: git -> 0.7Matthias Beyer
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-02-17Pass MergedStores to endpoint/jobs rather than only staging storeMatthias Beyer
This patch changes the code so that the MergedStores object is known in the endpoints and job execution code. This is necessary, because the jobs must be able to load artifacts from the release store as well, for example if a library was released in another submit and we can reuse it because the script for that library didn't change. For that, the interface of StoreRoot::join() was changed - -> Result<FullArtifactPath<'a>> + -> Result<Option<FullArtifactPath<'a>>> whereas it returns an Ok(None) if the artifact requested does not exist. This was necessary to try-joining a path on the store root of the staging store, and if there is no such file continue with try-joining the path on the release store. The calling code got a bit more complex with that change, though. Next, the MergedStores got a `derive(Clone)` because clone()ing it is cheap (two `Arc`s) and necessary to pass it easily to the jobs. Each instance of the code where the staging store was consulted was changed to consult the the release store as well. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-16Fix find_artifacts() to not fail if FS object movedMatthias Beyer
This patch fixes a bug in the crate::db::find_artifacts() implementation, which caused the function to fail if the filesystem object was removed. For example, if a package X was built and released, but then removed on the filesystem, this function would try to find that object but fail and cause butido to stop. Because that's not what we want, this function does not fail anymore but simply ignore the not-found object and return nothing for it. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-16Rewrite artifact searching in replacement algorithmMatthias Beyer
This patch rewrites the replacement searching algorithm, to try the staging store first and then the release store. It does so by sorting the artifacts by whether they are in the staging store or not (hence the FullArtifactPath::is_in_staging_store() function). It filters out not-found artifacts and returns only ones that were found in either store. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-15Fix: Pass additional environment from job definition when searching for ↵Matthias Beyer
replacement artifacts Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-15Add more trace output when filtering jobs by ENVMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-15No need to map () to ()Matthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
generated by cgit v1.2.3 (git 2.25.1) at 2024-07-02 10:40:38 +0000