summaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)Author
2021-09-16Add CLI parameters for tree-of subcommand to add condition-checking dataMatthias Beyer
This patch adds parameters for the tree-of subcommand which can be used to change the package-DAG-building based on conditional requirements. If a package has a dependency-condition that only includes a dependency if there is a certain image used to build, the "image" parameter can be used to inspect the package tree (DAG) build with that image in mind. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-16Add passing of data for condition-checkMatthias Beyer
This patch extends the interface for building the package DAG with a parameter for the data that is required to check conditions for conditional dependencies. A "DTO" type is added for this data. The actual condition-checking is not implemented in this patch. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-16Add conditional dependency support for runtime dependency typeMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-16Add conditional dependency support for build dependency typeMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-16Add getters for Condition membersMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-16Add conditional dependency variantMatthias Beyer
This patch adds a conditional dependency variant to the build/runtime dependency deserialization type(s). Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-16Add types for conditional dependenciesMatthias Beyer
This patch adds the first implementation for conditional dependencies in the dependency fields (that is the "build" and "runtime" keys in the package defintion). This is only the deserialization-interface, not the actual condition resolving code. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-16Rewrite Dependency type as enumMatthias Beyer
This patch reimplements the Dependency type as an enum. This is required because later we want to be able to use an "if" expression in the build dependencies, for example (not decided yet) something like runtime = [ { d = "libfoobar", if = [ { has_env = [ "TARGET_RH7" ]; } ] } ] (no valid toml here, formatting for readability) to make a dependency optional on a condition. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-16Add test: parsing DependencyMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-16Rewrite BuildDependency type as enumMatthias Beyer
This patch reimplements the BuildDependency type as an enum. This is required because later we want to be able to use an "if" expression in the build dependencies, for example (not decided yet) something like build = [ { d = "libfoobar", if = [ { has_env = [ "TARGET_RH7" ]; } ] } ] (no valid toml here, formatting for readability) to make a dependency optional on a condition. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-16Add test: parsing BuildDependencyMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-16Schedule to host where fewer containers are runningMatthias Beyer
This is not perfect, of course, as we rather should check the load on the host rather than some artificial number of running containers (more failed jobs result in more running containers, which results in more scheduled to the host), but still, this is an improvement over the previous state because utilization did not express enough. So the issue was that the utilization only represents the utilization of an endpoint _from the current process_. If butido is called three times, the same host was selected for a job because the utilization in each butido process resulted in the same values for this endpoint. In practice, this resulted in a build for "gcc" to be scheduled to the same host when butido was executed three times for "gcc" (e.g. for different platforms). The endresult was a load of ~70 on a host where 24 was "100% load". This is not ideal. Checking the load of a host is not possible via the docker API, so this is what I came up with. Now, builds are scheduled to endpoints which have fewer running containers. This might not result in a perfect distribution (or even a "good" one), but still it might result in a better distribution of workload than in before this patch. Some completely non-artificial testing resulted in the expected behaviour. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-16Add trace output when altering handle bookkeeping of running jobsMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-16Add fn to get number of running containers on endpointMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-16Add debug output of utilization of endpointMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-16Fix clippy: Remove needless borrowMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-16Fix clippy: Remove needless borrowMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-16Fix clippy: Remove needless borrowMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-09Merge branch 'rewrite-find-artifact'Matthias Beyer
2021-09-09Merge branch 'submits-with-pkg-name-version'Matthias Beyer
2021-09-09Merge branch 'submits-on-image'Matthias Beyer
2021-09-09Add --image flag to "db submits" command for filtering for a specific imageMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-09Add package name/version output in "db submits" commandMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-06Add CLI option to call find-artifacts subcommand with filter for imageMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-06Fix: Only reuse artifacts that were built on the same imageMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-06Add support for filtering by image nameMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-06Refactor: Use ? operator to remove indention levelMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-09-06Rewrite interface for finding artifacts with builderMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-08-31Pin dependency clapMatthias Beyer
Because we didn't pin the dependency in Cargo.toml, a `cargo update` or fresh build resulted in beta.4 being used and fails because a new feature was introduced in newer compilers. Hence, pin the clap version to exactly this version for now. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-08-31Add CHANGELOG for v0.2.1Matthias Beyer
Signed-off-by: Matthias Beyer <mail@beyermatthias.de> (cherry picked from commit 414fc4ff1f0aac20867728634af44ea92ea72b10)
2021-08-30Merge branch 'fix-patch-loading'Matthias Beyer
2021-08-30Fix clippy: Use or_insert_with() for less runtime costMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-08-30Fix clippy: Use character constantMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-08-30Fix: implement unimplemented!()Matthias Beyer
This fixes a missed unimplemented!() in the codebase. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-08-30Fix: Do not join patch path with root directory pathMatthias Beyer
This fixes the relative pathes for the patch files by not joining it with the root directory path. The exists-check was removed, because the filtering before that already does that. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-08-27Fix: Remove unused importsMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-08-27Merge branch 'fast-repository-loading'Matthias Beyer
After some completely non-scientific benchmarking, these patches bring down the loading time for a real-world repository from over 20 sec to about 17 sec on debug build of butido and to 5 sec in release build and even to 2 sec after the filesystem cache got a bit warm. So I guess we can safely merge this now and be happy with it (and go back to the more important issues).
2021-08-27Fix: Replace unimplemented!() match arm with bail!()Matthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-08-27Optimize: If there is a PathComponent::PkgToml, we can return true right awayMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-08-27Split repository::fs into modules and expose only necessary interfaceMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-08-27Add documentation to repository::fs moduleMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-08-26Reimplement Repository::load() with parallelismMatthias Beyer
A quick test showed that the parallel approach here reduced the loading time by approx 10%. Not good, but better than before. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-08-26Make sure we open as many files as possible in the recursionMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-08-26Fix: Do not push the full (absolute) path but only the relative oneMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-08-26Fix: do not filter out directoriesMatthias Beyer
Because the filter_entry() method is expected for the recursion to happen, we should not filter out directories here (except hidden ones). Rather filter them out later in the process. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-08-26Add getter for FileSystemRepresentation::rootMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-08-26Reimplement repository loadingMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-08-26Return the path to the file that was loaded as wellMatthias Beyer
Because we need that to find the patches the file may reference. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-08-26Add PathComponent::dir_name() helper fnMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-08-26Add getter for FileSystemRepresentation::filesMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-12 15:58:44 +0000