summaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)Author
2021-02-05Add testcase for diamond dependenciesMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-05Remove "tree" from submitMatthias Beyer
This removes the "tree" column from the "submits" table. This is because we do not store the build-tree in the database anymore. We don't actually need this feature and we can always re-build the tree from an old commit in the repository. Thus, this is not required anymore. Also, it is less easy to do as soon as the internal implementation changes from a "tree" structure to a "DAG" structure. Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-02-05Optimize: Don't duplicate job UUIDMatthias Beyer
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-02-04Merge branch 'test-chain-dependencies'Matthias Beyer
2021-02-04Fix: Ensure job UUIDs are propagated through whole treeMatthias Beyer
This patch changes the propagation of results, so that the UUIDs of the jobs producing the artifacts are propagated through the whole tree. This issue at hand was that when having a dependency tree like this: C -> B -> A The results from A were propagated to B and the results from B where propagated to C. But, because the implementation did not do this, the results from A where included in the results from B and the UUID from A was dropped. This was an issue because the implementation waited for _all_ dependencies (direct and transitive) by their job UUID. This means that C waited on a UUID that described the Job for A, but never received it, which caused everything to fail. This patch changes the algorithm, to not only report the own UUID and all artifacts of a job, but all artifacts with their UUID attached, which solves the issue. The root of the tree (the `Orchestrator`) simply drops the UUIDs before returning the artifacts to its caller. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-04Add testcase for chained dependenciesMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-03Be a bit more verbose in debug output hereMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-03Fix: Format UUIDs of missing job results to be human-readable before ↵Matthias Beyer
constructing error object Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-02Fix: Hash decoding for sha256Matthias Beyer
This seems strange, but it works. I don't know whether this is right, tho. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-02Add more error context informationMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-02Add bytes written/total bytes to status bar messageMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-02Fix: Download bar should be joined in blocking tokio taskMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-02Merge branch 'fix-progress'Matthias Beyer
2021-02-02Fix: Progress reportingMatthias Beyer
This patch fixes progress reporting. Because our progress-bar-creating helper initializes the bar with length 1, we have to set the length here manually. The bar has to be added to the multibar object right away, because otherwise it will be rendered to the output which gives us an ugly dead progress bar. If the length is set after the adding to the multibar object, this does not happen. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-02Fix: Make sure that bar is moved to LogReceiver, drop it afterwardsMatthias Beyer
This patch makes the bar moved to the LogReceiver instead of borrowing it to it. Because ProgressBar::clone() is cheap (according to indicatif documentation it is just an Arc<> holding the actual object), we can do this here without fearing the overhead. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-02Add network-mode setting for endpointsMatthias Beyer
This patch adds the ability to set network mode for an endpoint. This means that all containers on the endpoint are started with, for example, --net=host (if that is desired). Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-02Merge branch 'more-trace-output' into masterMatthias Beyer
2021-02-02Add trace output which packages will be verifiedMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-02Add tracing outputMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-02Add tracing outputMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-02Merge branch 'verification-async' into masterMatthias Beyer
2021-02-02Merge branch 'fixes' into masterMatthias Beyer
2021-02-02Make source verification completely asyncMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-02Fix: Receive artifacts _after_ checking whether jobs resulted in errorMatthias Beyer
This caused the program never to return if the running jobs resulted in an error and no artifact being sent to the parent - which caused the tokio::join!() to never return, thus the futures not to be pulled and thus the whole program sleep in an strange state which looked like some filesystem operations did not return. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-02Fix: fn does not have to be asyncMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-02Fix: wait properly for multibar joinMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-01Merge branch 'update-deps'Matthias Beyer
2021-01-30Update tokio: 0.2 -> 1.0, shipliftMatthias Beyer
Because tokio 1.0 does not ship with the Stream trait, this patch also introduces tokio_stream as new dependency. For more information, look here: https://docs.rs/tokio/1.0.3/tokio/stream/index.html Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-01-30typed-builder: 0.7 -> 0.8Matthias Beyer
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-01-30reqwest: 0.10 -> 0.11Matthias Beyer
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-01-30itertools: 0.9 -> 0.10Matthias Beyer
We don't need resiter::Map here anymore because itertools 0.10 provides a map_ok() extension. Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-01-29Merge branch 'doc-orchestrator'Matthias Beyer
2021-01-29Add documentation how Orchestrator worksMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-01-29CONTRIBUTING.md: delete trailing whitespaceLeon Schuermann
Signed-off-by: Leon Schuermann <leon@is.currently.online> Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-01-29README.md: fix glossary Markdown table formattingLeon Schuermann
With this change the table renders correctly in a variety of editors and applications, such as emacs markdown-mode, GitHub and dillinger.io. Signed-off-by: Leon Schuermann <leon@is.currently.online> Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-01-29Add dependency: aquamarineMatthias Beyer
MIT licensed documentation helper crate for drawing nice graphes. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-01-25Merge branch 'more-parallelism' into masterMatthias Beyer
2021-01-25Merge branch 'deny-some-warnings' into masterMatthias Beyer
2021-01-25Outsource receiving, ensure we received it allMatthias Beyer
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-01-25Make progress bar message format uniformMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-01-25Let the JobHandle::run() return a Vec<Artifact>Matthias Beyer
Before that change, it returned the dbmodels::Artifact objects, for which we needed to fetch the filestore::Artifact again. This change removes that restriction (improving runtime, of course). Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-01-25Reimplement Orchestrator::run()Matthias Beyer
This reimplements the Orchestrator::run() function _again_. Commit 889649ac16367fe671ce61363bb6ce82531e5a6b was the basis for this work, improving the baseline so we can take a step further in this commit. The approach before the change from 889649ac16367fe671ce61363bb6ce82531e5a6b had one flaw. In the following scenario: A / \ B E / \ \ C D F The nodes C, D and F are selected and then run. After they all succeeded, the next iteration is checked, and yields that B and E can be built. But if F takes extremely long, B and E both have to wait until it is ready (because that's how the implementation works), although B can be built as soon as C and D are ready. This patch changes the implementation to the following: 1. For each job, there is a task. 2. The task has a channel where it receives results from its dependencides. In above example, B would receive the results of the job runs for C and D, and E would receive the result from the job run of F. 3. The task also has a sender where it can send its resulting artifacts to a parent task. The task _also_ sends the results of its childs. This way we propagate the built artifacts up to the root node. All these tasks are started concurrently. The "root" task sends the result to the orchestrator. The task itself is responsible for sending the job to the scheduler and processing the result. If the job errored, the task sends that to its parent. If a child errored, the task aborts its own error and propagates that error. What does not yet work in this commit: * Artifacts that were built before the error occoured are not reported yet. * The staging/release stores may contain artifacts that can be re-used. They are completely ignored by now. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-01-25Do not deny missing docs, we havent written em yetMatthias Beyer
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-01-25Do not deny missing copy impls, we have to manyMatthias Beyer
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-01-25Do not deny missing debug impls, we have to manyMatthias Beyer
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-01-25Do not deny unused results, we have too manyMatthias Beyer
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-01-25Remove unused crateMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-01-25Remove unnecessary qualificationMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-01-25Fix: Use rand as _ to make lint happyMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-01-25Deny more things, sort denylistMatthias Beyer
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
generated by cgit v1.2.3 (git 2.25.1) at 2024-06-11 00:27:13 +0000