summaryrefslogtreecommitdiffstats
path: root/src/endpoint
AgeCommit message (Collapse)Author
2021-04-08Fix: Update the progress bar each secondMatthias Beyer
This patch tries to work around a longer-than-1-sec blocking future while waiting for the channel .recv() call in the LogReceiver. The issue at hand is: If the channel does not produce log output for longer than 1 second, the progress bar won't be `tick()`ed for that amount of time. That leads to progress bars that seem to block (no update of the time in the progress bar output), which might confuse users. This patch works around that by wrapping the recv() in a timeout future and then catch the timeout, ping the progress bar and try to `recv()` again. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-11Endpoint configuration as MapMatthias Beyer
This patch rewrites the endpoint configuration format to be a map. The problem here is, that a list of endpoints cannot easily be used with layered configuration, where a part of the configuration lives in the XDG config home of the user and the rest lives in the repository. With this patch, the endpoints are configured with a map instead of an array, which makes it less complicated to overwrite. The name of an endpoint is now the key in the map. A type `EndpointName` was introduced to take advantage of strong typing. Thus, the patch touches a bit more code to adapt to the new type in use. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-10Merge branch 'pathes-to-absolute'Matthias Beyer
2021-03-10Implement patch-to-container copyingMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-10Clippy fix: Remove unneeded lifetimesMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-10Refactor: Simplify impl of Endpoint::get_container_by_id()Matthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-07Add Endpoint::get_container_by_id()Matthias Beyer
This function returns a shiplift::Container rather than an own type, which is more convenient at this point, but might be rewritten if we provide more endpoints (something else than docker). Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-07Add Endpoint::has_container_with_id()Matthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-07Add Endpoint::container_stats() with helper typesMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-07Add Endpoint::stats() and helper type for gathering statisticsMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-05Add Endpoint::ping()Matthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-05Move endpoint connection setup to utility moduleMatthias Beyer
Because we can re-use this function in our commandline endpoint maintenance commands implementation. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-04Remove relative speed setting, select endpoint by utilization insteadMatthias Beyer
This patch removes the "speed" setting from the configuration, which was introduced to set a relative speed for each endpoint, with the idea that the scheduler then would select a faster node preferably. Instead, the utilization of an endpoint is now calculated (number of running jobs vs allowed maximum jobs on the endpoint), and the endpoint with lower utilization is selected. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-02Implement scheduling with max jobs per endpointMatthias Beyer
This patch implements support for max jobs per endpoint. The number of running jobs on one endpoint are tracked with a wrapper around the Endpoint object, which increases the job counter on allocation and decreases it on deallocation. This way, the scheduler can know how many jobs are running on one endpoint and select the next endpoint accordingly. The loading/comparing is not perfect, so it might happen that more jobs run on one endpoint than configured, but this is the first step into the right direction. Also, the selection happens on a tokio job which runs in a loop{}. Because this almost blocks the whole executor thread, we use `tokio::task::yield_now()` as soon as there is no free endpoint anymore, to yield the execution to another future to free resources for doing actual work, not scheduling. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-01Remove RwLock around EndpointMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-03-01Refactor: Environment variable aggregationMatthias Beyer
This patch refactors the collecting of the environment variables in the `RunnableJob::build_from_job()` implementation as well as in the `RunnableJob::environemtn()` implementation. This results in fewer allocations, especially but not only because the `RunnableJob::environment()` function returns an iterator now and all clone() calls were removed. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-25Merge branch 'multiple-release-stores'Matthias Beyer
2021-02-25Multiple release storesMatthias Beyer
This patch adds the ability to have more than one release store. With this patch, a user can (has to) configure release store names in the configuration file, and can then specify one of the configured names to release the artifacts to. This way, different release "channels" can be served, for example a stable channel and a rolling release channel (although "channel" is not in our wording). The code was adapted to be able to fetch releases from multiple release directories, in the crate::db::find_artifact implementation, so that re-using artifacts works across all release directories. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-24Move constants to dedicated moduleMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-22Remove MergedStoresMatthias Beyer
The concept of the MergedStores type was okay in the beginning, but it got more and more complex to use properly and most of the time, we used the release/staging stores directly anyways. So this removes the MergedStores type, which is a preparation for the change to have multiple release stores. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-17Pass MergedStores to endpoint/jobs rather than only staging storeMatthias Beyer
This patch changes the code so that the MergedStores object is known in the endpoints and job execution code. This is necessary, because the jobs must be able to load artifacts from the release store as well, for example if a library was released in another submit and we can reuse it because the script for that library didn't change. For that, the interface of StoreRoot::join() was changed - -> Result<FullArtifactPath<'a>> + -> Result<Option<FullArtifactPath<'a>>> whereas it returns an Ok(None) if the artifact requested does not exist. This was necessary to try-joining a path on the store root of the staging store, and if there is no such file continue with try-joining the path on the release store. The calling code got a bit more complex with that change, though. Next, the MergedStores got a `derive(Clone)` because clone()ing it is cheap (two `Arc`s) and necessary to pass it easily to the jobs. Each instance of the code where the staging store was consulted was changed to consult the the release store as well. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-15Add more error contextMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-15Fix: Return script errors properlyMatthias Beyer
This patch fixes error returning from the JobHandle::run() implementation. Somehow, in the many rewrites, the error returning resulted in code that did not differentiate between script run errors and scheduling errors. This patch fixes this by making the JobHandle::run() method return Result<Result<Vec<ArtifactPath>>> whereas the outer error is the scheduling result and the inner error is the script result. The calling code was adapted accordingly. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-08Colorize error messageMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-08Fix: Minify error messageMatthias Beyer
The information how to connect to a container is added to the error on the scheduler level. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-08Merge branch 'remove-artifact-type'Matthias Beyer
2021-02-08Fix: Do not try to fetch artifacts if container erroredMatthias Beyer
This patch fixes an issue where we tried to fetch the artifacts from the container if the container script exited unsuccessfully. This resulted in an "cannot fetch artifacts from container" error message, which shouldn't happen, as we shouldn't even try if the script failed. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-08Remove `Artifact` typeMatthias Beyer
This patch follows-up on the shrinking of the `Artifact` type and removes it entirely. The type is not needed. Only the `ArtifactPath` type is needed, which is a thin wrapper around `PathBuf`, ensuring that the path is relative to the store root. The `Artifact` type used `pom` to parse the name and version of the package from the `ArtifactPath` object it contained, which resulted in the restriction that the path must always be <name>-<version>... Which should not be a requirement and actually caused issues with a package named "foo-bar" (as an example). Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-06Fix: Do not set message when setting progressMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-02Merge branch 'fix-progress'Matthias Beyer
2021-02-02Fix: Make sure that bar is moved to LogReceiver, drop it afterwardsMatthias Beyer
This patch makes the bar moved to the LogReceiver instead of borrowing it to it. Because ProgressBar::clone() is cheap (according to indicatif documentation it is just an Arc<> holding the actual object), we can do this here without fearing the overhead. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-02-02Add network-mode setting for endpointsMatthias Beyer
This patch adds the ability to set network mode for an endpoint. This means that all containers on the endpoint are started with, for example, --net=host (if that is desired). Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-01-30Update tokio: 0.2 -> 1.0, shipliftMatthias Beyer
Because tokio 1.0 does not ship with the Stream trait, this patch also introduces tokio_stream as new dependency. For more information, look here: https://docs.rs/tokio/1.0.3/tokio/stream/index.html Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-01-30itertools: 0.9 -> 0.10Matthias Beyer
We don't need resiter::Map here anymore because itertools 0.10 provides a map_ok() extension. Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-01-25Merge branch 'more-parallelism' into masterMatthias Beyer
2021-01-25Make progress bar message format uniformMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-01-25Let the JobHandle::run() return a Vec<Artifact>Matthias Beyer
Before that change, it returned the dbmodels::Artifact objects, for which we needed to fetch the filestore::Artifact again. This change removes that restriction (improving runtime, of course). Signed-off-by: Matthias Beyer <matthias.beyer@atos.net> Tested-by: Matthias Beyer <matthias.beyer@atos.net>
2021-01-25Remove unnecessary qualificationMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-01-21Remove RunnableJob::package_environment()Matthias Beyer
This functionality is not required anymore, as we put the whole package definition in the job script interpolation anyways. Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-01-19Fix clippy: calling `as_bytes()` on a string literalMatthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-01-18Post-format clippy cleanupMatthias Beyer
This patch cleans some rustfmt code that was formatted, but where clippy had something to complain: a scope was unnecessary here, so remove it. Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-01-18Run `cargo fmt`Matthias Beyer
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
2021-01-15Fix clippy: using `clone` on a `Copy` typeMatthias Beyer
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-01-15Fix clippy: useless conversion to the same type: `anyhow::Error`Matthias Beyer
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-01-15Fix style: Remove trailing whitespaceMatthias Beyer
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-01-15Fix clippy: calls to `std::mem::drop` with a reference instead of an owned ↵Matthias Beyer
value. Dropping a reference does nothing Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-01-15Fix clippy: redundant closure foundMatthias Beyer
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-01-15Fix clippy: writing `&Vec<_>` instead of `&[_]` involves one more reference ↵Matthias Beyer
and cannot be used with non-Vec-based slices Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-01-15Fix clippy: explicit lifetimes given in parameter types where they could be ↵Matthias Beyer
elided Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
2021-01-15Fix clippy: this `.into_iter()` call is equivalent to `.iter()` and will not ↵Matthias Beyer
consume the `Vec` Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-19 09:57:54 +0000