| Age | Commit message (Collapse) | Author |
|---|
|
This patch reimplements the package orchestration functionality to rely
on a DAG rather than a tree.
A
/ \
B E
/ \ \
C D F
Before this change, the structure the packages were organized in for a
build was a tree.
That did work reasonable well for initial development of butido, because
this is a simple case and the implementation is rather simple, too.
But, packages and their dependencies are not always organized in a tree.
Most of the time, they are organized in a DAG:
.-> C -,
/ \
D > A
\ /
`-> B -ยด
This is a real-world example: A could be a common crypto-library that I
do not want to name here.
B and C could be libraries that use the said crypto-library and D could
be a program that use B and C.
Because said crypto-library builds rather long, building it twice and
throwing one result away is a no-go.
A DAG as organizational structure makes that issue go away entirely.
Also, we can later implement checks whether the DAG contains multiple
versions of the same library, if that is undesireable.
The change itself is rather big, frankly because it is a non-trivial
change the replace the whole data structure and its handling in the
orchestrator code.
First of all, we introduce the "daggy" library, which provides the DAG
implementation on top of the popular "petgraph" library.
The package `Tree` datastructure was replaced by a package `Dag`
datastructure. This type implements the heavy-lifting that is needed to
load a package and all its dependencies from the `Repository` object.
The `JobTree` was also reimplemented, but as `daggy::Dag` provides a
convenient `map()` function, its implementation which transforms the
package `Dag` into a job `Dag` is rather trivial.
`crate::job::Dag` then provides the convenience `iter()` function to
iterate over all elements in the DAG and providing a `JobDefinition`
object for each node.
The topology in which we traverse the DAG is not an issue, as we need to
create tasks for all `JobDefinition`s anyways, so we do not care about
traversal topology at all.
The `crate::package::Package` type got an `Hash` implementation, which
is necessary to keep track of the mappings while reading the DAG from
the repository.
The implementation does not create the edges between the nodes in the
DAG right when inserting, but afterwards.
To keep track of the `daggy::NodeIndex`es, it keeps a mapping
Package -> NodeIndex
in a Hashmap. Thus, `Package` must implement `std::hash::Hash`
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
Tested-by: Matthias Beyer <mail@beyermatthias.de>
squash! Reimplement as DAG
|
|
Because tokio 1.0 does not ship with the Stream trait, this patch also
introduces tokio_stream as new dependency.
For more information, look here:
https://docs.rs/tokio/1.0.3/tokio/stream/index.html
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
We don't need resiter::Map here anymore because itertools 0.10 provides
a map_ok() extension.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
MIT licensed documentation helper crate for drawing nice graphes.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This patch reimplements the tree-printing using the 'ptree' crate.
Because ptree wants the tree item to implement `Clone`, a wrapper type is
added which then implements `Clone` and `ptree::TreeItem`
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
This patch adds a compatibility setting in the configuration file, so that
butido does not fail late when parsing configuration parameters or package
definition files, but early, when the configured compatilibty does not match the
version of butido itself.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This reverts commit c3fc1281142ec10414197a31070cc45930a859e3
and commit 18b256e040881ac674463913b2a7e290125ea738.
The problem here is that we introduced "ring" to our dependencies, which has a
unappropriate license (ISC and openssl thing).
We should not depend on such a library, because it may conflict with our own EPL
2.0 (IANAL).
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
|
|
|
|
This patch re-implements hashing using the "ring" crypto library and
implementing a streaming hashing with it.
This way, we stream the file to the hasher rather than reading the full file to
memory.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This patch changes the (possibly heavy) File reading from disk to be executed on
a blocking tokio thread.
This way, we ensure we do not block until the `read()` operation is finished and
can continue executing async tasks on the runtime.
Because IO might be expensive (think of a 5GB file that is read to memory), this
is a good optimization.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
With this patch, we pin shiplift to a revision.
That is because shiplift may break from one revision to another on master (which
just happened and which is what we fixed in
cc8d3a7912203c90a6591e0d178c80f620bd61d0 ("Fix: Add type annotation").
We expect shiplift to hit tokio 1.0 anytime now and because of that we might
want to do a manual update of shiplift and tokio rather than an imperative
`cargo update` which may (short-term) break our codebase.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
We actually need to make an ugly (and very bad performing) hack here, to
get the `unindent` crate to actually unindent here.
text = format!("\n{}", text).unindent(),
the format!() call is necessary to tell `unindent::Unindent::unindent()`
that the intended indentation is actually zero.
If we do not add that `\n` here, the first line of the to-be-unindented
indented text stays unintentionally indented.
For example, unindenting the intentionally indented string:
" intentionally indented
intentionally indented
"
results unintentionally in the partially indented and partially
unindented text:
" intentionally indented
intentionally indented
"
Which is not our intention of course, although it seems to be the
intention of the unindent crate, as documented.
But with the intentionally added `\n` we unindent the indentation of all
indented "intentionally indented" lines.
Signed-off-by: Christoph Prokop <christoph.prokop@atos.net>
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
Tested-by: Christoph Prokop <christoph.prokop@atos.net>
Tested-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Because the scripts we run should not be able to access the logger
during handlbars rendering.
This is due to the fact that we need to have control over stdout/stderr
because of the progress bar printing, but also because there's
absolutely no need for the user to log during the handlebars rendering
process.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
This removes the ContainerError type for simplicity. We can call
anyhow!() and just work with that, no need to pull in thiserror just for
one type.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Sort and make indention nice.
Also dev-dependencies where removed because indoc is a normal dependency
anyways.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
This patch adds script linting via a configurable script.
With this patch, the configuration can point to a script or program that
gets the packaging script on STDIN and might exit with a nonzero value
on error.
The simplest script I can think of that adds value is:
#!/bin/bash
shellcheck -
to run the packaging script through shellcheck.
stdout and stderr of the linting script are printed in case of non-zero
return value.
Linting can be disabled by not setting the script in the config or by
passing a commandline flag to skip linting (a warning will be printed in
this case).
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
The licensing issue fixed by this patch is described in the comment in
the code.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
This patch updates the ascii_table dependency with a requirement to be
3.0.2 or newer.
This is because ascii_table was licensed as GPLv3 before that version,
and got relicensed to MIT in 3.0.2.
Because we do not want to depend on a library that is GPLv3, we asked
the author to relicense their crate, which is what they did for us.
This patch thus ensures that we do not link GPLv3 code.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
This patch implements error reporting if a container job did not end
successfully.
It does so by adding an error type `ContainerError`, which is either an
error that describes that a container did not exit with success, or an
anyhow::Error (that describes an error from the container management
code).
The algorithm of log-aggregation is now intercepted to catch any
exit-state log items.
If there is no exit-state from the container (No line with
"#BUTIDO:STATE:..."), no error is assumed.
Here could be a warning later on.
The so aggregated state is then passed up to the orchestrator, which
then collects the errors and prints them.
If the implementation is correct (which is not tested yet, because this
is rather difficult to test), all other containers should continue
operation until they are ready, before the errors are handled.
The code responsible for this (in the Orchestrator implementation) was
adapted to not collect until the first error, but collect everything and
then check for errors.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
This also needs the "uuid" and "chrono" diesel features.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
This imports the changes from the "db" branch, because the orchestrator
needs the database interface code to be implemented.
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
This changes the display_as_table() function to automatically ignore the
headers and not printing a database, but simply the values as a
whitespace seperated list when stdout is a pipe.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
