Age | Commit message (Collapse) | Author |
|
This fixes a nasty bug; because we used the LogItem::display() function to build
the log string that was put into the database, the database contained
_colorized_ log lines. Because of this, our parsing of the log text later (when
reading historical data from the database) failed to parse whether the job was
successfull or not.
The issue was actually introduced in b3a6458ce34e3065192208826b2fc85edd4761f9,
where we changed the `LogItem::display()` implementation.
With this fix in place, the log text in the database is corrected again (only
for new logs that get put into the database).
Fixes: b3a6458ce34e3065192208826b2fc85edd4761f9 ("Add color support in LogItem::display() impl")
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
Tested-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
0.16.1 has been published which fixes the bug introduced by 0.16.0, hence update
this dependency.
This reverts commit ddbd9629b3188c9f08d023829683c40ab9e1448b.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This patch changes the LogReceiver to preallocate a reasonable large buffer for
holding logs in memory.
Because of the fact that we have to write the log _in one piece_ to the
database (there might be a way to append to the log field, but that's
optimization of a problem we shouldn't have - we rather should use another
approach for storing logs), we have to keep the log in memory until the job is
finished.
For keeping the log of a job in memory, we use a `Vec`. After consulting the
documentation
https://doc.rust-lang.org/stable/std/collections/index.html#when-should-you-use-which-collection
https://doc.rust-lang.org/stable/std/collections/index.html#sequences
we found that `Vec` should be appropriate here, although `VecDeque` might be an
option as well because of O(1) insertion time (and we're only inserting).
Still, because `Vec` has amortized constant time for adding elements at the end,
this should be sufficient.
Preallocating a reasonable large amount of elements could yield big benefits
with only minor disadvantage. If the job fails early (after only a few lines of
log output), there's memory wasted.
Also, if we have a large number of jobs, we allocate a huge amount of memory
before filling it up.
Because we need to have that memory anyways (considering all jobs succeed), that
is not really a disadvantage.
Now, what is "reasonable large"? This value might be changed later on, but for
now, I took this value from experience we had when using butido in practice:
select
AVG(length.l), MAX(length.l), MIN(length.l)
FROM
(
SELECT
LENGTH(log_text) - LENGTH(REPLACE(log_text, chr(10), '')) AS l
FROM
jobs
) AS length
+-----------------------+--------+-------+
| avg | max | min |
|-----------------------+--------+-------|
| 2863.0497427101200686 | 165213 | 11 |
+-----------------------+--------+-------+
The max and min values seem to be extreme. Now, the values contain a lot of
failed jobs and the maximum value (165k log lines is _huge_!) was also a bogus
setup.
Removing these from the equation, using only the successful jobs, gives us a
not-so-different number:
SELECT
AVG(l.len), MAX(l.len), MIN(l.len)
FROM
(
SELECT
LENGTH(log_text) - LENGTH(REPLACE(log_text, CHR(10), '')) AS len,
STRPOS(log_text, 'BUTIDO:STATE:OK') AS okpos
FROM JOBS
) AS l
WHERE
l.okpos != 0
AND
l.len != 165213
+-----------------------+-------+-------+
| avg | max | min |
|-----------------------+-------+-------|
| 2661.7306791569086651 | 55422 | 66 |
+-----------------------+-------+-------+
Using the average (arithmetic mean) as a baseline, we decided to go for 4096
(2^12) preallocated elements in the buffer, which should be reasonable.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This reverts the dependency update because indicatif 0.16 introduced a
reachable `unreachable!()` statement in their sourcecode that we
actually hit if we `--hide-bars`.
This reverts commit 6ceccb679d9c2d19389c6c6eef792d8db9086f31.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
This patch adds the endpoint name and a (7 character) appreviated container id
in the progress message, so that the user can see where the package is built.
This reduces the compile+debug loop by one step in the case where the user does
not want to inspect the logs anyways, because they can jump to the right
container right away and do not have to `butido db job -sL <job id>` to find out
where the job was running.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
Tested-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Because the interfaces of indicatif have changed, this PR changes a lot of
calls in the codebase. (Yay, moving all the things!)
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This patch adds a configurable timeout value (default 10 seconds)
in the Endpoint struct.
This way we get an error if the connection to the endpoint is stalled.
Signed-off-by: Christoph Prokop <christoph.prokop@atos.net>
Pair-programmed-with: Matthias Beyer <matthias.beyer@atos.net>
Suggested-by: Matthias Beyer <matthias.beyer@atos.net>
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
This patch tries to work around a longer-than-1-sec blocking future while
waiting for the channel .recv() call in the LogReceiver.
The issue at hand is: If the channel does not produce log output for longer than
1 second, the progress bar won't be `tick()`ed for that amount of time.
That leads to progress bars that seem to block (no update of the time in the
progress bar output), which might confuse users.
This patch works around that by wrapping the recv() in a timeout future and then
catch the timeout, ping the progress bar and try to `recv()` again.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This patch rewrites the endpoint configuration format to be a map.
The problem here is, that a list of endpoints cannot easily be used with layered
configuration, where a part of the configuration lives in the XDG config home of
the user and the rest lives in the repository.
With this patch, the endpoints are configured with a map instead of an array,
which makes it less complicated to overwrite.
The name of an endpoint is now the key in the map.
A type `EndpointName` was introduced to take advantage of strong typing. Thus,
the patch touches a bit more code to adapt to the new type in use.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
Tested-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This function returns a shiplift::Container rather than an own type, which is
more convenient at this point, but might be rewritten if we provide more
endpoints (something else than docker).
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Because we can re-use this function in our commandline endpoint maintenance
commands implementation.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This patch removes the "speed" setting from the configuration, which was
introduced to set a relative speed for each endpoint, with the idea that the
scheduler then would select a faster node preferably.
Instead, the utilization of an endpoint is now calculated (number of running
jobs vs allowed maximum jobs on the endpoint), and the endpoint with lower
utilization is selected.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This patch implements support for max jobs per endpoint.
The number of running jobs on one endpoint are tracked with a wrapper around the
Endpoint object, which increases the job counter on allocation and decreases it
on deallocation.
This way, the scheduler can know how many jobs are running on one endpoint and
select the next endpoint accordingly.
The loading/comparing is not perfect, so it might happen that more jobs run on
one endpoint than configured, but this is the first step into the right
direction.
Also, the selection happens on a tokio job which runs in a loop{}. Because this
almost blocks the whole executor thread, we use `tokio::task::yield_now()` as
soon as there is no free endpoint anymore, to yield the execution to another
future to free resources for doing actual work, not scheduling.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
Tested-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This patch refactors the collecting of the environment variables in the
`RunnableJob::build_from_job()` implementation as well as in the
`RunnableJob::environemtn()` implementation.
This results in fewer allocations, especially but not only because the
`RunnableJob::environment()` function returns an iterator now and all clone()
calls were removed.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
|
|
This patch adds the ability to have more than one release store.
With this patch, a user can (has to) configure release store names in the
configuration file, and can then specify one of the configured names to release
the artifacts to.
This way, different release "channels" can be served, for example a stable
channel and a rolling release channel (although "channel" is not in our wording).
The code was adapted to be able to fetch releases from multiple release
directories, in the crate::db::find_artifact implementation, so that re-using
artifacts works across all release directories.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
The concept of the MergedStores type was okay in the beginning, but it got more
and more complex to use properly and most of the time, we used the
release/staging stores directly anyways.
So this removes the MergedStores type, which is a preparation for the change to
have multiple release stores.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This patch changes the code so that the MergedStores object is known in the
endpoints and job execution code.
This is necessary, because the jobs must be able to load artifacts from the
release store as well, for example if a library was released in another submit
and we can reuse it because the script for that library didn't change.
For that, the interface of StoreRoot::join() was changed
- -> Result<FullArtifactPath<'a>>
+ -> Result<Option<FullArtifactPath<'a>>>
whereas it returns an Ok(None) if the artifact requested does not exist.
This was necessary to try-joining a path on the store root of the staging store,
and if there is no such file continue with try-joining the path on the release
store.
The calling code got a bit more complex with that change, though.
Next, the MergedStores got a `derive(Clone)` because clone()ing it is cheap
(two `Arc`s) and necessary to pass it easily to the jobs.
Each instance of the code where the staging store was consulted was changed to
consult the the release store as well.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This patch fixes error returning from the JobHandle::run() implementation.
Somehow, in the many rewrites, the error returning resulted in code that did not
differentiate between script run errors and scheduling errors.
This patch fixes this by making the JobHandle::run() method return
Result<Result<Vec<ArtifactPath>>>
whereas the outer error is the scheduling result and the inner error is the
script result.
The calling code was adapted accordingly.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
The information how to connect to a container is added to the error on the
scheduler level.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
Tested-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
|
|
This patch fixes an issue where we tried to fetch the artifacts from the
container if the container script exited unsuccessfully.
This resulted in an "cannot fetch artifacts from container" error message, which
shouldn't happen, as we shouldn't even try if the script failed.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This patch follows-up on the shrinking of the `Artifact` type and removes it
entirely.
The type is not needed. Only the `ArtifactPath` type is needed, which is a thin
wrapper around `PathBuf`, ensuring that the path is relative to the store root.
The `Artifact` type used `pom` to parse the name and version of the package from
the `ArtifactPath` object it contained, which resulted in the restriction that
the path must always be
<name>-<version>...
Which should not be a requirement and actually caused issues with a package
named "foo-bar" (as an example).
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
Tested-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
|
|
This patch makes the bar moved to the LogReceiver instead of borrowing it to it.
Because ProgressBar::clone() is cheap (according to indicatif documentation it
is just an Arc<> holding the actual object), we can do this here without fearing
the overhead.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This patch adds the ability to set network mode for an endpoint. This means that
all containers on the endpoint are started with, for example, --net=host (if
that is desired).
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
Tested-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Because tokio 1.0 does not ship with the Stream trait, this patch also
introduces tokio_stream as new dependency.
For more information, look here:
https://docs.rs/tokio/1.0.3/tokio/stream/index.html
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
We don't need resiter::Map here anymore because itertools 0.10 provides
a map_ok() extension.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
Tested-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Before that change, it returned the dbmodels::Artifact objects, for which we
needed to fetch the filestore::Artifact again.
This change removes that restriction (improving runtime, of course).
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
Tested-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This functionality is not required anymore, as we put the whole package
definition in the job script interpolation anyways.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This patch cleans some rustfmt code that was formatted, but where clippy
had something to complain: a scope was unnecessary here, so remove it.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|