| Age | Commit message (Collapse) | Author |
|---|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
butido should not fail if the source cache directory already exists (which is
the case when downloading two source files for one package).
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
Tested-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Because we can just use the actual interface function here.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
It is easier for the caller (because more visible what happens) to call
`Source::path().exists()`.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This patch re-implements hashing using streams and buffered readers instead of
reading a full file to RAM before hashing it.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Because we continuously get blocking filesystem operations when the
implementation of the verification is async, simply remove the asyncness here
now.
This does not decrease performance (yet), because the function is called
concurrently with other calls anyways.
It blocks the tokio worker thread tho, thus maximum parallelism might be =
number of cores. :-(
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This patch adds a feature where a source entry in a package can be marked for
manual download.
This gives us the ability to mask downloads which are hidden behind cruel
JavaScript bullshit bloat where a `curl` cannot access the remote file.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
This reverts commit c3fc1281142ec10414197a31070cc45930a859e3
and commit 18b256e040881ac674463913b2a7e290125ea738.
The problem here is that we introduced "ring" to our dependencies, which has a
unappropriate license (ISC and openssl thing).
We should not depend on such a library, because it may conflict with our own EPL
2.0 (IANAL).
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
This patch, as a followup to 18b256e040881ac674463913b2a7e290125ea738
("Reimplement hash verification") moves the hashing itself into the blocking
closure, so that we get maximum parallelism here.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This patch re-implements hashing using the "ring" crypto library and
implementing a streaming hashing with it.
This way, we stream the file to the hasher rather than reading the full file to
memory.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
This patch changes the (possibly heavy) File reading from disk to be executed on
a blocking tokio thread.
This way, we ensure we do not block until the `read()` operation is finished and
can continue executing async tasks on the runtime.
Because IO might be expensive (think of a 5GB file that is read to memory), this
is a good optimization.
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Because tokio seems to block forever here when "read()"ing the file to
memory, we use the `std::fs` API here.
This blocks, but is called async in tokio anyways, so we at least get
_some_ parallelization.
It is at least "a bit" parallelized, as this only allows parallelization
up to N `read()`s, where N is the number of tokio runtime threads, which
is the number of cores of the machine this is called on.
The `tokio::fs` API is a wrapper around `std::fs` anyways, right now.
So until tokio can work with async OS API calls (io_uring anyone?),
this is not worse as `tokio::fs`.
Signed-off-by: Christoph Prokop <christoph.prokop@atos.net>
Signed-off-by: Matthias Beyer <matthias.beyer@atos.net>
Tested-by: Christoph Prokop <christoph.prokop@atos.net>
Tested-by: Matthias Beyer <matthias.beyer@atos.net>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
This changes the source file hash verification mechanism to return an
Ok(()) on success and an Err(_) on failure.
Returning a bool seems to be the right way, but it is not because if the
verification fails, we print the error message anyways, so there is no
point in having a bool around.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Because the source is downloaded to a directory instead of a flat file
in the source cache root, we should ensure this directory exists.
We do not, though, create the source cache root directory, but only
check whether it is there.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
This patch changes the sources to be named.
This is required for nice interpolation when layering pkg.toml files.
Before this, we had to make sure that the `[[sources]]` array element
was on the right position.
For example, consider two files:
/pkg.toml
/package/pkg.toml
in the first:
[[sources]]
url = "some/thing"
in the second:
[[sources]
hash.type = "sha1"
hash.value = "asdf..."
When merged:
[[sources]
url = "some/thing"
hash.type = "sha1"
hash.value = "asdf..."
but because the _position_ was responsible for merging these elements,
adding a new source file _before_ the existing on in /pkg.toml would
yield the /package/pkg.toml incomplete when merged onto /pkg.toml, thus
resulting in an error.
With named source entries, this is less likely.
The source file name includes the source name as well, of course.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
This patch implements multiple (unnamed) sources per package.
This means that a package can have an array of sources.
What was adapted to allow multiple sources per package:
* Downloads are made in parallel now
* The cache structure was changed to /<package>-<version>/<hash>.source
* The UI was changed to contain the full `Package` struct (as JSON
object) in a UI format string
Tests were adapted.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
This patch cleans the imports, removes the unused ones and moves
imports, wherever possible, to the outer scope.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
|
This adds a module for source cache handling.
Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
|
