summaryrefslogtreecommitdiffstats
path: root/src/libstore/download.cc
AgeCommit message (Collapse)Author
2020-04-08Rename src/lib/download.* to src/lib/datatransfer.*Nikola Knezevic
2020-04-08DownloadRequest -> DataTransferRequestNikola Knezevic
2020-04-08Downloader: Only write data to the sink on a 200 responseEelco Dolstra
Hopefully fixes #3278.
2020-04-07CleanupEelco Dolstra
2020-04-07Backport libfetchers from the flakes branchEelco Dolstra
This provides a pluggable mechanism for defining new fetchers. It adds a builtin function 'fetchTree' that generalizes existing fetchers like 'fetchGit', 'fetchMercurial' and 'fetchTarball'. 'fetchTree' takes a set of attributes, e.g. fetchTree { type = "git"; url = "https://example.org/repo.git"; ref = "some-branch"; rev = "abcdef..."; } The existing fetchers are just wrappers around this. Note that the input attributes to fetchTree are the same as flake input specifications and flake lock file entries. All fetchers share a common cache stored in ~/.cache/nix/fetcher-cache-v1.sqlite. This replaces the ad hoc caching mechanisms in fetchGit and download.cc (e.g. ~/.cache/nix/{tarballs,git-revs*}). This also adds support for Git worktrees (c169ea59049f861aaba429f48b828d0820b74d1d).
2020-04-05Don't retry on "unsupported protocol" errorCole Helbling
When encountering an unsupported protocol, there's no need to retry. Chances are, it won't suddenly be supported between retry attempts; error instead. Otherwise, you see something like the following: $ nix-env -i -f git://git@github.com/foo/bar warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 335 ms warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 604 ms warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 1340 ms warning: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1); retrying in 2685 ms With this change, you now see: $ nix-env -i -f git://git@github.com/foo/bar error: unable to download 'git://git@github.com/foo/bar': Unsupported protocol (1)
2020-01-29Merge pull request #3335 from domenkozar/retry-429Eelco Dolstra
retry on HTTP status code 429
2020-01-29retry on HTTP status code 429Domen Kožar
2019-12-19Merge branch 'libarchive' of https://github.com/yorickvP/nixEelco Dolstra
2019-12-10Make the Store API more type-safeEelco Dolstra
Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-07libarchive proof of conceptYorick van Pelt
2019-11-26Drop remaining uses of external "tar"Eelco Dolstra
Also, fetchGit now runs in O(1) memory since we pipe the output of 'git archive' directly into unpackTarball() (rather than first reading it all into memory).
2019-11-26Merge pull request #3141 from xbreak/nocafileEelco Dolstra
Downloader: Log configured CA file
2019-11-07Remove #includeEelco Dolstra
2019-10-18Downloader: Warn if no trusted CA file has been configuredxbreak
2019-09-03Ensure that Callback is called only onceEelco Dolstra
Also, make Callback movable but uncopyable.
2019-09-03Downloader: Remove a possible double call to CallbackEelco Dolstra
2019-08-08conf: stalled-download-timeout: make tunableGraham Christensen
Make curl's low speed limit configurable via stalled-download-timeout. Before, this limit was five minutes without receiving a single byte. This is much too long as if the remote end may not have even acknowledged the HTTP request.
2019-07-10Resume NAR downloadsEelco Dolstra
This is a much simpler fix to the 'error 9 while decompressing xz file' problem than 78fa47a7f08a4cb6ee7061bf0bd86a40e1d6dc91. We just do a ranged HTTP request starting after the data that we previously wrote into the sink. Fixes #2952, #379.
2019-07-10Downloader: Use warn()Eelco Dolstra
2019-07-10Revert "Fix 'error 9 while decompressing xz file'"Eelco Dolstra
This reverts commit 78fa47a7f08a4cb6ee7061bf0bd86a40e1d6dc91.
2019-07-02build: add exit code for hash and check mismatchesDaiderd Jordan
Makes it easier to identify the failure reason in other tooling, eg. differentiate between a non-deterministic --check vs a failed build. $ nix-build '<nix/fetchurl.nix>' --argstr url http://example.org --argstr sha256 0000000000000000000000000000000000000000000000000000 hash mismatch in fixed-output derivation '/nix/store/nzi9ck45rwlxzcwr25is7qlf3hs5xl83-example.org': wanted: sha256:0000000000000000000000000000000000000000000000000000 got: sha256:08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm $ echo $? 102 $ nix-build -E 'with import <nixpkgs> {}; runCommand "foo" {} "date +%s > $out"' --check warning: rewriting hashes in '/nix/store/g3k47g0399fvjmbm0p0mnad74k4w8vkz-foo'; cross fingers error: derivation '/nix/store/mggc8dz13ackb49qca6m23zq4fpq132q-foo.drv' may not be deterministic: output '/nix/store/g3k47g0399fvjmbm0p0mnad74k4w8vkz-foo' differs $ echo $? 104
2019-06-28Fix `http2 = false` having no effect. Fixes #2971.Aniket Deshpande
Setting `http2 = false` in nix config (e.g. /etc/nix/nix.conf) had no effect, and `nix-env -vvvvv -i hello` still downloaded .nar packages using HTTP/2. In `src/libstore/download.cc`, the `CURL_HTTP_VERSION_2TLS` option was being explicitly set when `downloadSettings.enableHttp2` was `true`, but, `CURL_HTTP_VERSION_1_1` option was not being explicitly set when `downloadSettings.enableHttp2` was `false`. This may be because `https://curl.haxx.se/libcurl/c/libcurl-env.html` states: "You have to set this option if you want to use libcurl's HTTP/2 support." but, also, in the changelog, states: "DEFAULT Since curl 7.62.0: CURL_HTTP_VERSION_2TLS Before that: CURL_HTTP_VERSION_1_1" So, the default setting for `libcurl` is HTTP/2 for version >= 7.62.0. In this commit, option `CURLOPT_HTTP_VERSION` is explicitly set to `CURL_HTTP_VERSION_1_1` when `downloadSettings.enableHttp2` nix config setting is `false`. This can be tested by running `nix-env -vvvvv -i hello | grep HTTP`
2019-06-24Fix 32-bit overflow with --no-netEelco Dolstra
--no-net causes tarballTtl to be set to the largest 32-bit integer, which causes comparison like 'time + tarballTtl < other_time' to fail on 32-bit systems. So cast them to 64-bit first. https://hydra.nixos.org/build/95076624 (cherry picked from commit 29ccb2e9697ee2184012dd13854e487928ae4441)
2019-06-24Refactor downloadCached() interfaceEelco Dolstra
(cherry picked from commit df3f5a78d5ab0a1f2dc9d288b271b38a9b8b33b5)
2019-06-24Add '--no-net' convenience flagEelco Dolstra
This flag * Disables substituters. * Sets the tarball-ttl to infinity (ensuring e.g. that the flake registry and any downloaded flakes are considered current). * Disables retrying downloads and sets the connection timeout to the minimum. (So it doesn't completely disable downloads at the moment.) (cherry picked from commit 8ea842260b4fd93315d35c5ba94b1ff99ab391d8)
2019-06-24Fix 'error 9 while decompressing xz file'Eelco Dolstra
Once we've started writing data to a Sink, we can't restart a download request, because then we end up writing duplicate data to the Sink. Therefore we shouldn't handle retries in Downloader but at a higher level (in particular, in copyStorePath()). Fixes #2952. (cherry picked from commit a67cf5a3585c41dd9f219a2c7aa9cf67fa69520b)
2019-06-24Downloader: Propagate exceptions from decompressionSink->finish()Eelco Dolstra
(cherry picked from commit 15fa70cd1b853f5e62662b99ccb9ef3da6cfadff)
2019-06-24CachedDownloadResult: Include store pathEelco Dolstra
Also, make fetchGit and fetchMercurial update allowedPaths properly. (Maybe the evaluator, rather than the caller of the evaluator, should apply toRealPath(), but that's a bigger change.) (cherry picked from commit 5c34d665386f4053d666b0899ecca0639e500fbd)
2019-06-24downloadCached: Return ETagEelco Dolstra
(cherry picked from commit 529add316c5356a8060c35f987643b7bf5c796dc)
2019-02-26fix indentationDzmitry Zaitsau
2019-02-25Apply param values on S3Helper initializationDzmitry Zaitsau
2019-02-25Extend S3 URL parsing with parameters extractionDzmitry Zaitsau
2018-12-12libstore: improve hash mismatch error messageszimbatm
Use the same output ordering and format everywhere. This is such a common issue that we trade the single-line error message for more readability. Old message: ``` fixed-output derivation produced path '/nix/store/d4nw9x2sy9q3r32f3g5l5h1k833c01vq-example.com' with sha256 hash '08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm' instead of the expected hash '1xzwnipjd54wl8g93vpw6hxnpmdabq0wqywriiwmh7x8k0lvpq5m' ``` New message: ``` hash mismatch in fixed-output derivation '/nix/store/d4nw9x2sy9q3r32f3g5l5h1k833c01vq-example.com': wanted: sha256:1xzwnipjd54wl8g93vpw6hxnpmdabq0wqywriiwmh7x8k0lvpq5m got: sha256:08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm ```
2018-12-07s3: make scheme configurableDaiderd Jordan
This enables using for http for S3 request for debugging or implementations that don't have https configured. This is not a problem for binary caches since they should not contain sensitive information. Both package signatures and AWS auth already protect against tampering.
2018-11-20Merge pull request #2216 from dtzWill/fix/curl-760-timeoutEelco Dolstra
download: if there are active requests, never sleep for 10s
2018-09-26Ensure download thread livenessEelco Dolstra
* Don't wait forever for the client to remove data from the buffer. This does mean that the buffer can grow without bounds (e.g. when downloading is faster than writing to disk), but meh. * Don't hold the state lock while calling the sink. The sink could take any amount of time to process the data (in particular when it's actually a coroutine), so we don't want to block the download thread.
2018-09-03download: fix size reported to progress barWill Dietz
2018-08-06Fix Brotli decompression in 'nix log'Eelco Dolstra
This didn't work anymore since decompression was only done in the non-coroutine case. Decompressors are now sinks, just like compressors. Also fixed a bug in bzip2 API handling (we have to handle BZ_RUN_OK rather than BZ_OK), which we didn't notice because there was a missing 'throw': if (ret != BZ_OK) CompressionError("error while compressing bzip2 file");
2018-08-06Fix reporting of HTTP body size when a result callback is usedEelco Dolstra
2018-08-02s3 binary cache: support specifying an endpointGraham Christensen
Works for uploading and not downloading.
2018-07-31Fix race downloading into a sinkEelco Dolstra
This fixes 'error 10 while decompressing xz file'. https://hydra.nixos.org/build/78308551
2018-06-18Detect redirect loopsEelco Dolstra
Fixes #2225.
2018-06-08download: if there are active requests, never sleep for 10sWill Dietz
In some versions/configurations libcurl doesn't handle timeouts (especially DNS timeouts) in a way that wakes curl_multi_wait. This doesn't appear to be a problem if using c-ares, FWIW.
2018-06-05Handle HTTP 410 Gone errorsEelco Dolstra
2018-06-05Improve binary cache fallback testEelco Dolstra
2018-06-05Further improve upload messagesEelco Dolstra
2018-06-01download.cc: fix readCallback return type (int -> size_t)Will Dietz
I'm not sure if curl ever asks for enough data at once for truncation to occur but better safe than sorry.
2018-06-01Improve binary cache upload messagesEelco Dolstra
Don't say "download" when we mean "upload".
2018-05-30getDownloader(): SimplifyEelco Dolstra
generated by cgit v1.2.3 (git 2.25.1) at 2024-09-09 05:46:15 +0000