diff options
Diffstat (limited to 'docs')
74 files changed, 781 insertions, 1041 deletions
diff --git a/docs/changes.rst b/docs/changes.rst index 90be2b0bb..3a330eedc 100644 --- a/docs/changes.rst +++ b/docs/changes.rst @@ -12,8 +12,8 @@ This section provides information about security and corruption issues. Upgrade Notes ============= -borg 1.2.x to borg 2.0 ----------------------- +borg 1.2.x/1.4.x to borg 2.0 +---------------------------- Compatibility notes: @@ -21,11 +21,11 @@ Compatibility notes: We tried to put all the necessary "breaking" changes into this release, so we hopefully do not need another breaking release in the near future. The changes - were necessary for improved security, improved speed, unblocking future - improvements, getting rid of legacy crap / design limitations, having less and - simpler code to maintain. + were necessary for improved security, improved speed and parallelism, + unblocking future improvements, getting rid of legacy crap and design + limitations, having less and simpler code to maintain. - You can use "borg transfer" to transfer archives from borg 1.1/1.2 repos to + You can use "borg transfer" to transfer archives from borg 1.2/1.4 repos to a new borg 2.0 repo, but it will need some time and space. Before using "borg transfer", you must have upgraded to borg >= 1.2.6 (or @@ -84,6 +84,7 @@ Compatibility notes: - removed --nobsdflags (use --noflags) - removed --noatime (default now, see also --atime) - removed --save-space option (does not change behaviour) +- removed --bypass-lock option - using --list together with --progress is now disallowed (except with --log-json), #7219 - the --glob-archives option was renamed to --match-archives (the short option name -a is unchanged) and extended to support different pattern styles: @@ -114,12 +115,61 @@ Compatibility notes: fail now that somehow "worked" before (but maybe didn't work as intended due to the contradicting options). - .. _changelog: Change Log 2.x ============== +Version 2.0.0b10 (2024-09-09) +----------------------------- + +TL;DR: this is a huge change and the first very fundamental change in how borg +works since ever: + +- you will need to create new repos. +- likely more exciting than previous betas, definitely not for production. + +New features: + +- borgstore based repository, file:, ssh: and sftp: for now, more possible. +- repository stores objects separately now, not using segment files. + this has more fs overhead, but needs much less I/O because no segment + files compaction is required anymore. also, no repository index is + needed anymore because we can directly find the objects by their ID. +- locking: new borgstore based repository locking with automatic stale + lock removal (if lock does not get refreshed, if lock owner process is dead). +- simultaneous repository access for many borg commands except check/compact. + the cache lock for adhocwithfiles is still exclusive though, so use + BORG_CACHE_IMPL=adhoc if you want to try that out using only 1 machine + and 1 user (that implementation doesn't use a cache lock). When using + multiple client machines or users, it also works with the default cache. +- delete/prune: much quicker now and can be undone. +- check --repair --undelete-archives: bring archives back from the dead. +- rspace: manage reserved space in repository (avoid dead-end situation if + repository fs runs full). + +Bugs/issues fixed: + +- a lot! all linked from PR #8332. + +Other changes: + +- repository: remove transactions, solved differently and much simpler now + (convergence and write order primarily). +- repository: replaced precise reference counting with "object exists in repo?" + and "garbage collection of unused objects". +- cache: remove transactions, remove chunks cache. + removed LocalCache, BORG_CACHE_IMPL=local, solving all related issues. + as in beta 9, adhowwithfiles is the default implementation. +- compact: needs the borg key now (run it clientside), -v gives nice stats. +- transfer: archive transfers from borg 1.x need the --from-borg1 option +- check: reimplemented / bigger changes. +- code: got rid of a metric ton of not needed complexity. + when borg does not need to read borg 1.x repos/archives anymore, after + users have transferred their archives, even much more can be removed. +- docs: updated / removed outdated stuff + + Version 2.0.0b9 (2024-07-20) ---------------------------- diff --git a/docs/changes_1.x.rst b/docs/changes_1.x.rst index 3366a90b8..714726c1c 100644 --- a/docs/changes_1.x.rst +++ b/docs/changes_1.x.rst @@ -3469,7 +3469,7 @@ Other changes: - archiver tests: add check_cache tool - lints refcounts - fixed cache sync performance regression from 1.1.0b1 onwards, #1940 -- syncing the cache without chunks.archive.d (see :ref:`disable_archive_chunks`) +- syncing the cache without chunks.archive.d now avoids any merges and is thus faster, #1940 - borg check --verify-data: faster due to linear on-disk-order scan - borg debug-xxx commands removed, we use "debug xxx" subcommands now, #1627 diff --git a/docs/deployment/automated-local.rst b/docs/deployment/automated-local.rst index d34a70a7f..dbc871511 100644 --- a/docs/deployment/automated-local.rst +++ b/docs/deployment/automated-local.rst @@ -105,7 +105,7 @@ modify it to suit your needs (e.g. more backup sets, dumping databases etc.). # # Options for borg create - BORG_OPTS="--stats --one-file-system --compression lz4 --checkpoint-interval 86400" + BORG_OPTS="--stats --one-file-system --compression lz4" # Set BORG_PASSPHRASE or BORG_PASSCOMMAND somewhere around here, using export, # if encryption is used. diff --git a/docs/deployment/hosting-repositories.rst b/docs/deployment/hosting-repositories.rst index 55fd3e15e..b0efbf696 100644 --- a/docs/deployment/hosting-repositories.rst +++ b/docs/deployment/hosting-repositories.rst @@ -68,8 +68,6 @@ can be filled to the specified quota. If storage quotas are used, ensure that all deployed Borg releases support storage quotas. -Refer to :ref:`internals_storage_quota` for more details on storage quotas. - **Specificities: Append-only repositories** Running ``borg init`` via a ``borg serve --append-only`` server will **not** diff --git a/docs/faq.rst b/docs/faq.rst index 0daa226ca..e36fa3120 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -14,7 +14,7 @@ What is the difference between a repo on an external hard drive vs. repo on a se If Borg is running in client/server mode, the client uses SSH as a transport to talk to the remote agent, which is another Borg process (Borg is installed on the server, too) started automatically by the client. The Borg server is doing -storage-related low-level repo operations (get, put, commit, check, compact), +storage-related low-level repo operations (list, load and store objects), while the Borg client does the high-level stuff: deduplication, encryption, compression, dealing with archives, backups, restores, etc., which reduces the amount of data that goes over the network. @@ -27,17 +27,7 @@ which is slower. Can I back up from multiple servers into a single repository? ------------------------------------------------------------- -In order for the deduplication used by Borg to work, it -needs to keep a local cache containing checksums of all file -chunks already stored in the repository. This cache is stored in -``~/.cache/borg/``. If Borg detects that a repository has been -modified since the local cache was updated it will need to rebuild -the cache. This rebuild can be quite time consuming. - -So, yes it's possible. But it will be most efficient if a single -repository is only modified from one place. Also keep in mind that -Borg will keep an exclusive lock on the repository while creating -or deleting archives, which may make *simultaneous* backups fail. +Yes, you can! Even simultaneously. Can I back up to multiple, swapped backup targets? -------------------------------------------------- @@ -124,50 +114,31 @@ Are there other known limitations? remove files which are in the destination, but not in the archive. See :issue:`4598` for a workaround and more details. -.. _checkpoints_parts: +.. _interrupted_backup: If a backup stops mid-way, does the already-backed-up data stay there? ---------------------------------------------------------------------- -Yes, Borg supports resuming backups. - -During a backup, a special checkpoint archive named ``<archive-name>.checkpoint`` -is saved at every checkpoint interval (the default value for this is 30 -minutes) containing all the data backed-up until that point. - -This checkpoint archive is a valid archive, but it is only a partial backup -(not all files that you wanted to back up are contained in it and the last file -in it might be a partial file). Having it in the repo until a successful, full -backup is completed is useful because it references all the transmitted chunks up -to the checkpoint. This means that in case of an interruption, you only need to -retransfer the data since the last checkpoint. +Yes, the data transferred into the repo stays there - just avoid running +``borg compact`` before you completed the backup, because that would remove +chunks that were already transferred to the repo, but not (yet) referenced +by an archive. If a backup was interrupted, you normally do not need to do anything special, -just invoke ``borg create`` as you always do. If the repository is still locked, -you may need to run ``borg break-lock`` before the next backup. You may use the -same archive name as in previous attempt or a different one (e.g. if you always -include the current datetime), it does not matter. +just invoke ``borg create`` as you always do. You may use the same archive name +as in previous attempt or a different one (e.g. if you always include the +current datetime), it does not matter. Borg always does full single-pass backups, so it will start again from the beginning - but it will be much faster, because some of the data was -already stored into the repo (and is still referenced by the checkpoint -archive), so it does not need to get transmitted and stored again. - -Once your backup has finished successfully, you can delete all -``<archive-name>.checkpoint`` archives. If you run ``borg prune``, it will -also care for deleting unneeded checkpoints. - -Note: the checkpointing mechanism may create a partial (truncated) last file -in a checkpoint archive named ``<filename>.borg_part``. Such partial files -won't be contained in the final archive. -This is done so that checkpoints work cleanly and promptly while a big -file is being processed. +already stored into the repo, so it does not need to get transmitted and stored +again. How can I back up huge file(s) over a unstable connection? ---------------------------------------------------------- -Yes. For more details, see :ref:`checkpoints_parts`. +Yes. For more details, see :ref:`interrupted_backup`. How can I restore huge file(s) over an unstable connection? ----------------------------------------------------------- @@ -220,23 +191,6 @@ Yes, if you want to detect accidental data damage (like bit rot), use the If you want to be able to detect malicious tampering also, use an encrypted repo. It will then be able to check using CRCs and HMACs. -Can I use Borg on SMR hard drives? ----------------------------------- - -SMR (shingled magnetic recording) hard drives are very different from -regular hard drives. Applications have to behave in certain ways or -performance will be heavily degraded. - -Borg ships with default settings suitable for SMR drives, -and has been successfully tested on *Seagate Archive v2* drives -using the ext4 file system. - -Some Linux kernel versions between 3.19 and 4.5 had various bugs -handling device-managed SMR drives, leading to IO errors, unresponsive -drives and unreliable operation in general. - -For more details, refer to :issue:`2252`. - .. _faq-integrityerror: I get an IntegrityError or similar - what now? @@ -355,7 +309,7 @@ Why is the time elapsed in the archive stats different from wall clock time? ---------------------------------------------------------------------------- Borg needs to write the time elapsed into the archive metadata before finalizing -the archive and committing the repo & cache. +the archive and saving the files cache. This means when Borg is run with e.g. the ``time`` command, the duration shown in the archive stats may be shorter than the full time the command runs for. @@ -391,8 +345,7 @@ will of course delete everything in the archive, not only some files. :ref:`borg_recreate` command to rewrite all archives with a different ``--exclude`` pattern. See the examples in the manpage for more information. -Finally, run :ref:`borg_compact` with the ``--threshold 0`` option to delete the -data chunks from the repository. +Finally, run :ref:`borg_compact` to delete the data chunks from the repository. Can I safely change the compression level or algorithm? -------------------------------------------------------- @@ -402,6 +355,7 @@ are calculated *before* compression. New compression settings will only be applied to new chunks, not existing chunks. So it's safe to change them. +Use ``borg rcompress`` to efficiently recompress a complete repository. Security ######## @@ -704,38 +658,6 @@ serialized way in a single script, you need to give them ``--lock-wait N`` (with being a bit more than the time the server needs to terminate broken down connections and release the lock). -.. _disable_archive_chunks: - -The borg cache eats way too much disk space, what can I do? ------------------------------------------------------------ - -This may especially happen if borg needs to rebuild the local "chunks" index - -either because it was removed, or because it was not coherent with the -repository state any more (e.g. because another borg instance changed the -repository). - -To optimize this rebuild process, borg caches per-archive information in the -``chunks.archive.d/`` directory. It won't help the first time it happens, but it -will make the subsequent rebuilds faster (because it needs to transfer less data -from the repository). While being faster, the cache needs quite some disk space, -which might be unwanted. - -You can disable the cached archive chunk indexes by setting the environment -variable ``BORG_USE_CHUNKS_ARCHIVE`` to ``no``. - -This has some pros and cons, though: - -- much less disk space needs for ~/.cache/borg. -- chunk cache resyncs will be slower as it will have to transfer chunk usage - metadata for all archives from the repository (which might be slow if your - repo connection is slow) and it will also have to build the hashtables from - that data. - chunk cache resyncs happen e.g. if your repo was written to by another - machine (if you share same backup repo between multiple machines) or if - your local chunks cache was lost somehow. - -The long term plan to improve this is called "borgception", see :issue:`474`. - Can I back up my root partition (/) with Borg? ---------------------------------------------- @@ -779,7 +701,7 @@ This can make creation of the first archive slower, but saves time and disk space on subsequent runs. Here what Borg does when you run ``borg create``: - Borg chunks the file (using the relatively expensive buzhash algorithm) -- It then computes the "id" of the chunk (hmac-sha256 (often slow, except +- It then computes the "id" of the chunk (hmac-sha256 (slow, except if your CPU has sha256 acceleration) or blake2b (fast, in software)) - Then it checks whether this chunk is already in the repo (local hashtable lookup, fast). If so, the processing of the chunk is completed here. Otherwise it needs to @@ -790,9 +712,8 @@ and disk space on subsequent runs. Here what Borg does when you run ``borg creat - Transmits to repo. If the repo is remote, this usually involves an SSH connection (does its own encryption / authentication). - Stores the chunk into a key/value store (the key is the chunk id, the value - is the data). While doing that, it computes CRC32 / XXH64 of the data (repo low-level - checksum, used by borg check --repository) and also updates the repo index - (another hashtable). + is the data). While doing that, it computes XXH64 of the data (repo low-level + checksum, used by borg check --repository). Subsequent backups are usually very fast if most files are unchanged and only a few are new or modified. The high performance on unchanged files primarily depends @@ -826,10 +747,9 @@ If you feel your Borg backup is too slow somehow, here is what you can do: - Don't use any expensive compression. The default is lz4 and super fast. Uncompressed is often slower than lz4. - Just wait. You can also interrupt it and start it again as often as you like, - it will converge against a valid "completed" state (see ``--checkpoint-interval``, - maybe use the default, but in any case don't make it too short). It is starting + it will converge against a valid "completed" state. It is starting from the beginning each time, but it is still faster then as it does not store - data into the repo which it already has there from last checkpoint. + data into the repo which it already has there. - If you don’t need additional file attributes, you can disable them with ``--noflags``, ``--noacls``, ``--noxattrs``. This can lead to noticeable performance improvements when your backup consists of many small files. @@ -1021,6 +941,12 @@ To achieve this, run ``borg create`` within the mountpoint/snapshot directory: cd /mnt/rootfs borg create rootfs_backup . +Another way (without changing the directory) is to use the slashdot hack: + +:: + + borg create rootfs_backup /mnt/rootfs/./ + I am having troubles with some network/FUSE/special filesystem, why? -------------------------------------------------------------------- @@ -1100,16 +1026,6 @@ to make it behave correctly:: .. _workaround: https://unix.stackexchange.com/a/123236 -Can I disable checking for free disk space? -------------------------------------------- - -In some cases, the free disk space of the target volume is reported incorrectly. -This can happen for CIFS- or FUSE shares. If you are sure that your target volume -will always have enough disk space, you can use the following workaround to disable -checking for free disk space:: - - borg config -- additional_free_space -2T - How do I rename a repository? ----------------------------- @@ -1126,26 +1042,6 @@ It may be useful to set ``BORG_RELOCATED_REPO_ACCESS_IS_OK=yes`` to avoid the prompts when renaming multiple repositories or in a non-interactive context such as a script. See :doc:`deployment` for an example. -The repository quota size is reached, what can I do? ----------------------------------------------------- - -The simplest solution is to increase or disable the quota and resume the backup: - -:: - - borg config /path/to/repo storage_quota 0 - -If you are bound to the quota, you have to free repository space. The first to -try is running :ref:`borg_compact` to free unused backup space (see also -:ref:`separate_compaction`): - -:: - - borg compact /path/to/repo - -If your repository is already compacted, run :ref:`borg_prune` or -:ref:`borg_delete` to delete archives that you do not need anymore, and then run -``borg compact`` again. My backup disk is full, what can I do? -------------------------------------- @@ -1159,11 +1055,6 @@ conditions, but generally this should be avoided. If your backup disk is already full when Borg starts a write command like `borg create`, it will abort immediately and the repository will stay as-is. -If you run a backup that stops due to a disk running full, Borg will roll back, -delete the new segment file and thus freeing disk space automatically. There -may be a checkpoint archive left that has been saved before the disk got full. -You can keep it to speed up the next backup or delete it to get back more disk -space. Miscellaneous ############# diff --git a/docs/internals/compaction.odg b/docs/internals/compaction.odg Binary files differdeleted file mode 100644 index 8d193e009..000000000 --- a/docs/internals/compaction.odg +++ /dev/null diff --git a/docs/internals/compaction.png b/docs/internals/compaction.png Binary files differdeleted file mode 100644 index d5c53a680..000000000 --- a/docs/internals/compaction.png +++ /dev/null diff --git a/docs/internals/data-structures.rst b/docs/internals/data-structures.rst index 8d1562ff2..45de9ee34 100644 --- a/docs/internals/data-structures.rst +++ b/docs/internals/data-structures.rst @@ -19,63 +19,51 @@ discussion about internals`_ and also on static code analysis. Repository ---------- -.. Some parts of this description were taken from the Repository docstring - -Borg stores its data in a `Repository`, which is a file system based -transactional key-value store. Thus the repository does not know about -the concept of archives or items. - -Each repository has the following file structure: - -README - simple text file telling that this is a Borg repository - -config - repository configuration +Borg stores its data in a `Repository`, which is a key-value store and has +the following structure: + +config/ + readme + simple text object telling that this is a Borg repository + id + the unique repository ID encoded as hexadecimal number text + version + the repository version encoded as decimal number text + manifest + some data about the repository, binary + last-key-checked + repository check progress (partial checks, full checks' checkpointing), + path of last object checked as text + space-reserve.N + purely random binary data to reserve space, e.g. for disk-full emergencies + +There is a list of pointers to archive objects in this directory: + +archives/ + 0000... .. ffff... + +The actual data is stored into a nested directory structure, using the full +object ID as name. Each (encrypted and compressed) object is stored separately. data/ - directory where the actual data is stored - -hints.%d - hints for repository compaction - -index.%d - repository index - -lock.roster and lock.exclusive/* - used by the locking system to manage shared and exclusive locks - -Transactionality is achieved by using a log (aka journal) to record changes. The log is a series of numbered files -called segments_. Each segment is a series of log entries. The segment number together with the offset of each -entry relative to its segment start establishes an ordering of the log entries. This is the "definition" of -time for the purposes of the log. - -.. _config-file: - -Config file -~~~~~~~~~~~ + 00/ .. ff/ + 00/ .. ff/ + 0000... .. ffff... -Each r |
