summaryrefslogtreecommitdiffstats
path: root/docs/usage
diff options
context:
space:
mode:
authorThomas Waldmann <tw@waldmann-edv.de>2023-07-20 18:57:22 +0200
committerThomas Waldmann <tw@waldmann-edv.de>2023-07-20 18:57:22 +0200
commitb419b6ad032bebc5e8e5525826f7508a88f1719f (patch)
tree0b37758f047e506ed81076a71b616ddec8a42017 /docs/usage
parent65a2fe8502df375828b2b527106bce454edc7671 (diff)
build_man / build_usage
Diffstat (limited to 'docs/usage')
-rw-r--r--docs/usage/check.rst.inc180
-rw-r--r--docs/usage/create.rst.inc224
-rw-r--r--docs/usage/diff.rst.inc4
-rw-r--r--docs/usage/export-tar.rst.inc4
-rw-r--r--docs/usage/extract.rst.inc4
-rw-r--r--docs/usage/help.rst.inc150
-rw-r--r--docs/usage/init.rst.inc2
-rw-r--r--docs/usage/list.rst.inc4
-rw-r--r--docs/usage/mount.rst.inc4
-rw-r--r--docs/usage/prune.rst.inc15
-rw-r--r--docs/usage/recreate.rst.inc4
11 files changed, 339 insertions, 256 deletions
diff --git a/docs/usage/check.rst.inc b/docs/usage/check.rst.inc
index eebfc5652..71922bd87 100644
--- a/docs/usage/check.rst.inc
+++ b/docs/usage/check.rst.inc
@@ -85,75 +85,115 @@ borg check
Description
~~~~~~~~~~~
-The check command verifies the consistency of a repository and the corresponding archives.
-
-check --repair is a potentially dangerous function and might lead to data loss
-(for kinds of corruption it is not capable of dealing with). BE VERY CAREFUL!
+The check command verifies the consistency of a repository and its archives.
+It consists of two major steps:
+
+1. Checking the consistency of the repository itself. This includes checking
+ the segment magic headers, and both the metadata and data of all objects in
+ the segments. The read data is checked by size and CRC. Bit rot and other
+ types of accidental damage can be detected this way. Running the repository
+ check can be split into multiple partial checks using ``--max-duration``.
+ When checking a remote repository, please note that the checks run on the
+ server and do not cause significant network traffic.
+
+2. Checking consistency and correctness of the archive metadata and optionally
+ archive data (requires ``--verify-data``). This includes ensuring that the
+ repository manifest exists, the archive metadata chunk is present, and that
+ all chunks referencing files (items) in the archive exist. This requires
+ reading archive and file metadata, but not data. To cryptographically verify
+ the file (content) data integrity pass ``--verify-data``, but keep in mind
+ that this requires reading all data and is hence very time consuming. When
+ checking archives of a remote repository, archive checks run on the client
+ machine because they require decrypting data and therefore the encryption
+ key.
+
+Both steps can also be run independently. Pass ``--repository-only`` to run the
+repository checks only, or pass ``--archives-only`` to run the archive checks
+only.
+
+The ``--max-duration`` option can be used to split a long-running repository
+check into multiple partial checks. After the given number of seconds the check
+is interrupted. The next partial check will continue where the previous one
+stopped, until the full repository has been checked. Assuming a complete check
+would take 7 hours, then running a daily check with ``--max-duration=3600``
+(1 hour) would result in one full repository check per week. Doing a full
+repository check aborts any previous partial check; the next partial check will
+restart from the beginning. With partial repository checks you can run neither
+archive checks, nor enable repair mode. Consequently, if you want to use
+``--max-duration`` you must also pass ``--repository-only``, and must not pass
+``--archives-only``, nor ``--repair``.
+
+**Warning:** Please note that partial repository checks (i.e. running it with
+``--max-duration``) can only perform non-cryptographic checksum checks on the
+segment files. A full repository check (i.e. without ``--max-duration``) can
+also do a repository index check. Enabling partial repository checks excepts
+archive checks for the same reason. Therefore partial checks may be useful with
+very large repositories only where a full check would take too long.
+
+The ``--verify-data`` option will perform a full integrity verification (as
+opposed to checking the CRC32 of the segment) of data, which means reading the
+data from the repository, decrypting and decompressing it. It is a complete
+cryptographic verification and hence very time consuming, but will detect any
+accidental and malicious corruption. Tamper-resistance is only guaranteed for
+encrypted repositories against attackers without access to the keys. You can
+not use ``--verify-data`` with ``--repository-only``.
+
+About repair mode
++++++++++++++++++
+
+The check command is a readonly task by default. If any corruption is found,
+Borg will report the issue and proceed with checking. To actually repair the
+issues found, pass ``--repair``.
+
+.. note::
+
+ ``--repair`` is a **POTENTIALLY DANGEROUS FEATURE** and might lead to data
+ loss! This does not just include data that was previously lost anyway, but
+ might include more data for kinds of corruption it is not capable of
+ dealing with. **BE VERY CAREFUL!**
Pursuant to the previous warning it is also highly recommended to test the
-reliability of the hardware running this software with stress testing software
-such as memory testers. Unreliable hardware can also lead to data loss especially
-when this command is run in repair mode.
-
-First, the underlying repository data files are checked:
-
-- For all segments, the segment magic header is checked.
-- For all objects stored in the segments, all metadata (e.g. CRC and size) and
- all data is read. The read data is checked by size and CRC. Bit rot and other
- types of accidental damage can be detected this way.
-- In repair mode, if an integrity error is detected in a segment, try to recover
- as many objects from the segment as possible.
-- In repair mode, make sure that the index is consistent with the data stored in
- the segments.
-- If checking a remote repo via ``ssh:``, the repo check is executed on the server
- without causing significant network traffic.
-- The repository check can be skipped using the ``--archives-only`` option.
-- A repository check can be time consuming. Partial checks are possible with the
- ``--max-duration`` option.
-
-Second, the consistency and correctness of the archive metadata is verified:
-
-- Is the repo manifest present? If not, it is rebuilt from archive metadata
- chunks (this requires reading and decrypting of all metadata and data).
-- Check if archive metadata chunk is present; if not, remove archive from manifest.
-- For all files (items) in the archive, for all chunks referenced by these
- files, check if chunk is present. In repair mode, if a chunk is not present,
- replace it with a same-size replacement chunk of zeroes. If a previously lost
- chunk reappears (e.g. via a later backup), in repair mode the all-zero replacement
- chunk will be replaced by the correct chunk. This requires reading of archive and
- file metadata, but not data.
-- In repair mode, when all the archives were checked, orphaned chunks are deleted
- from the repo. One cause of orphaned chunks are input file related errors (like
- read errors) in the archive creation process.
-- In verify-data mode, a complete cryptographic verification of the archive data
- integrity is performed. This conflicts with ``--repository-only`` as this mode
- only makes sense if the archive checks are enabled. The full details of this mode
- are documented below.
-- If checking a remote repo via ``ssh:``, the archive check is executed on the
- client machine because it requires decryption, and this is always done client-side
- as key access is needed.
-- The archive checks can be time consuming; they can be skipped using the
- ``--repository-only`` option.
-
-The ``--max-duration`` option can be used to split a long-running repository check
-into multiple partial checks. After the given number of seconds the check is
-interrupted. The next partial check will continue where the previous one stopped,
-until the complete repository has been checked. Example: Assuming a complete check took 7
-hours, then running a daily check with --max-duration=3600 (1 hour) resulted in one
-completed check per week.
-
-Attention: A partial --repository-only check can only do way less checking than a full
---repository-only check: only the non-cryptographic checksum checks on segment file
-entries are done, while a full --repository-only check would also do a repo index check.
-A partial check cannot be combined with the ``--repair`` option. Partial checks
-may therefore be useful only with very large repositories where a full check would take
-too long.
-Doing a full repository check aborts a partial check; the next partial check will restart
-from the beginning.
-
-The ``--verify-data`` option will perform a full integrity verification (as opposed to
-checking the CRC32 of the segment) of data, which means reading the data from the
-repository, decrypting and decompressing it. This is a cryptographic verification,
-which will detect (accidental) corruption. For encrypted repositories it is
-tamper-resistant as well, unless the attacker has access to the keys. It is also very
-slow. \ No newline at end of file
+reliability of the hardware running Borg with stress testing software. This
+especially includes storage and memory testers. Unreliable hardware might lead
+to additional data loss.
+
+It is highly recommended to create a backup of your repository before running
+in repair mode (i.e. running it with ``--repair``).
+
+Repair mode will attempt to fix any corruptions found. Fixing corruptions does
+not mean recovering lost data: Borg can not magically restore data lost due to
+e.g. a hardware failure. Repairing a repository means sacrificing some data
+for the sake of the repository as a whole and the remaining data. Hence it is,
+by definition, a potentially lossy task.
+
+In practice, repair mode hooks into both the repository and archive checks:
+
+1. When checking the repository's consistency, repair mode will try to recover
+ as many objects from segments with integrity errors as possible, and ensure
+ that the index is consistent with the data stored in the segments.
+
+2. When checking the consistency and correctness of archives, repair mode might
+ remove whole archives from the manifest if their archive metadata chunk is
+ corrupt or lost. On a chunk level (i.e. the contents of files), repair mode
+ will replace corrupt or lost chunks with a same-size replacement chunk of
+ zeroes. If a previously zeroed chunk reappears, repair mode will restore
+ this lost chunk using the new chunk. Lastly, repair mode will also delete
+ orphaned chunks (e.g. caused by read errors while creating the archive).
+
+Most steps taken by repair mode have a one-time effect on the repository, like
+removing a lost archive from the repository. However, replacing a corrupt or
+lost chunk with an all-zero replacement will have an ongoing effect on the
+repository: When attempting to extract a file referencing an all-zero chunk,
+the ``extract`` command will distinctly warn about it. The FUSE filesystem
+created by the ``mount`` command will reject reading such a "zero-patched"
+file unless a special mount option is given.
+
+As mentioned earlier, Borg might be able to "heal" a "zero-patched" file in
+repair mode, if all its previously lost chunks reappear (e.g. via a later
+backup). This is achieved by Borg not only keeping track of the all-zero
+replacement chunks, but also by keeping metadata about the lost chunks. In
+repair mode Borg will check whether a previously lost chunk reappeared and will
+replace the all-zero replacement chunk by the reappeared chunk. If all lost
+chunks of a "zero-patched" file reappear, this effectively "heals" the file.
+Consequently, if lost chunks were repaired earlier, it is advised to run
+``--repair`` a second time after creating some new backups. \ No newline at end of file
diff --git a/docs/usage/create.rst.inc b/docs/usage/create.rst.inc
index 68fa29184..3059d4a8f 100644
--- a/docs/usage/create.rst.inc
+++ b/docs/usage/create.rst.inc
@@ -12,107 +12,107 @@ borg create
.. class:: borg-options-table
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | **positional arguments** |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``ARCHIVE`` | name of archive to create (must be also a valid directory name) |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``PATH`` | paths to archive |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | **options** |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``-n``, ``--dry-run`` | do not create a backup archive |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``-s``, ``--stats`` | print statistics for the created archive |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--list`` | output verbose list of items (files, dirs, ...) |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--filter STATUSCHARS`` | only display items with the given status characters (see description) |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--json`` | output stats as JSON. Implies ``--stats``. |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--no-cache-sync`` | experimental: do not synchronize the cache. Implies not using the files cache. |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--stdin-name NAME`` | use NAME in archive for stdin data (default: 'stdin') |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--stdin-user USER`` | set user USER in archive for stdin data (default: 'root') |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--stdin-group GROUP`` | set group GROUP in archive for stdin data (default: 'wheel') |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--stdin-mode M`` | set mode to M in archive for stdin data (default: 0660) |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--content-from-command`` | interpret PATH as command and store its stdout. See also section Reading from stdin below. |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--paths-from-stdin`` | read DELIM-separated list of paths to backup from stdin. Will not recurse into directories. |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--paths-from-command`` | interpret PATH as command and treat its output as ``--paths-from-stdin`` |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--paths-delimiter DELIM`` | set path delimiter for ``--paths-from-stdin`` and ``--paths-from-command`` (default: \n) |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | .. class:: borg-common-opt-ref |
- | |
- | :ref:`common_options` |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | **Exclusion options** |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``-e PATTERN``, ``--exclude PATTERN`` | exclude paths matching PATTERN |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--exclude-from EXCLUDEFILE`` | read exclude patterns from EXCLUDEFILE, one per line |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--pattern PATTERN`` | include/exclude paths matching PATTERN |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--patterns-from PATTERNFILE`` | read include/exclude patterns from PATTERNFILE, one per line |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--exclude-caches`` | exclude directories that contain a CACHEDIR.TAG file (http://www.bford.info/cachedir/spec.html) |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--exclude-if-present NAME`` | exclude directories that are tagged by containing a filesystem object with the given NAME |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--keep-exclude-tags`` | if tag objects are specified with ``--exclude-if-present``, don't omit the tag objects themselves from the backup archive |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--exclude-nodump`` | exclude files flagged NODUMP |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | **Filesystem options** |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``-x``, ``--one-file-system`` | stay in the same file system and do not store mount points of other file systems. This might behave different from your expectations, see the docs. |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--numeric-owner`` | deprecated, use ``--numeric-ids`` instead |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--numeric-ids`` | only store numeric user and group identifiers |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--noatime`` | do not store atime into archive |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--atime`` | do store atime into archive |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--noctime`` | do not store ctime into archive |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--nobirthtime`` | do not store birthtime (creation date) into archive |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--nobsdflags`` | deprecated, use ``--noflags`` instead |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--noflags`` | do not read and store flags (e.g. NODUMP, IMMUTABLE) into archive |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--noacls`` | do not read and store ACLs into archive |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--noxattrs`` | do not read and store xattrs into archive |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--sparse`` | detect sparse holes in input (supported only by fixed chunker) |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--files-cache MODE`` | operate files cache in MODE. default: ctime,size,inode |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--read-special`` | open and read block and char device files as well as FIFOs as if they were regular files. Also follows symlinks pointing to these kinds of files. |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | **Archive options** |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--comment COMMENT`` | add a comment text to the archive |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--timestamp TIMESTAMP`` | manually specify the archive creation date/time (UTC, yyyy-mm-ddThh:mm:ss format). Alternatively, give a reference file/directory. |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``-c SECONDS``, ``--checkpoint-interval SECONDS`` | write checkpoint every SECONDS seconds (Default: 1800) |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``--chunker-params PARAMS`` | specify the chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE). default: buzhash,19,23,21,4095 |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
- | | ``-C COMPRESSION``, ``--compression COMPRESSION`` | select compression algorithm, see the output of the "borg help compression" command for details. |
- +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
+ +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | **positional arguments** |
+ +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | | ``ARCHIVE`` | name of archive to create (must be also a valid directory name) |
+ +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | | ``PATH`` | paths to archive |
+ +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | **options** |
+ +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | | ``-n``, ``--dry-run`` | do not create a backup archive |
+ +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | | ``-s``, ``--stats`` | print statistics for the created archive |
+ +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | | ``--list`` | output verbose list of items (files, dirs, ...) |
+ +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | | ``--filter STATUSCHARS`` | only display items with the given status characters (see description) |
+ +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | | ``--json`` | output stats as JSON. Implies ``--stats``. |
+ +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | | ``--no-cache-sync`` | experimental: do not synchronize the cache. Implies not using the files cache. |
+ +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | | ``--stdin-name NAME`` | use NAME in archive for stdin data (default: 'stdin') |
+ +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | | ``--stdin-user USER`` | set user USER in archive for stdin data (default: 'root') |
+ +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | | ``--stdin-group GROUP`` | set group GROUP in archive for stdin data (default: 'wheel') |
+ +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | | ``--stdin-mode M`` | set mode to M in archive for stdin data (default: 0660) |
+ +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | | ``--content-from-command`` | interpret PATH as command and store its stdout. See also section Reading from stdin below. |
+ +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------