From ab31dc476e8c6e239a0d9bcbdfa076ec5f554c8a Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Sun, 4 Oct 2020 20:32:38 +0200 Subject: build_usage --- docs/usage/check.rst.inc | 3 + docs/usage/common-options.rst.inc | 2 +- docs/usage/create.rst.inc | 48 +++++++++++++-- docs/usage/help.rst.inc | 2 +- docs/usage/key_import.rst.inc | 11 +++- docs/usage/list.rst.inc | 8 +++ docs/usage/prune.rst.inc | 4 +- docs/usage/recreate.rst.inc | 120 +++++++++++++++++++------------------- 8 files changed, 130 insertions(+), 68 deletions(-) diff --git a/docs/usage/check.rst.inc b/docs/usage/check.rst.inc index e6c2717a0..9dce5c204 100644 --- a/docs/usage/check.rst.inc +++ b/docs/usage/check.rst.inc @@ -87,6 +87,9 @@ Description The check command verifies the consistency of a repository and the corresponding archives. +check --repair is a potentially dangerous function and might lead to data loss +(for kinds of corruption it is not capable of dealing with). BE VERY CAREFUL! + First, the underlying repository data files are checked: - For all segments, the segment magic header is checked. diff --git a/docs/usage/common-options.rst.inc b/docs/usage/common-options.rst.inc index 41fededf1..6a2be732c 100644 --- a/docs/usage/common-options.rst.inc +++ b/docs/usage/common-options.rst.inc @@ -11,7 +11,7 @@ --bypass-lock Bypass locking mechanism --show-version show/log the borg version --show-rc show/log the return code (rc) ---umask M set umask to M (local and remote, default: 0077) +--umask M set umask to M (local only, default: 0077) --remote-path PATH use PATH as borg executable on the remote (default: "borg") --remote-ratelimit RATE set remote network upload rate limit in kiByte/s (default: 0=unlimited) --consider-part-files treat part files like normal files (e.g. to list/extract them) diff --git a/docs/usage/create.rst.inc b/docs/usage/create.rst.inc index effab7abe..e4bdfd677 100644 --- a/docs/usage/create.rst.inc +++ b/docs/usage/create.rst.inc @@ -33,7 +33,9 @@ borg create +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``--no-cache-sync`` | experimental: do not synchronize the cache. Implies not using the files cache. | +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``--stdin-name NAME`` | use NAME in archive for stdin data (default: "stdin") | + | | ``--stdin-name NAME`` | use NAME in archive for stdin data (default: 'stdin') | + +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--content-from-command`` | interpret PATH as command and store its stdout. See also section Reading from stdin below. | +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ | .. class:: borg-common-opt-ref | | | @@ -115,7 +117,8 @@ borg create --filter STATUSCHARS only display items with the given status characters (see description) --json output stats as JSON. Implies ``--stats``. --no-cache-sync experimental: do not synchronize the cache. Implies not using the files cache. - --stdin-name NAME use NAME in archive for stdin data (default: "stdin") + --stdin-name NAME use NAME in archive for stdin data (default: 'stdin') + --content-from-command interpret PATH as command and store its stdout. See also section Reading from stdin below. :ref:`common_options` @@ -162,7 +165,9 @@ that means if relative paths are desired, the command has to be run from the cor directory. When giving '-' as path, borg will read data from standard input and create a -file 'stdin' in the created archive from that data. +file 'stdin' in the created archive from that data. In some cases it's more +appropriate to use --content-from-command, however. See section *Reading from +stdin* below for details. The archive will consume almost no disk space for files or parts of files that have already been stored in other archives. @@ -243,6 +248,8 @@ only include the objects specified by ``--exclude-if-present`` in your backup, and not include any other contents of the containing folder, this can be enabled through using the ``--keep-exclude-tags`` option. +.. _list_item_flags: + Item flags ++++++++++ @@ -281,4 +288,37 @@ Other flags used include: - 'i' = backup data was read from standard input (stdin) - '-' = dry run, item was *not* backed up - 'x' = excluded, item was *not* backed up -- '?' = missing status code (if you see this, please file a bug report!) \ No newline at end of file +- '?' = missing status code (if you see this, please file a bug report!) + +Reading from stdin +++++++++++++++++++ + +There are two methods to read from stdin. Either specify ``-`` as path and +pipe directly to borg:: + + backup-vm --id myvm --stdout | borg create REPO::ARCHIVE - + +Or use ``--content-from-command`` to have Borg manage the execution of the +command and piping. If you do so, the first PATH argument is interpreted +as command to execute and any further arguments are treated as arguments +to the command:: + + borg create --content-from-command REPO::ARCHIVE -- backup-vm --id myvm --stdout + +``--`` is used to ensure ``--id`` and ``--stdout`` are **not** considered +arguments to ``borg`` but rather ``backup-vm``. + +The difference between the two approaches is that piping to borg creates an +archive even if the command piping to borg exits with a failure. In this case, +**one can end up with truncated output being backed up**. Using +``--content-from-command``, in contrast, borg is guaranteed to fail without +creating an archive should the command fail. The command is considered failed +when it returned a non-zero exit code. + +Reading from stdin yields just a stream of data without file metadata +associated with it, and the files cache is not needed at all. So it is +safe to disable it via ``--files-cache disabled`` and speed up backup +creation a bit. + +By default, the content read from stdin is stored in a file called 'stdin'. +Use ``--stdin-name`` to change the name. \ No newline at end of file diff --git a/docs/usage/help.rst.inc b/docs/usage/help.rst.inc index 8ac089cfb..c78482667 100644 --- a/docs/usage/help.rst.inc +++ b/docs/usage/help.rst.inc @@ -77,7 +77,7 @@ Path full-match, selector `pf:` This pattern style is (only) useful to match full paths. This is kind of a pseudo pattern as it can not have any variable or unspecified parts - the full path must be given. `pf:root/file.ext` matches - `root/file.txt` only. A leading path separator is always removed. + `root/file.ext` only. A leading path separator is always removed. Implementation note: this is implemented via very time-efficient O(1) hashtable lookups (this means you can have huge amounts of such patterns diff --git a/docs/usage/key_import.rst.inc b/docs/usage/key_import.rst.inc index ab997d44c..a0890ff38 100644 --- a/docs/usage/key_import.rst.inc +++ b/docs/usage/key_import.rst.inc @@ -58,4 +58,13 @@ This command restores a key previously backed up with the export command. If the ``--paper`` option is given, the import will be an interactive process in which each line is checked for plausibility before -proceeding to the next line. For this format PATH must not be given. \ No newline at end of file +proceeding to the next line. For this format PATH must not be given. + +For repositories using keyfile encryption, the key file which ``borg key +import`` writes to depends on several factors. If the ``BORG_KEY_FILE`` +environment variable is set and non-empty, ``borg key import`` creates +or overwrites that file named by ``$BORG_KEY_FILE``. Otherwise, ``borg +key import`` searches in the ``$BORG_KEYS_DIR`` directory for a key file +associated with the repository. If a key file is found in +``$BORG_KEYS_DIR``, ``borg key import`` overwrites it; otherwise, ``borg +key import`` creates a new key file in ``$BORG_KEYS_DIR``. \ No newline at end of file diff --git a/docs/usage/list.rst.inc b/docs/usage/list.rst.inc index 709ca0bc7..f26728678 100644 --- a/docs/usage/list.rst.inc +++ b/docs/usage/list.rst.inc @@ -164,12 +164,20 @@ Keys for listing archive files: - isoctime - isoatime +- blake2b +- blake2s - md5 - sha1 - sha224 - sha256 - sha384 +- sha3_224 +- sha3_256 +- sha3_384 +- sha3_512 - sha512 +- shake_128 +- shake_256 - xxh64: XXH64 checksum of this file (note: this is NOT a cryptographic hash!) - archiveid diff --git a/docs/usage/prune.rst.inc b/docs/usage/prune.rst.inc index dd82bf0ef..ffe48429c 100644 --- a/docs/usage/prune.rst.inc +++ b/docs/usage/prune.rst.inc @@ -21,7 +21,7 @@ borg prune +-----------------------------------------------------------------------------+---------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``-n``, ``--dry-run`` | do not change repository | +-----------------------------------------------------------------------------+---------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``--force`` | force pruning of corrupted archives | + | | ``--force`` | force pruning of corrupted archives, use ``--force --force`` in case ``--force`` does not work. | +-----------------------------------------------------------------------------+---------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``-s``, ``--stats`` | print statistics for the deleted archive | +-----------------------------------------------------------------------------+---------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+ @@ -72,7 +72,7 @@ borg prune optional arguments -n, --dry-run do not change repository - --force force pruning of corrupted archives + --force force pruning of corrupted archives, use ``--force --force`` in case ``--force`` does not work. -s, --stats print statistics for the deleted archive --list output verbose list of archives it keeps/prunes --keep-within INTERVAL keep all archives within this time interval diff --git a/docs/usage/recreate.rst.inc b/docs/usage/recreate.rst.inc index de09ccbcc..e45784fae 100644 --- a/docs/usage/recreate.rst.inc +++ b/docs/usage/recreate.rst.inc @@ -12,59 +12,59 @@ borg recreate .. class:: borg-options-table - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | **positional arguments** | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``REPOSITORY_OR_ARCHIVE`` | repository or archive to recreate | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``PATH`` | paths to recreate; patterns are supported | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | **optional arguments** | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``--list`` | output verbose list of items (files, dirs, ...) | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``--filter STATUSCHARS`` | only display items with the given status characters (listed in borg create --help) | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``-n``, ``--dry-run`` | do not change anything | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``-s``, ``--stats`` | print statistics at end | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | .. class:: borg-common-opt-ref | - | | - | :ref:`common_options` | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | **Exclusion options** | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``-e PATTERN``, ``--exclude PATTERN`` | exclude paths matching PATTERN | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``--exclude-from EXCLUDEFILE`` | read exclude patterns from EXCLUDEFILE, one per line | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``--pattern PATTERN`` | experimental: include/exclude paths matching PATTERN | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``--patterns-from PATTERNFILE`` | experimental: read include/exclude patterns from PATTERNFILE, one per line | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``--exclude-caches`` | exclude directories that contain a CACHEDIR.TAG file (http://www.bford.info/cachedir/spec.html) | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``--exclude-if-present NAME`` | exclude directories that are tagged by containing a filesystem object with the given NAME | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``--keep-exclude-tags`` | if tag objects are specified with ``--exclude-if-present``, don't omit the tag objects themselves from the backup archive | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | **Archive options** | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``--target TARGET`` | create a new archive with the name ARCHIVE, do not replace existing archive (only applies for a single archive) | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``-c SECONDS``, ``--checkpoint-interval SECONDS`` | write checkpoint every SECONDS seconds (Default: 1800) | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``--comment COMMENT`` | add a comment text to the archive | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``--timestamp TIMESTAMP`` | manually specify the archive creation date/time (UTC, yyyy-mm-ddThh:mm:ss format). alternatively, give a reference file/directory. | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``-C COMPRESSION``, ``--compression COMPRESSION`` | select compression algorithm, see the output of the "borg help compression" command for details. | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``--recompress MODE`` | recompress data chunks according to ``--compression``. MODE `if-different`: recompress if current compression is with a different compression algorithm (the level is not considered). MODE `always`: recompress even if current compression is with the same compression algorithm (use this to change the compression level). MODE `never` (default): do not recompress. | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``--chunker-params PARAMS`` | specify the chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE) or `default` to use the current defaults. default: buzhash,19,23,21,4095 | - +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | **positional arguments** | + +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``REPOSITORY_OR_ARCHIVE`` | repository or archive to recreate | + +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``PATH`` | paths to recreate; patterns are supported | + +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | **optional arguments** | + +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--list`` | output verbose list of items (files, dirs, ...) | + +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--filter STATUSCHARS`` | only display items with the given status characters (listed in borg create --help) | + +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``-n``, ``--dry-run`` | do not change anything | + +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``-s``, ``--stats`` | print statistics at end | + +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | .. class:: borg-common-opt-ref | + | | + | :ref:`common_options` | + +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | **Exclusion options** | + +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``-e PATTERN``, ``--exclude PATTERN`` | exclude paths matching PATTERN | + +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--exclude-from EXCLUDEFILE`` | read exclude patterns from EXCLUDEFILE, one per line | + +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--pattern PATTERN`` | experimental: include/exclude paths matching PATTERN | + +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--patterns-from PATTERNFILE`` | experimental: read include/exclude patterns from PATTERNFILE, one per line | + +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--exclude-caches`` | exclude directories that contain a CACHEDIR.TAG file (http://www.bford.info/cachedir/spec.html) | + +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--exclude-if-present NAME`` | exclude directories that are tagged by containing a filesystem object with the given NAME | + +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--keep-exclude-tags`` | if tag objects are specified with ``--exclude-if-present``, don't omit the tag objects themselves from the backup archive | + +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | **Archive options** | + +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--target TARGET`` | create a new archive with the name ARCHIVE, do not replace existing archive (only applies for a single archive)