build_usage

8 files changed, 130 insertions, 68 deletions

diff --git a/docs/usage/check.rst.inc b/docs/usage/check.rst.inc
index e6c2717a0..9dce5c204 100644
--- a/docs/usage/check.rst.inc
+++ b/docs/usage/check.rst.inc
@@ -87,6 +87,9 @@ Description
 
 The check command verifies the consistency of a repository and the corresponding archives.
 
+check --repair is a potentially dangerous function and might lead to data loss
+(for kinds of corruption it is not capable of dealing with). BE VERY CAREFUL!
+
 First, the underlying repository data files are checked:
 
 - For all segments, the segment magic header is checked.
diff --git a/docs/usage/common-options.rst.inc b/docs/usage/common-options.rst.inc
index 41fededf1..6a2be732c 100644
--- a/docs/usage/common-options.rst.inc
+++ b/docs/usage/common-options.rst.inc
@@ -11,7 +11,7 @@
 --bypass-lock      Bypass locking mechanism
 --show-version     show/log the borg version
 --show-rc          show/log the return code (rc)
---umask M          set umask to M (local and remote, default: 0077)
+--umask M          set umask to M (local only, default: 0077)
 --remote-path PATH    use PATH as borg executable on the remote (default: "borg")
 --remote-ratelimit RATE    set remote network upload rate limit in kiByte/s (default: 0=unlimited)
 --consider-part-files    treat part files like normal files (e.g. to list/extract them)
diff --git a/docs/usage/create.rst.inc b/docs/usage/create.rst.inc
index effab7abe..e4bdfd677 100644
--- a/docs/usage/create.rst.inc
+++ b/docs/usage/create.rst.inc
@@ -33,7 +33,9 @@ borg create
     +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
     |                                                       | ``--no-cache-sync``                               | experimental: do not synchronize the cache. Implies not using the files cache.                                                                    |
     +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--stdin-name NAME``                             | use NAME in archive for stdin data (default: "stdin")                                                                                             |
+    |                                                       | ``--stdin-name NAME``                             | use NAME in archive for stdin data (default: 'stdin')                                                                                             |
+    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``--content-from-command``                        | interpret PATH as command and store its stdout. See also section Reading from stdin below.                                                        |
     +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
     | .. class:: borg-common-opt-ref                                                                                                                                                                                                                                |
     |                                                                                                                                                                                                                                                               |
@@ -115,7 +117,8 @@ borg create
         --filter STATUSCHARS    only display items with the given status characters (see description)
         --json          output stats as JSON. Implies ``--stats``.
         --no-cache-sync    experimental: do not synchronize the cache. Implies not using the files cache.
-        --stdin-name NAME    use NAME in archive for stdin data (default: "stdin")
+        --stdin-name NAME    use NAME in archive for stdin data (default: 'stdin')
+        --content-from-command    interpret PATH as command and store its stdout. See also section Reading from stdin below.
 
 
     :ref:`common_options`
@@ -162,7 +165,9 @@ that means if relative paths are desired, the command has to be run from the cor
 directory.
 
 When giving '-' as path, borg will read data from standard input and create a
-file 'stdin' in the created archive from that data.
+file 'stdin' in the created archive from that data. In some cases it's more
+appropriate to use --content-from-command, however. See section *Reading from
+stdin* below for details.
 
 The archive will consume almost no disk space for files or parts of files that
 have already been stored in other archives.
@@ -243,6 +248,8 @@ only include the objects specified by ``--exclude-if-present`` in your backup,
 and not include any other contents of the containing folder, this can be enabled
 through using the ``--keep-exclude-tags`` option.
 
+.. _list_item_flags:
+
 Item flags
 ++++++++++
 
@@ -281,4 +288,37 @@ Other flags used include:
 - 'i' = backup data was read from standard input (stdin)
 - '-' = dry run, item was *not* backed up
 - 'x' = excluded, item was *not* backed up
-- '?' = missing status code (if you see this, please file a bug report!)
\ No newline at end of file
+- '?' = missing status code (if you see this, please file a bug report!)
+
+Reading from stdin
+++++++++++++++++++
+
+There are two methods to read from stdin. Either specify ``-`` as path and
+pipe directly to borg::
+
+    backup-vm --id myvm --stdout | borg create REPO::ARCHIVE -
+
+Or use ``--content-from-command`` to have Borg manage the execution of the
+command and piping. If you do so, the first PATH argument is interpreted
+as command to execute and any further arguments are treated as arguments
+to the command::
+
+    borg create --content-from-command REPO::ARCHIVE -- backup-vm --id myvm --stdout
+
+``--`` is used to ensure ``--id`` and ``--stdout`` are **not** considered
+arguments to ``borg`` but rather ``backup-vm``.
+
+The difference between the two approaches is that piping to borg creates an
+archive even if the command piping to borg exits with a failure. In this case,
+**one can end up with truncated output being backed up**. Using
+``--content-from-command``, in contrast, borg is guaranteed to fail without
+creating an archive should the command fail. The command is considered failed
+when it returned a non-zero exit code.
+
+Reading from stdin yields just a stream of data without file metadata
+associated with it, and the files cache is not needed at all. So it is
+safe to disable it via ``--files-cache disabled`` and speed up backup
+creation a bit.
+
+By default, the content read from stdin is stored in a file called 'stdin'.
+Use ``--stdin-name`` to change the name.
\ No newline at end of file
diff --git a/docs/usage/help.rst.inc b/docs/usage/help.rst.inc
index 8ac089cfb..c78482667 100644
--- a/docs/usage/help.rst.inc
+++ b/docs/usage/help.rst.inc
@@ -77,7 +77,7 @@ Path full-match, selector `pf:`
     This pattern style is (only) useful to match full paths.
     This is kind of a pseudo pattern as it can not have any variable or
     unspecified parts - the full path must be given. `pf:root/file.ext` matches
-    `root/file.txt` only. A leading path separator is always removed.
+    `root/file.ext` only. A leading path separator is always removed.
 
     Implementation note: this is implemented via very time-efficient O(1)
     hashtable lookups (this means you can have huge amounts of such patterns
diff --git a/docs/usage/key_import.rst.inc b/docs/usage/key_import.rst.inc
index ab997d44c..a0890ff38 100644
--- a/docs/usage/key_import.rst.inc
+++ b/docs/usage/key_import.rst.inc
@@ -58,4 +58,13 @@ This command restores a key previously backed up with the export command.
 
 If the ``--paper`` option is given, the import will be an interactive
 process in which each line is checked for plausibility before
-proceeding to the next line. For this format PATH must not be given.
\ No newline at end of file
+proceeding to the next line. For this format PATH must not be given.
+
+For repositories using keyfile encryption, the key file which ``borg key
+import`` writes to depends on several factors. If the ``BORG_KEY_FILE``
+environment variable is set and non-empty, ``borg key import`` creates
+or overwrites that file named by ``$BORG_KEY_FILE``. Otherwise, ``borg
+key import`` searches in the ``$BORG_KEYS_DIR`` directory for a key file
+associated with the repository. If a key file is found in
+``$BORG_KEYS_DIR``, ``borg key import`` overwrites it; otherwise, ``borg
+key import`` creates a new key file in ``$BORG_KEYS_DIR``.
\ No newline at end of file
diff --git a/docs/usage/list.rst.inc b/docs/usage/list.rst.inc
index 709ca0bc7..f26728678 100644
--- a/docs/usage/list.rst.inc
+++ b/docs/usage/list.rst.inc
@@ -164,12 +164,20 @@ Keys for listing archive files:
 - isoctime
 - isoatime
 
+- blake2b
+- blake2s
 - md5
 - sha1
 - sha224
 - sha256
 - sha384
+- sha3_224
+- sha3_256
+- sha3_384
+- sha3_512
 - sha512
+- shake_128
+- shake_256
 - xxh64: XXH64 checksum of this file (note: this is NOT a cryptographic hash!)
 
 - archiveid
diff --git a/docs/usage/prune.rst.inc b/docs/usage/prune.rst.inc
index dd82bf0ef..ffe48429c 100644
--- a/docs/usage/prune.rst.inc
+++ b/docs/usage/prune.rst.inc
@@ -21,7 +21,7 @@ borg prune
     +-----------------------------------------------------------------------------+---------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
     |                                                                             | ``-n``, ``--dry-run``                 | do not change repository                                                                                                                               |
     +-----------------------------------------------------------------------------+---------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                                             | ``--force``                           | force pruning of corrupted archives                                                                                                                    |
+    |                                                                             | ``--force``                           | force pruning of corrupted archives, use ``--force --force`` in case ``--force`` does not work.                                                        |
     +-----------------------------------------------------------------------------+---------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
     |                                                                             | ``-s``, ``--stats``                   | print statistics for the deleted archive                                                                                                               |
     +-----------------------------------------------------------------------------+---------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
@@ -72,7 +72,7 @@ borg prune
 
     optional arguments
         -n, --dry-run         do not change repository
-        --force               force pruning of corrupted archives
+        --force               force pruning of corrupted archives, use ``--force --force`` in case ``--force`` does not work.
         -s, --stats           print statistics for the deleted archive
         --list                output verbose list of archives it keeps/prunes
         --keep-within INTERVAL    keep all archives within this time interval
diff --git a/docs/usage/recreate.rst.inc b/docs/usage/recreate.rst.inc
index de09ccbcc..e45784fae 100644
--- a/docs/usage/recreate.rst.inc
+++ b/docs/usage/recreate.rst.inc
@@ -12,59 +12,59 @@ borg recreate
 
     .. class:: borg-options-table
 
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    | **positional arguments**                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``REPOSITORY_OR_ARCHIVE``                         | repository or archive to recreate                                                                                                                                                                                                                                                                                                                                          |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``PATH``                                          | paths to recreate; patterns are supported                                                                                                                                                                                                                                                                                                                                  |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    | **optional arguments**                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--list``                                        | output verbose list of items (files, dirs, ...)                                                                                                                                                                                                                                                                                                                            |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--filter STATUSCHARS``                          | only display items with the given status characters (listed in borg create --help)                                                                                                                                                                                                                                                                                         |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``-n``, ``--dry-run``                             | do not change anything                                                                                                                                                                                                                                                                                                                                                     |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``-s``, ``--stats``                               | print statistics at end                                                                                                                                                                                                                                                                                                                                                    |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    | .. class:: borg-common-opt-ref                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
-    |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
-    | :ref:`common_options`                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    | **Exclusion options**                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``-e PATTERN``, ``--exclude PATTERN``             | exclude paths matching PATTERN                                                                                                                                                                                                                                                                                                                                             |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--exclude-from EXCLUDEFILE``                    | read exclude patterns from EXCLUDEFILE, one per line                                                                                                                                                                                                                                                                                                                       |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--pattern PATTERN``                             | experimental: include/exclude paths matching PATTERN                                                                                                                                                                                                                                                                                                                       |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--patterns-from PATTERNFILE``                   | experimental: read include/exclude patterns from PATTERNFILE, one per line                                                                                                                                                                                                                                                                                                 |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--exclude-caches``                              | exclude directories that contain a CACHEDIR.TAG file (http://www.bford.info/cachedir/spec.html)                                                                                                                                                                                                                                                                            |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--exclude-if-present NAME``                     | exclude directories that are tagged by containing a filesystem object with the given NAME                                                                                                                                                                                                                                                                                  |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--keep-exclude-tags``                           | if tag objects are specified with ``--exclude-if-present``, don't omit the tag objects themselves from the backup archive                                                                                                                                                                                                                                                  |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    | **Archive options**                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--target TARGET``                               | create a new archive with the name ARCHIVE, do not replace existing archive (only applies for a single archive)                                                                                                                                                                                                                                                            |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``-c SECONDS``, ``--checkpoint-interval SECONDS`` | write checkpoint every SECONDS seconds (Default: 1800)                                                                                                                                                                                                                                                                                                                     |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--comment COMMENT``                             | add a comment text to the archive                                                                                                                                                                                                                                                                                                                                          |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--timestamp TIMESTAMP``                         | manually specify the archive creation date/time (UTC, yyyy-mm-ddThh:mm:ss format). alternatively, give a reference file/directory.                                                                                                                                                                                                                                         |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``-C COMPRESSION``, ``--compression COMPRESSION`` | select compression algorithm, see the output of the "borg help compression" command for details.                                                                                                                                                                                                                                                                           |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--recompress MODE``                             | recompress data chunks according to ``--compression``. MODE `if-different`: recompress if current compression is with a different compression algorithm (the level is not considered). MODE `always`: recompress even if current compression is with the same compression algorithm (use this to change the compression level). MODE `never` (default): do not recompress. |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-    |                                                       | ``--chunker-params PARAMS``                       | specify the chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE) or `default` to use the current defaults. default: buzhash,19,23,21,4095                                                                                                                                                                                             |
-    +-------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    | **positional arguments**                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
+    +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``REPOSITORY_OR_ARCHIVE``                         | repository or archive to recreate                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
+    +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``PATH``                                          | paths to recreate; patterns are supported                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
+    +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    | **optional arguments**                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
+    +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``--list``                                        | output verbose list of items (files, dirs, ...)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+    +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``--filter STATUSCHARS``                          | only display items with the given status characters (listed in borg create --help)                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
+    +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``-n``, ``--dry-run``                             | do not change anything                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+    +-------------------------------------------------------+---------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+    |                                                       | ``-s``, ``--stats``                               | print statistics at end                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
+    +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------