From 9a8eec6af1c2916e0cd52525cf97383631af3598 Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Sat, 19 Feb 2022 18:47:47 +0100 Subject: build_usage --- docs/usage/create.rst.inc | 7 +- docs/usage/delete.rst.inc | 2 +- docs/usage/diff.rst.inc | 2 +- docs/usage/export-tar.rst.inc | 2 +- docs/usage/extract.rst.inc | 2 +- docs/usage/help.rst.inc | 148 +++++++++++++++++++++++++----------------- docs/usage/init.rst.inc | 68 ++++++++++++------- docs/usage/list.rst.inc | 2 +- 8 files changed, 138 insertions(+), 95 deletions(-) diff --git a/docs/usage/create.rst.inc b/docs/usage/create.rst.inc index e6fa4e3a1..cb377b2ac 100644 --- a/docs/usage/create.rst.inc +++ b/docs/usage/create.rst.inc @@ -260,16 +260,15 @@ how much your repository will grow. Please note that the "All archives" stats re the state after creation. Also, the ``--stats`` and ``--dry-run`` options are mutually exclusive because the data is not actually compressed and deduplicated during a dry run. -See the output of the "borg help patterns" command for more help on exclude patterns. +For more help on include/exclude patterns, see the :ref:`borg_patterns` command output. -See the output of the "borg help placeholders" command for more help on placeholders. +For more help on placeholders, see the :ref:`borg_placeholders` command output. .. man NOTES The ``--exclude`` patterns are not like tar. In tar ``--exclude`` .bundler/gems will exclude foo/.bundler/gems. In borg it will not, you need to use ``--exclude`` -'\*/.bundler/gems' to get the same effect. See ``borg help patterns`` for -more information. +'\*/.bundler/gems' to get the same effect. In addition to using ``--exclude`` patterns, it is possible to use ``--exclude-if-present`` to specify the name of a filesystem object (e.g. a file diff --git a/docs/usage/delete.rst.inc b/docs/usage/delete.rst.inc index e058804f1..0cf94191f 100644 --- a/docs/usage/delete.rst.inc +++ b/docs/usage/delete.rst.inc @@ -112,7 +112,7 @@ Please note that the "All archives" stats refer to the state after deletion. You can delete multiple archives by specifying their common prefix, if they have one, using the ``--prefix PREFIX`` option. You can also specify a shell pattern to match multiple archives using the ``--glob-archives GLOB`` option -(for more info on these patterns, see ``borg help patterns``). Note that these +(for more info on these patterns, see :ref:`borg_patterns`). Note that these two options are mutually exclusive. To avoid accidentally deleting archives, especially when using glob patterns, diff --git a/docs/usage/diff.rst.inc b/docs/usage/diff.rst.inc index ec3bffc29..267aad605 100644 --- a/docs/usage/diff.rst.inc +++ b/docs/usage/diff.rst.inc @@ -102,4 +102,4 @@ If you did not create the archives with different chunker params, pass ``--same-chunker-params``. Note that the chunker params changed from Borg 0.xx to 1.0. -See the output of the "borg help patterns" command for more help on exclude patterns. \ No newline at end of file +For more help on include/exclude patterns, see the :ref:`borg_patterns` command output. \ No newline at end of file diff --git a/docs/usage/export-tar.rst.inc b/docs/usage/export-tar.rst.inc index c898d7b51..c02a04ee6 100644 --- a/docs/usage/export-tar.rst.inc +++ b/docs/usage/export-tar.rst.inc @@ -112,7 +112,7 @@ By default the entire archive is extracted but a subset of files and directories can be selected by passing a list of ``PATHs`` as arguments. The file selection can further be restricted by using the ``--exclude`` option. -See the output of the "borg help patterns" command for more help on exclude patterns. +For more help on include/exclude patterns, see the :ref:`borg_patterns` command output. ``--progress`` can be slower than no progress display, since it makes one additional pass over the archive metadata. \ No newline at end of file diff --git a/docs/usage/extract.rst.inc b/docs/usage/extract.rst.inc index ce6ede4cc..c157c768a 100644 --- a/docs/usage/extract.rst.inc +++ b/docs/usage/extract.rst.inc @@ -106,7 +106,7 @@ archive is extracted but a subset of files and directories can be selected by passing a list of ``PATHs`` as arguments. The file selection can further be restricted by using the ``--exclude`` option. -See the output of the "borg help patterns" command for more help on exclude patterns. +For more help on include/exclude patterns, see the :ref:`borg_patterns` command output. By using ``--dry-run``, you can do all extraction steps except actually writing the output data: reading metadata and data chunks from the repo, checking the hash/hmac, diff --git a/docs/usage/help.rst.inc b/docs/usage/help.rst.inc index cd7682db0..39dcca80f 100644 --- a/docs/usage/help.rst.inc +++ b/docs/usage/help.rst.inc @@ -20,10 +20,15 @@ store all files as `some/path/.../file.ext` and ``borg create /path/to/repo /home/user`` will store all files as `home/user/.../file.ext`. +A directory exclusion pattern can end either with or without a slash ('/'). +If it ends with a slash, such as `some/path/`, the directory will be +included but not its content. If it does not end with a slash, such as +`some/path`, both the directory and content will be excluded. + File patterns support these styles: fnmatch, shell, regular expressions, path prefixes and path full-matches. By default, fnmatch is used for ``--exclude`` patterns and shell-style is used for the ``--pattern`` - option. For commands that support patterns in their ``PATH`` argument +option. For commands that support patterns in their ``PATH`` argument like (``borg list``), the default pattern is path prefix. Starting with Borg 1.2, for all but regular expression pattern matching @@ -144,64 +149,85 @@ Examples:: EOF $ borg create --exclude-from exclude.txt backup / - A more general and easier to use way to define filename matching patterns exists - with the ``--pattern`` and ``--patterns-from`` options. Using these, you may - specify the backup roots (starting points) and patterns for inclusion/exclusion. - A root path starts with the prefix `R`, followed by a path (a plain path, not a - file pattern). An include rule starts with the prefix +, an exclude rule starts - with the prefix -, an exclude-norecurse rule starts with !, all followed by a pattern. - - .. note:: - - Via ``--pattern`` or ``--patterns-from`` you can define BOTH inclusion and exclusion - of files using pattern prefixes ``+`` and ``-``. With ``--exclude`` and - ``--exclude-from`` ONLY excludes are defined. - - Inclusion patterns are useful to include paths that are contained in an excluded - path. The first matching pattern is used so if an include pattern matches before - an exclude pattern, the file is backed up. If an exclude-norecurse pattern matches - a directory, it won't recurse into it and won't discover any potential matches for - include rules below that directory. - - .. note:: - - It's possible that a sub-directory/file is matched while parent directories are not. - In that case, parent directories are not backed up thus their user, group, permission, - etc. can not be restored. - - Note that the default pattern style for ``--pattern`` and ``--patterns-from`` is - shell style (`sh:`), so those patterns behave similar to rsync include/exclude - patterns. The pattern style can be set via the `P` prefix. - - Patterns (``--pattern``) and excludes (``--exclude``) from the command line are - considered first (in the order of appearance). Then patterns from ``--patterns-from`` - are added. Exclusion patterns from ``--exclude-from`` files are appended last. - - Examples:: - - # backup pics, but not the ones from 2018, except the good ones: - # note: using = is essential to avoid cmdline argument parsing issues. - borg create --pattern=+pics/2018/good --pattern=-pics/2018 repo::arch pics - - # use a file with patterns: - borg create --patterns-from patterns.lst repo::arch - - The patterns.lst file could look like that:: - - # "sh:" pattern style is the default, so the following line is not needed: - P sh - R / - # can be rebuild - - /home/*/.cache - # they're downloads for a reason - - /home/*/Downloads - # susan is a nice person - # include susans home - + /home/susan - # don't backup the other home directories - - /home/* - # don't even look in /proc - ! /proc +A more general and easier to use way to define filename matching patterns exists +with the ``--pattern`` and ``--patterns-from`` options. Using these, you may +specify the backup roots (starting points) and patterns for inclusion/exclusion. +A root path starts with the prefix `R`, followed by a path (a plain path, not a +file pattern). An include rule starts with the prefix +, an exclude rule starts +with the prefix -, an exclude-norecurse rule starts with !, all followed by a pattern. + +.. note:: + + Via ``--pattern`` or ``--patterns-from`` you can define BOTH inclusion and exclusion + of files using pattern prefixes ``+`` and ``-``. With ``--exclude`` and + ``--exclude-from`` ONLY excludes are defined. + +Inclusion patterns are useful to include paths that are contained in an excluded +path. The first matching pattern is used so if an include pattern matches before +an exclude pattern, the file is backed up. If an exclude-norecurse pattern matches +a directory, it won't recurse into it and won't discover any potential matches for +include rules below that directory. + +.. note:: + + It's possible that a sub-directory/file is matched while parent directories are not. + In that case, parent directories are not backed up thus their user, group, permission, + etc. can not be restored. + +Note that the default pattern style for ``--pattern`` and ``--patterns-from`` is +shell style (`sh:`), so those patterns behave similar to rsync include/exclude +patterns. The pattern style can be set via the `P` prefix. + +Patterns (``--pattern``) and excludes (``--exclude``) from the command line are +considered first (in the order of appearance). Then patterns from ``--patterns-from`` +are added. Exclusion patterns from ``--exclude-from`` files are appended last. + +Examples:: + + # backup pics, but not the ones from 2018, except the good ones: + # note: using = is essential to avoid cmdline argument parsing issues. + borg create --pattern=+pics/2018/good --pattern=-pics/2018 repo::arch pics + + # use a file with patterns: + borg create --patterns-from patterns.lst repo::arch + +The patterns.lst file could look like that:: + + # "sh:" pattern style is the default, so the following line is not needed: + P sh + R / + # can be rebuild + - /home/*/.cache + # they're downloads for a reason + - /home/*/Downloads + # susan is a nice person + # include susans home + + /home/susan + # also back up this exact file + + pf:/home/bobby/specialfile.txt + # don't backup the other home directories + - /home/* + # don't even look in /proc + ! /proc + +You can specify recursion roots either on the command line or in a patternfile:: + + # these two commands do the same thing + borg create --exclude /home/bobby/junk repo::arch /home/bobby /home/susan + borg create --patterns-from patternfile.lst repo::arch + +The patternfile:: + + # note that excludes use fm: by default and patternfiles use sh: by default. + # therefore, we need to specify fm: to have the same exact behavior. + P fm + R /home/bobby + R /home/susan + + - /home/bobby/junk + +This allows you to share the same patterns between multiple repositories +without needing to specify them on the command line. .. _borg_placeholders: @@ -223,11 +249,11 @@ and ``--remote-path`` values support these placeholders: {now} The current local date and time, by default in ISO-8601 format. - You can also supply your own `format string `_, e.g. {now:%Y-%m-%d_%H:%M:%S} + You can also supply your own `format string `_, e.g. {now:%Y-%m-%d_%H:%M:%S} {utcnow} The current UTC date and time, by default in ISO-8601 format. - You can also supply your own `format string `_, e.g. {utcnow:%Y-%m-%d_%H:%M:%S} + You can also supply your own `format string `_, e.g. {utcnow:%Y-%m-%d_%H:%M:%S} {user} The user name (or UID, if no name is available) of the user running borg. diff --git a/docs/usage/init.rst.inc b/docs/usage/init.rst.inc index aa95281aa..d3d3f301d 100644 --- a/docs/usage/init.rst.inc +++ b/docs/usage/init.rst.inc @@ -62,31 +62,41 @@ Description This command initializes an empty repository. A repository is a filesystem directory containing the deduplicated data from zero or more archives. -Encryption can be enabled at repository init time. It cannot be changed later. +Encryption mode TLDR +++++++++++++++++++++ -It is not recommended to work without encryption. Repository encryption protects -you e.g. against the case that an attacker has access to your backup repository. +The encryption mode can only be configured when creating a new repository - +you can neither configure it on a per-archive basis nor change the +encryption mode of an existing repository. -Borg relies on randomly generated key material and uses that for chunking, id -generation, encryption and authentication. The key material is encrypted using -the passphrase you give before it is stored on-disk. +Use ``repokey``:: -You need to be careful with the key / the passphrase: + borg init --encryption repokey /path/to/repo -If you want "passphrase-only" security, use one of the repokey modes. The -key will be stored inside the repository (in its "config" file). In above -mentioned attack scenario, the attacker will have the key (but not the -passphrase). +Or ``repokey-blake2`` depending on which is faster on your client machines (see below):: -If you want "passphrase and having-the-key" security, use one of the keyfile -modes. The key will be stored in your home directory (in .config/borg/keys). -In the attack scenario, the attacker who has just access to your repo won't -have the key (and also not the passphrase). + borg init --encryption repokey-blake2 /path/to/repo -Make a backup copy of the key file (keyfile mode) or repo config file -(repokey mode) and keep it at a safe place, so you still have the key in -case it gets corrupted or lost. Also keep the passphrase at a safe place. -The backup that is encrypted with that key won't help you with that, of course. +Borg will: + +1. Ask you to come up with a passphrase. +2. Create a borg key (which contains 3 random secrets. See :ref:`key_files`). +3. Encrypt the key with your passphrase. +4. Store the encrypted borg key inside the repository directory (in the repo config). + This is why it is essential to use a secure passphrase. +5. Encrypt and sign your backups to prevent anyone from reading or forging them unless they + have the key and know the passphrase. Make sure to keep a backup of + your key **outside** the repository - do not lock yourself out by + "leaving your keys inside your car" (see :ref:`borg_key_export`). + For remote backups the encryption is done locally - the remote machine + never sees your passphrase, your unencrypted key or your unencrypted files. + Chunking and id generation are also based on your key to improve + your privacy. +6. Use the key when extracting files to decrypt them and to verify that the contents of + the backups have not been accidentally or maliciously altered. + +Picking a passphrase +++++++++++++++++++++ Make sure you use a good passphrase. Not too short, not too simple. The real encryption / decryption key is encrypted with / locked by your passphrase. @@ -106,14 +116,22 @@ a different keyboard layout. You can change your passphrase for existing repos at any time, it won't affect the encryption/decryption key or other secrets. -Encryption modes -++++++++++++++++ +More encryption modes ++++++++++++++++++++++ + +Only use ``--encryption none`` if you are OK with anyone who has access to +your repository being able to read your backups and tamper with their +contents without you noticing. + +If you want "passphrase and having-the-key" security, use ``--encryption keyfile``. +The key will be stored in your home directory (in ``~/.config/borg/keys``). -You can choose from the encryption modes seen in the table below on a per-repo -basis. The mode determines encryption algorithm, hash/MAC algorithm and also the -key storage location. +If you do **not** want to encrypt the contents of your backups, but still +want to detect malicious tampering use ``--encryption authenticated``. -Example: `borg init --encryption repokey ...` +If ``BLAKE2b`` is faster than ``SHA-256`` on your hardware, use ``--encryption authenticated-blake2``, +``--encryption repokey-blake2`` or ``--encryption keyfile-blake2``. Note: for remote backups +the hashing is done on your local machine. .. nanorst: inline-fill diff --git a/docs/usage/list.rst.inc b/docs/usage/list.rst.inc index 456f42a66..2c1bdc14e 100644 --- a/docs/usage/list.rst.inc +++ b/docs/usage/list.rst.inc @@ -105,7 +105,7 @@ Description This command lists the contents of a repository or an archive. -See the "borg help patterns" command for more help on exclude patterns. +For more help on include/exclude patterns, see the :ref:`borg_patterns` command output. .. man NOTES -- cgit v1.2.3