diff options
Diffstat (limited to 'docs/faq.rst')
| -rw-r--r-- | docs/faq.rst | 649 |
1 files changed, 523 insertions, 126 deletions
diff --git a/docs/faq.rst b/docs/faq.rst index 3eee339a2..442df6f66 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -5,6 +5,9 @@ Frequently asked questions ========================== +Usage & Limitations +################### + Can I backup VM disk images? ---------------------------- @@ -12,6 +15,39 @@ Yes, the `deduplication`_ technique used by |project_name| makes sure only the modified parts of the file are stored. Also, we have optional simple sparse file support for extract. +If you use non-snapshotting backup tools like Borg to back up virtual machines, +then the VMs should be turned off for the duration of the backup. Backing up live VMs can (and will) +result in corrupted or inconsistent backup contents: a VM image is just a regular file to +Borg with the same issues as regular files when it comes to concurrent reading and writing from +the same file. + +For backing up live VMs use file system snapshots on the VM host, which establishes +crash-consistency for the VM images. This means that with most file systems +(that are journaling) the FS will always be fine in the backup (but may need a +journal replay to become accessible). + +Usually this does not mean that file *contents* on the VM are consistent, since file +contents are normally not journaled. Notable exceptions are ext4 in data=journal mode, +ZFS and btrfs (unless nodatacow is used). + +Applications designed with crash-consistency in mind (most relational databases +like PostgreSQL, SQLite etc. but also for example Borg repositories) should always +be able to recover to a consistent state from a backup created with +crash-consistent snapshots (even on ext4 with data=writeback or XFS). + +Hypervisor snapshots capturing most of the VM's state can also be used for backups +and can be a better alternative to pure file system based snapshots of the VM's disk, +since no state is lost. Depending on the application this can be the easiest and most +reliable way to create application-consistent backups. + +Other applications may require a lot of work to reach application-consistency: +It's a broad and complex issue that cannot be explained in entirety here. + +Borg doesn't intend to address these issues due to their huge complexity +and platform/software dependency. Combining Borg with the mechanisms provided +by the platform (snapshots, hypervisor features) will be the best approach +to start tackling them. + Can I backup from multiple servers into a single repository? ------------------------------------------------------------ @@ -31,7 +67,7 @@ Can I copy or synchronize my repo to another location? ------------------------------------------------------ Yes, you could just copy all the files. Make sure you do that while no -backup is running. So what you get here is this: +backup is running (use `borg with-lock ...`). So what you get here is this: - client machine ---borg create---> repo1 - repo1 ---copy---> repo2 @@ -47,25 +83,6 @@ If you want to have 2 independent backups, it is better to do it like this: - client machine ---borg create---> repo1 - client machine ---borg create---> repo2 -Which file types, attributes, etc. are preserved? -------------------------------------------------- - - * Directories - * Regular files - * Hardlinks (considering all files in the same archive) - * Symlinks (stored as symlink, the symlink is not followed) - * Character and block device files - * FIFOs ("named pipes") - * Name - * Contents - * Timestamps in nanosecond precision: mtime, atime, ctime - * IDs of owning user and owning group - * Names of owning user and owning group (if the IDs can be resolved) - * Unix Mode/Permissions (u/g/o permissions, suid, sgid, sticky) - * Extended Attributes (xattrs) on Linux, OS X and FreeBSD - * Access Control Lists (ACL_) on Linux, OS X and FreeBSD - * BSD flags on OS X and FreeBSD - Which file types, attributes, etc. are *not* preserved? ------------------------------------------------------- @@ -79,34 +96,215 @@ Which file types, attributes, etc. are *not* preserved? backed up as (deduplicated and compressed) runs of zero bytes. Archive extraction has optional support to extract all-zero chunks as holes in a sparse file. - * filesystem specific attributes, like ext4 immutable bit, see :issue:`618`. + * Some filesystem specific attributes, like btrfs NOCOW, see :ref:`platforms`. + * For hardlinked symlinks, the hardlinking can not be archived (and thus, + the hardlinking will not be done at extraction time). The symlinks will + be archived and extracted as non-hardlinked symlinks, see :issue:`2379`. -Why is my backup bigger than with attic? Why doesn't |project_name| do compression by default? ----------------------------------------------------------------------------------------------- +Are there other known limitations? +---------------------------------- -Attic was rather unflexible when it comes to compression, it always -compressed using zlib level 6 (no way to switch compression off or -adjust the level or algorithm). +- A single archive can only reference a limited volume of file/dir metadata, + usually corresponding to tens or hundreds of millions of files/dirs. + When trying to go beyond that limit, you will get a fatal IntegrityError + exception telling that the (archive) object is too big. + An easy workaround is to create multiple archives with less items each. + See also the :ref:`archive_limitation` and :issue:`1452`. + + :ref:`borg_info` shows how large (relative to the maximum size) existing + archives are. + +.. _checkpoints_parts: + +If a backup stops mid-way, does the already-backed-up data stay there? +---------------------------------------------------------------------- + +Yes, |project_name| supports resuming backups. + +During a backup a special checkpoint archive named ``<archive-name>.checkpoint`` +is saved every checkpoint interval (the default value for this is 30 +minutes) containing all the data backed-up until that point. + +This checkpoint archive is a valid archive, +but it is only a partial backup (not all files that you wanted to backup are +contained in it). Having it in the repo until a successful, full backup is +completed is useful because it references all the transmitted chunks up +to the checkpoint. This means that in case of an interruption, you only need to +retransfer the data since the last checkpoint. + +If a backup was interrupted, you do not need to do any special considerations, +just invoke ``borg create`` as you always do. You may use the same archive name +as in previous attempt or a different one (e.g. if you always include the current +datetime), it does not matter. + +|project_name| always does full single-pass backups, so it will start again +from the beginning - but it will be much faster, because some of the data was +already stored into the repo (and is still referenced by the checkpoint +archive), so it does not need to get transmitted and stored again. + +Once your backup has finished successfully, you can delete all +``<archive-name>.checkpoint`` archives. If you run ``borg prune``, it will +also care for deleting unneeded checkpoints. + +Note: the checkpointing mechanism creates hidden, partial files in an archive, +so that checkpoints even work while a big file is being processed. +They are named ``<filename>.borg_part_<N>`` and all operations usually ignore +these files, but you can make them considered by giving the option +``--consider-part-files``. You usually only need that option if you are +really desperate (e.g. if you have no completed backup of that file and you'ld +rather get a partial file extracted than nothing). You do **not** want to give +that option under any normal circumstances. + +How can I backup huge file(s) over a unstable connection? +--------------------------------------------------------- + +This is not a problem anymore. + +For more details, see :ref:`checkpoints_parts`. + +How can I restore huge file(s) over an unstable connection? +----------------------------------------------------------- + +If you cannot manage to extract the whole big file in one go, you can extract +all the part files and manually concatenate them together. + +For more details, see :ref:`checkpoints_parts`. + +Can |project_name| add redundancy to the backup data to deal with hardware malfunction? +--------------------------------------------------------------------------------------- + +No, it can't. While that at first sounds like a good idea to defend against +some defect HDD sectors or SSD flash blocks, dealing with this in a +reliable way needs a lot of low-level storage layout information and +control which we do not have (and also can't get, even if we wanted). + +So, if you need that, consider RAID or a filesystem that offers redundant +storage or just make backups to different locations / different hardware. + +See also :issue:`225`. + +Can |project_name| verify data integrity of a backup archive? +------------------------------------------------------------- + +Yes, if you want to detect accidental data damage (like bit rot), use the +``check`` operation. It will notice corruption using CRCs and hashes. +If you want to be able to detect malicious tampering also, use an encrypted +repo. It will then be able to check using CRCs and HMACs. + +Can I use Borg on SMR hard drives? +---------------------------------- + +SMR (shingled magnetic recording) hard drives are very different from +regular hard drives. Applications have to behave in certain ways or +performance will be heavily degraded. + +Borg 1.1 ships with default settings suitable for SMR drives, +and has been successfully tested on *Seagate Archive v2* drives +using the ext4 file system. + +Some Linux kernel versions between 3.19 and 4.5 had various bugs +handling device-managed SMR drives, leading to IO errors, unresponsive +drives and unreliable operation in general. + +For more details, refer to :issue:`2252`. + +.. _faq-integrityerror: + +I get an IntegrityError or similar - what now? +---------------------------------------------- + +A single error does not necessarily indicate bad hardware or a Borg +bug. All hardware exhibits a bit error rate (BER). Hard drives are typically +specified as exhibiting less than one error every 12 to 120 TB +(one bit error in 10e14 to 10e15 bits). The specification is often called +*unrecoverable read error rate* (URE rate). + +Apart from these very rare errors there are two main causes of errors: + +(i) Defective hardware: described below. +(ii) Bugs in software (Borg, operating system, libraries): + Ensure software is up to date. + Check whether the issue is caused by any fixed bugs described in :ref:`important_notes`. + + +.. rubric:: Finding defective hardware + +.. note:: + + Hardware diagnostics are operating system dependent and do not + apply universally. The commands shown apply for popular Unix-like + systems. Refer to your operating system's manual. + +Checking hard drives + Find the drive containing the repository and use *findmnt*, *mount* or *lsblk* + to learn the device path (typically */dev/...*) of the drive. + Then, smartmontools can retrieve self-diagnostics of the drive in question:: + + # smartctl -a /dev/sdSomething + + The *Offline_Uncorrectable*, *Current_Pending_Sector* and *Reported_Uncorrect* + attributes indicate data corruption. A high *UDMA_CRC_Error_Count* usually + indicates a bad cable. -|project_name| offers a lot of different compression algorithms and -levels. Which of them is the best for you pretty much depends on your -use case, your data, your hardware -- so you need to do an informed -decision about whether you want to use compression, which algorithm -and which level you want to use. This is why compression defaults to -none. + I/O errors logged by the system (refer to the system journal or + dmesg) can point to issues as well. I/O errors only affecting the + file system easily go unnoticed, since they are not reported to + applications (e.g. Borg), while these errors can still corrupt data. + + Drives can corrupt some sectors in one event, while remaining + reliable otherwise. Conversely, drives can fail completely with no + advance warning. If in doubt, copy all data from the drive in + question to another drive -- just in case it fails completely. + + If any of these are suspicious, a self-test is recommended:: + + # smartctl -t long /dev/sdSomething + + Running ``fsck`` if not done already might yield further insights. + +Checking memory + Intermittent issues, such as ``borg check`` finding errors + inconsistently between runs, are frequently caused by bad memory. + + Run memtest86+ (or an equivalent memory tester) to verify that + the memory subsystem is operating correctly. + +Checking processors + Processors rarely cause errors. If they do, they are usually overclocked + or otherwise operated outside their specifications. We do not recommend to + operate hardware outside its specifications for productive use. + + Tools to verify correct processor operation include Prime95 (mprime), linpack, + and the `Intel Processor Diagnostic Tool + <https://downloadcenter.intel.com/download/19792/Intel-Processor-Diagnostic-Tool>`_ + (applies only to Intel processors). + +.. rubric:: Repairing a damaged repository + +With any defective hardware found and replaced, the damage done to the repository +needs to be ascertained and fixed. + +:ref:`borg_check` provides diagnostics and ``--repair`` options for repositories with +issues. We recommend to first run without ``--repair`` to assess the situation. +If the found issues and proposed repairs seem right, re-run "check" with ``--repair`` enabled. + +Security +######## How can I specify the encryption passphrase programmatically? ------------------------------------------------------------- -The encryption passphrase can be specified programmatically using the -`BORG_PASSPHRASE` environment variable. This is convenient when setting up -automated encrypted backups. Another option is to use -key file based encryption with a blank passphrase. See -:ref:`encrypted_repos` for more details. +There are several ways to specify a passphrase without human intervention: -.. _password_env: -.. note:: Be careful how you set the environment; using the ``env`` +Setting ``BORG_PASSPHRASE`` + The passphrase can be specified using the ``BORG_PASSPHRASE`` enviroment variable. + This is often the simplest option, but can be insecure if the script that sets it + is world-readable. + + .. _password_env: + .. note:: Be careful how you set the environment; using the ``env`` command, a ``system()`` call or using inline shell scripts + (e.g. ``BORG_PASSPHRASE=hunter2 borg ...``) might expose the credentials in the process list directly and they will be readable to all users on a system. Using ``export`` in a shell script file should be safe, however, as @@ -114,6 +312,84 @@ key file based encryption with a blank passphrase. See user <https://security.stackexchange.com/questions/14000/environment-variable-accessibility-in-linux/14009#14009>`_. +Using ``BORG_PASSCOMMAND`` with a properly permissioned file + Another option is to create a file with a password in it in your home + directory and use permissions to keep anyone else from reading it. For + example, first create a key:: + + head -c 1024 /dev/urandom | base64 > ~/.borg-passphrase + chmod 400 ~/.borg-passphrase + + Then in an automated script one can put:: + + export BORG_PASSCOMMAND="cat ~/.borg-passphrase" + + and Borg will automatically use that passphrase. + +Using keyfile-based encryption with a blank passphrase + It is possible to encrypt your repository in ``keyfile`` mode instead of the default + ``repokey`` mode and use a blank passphrase for the key file (simply press Enter twice + when ``borg init`` asks for the password). See :ref:`encrypted_repos` + for more details. + +Using ``BORG_PASSCOMMAND`` with macOS Keychain + macOS has a native manager for secrets (such as passphrases) which is safer + than just using a file as it is encrypted at rest and unlocked manually + (fortunately, the login keyring automatically unlocks when you login). With + the built-in ``security`` command, you can access it from the command line, + making it useful for ``BORG_PASSCOMMAND``. + + First generate a passphrase and use ``security`` to save it to your login + (default) keychain:: + + security add-generic-password -D secret -U -a $USER -s borg-passphrase -w $(head -c 1024 /dev/urandom | base64) + + In your backup script retrieve it in the ``BORG_PASSCOMMAND``:: + + export BORG_PASSCOMMAND="security find-generic-password -a $USER -s borg-passphrase -w" + +Using ``BORG_PASSCOMMAND`` with GNOME Keyring + GNOME also has a keyring daemon that can be used to store a Borg passphrase. + First ensure ``libsecret-tools``, ``gnome-keyring`` and ``libpam-gnome-keyring`` + are installed. If ``libpam-gnome-keyring`` wasn't already installed, ensure it + runs on login:: + + sudo sh -c "echo session optional pam_gnome_keyring.so auto_start >> /etc/pam.d/login" + sudo sh -c "echo password optional pam_gnome_keyring.so >> /etc/pam.d/passwd" + # you may need to relogin afterwards to activate the login keyring + + Then add a secret to the login keyring:: + + head -c 1024 /dev/urandom | base64 | secret-tool store borg-repository repo-name --label="Borg Passphrase" + + If a dialog box pops up prompting you to pick a password for a new keychain, use your + login password. If there is a checkbox for automatically unlocking on login, check it + to allow backups without any user intervention whatsoever. + + Once the secret is saved, retrieve it in a backup script using ``BORG_PASSCOMMAND``:: + + export BORG_PASSCOMMAND="secret-tool lookup borg-repository repo-name" + + .. note:: For this to automatically unlock the keychain it must be run + in the ``dbus`` session of an unlocked terminal; for example, running a backup + script as a ``cron`` job might not work unless you also ``export DISPLAY=:0`` + so ``secret-tool`` can pick up your open session. `It gets even more complicated`__ + when you are running the tool as a different user (e.g. running a backup as root + with the password stored in the user keyring). + +__ https://github.com/borgbackup/borg/pull/2837#discussion_r127641330 + +Using ``BORG_PASSCOMMAND`` with KWallet + KDE also has a keychain feature in the form of KWallet. The command-line tool + ``kwalletcli`` can be used to store and retrieve secrets. Ensure ``kwalletcli`` + is installed, generate a passphrase, and store it in your "wallet":: + + head -c 1024 /dev/urandom | base64 | kwalletcli -Pe borg-passphrase -f Passwords + + Once the secret is saved, retrieve it in a backup script using ``BORG_PASSCOMMAND``:: + + export BORG_PASSCOMMAND="kwalletcli -e borg-passphrase -f Passwords" + When backing up to remote encrypted repos, is encryption done locally? ---------------------------------------------------------------------- @@ -178,6 +454,16 @@ Thus: - have media at another place - have a relatively recent backup on your media +How do I report a security issue with Borg? +------------------------------------------- + +Send a private email to the :ref:`security contact <security-contact>` +if you think you have discovered a security issue. +Please disclose security issues responsibly. + +Common issues +############# + Why do I get "connection closed by remote" after a while? --------------------------------------------------------- @@ -201,10 +487,12 @@ yet noticed on the server. Try these settings: ClientAliveCountMax 3 If you have multiple borg create ... ; borg create ... commands in a already -serialized way in a single script, you need to give them --lock-wait N (with N +serialized way in a single script, you need to give them ``--lock-wait N`` (with N being a bit more than the time the server needs to terminate broken down connections and release the lock). +.. _disable_archive_chunks: + The borg cache eats way too much disk space, what can I do? ----------------------------------------------------------- @@ -235,60 +523,13 @@ This has some pros and cons, though: The long term plan to improve this is called "borgception", see :issue:`474`. -If a backup stops mid-way, does the already-backed-up data stay there? ----------------------------------------------------------------------- +Can I backup my root partition (/) with Borg? +--------------------------------------------- -Yes, |project_name| supports resuming backups. - -During a backup a special checkpoint archive named ``<archive-name>.checkpoint`` -is saved every checkpoint interval (the default value for this is 5 -minutes) containing all the data backed-up until that point. - -Checkpoints only happen between files (so they don't help for interruptions -happening while a very large file is being processed). - -This checkpoint archive is a valid archive (all files in it are valid and complete), -but it is only a partial backup (not all files that you wanted to backup are -contained in it). Having it in the repo until a successful, full backup is -completed is useful because it references all the transmitted chunks up -to the checkpoint. This means that in case of an interruption, you only need to -retransfer the data since the last checkpoint. - -If a backup was interrupted, you do not need to do any special considerations, -just invoke ``borg create`` as you always do. You may use the same archive name -as in previous attempt or a different one (e.g. if you always include the current -datetime), it does not matter. - -|project_name| always does full single-pass backups, so it will start again -from the beginning - but it will be much faster, because some of the data was -already stored into the repo (and is still referenced by the checkpoint -archive), so it does not need to get transmitted and stored again. - -Once your backup has finished successfully, you can delete all -``<archive-name>.checkpoint`` archives. If you run ``borg prune``, it will -also care for deleting unneeded checkpoints. - -How can I backup huge file(s) over a unstable connection? ---------------------------------------------------------- - -You can use this "split trick" as a workaround for the in-between-files-only -checkpoints (see above), huge files and a instable connection to the repository: - -Split the huge file(s) into parts of manageable size (e.g. 100MB) and create -a temporary archive of them. Borg will create checkpoints now more frequently -than if you try to backup the files in their original form (e.g. 100GB). - -After that, you can remove the parts again and backup the huge file(s) in -their original form. This will now work a lot faster as a lot of content chunks -are already in the repository. - -After you have successfully backed up the huge original file(s), you can remove -the temporary archive you made from the parts. - -We realize that this is just a better-than-nothing workaround, see :issue:`1198` -for a potential solution. - -Please note that this workaround only helps you for backup, not for restore. +Backing up your entire root partition works just fine, but remember to +exclude directories that make no sense to backup, such as /dev, /proc, +/sys, /tmp and /run, and to use ``--one-file-system`` if you only want to +backup the root partition (and not any mounted devices e.g.). If it crashes with a UnicodeError, what can I do? ------------------------------------------------- @@ -309,39 +550,15 @@ If you run into that, try this: the parent directory (or even everything) - mount the repo using FUSE and use some file manager -Can |project_name| add redundancy to the backup data to deal with hardware malfunction? ---------------------------------------------------------------------------------------- - -No, it can't. While that at first sounds like a good idea to defend against -some defect HDD sectors or SSD flash blocks, dealing with this in a -reliable way needs a lot of low-level storage layout information and -control which we do not have (and also can't get, even if we wanted). - -So, if you need that, consider RAID or a filesystem that offers redundant -storage or just make backups to different locations / different hardware. - -See also :issue:`225`. - -Can |project_name| verify data integrity of a backup archive? -------------------------------------------------------------- - -Yes, if you want to detect accidental data damage (like bit rot), use the -``check`` operation. It will notice corruption using CRCs and hashes. -If you want to be able to detect malicious tampering also, use an encrypted -repo. It will then be able to check using CRCs and HMACs. - .. _a_status_oddity: -I am seeing 'A' (added) status for a unchanged file!? ------------------------------------------------------ +I am seeing 'A' (added) status for an unchanged file!? +------------------------------------------------------ The files cache is used to determine whether |project_name| already "knows" / has backed up a file and if so, to skip the file from -chunking. It does intentionally *not* contain files that: - -- have >= 10 as "entry age" (|project_name| has not seen this file for a while) -- have a modification time (mtime) same as the newest mtime in the created - archive +chunking. It does intentionally *not* contain files that have a modification +time (mtime) same as the newest mtime in the created archive. So, if you see an 'A' status for unchanged file(s), they are likely the files with the most recent mtime in that archive. @@ -368,23 +585,58 @@ those files are reported as being added when, really, chunks are already used. +.. _always_chunking: + +It always chunks all my files, even unchanged ones! +--------------------------------------------------- + +|project_name| maintains a files cache where it remembers the mtime, size and +inode of files. When |project_name| does a new backup and starts processing a +file, it first looks whether the file has changed (compared to the values +stored in the files cache). If the values are the same, the file is assumed +unchanged and thus its contents won't get chunked (again). + +|project_name| can't keep an infinite history of files of course, thus entries +in the files cache have a "maximum time to live" which is set via the +environment variable BORG_FILES_CACHE_TTL (and defaults to 20). +Every time you do a backup (on the same machine, using the same user), the +cache entries' ttl values of files that were not "seen" are incremented by 1 +and if they reach BORG_FILES_CACHE_TTL, the entry is removed from the cache. + +So, for example, if you do daily backups of 26 different data sets A, B, +C, ..., Z on one machine (using the default TTL), the files from A will be +already forgotten when you repeat the same backups on the next day and it +will be slow because it would chunk all the files each time. If you set +BORG_FILES_CACHE_TTL to at least 26 (or maybe even a small multiple of that), +it would be much faster. + +Another possible reason is that files don't always have the same path, for +example if you mount a filesystem without stable mount points for each backup or if you are running the backup from a filesystem snapshot whose name is not stable. +If the directory where you mount a filesystem is different every time, +|project_name| assume they are different files. + + Is there a way to limit bandwidth with |project_name|? ------------------------------------------------------ -There is no command line option to limit bandwidth with |project_name|, but -bandwidth limiting can be accomplished with pipeviewer_: +To limit upload (i.e. :ref:`borg_create`) bandwidth, use the +``--remote-ratelimit`` option. + +There is no built-in way to limit *download* +(i.e. :ref:`borg_extract`) bandwidth, but limiting download bandwidth +can be accomplished with pipeviewer_: Create a wrapper script: /usr/local/bin/pv-wrapper :: - #!/bin/bash + #!/bin/sh ## -q, --quiet do not output any transfer information at all ## -L, --rate-limit RATE limit transfer to RATE bytes per second - export RATE=307200 + RATE=307200 pv -q -L $RATE | "$@" Add BORG_RSH environment variable to use pipeviewer wrapper script with ssh. :: - export BORG_RSH='/usr/local/bin/pv-wrapper.sh ssh' + export BORG_RSH='/usr/local/bin/pv-wrapper ssh' Now |project_name| will be bandwidth limited. Nice thing about pv is that you can change rate-limit on the fly: :: @@ -413,6 +665,74 @@ If you can reproduce the issue with the proven filesystem, please file an issue in the |project_name| issue tracker about that. +Why does running 'borg check --repair' warn about data loss? +------------------------------------------------------------ + +Repair usually works for recovering data in a corrupted archive. However, +it's impossible to predict all modes of corruption. In some very rare +instances, such as malfunctioning storage hardware, additional repo +corruption may occur. If you can't afford to lose the repo, it's strongly +recommended that you perform repair on a copy of the repo. + +In other words, the warning is there to emphasize that |project_name|: + - Will perform automated routines that modify your backup repository + - Might not actually fix the problem you are experiencing + - Might, in very rare cases, further corrupt your repository + +In the case of malfunctioning hardware, such as a drive or USB hub +corrupting data when read or written, it's best to diagnose and fix the +cause of the initial corruption before attempting to repair the repo. If +the corruption is caused by a one time event such as a power outage, +running `borg check --repair` will fix most problems. + + +Why isn't there more progress / ETA information displayed? +---------------------------------------------------------- + +Some borg runs take quite a bit, so it would be nice to see a progress display, +maybe even including a ETA (expected time of "arrival" [here rather "completion"]). + +For some functionality, this can be done: if the total amount of work is more or +less known, we can display progress. So check if there is a ``--progress`` option. + +But sometimes, the total amount is unknown (e.g. for ``borg create`` we just do +a single pass over the filesystem, so we do not know the total file count or data +volume before reaching the end). Adding another pass just to determine that would +take additional time and could be incorrect, if the filesystem is changing. + +Even if the fs does not change and we knew count and size of all files, we still +could not compute the ``borg create`` ETA as we do not know the amount of changed +chunks, how the bandwidth of source and destination or system performance might +fluctuate. + +You see, trying to display ETA would be futile. The borg developers prefer to +rather not implement progress / ETA display than doing futile attempts. + +See also: https://xkcd.com/612/ + + +Miscellaneous +############# + +Requirements for the borg single-file binary, esp. (g)libc? +----------------------------------------------------------- + +We try to build the binary on old, but still supported systems - to keep the +minimum requirement for the (g)libc low. The (g)libc can't be bundled into +the binary as it needs to fit your kernel and OS, but Python and all other +required libraries will be bundled into the binary. + +If your system fulfills the minimum (g)libc requirement (see the README that +is released with the binary), there should be no problem. If you are slightly +below the required version, maybe just try. Due to the dynamic loading (or not +loading) of some shared libraries, it might still work depending on what +libraries are actually loaded and used. + +In the borg git repository, there is scripts/glibc_check.py that can determine +(based on the symbols' versions they want to link to) whether a set of given +(Linux) binaries works with a given glibc version. + + Why was Borg forked from Attic? ------------------------------- @@ -441,3 +761,80 @@ Borg intends to be: * do not break compatibility accidentally, without a good reason or without warning. allow compatibility breaking for other cases. * if major version number changes, it may have incompatible changes + +Migrating from Attic +#################### + +What are the differences between Attic and Borg? +------------------------------------------------ + +Borg is a fork of `Attic`_ and maintained by "`The Borg collective`_". + +.. _Attic: https://github.com/jborg/attic +.. _The Borg collective: https://borgbackup.readthedocs.org/en/latest/authors.html + +Here's a (incomplete) list of some major changes: + +* lots of attic issues fixed (see `issue #5 <https://github.com/borgbackup/borg/issues/5>`_), + including critical data corruption bugs and security issues. +* more open, faster paced development (see `issue #1 <https://github.com/borgbackup/borg/issues/1>`_) +* less chunk management overhead (less memory and disk usage for chunks index) +* faster remote cache resync (useful when backing up multiple machines into same repo) +* compression: no, lz4, zlib or lzma compression, adjustable compression levels +* repokey replaces problematic passphrase mode (you can't change the passphrase nor the pbkdf2 iteration count in "passphrase" mode) +* simple sparse file support, great for virtual machine disk files +* can read special files (e.g. block devices) or from stdin, write to stdout +* mkdir-based locking is more compatible than attic's posix locking +* uses fadvise to not spoil / blow up the fs cache +* better error messages / exception handling +* better logging, screen output, progress indication +* tested on misc. Linux systems, 32 and 64bit, FreeBSD, OpenBSD, NetBSD, macOS + +Please read the :ref:`changelog` (or ``docs/changes.rst`` in the source distribution) for more +information. + +Borg is not compatible with original Attic (but there is a one-way conversion). + +How do I migrate from Attic to Borg? +------------------------------------ + +Use :ref:`borg_upgrade`. This is a one-way process that cannot be reversed. + +There are some caveats: + +- The upgrade can only be performed on local repositories. + It cannot be performed on remote repositories. + +- If the repository is in "keyfile" encryption mode, the keyfile must + exist locally or it must be manually moved after performing the upgrade: + + 1. Locate the repository ID, contained in the ``config`` file in the repository. + 2. Locate the attic key file at ``~/.attic/keys/``. The correct key for the + repository starts with the line ``ATTIC_KEY <repository id>``. + 3. Copy the attic key file to ``~/.config/borg/keys/`` + 4. Change the first line from ``ATTIC_KEY ...`` to ``BORG_KEY ...``. + 5. Verify that the repository is now accessible (e.g. ``borg list <repository>``). +- Attic and Borg use different :ref:`"chunker params" <chunker-params>`. + This means that data added by Borg won't deduplicate with the existing data + stored by Attic. The effect is lessened if the files cache is used with Borg. +- Repositories in "passphrase" mode *must* be migrated to "repokey" mode using + :ref:`borg_key_migrate-to-repokey`. Borg does not support the "passphrase" mode + any other way. + +Why is my backup bigger than with attic? +---------------------------------------- + +Attic was rather unflexible when it comes to compression, it always +compressed using zlib level 6 (no way to switch compression off or +adjust the level or algorithm). + +The default in Borg is lz4, which is fast enough to not use significant CPU time +in most cases, but can only achieve modest compression. It still compresses +easily compressed data fairly well. + +zlib compression with all levels (1-9) as well as LZMA (1-6) are available +as well, for cases where they are worth it. + +Which choice is the best option depends on a number of factors, like +bandwidth to the repository, how well the data compresses, available CPU +power and so on. |