API Documentation

class borg.archiver.Archiver(lock_wait=None)[source]
build_parser(args=None, prog=None)[source]
do_break_lock(args)[source]

Break the repository lock (e.g. in case it was left by a dead borg.

do_change_passphrase(args)[source]

Change repository key file passphrase

do_check(args)[source]

Check repository consistency

do_create(args)[source]

Create new archive

do_debug_delete_obj(args)[source]

delete the objects with the given IDs from the repo

do_debug_dump_archive_items(args)[source]

dump (decrypted, decompressed) archive items metadata (not: data)

do_debug_get_obj(args)[source]

get object contents from the repository and write it into file

do_debug_put_obj(args)[source]

put file(s) contents into the repository

do_delete(args)[source]

Delete an existing repository or archive

do_extract(args)[source]

Extract archive contents

do_help(parser, commands, args)[source]
do_info(args)[source]

Show archive details such as disk space used

do_init(args)[source]

Initialize an empty repository

do_list(args)[source]

List archive or repository contents

do_migrate_to_repokey(args)[source]

Migrate passphrase -> repokey

do_mount(args)[source]

Mount archive or an entire repository as a FUSE fileystem

do_prune(args)[source]

Prune repository archives according to specified rules

do_rename(args)[source]

Rename an existing archive

do_serve(args)[source]

Start in server mode. This command is usually not used manually.

do_upgrade(args)[source]

upgrade a repository from a previous version

get_args(argv, cmd)[source]

usually, just returns argv, except if we deal with a ssh forced command for borg serve.

helptext = {'patterns': "\nExclusion patterns support four separate styles, fnmatch, shell, regular\nexpressions and path prefixes. If followed by a colon (':') the first two\ncharacters of a pattern are used as a style selector. Explicit style\nselection is necessary when a non-default style is desired or when the\ndesired pattern starts with two alphanumeric characters followed by a colon\n(i.e. `aa:something/*`).\n\n`Fnmatch <https://docs.python.org/3/library/fnmatch.html>`_, selector `fm:`\n\n These patterns use a variant of shell pattern syntax, with '*' matching\n any number of characters, '?' matching any single character, '[...]'\n matching any single character specified, including ranges, and '[!...]'\n matching any character not specified. For the purpose of these patterns,\n the path separator ('\\' for Windows and '/' on other systems) is not\n treated specially. Wrap meta-characters in brackets for a literal match\n (i.e. `[?]` to match the literal character `?`). For a path to match\n a pattern, it must completely match from start to end, or must match from\n the start to just before a path separator. Except for the root path,\n paths will never end in the path separator when matching is attempted.\n Thus, if a given pattern ends in a path separator, a '*' is appended\n before matching is attempted.\n\nShell-style patterns, selector `sh:`\n\n Like fnmatch patterns these are similar to shell patterns. The difference\n is that the pattern may include `**/` for matching zero or more directory\n levels, `*` for matching zero or more arbitrary characters with the\n exception of any path separator.\n\nRegular expressions, selector `re:`\n\n Regular expressions similar to those found in Perl are supported. Unlike\n shell patterns regular expressions are not required to match the complete\n path and any substring match is sufficient. It is strongly recommended to\n anchor patterns to the start ('^'), to the end ('$') or both. Path\n separators ('\\' for Windows and '/' on other systems) in paths are\n always normalized to a forward slash ('/') before applying a pattern. The\n regular expression syntax is described in the `Python documentation for\n the re module <https://docs.python.org/3/library/re.html>`_.\n\nPrefix path, selector `pp:`\n\n This pattern style is useful to match whole sub-directories. The pattern\n `pp:/data/bar` matches `/data/bar` and everything therein.\n\nExclusions can be passed via the command line option `--exclude`. When used\nfrom within a shell the patterns should be quoted to protect them from\nexpansion.\n\nThe `--exclude-from` option permits loading exclusion patterns from a text\nfile with one pattern per line. Lines empty or starting with the number sign\n('#') after removing whitespace on both ends are ignored. The optional style\nselector prefix is also supported for patterns loaded from a file. Due to\nwhitespace removal paths with whitespace at the beginning or end can only be\nexcluded using regular expressions.\n\nExamples:\n\n# Exclude '/home/user/file.o' but not '/home/user/file.odt':\n$ borg create -e '*.o' backup /\n\n# Exclude '/home/user/junk' and '/home/user/subdir/junk' but\n# not '/home/user/importantjunk' or '/etc/junk':\n$ borg create -e '/home/*/junk' backup /\n\n# Exclude the contents of '/home/user/cache' but not the directory itself:\n$ borg create -e /home/user/cache/ backup /\n\n# The file '/home/user/cache/important' is *not* backed up:\n$ borg create -e /home/user/cache/ backup / /home/user/cache/important\n\n# The contents of directories in '/home' are not backed up when their name\n# ends in '.tmp'\n$ borg create --exclude 're:^/home/[^/]+\\.tmp/' backup /\n\n# Load exclusions from file\n$ cat >exclude.txt <<EOF\n# Comment line\n/home/*/junk\n*.tmp\nfm:aa:something/*\nre:^/home/[^/]\\.tmp/\nsh:/home/*/.thumbnails\nEOF\n$ borg create --exclude-from exclude.txt backup /\n"}
open_repository(args, create=False, exclusive=False, lock=True)[source]
parse_args(args=None)[source]
preprocess_args(args)[source]
print_error(msg, *args)[source]
print_file_status(status, path)[source]
print_warning(msg, *args)[source]
run(args)[source]
class borg.archiver.ToggleAction(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)[source]

argparse action to handle “toggle” flags easily

toggle flags are in the form of --foo, --no-foo.

the --no-foo argument still needs to be passed to the add_argument() call, but it simplifies the --no detection.

borg.archiver.main()[source]
borg.archiver.setup_signal_handlers()[source]
borg.archiver.sig_info_handler(signum, stack)[source]

search the stack for infos about the currently processed file and print them

class borg.upgrader.AtticKeyfileKey(repository)[source]

backwards compatible Attic key file parser

FILE_ID = 'ATTIC KEY'
classmethod find_key_file(repository)[source]

copy of attic’s `find_key_file`_

this has two small modifications:

  1. it uses the above `get_keys_dir`_ instead of the global one, assumed to be borg’s
  2. it uses `repository.path`_ instead of `repository._location.canonical_path`_ because we can’t assume the repository has been opened by the archiver yet
static get_keys_dir()[source]

Determine where to repository keys and cache

class borg.upgrader.AtticRepositoryUpgrader(*args, **kw)[source]
borg_readme()[source]
convert_cache(dryrun)[source]

convert caches from attic to borg

those are all hash indexes, so we need to s/ATTICIDX/BORG_IDX/ in a few locations:

  • the files and chunks cache (in $ATTIC_CACHE_DIR or $HOME/.cache/attic/<repoid>/), which we could just drop, but if we’d want to convert, we could open it with the Cache.open(), edit in place and then Cache.close() to make sure we have locking right
static convert_keyfiles(keyfile, dryrun)[source]

convert key files from attic to borg

replacement pattern is s/ATTIC KEY/BORG_KEY/ in get_keys_dir(), that is $ATTIC_KEYS_DIR or $HOME/.attic/keys, and moved to $BORG_KEYS_DIR or $HOME/.config/borg/keys.

no need to decrypt to convert. we need to rewrite the whole key file because magic string length changed, but that’s not a problem because the keyfiles are small (compared to, say, all the segments).

convert_repo_index(dryrun, inplace)[source]

convert some repo files

those are all hash indexes, so we need to s/ATTICIDX/BORG_IDX/ in a few locations:

  • the repository index (in $ATTIC_REPO/index.%d, where %d is the Repository.get_index_transaction_id()), which we should probably update, with a lock, see Repository.open(), which i’m not sure we should use because it may write data on Repository.close()...
static convert_segments(segments, dryrun=True, inplace=False, progress=False)[source]

convert repository segments from attic to borg

replacement pattern is s/ATTICSEG/BORG_SEG/ in files in $ATTIC_REPO/data/**.

luckily the magic string length didn’t change so we can just replace the 8 first bytes of all regular files in there.

find_attic_keyfile()[source]

find the attic keyfiles

the keyfiles are loaded by KeyfileKey.find_key_file(). that finds the keys with the right identifier for the repo.

this is expected to look into $HOME/.attic/keys or $ATTIC_KEYS_DIR for key files matching the given Borg repository.

it is expected to raise an exception (KeyfileNotFoundError) if no key is found. whether that exception is from Borg or Attic is unclear.

this is split in a separate function in case we want to use the attic code here directly, instead of our local implementation.

static header_replace(filename, old_magic, new_magic, inplace=True)[source]
upgrade(dryrun=True, inplace=False, progress=False)[source]

convert an attic repository to a borg repository

those are the files that need to be upgraded here, from most important to least important: segments, key files, and various caches, the latter being optional, as they will be rebuilt if missing.

we nevertheless do the order in reverse, as we prefer to do the fast stuff first, to improve interactivity.

class borg.upgrader.Borg0xxKeyfileKey(repository)[source]

backwards compatible borg 0.xx key file parser

classmethod find_key_file(repository)[source]
static get_keys_dir()[source]
class borg.upgrader.BorgRepositoryUpgrader(path, create=False, exclusive=False, lock_wait=None, lock=True)[source]
find_borg0xx_keyfile()[source]
move_keyfiles(keyfile, dryrun)[source]
upgrade(dryrun=True, inplace=False, progress=False)[source]

convert an old borg repository to a current borg repository

class borg.archive.Archive(repository, key, manifest, name, cache=None, create=False, checkpoint_interval=300, numeric_owner=False, progress=False, chunker_params=(19, 23, 21, 4095), start=datetime.datetime(2016, 3, 9, 16, 42, 5, 494555), end=datetime.datetime(2016, 3, 9, 16, 42, 5, 494565))[source]
exception AlreadyExists[source]

Archive {} already exists

exception Archive.DoesNotExist[source]

Archive {} does not exist

exception Archive.IncompatibleFilesystemEncodingError[source]

Failed to encode filename “{}” into file system encoding “{}”. Consider configuring the LANG environment variable.

Archive.add_item(item)[source]
Archive.calc_stats(cache)[source]
Archive.delete(stats, progress=False)[source]
Archive.duration[source]
Archive.extract_item(item, restore_attrs=True, dry_run=False, stdout=False, sparse=False)[source]
Archive.fpr[source]
Archive.iter_items(filter=None, preload=False)[source]
static Archive.list_archives(repository, key, manifest, cache=None)[source]
Archive.load(id)[source]
Archive.process_dev(path, st)[source]
Archive.process_dir(path, st)[source]
Archive.process_fifo(path, st)[source]
Archive.process_file(path, st, cache)[source]
Archive.process_stdin(path, cache)[source]
Archive.rename(name)[source]
Archive.restore_attrs(path, item, symlink=False, fd=None)[source]
Archive.save(name=None, timestamp=None)[source]
Archive.stat_attrs(st, path)[source]
Archive.ts[source]

Timestamp of archive creation (start) in UTC

Archive.ts_end[source]

Timestamp of archive creation (end) in UTC

Archive.write_checkpoint()[source]
class borg.archive.ArchiveChecker[source]
check(repository, repair=False, archive=None, last=None, prefix=None, save_space=False)[source]
finish(save_space=False)[source]
identify_key(repository)[source]
init_chunks()[source]

Fetch a list of all object keys from repository

orphan_chunks_check()[source]
rebuild_manifest()[source]

Rebuild the manifest object if it is missing

Iterates through all objects in the repository looking for archive metadata blocks.

rebuild_refcounts(archive=None, last=None, prefix=None)[source]

Rebuild object reference counts by walking the metadata

Missing and/or incorrect data is repaired when detected

class borg.archive.CacheChunkBuffer(cache, key, stats, chunker_params=(12, 16, 14, 4095))[source]
write_chunk(chunk)[source]
class borg.archive.ChunkBuffer(key, chunker_params=(12, 16, 14, 4095))[source]
BUFFER_SIZE = 1048576
add(item)[source]
flush(flush=False)[source]
is_full()[source]
write_chunk(chunk)[source]
class borg.archive.DownloadPipeline(repository, key)[source]
fetch_many(ids, is_preloaded=False)[source]
unpack_many(ids, filter=None, preload=False)[source]
class borg.archive.RobustUnpacker(validator)[source]

A restartable/robust version of the streaming msgpack unpacker

feed(data)[source]
resync()[source]
class borg.fuse.FuseOperations(key, repository, manifest, archive, cached_repo)[source]

Export archive as a fuse filesystem

allocate_inode()[source]
get_item(inode)[source]
getattr(inode, ctx=None)[source]
getxattr(inode, name, ctx=None)[source]
listxattr(inode, ctx=None)[source]
lookup(parent_inode, name, ctx=None)[source]
mount(mountpoint, extra_options, foreground=False)[source]
open(inode, flags, ctx=None)[source]
opendir(inode, ctx=None)[source]
process_archive(archive, prefix=[])[source]

Build fuse inode hierarchy from archive metadata

read(fh, offset, size)[source]
readdir(fh, off)[source]
statfs(ctx=None)[source]
class borg.fuse.ItemCache[source]
add(item)[source]
get(inode)[source]
borg.fuse.fuse_main()[source]
class borg.locking.ExclusiveLock(path, timeout=None, sleep=None, id=None)[source]

An exclusive Lock based on mkdir fs operation being atomic.

If possible, try to use the contextmanager here like: with ExclusiveLock(...) as lock:

...

This makes sure the lock is released again if the block is left, no matter how (e.g. if an exception occurred).

acquire(timeout=None, sleep=None)[source]
break_lock()[source]
by_me()[source]
is_locked()[source]
release()[source]
exception borg.locking.LockError[source]

Failed to acquire the lock {}.

exception borg.locking.LockErrorT[source]

Failed to acquire the lock {}.

exception borg.locking.LockFailed[source]

Failed to create/acquire the lock {} ({}).

class borg.locking.LockRoster(path, id=None)[source]

A Lock Roster to track shared/exclusive lockers.

Note: you usually should call the methods with an exclusive lock held, to avoid conflicting access by multiple threads/processes/machines.

get(key)[source]
load()[source]
modify(key, op)[source]
remove()[source]
save(data)[source]
exception borg.locking.LockTimeout[source]

Failed to create/acquire the lock {} (timeout).

exception borg.locking.NotLocked[source]

Failed to release the lock {} (was not locked).

exception borg.locking.NotMyLock[source]

Failed to release the lock {} (was/is locked, but not by me).

class borg.locking.TimeoutTimer(timeout=None, sleep=None)[source]

A timer for timeout checks (can also deal with no timeout, give timeout=None [default]). It can also compute and optionally execute a reasonable sleep time (e.g. to avoid polling too often or to support thread/process rescheduling).

sleep()[source]
start()[source]
timed_out()[source]
timed_out_or_sleep()[source]
class borg.locking.UpgradableLock(path, exclusive=False, sleep=None, timeout=None, id=None)[source]

A Lock for a resource that can be accessed in a shared or exclusive way. Typically, write access to a resource needs an exclusive lock (1 writer, noone is allowed reading) and read access to a resource needs a shared lock (multiple readers are allowed).

If possible, try to use the contextmanager here like: with UpgradableLock(...) as lock:

...

This makes sure the lock is released again if the block is left, no matter how (e.g. if an exception occurred).

acquire(exclusive=None, remove=None, sleep=None)[source]
break_lock()[source]
downgrade()[source]
release()[source]
upgrade()[source]
borg.locking.get_id()[source]

Get identification tuple for ‘us’

borg.shellpattern.translate(pat)[source]

Translate a shell-style pattern to a regular expression.

The pattern may include “**<sep>” (<sep> stands for the platform-specific path separator; “/” on POSIX systems) for matching zero or more directory levels and “*” for matching zero or more arbitrary characters with the exception of any path separator. Wrap meta-characters in brackets for a literal match (i.e. “[?]” to match the literal character ”?”).

This function is derived from the “fnmatch” module distributed with the Python standard library.

Copyright (C) 2001-2016 Python Software Foundation. All rights reserved.

TODO: support {alt1,alt2} shell-style alternatives

class borg.repository.LoggedIO(path, limit, segments_per_dir, capacity=90)[source]
COMMIT = b'@\xf4<%\t\x00\x00\x00\x02'
exception SegmentFull[source]

raised when a segment is full, before opening next

LoggedIO.cleanup(transaction_id)[source]

Delete segment files left by aborted transactions

LoggedIO.close()[source]
LoggedIO.close_segment()[source]
LoggedIO.crc_fmt = <Struct object at 0x7ff633a4c6c0>
LoggedIO.delete_segment(segment)[source]
LoggedIO.get_fd(segment)[source]
LoggedIO.get_latest_segment()[source]
LoggedIO.get_segments_transaction_id()[source]

Verify that the transaction id is consistent with the index transaction id

LoggedIO.get_write_fd(no_new=False, raise_full=False)[source]
LoggedIO.header_fmt = <Struct object at 0x7ff633a4c618>
LoggedIO.header_no_crc_fmt = <Struct object at 0x7ff633a4c688>
LoggedIO.is_committed_segment(filename)[source]

Check if segment ends with a COMMIT_TAG tag

LoggedIO.iter_objects(segment, include_data=False)[source]
LoggedIO.put_header_fmt = <Struct object at 0x7ff633a4c650>
LoggedIO.read(segment, offset, id)[source]
LoggedIO.recover_segment(segment, filename)[source]
LoggedIO.segment_exists(segment)[source]
LoggedIO.segment_filename(segment)[source]
LoggedIO.segment_iterator(reverse=False)[source]
LoggedIO.write_commit()[source]
LoggedIO.write_delete(id, raise_full=False)[source]
LoggedIO.write_put(id, data, raise_full=False)[source]
class borg.repository.Repository(path, create=False, exclusive=False, lock_wait=None, lock=True)[source]

Filesystem based transactional key value store

On disk layout: dir/README dir/config dir/data/<X / SEGMENTS_PER_DIR>/<X> dir/index.X dir/hints.X

exception AlreadyExists[source]

Repository {} already exists.

exception Repository.CheckNeeded[source]

Inconsistency detected. Please run “borg check {}”.

Repository.DEFAULT_MAX_SEGMENT_SIZE = 5242880
Repository.DEFAULT_SEGMENTS_PER_DIR = 10000
exception Repository.DoesNotExist[source]

Repository {} does not exist.

exception Repository.InvalidRepository[source]

{} is not a valid repository. Check repo config.

exception Repository.ObjectNotFound[source]

Object with key {} not found in repository {}.

Repository.break_lock()[source]
Repository.check(repair=False, save_space=False)[source]

Check repository consistency

This method verifies all segment checksums and makes sure the index is consistent with the data stored in the segments.

Repository.close()[source]
Repository.commit(save_space=False)[source]

Commit transaction

Repository.compact_segments(save_space=False)[source]

Compact sparse segments by copying data into new segments

Repository.create(path)[source]

Create a new empty repository at path

Repository.delete(id, wait=True)[source]
Repository.destroy()[source]

Destroy the repository at self.path

Repository.get(id_)[source]
Repository.get_index_transaction_id()[source]
Repository.get_many(ids, is_preloaded=False)[source]
Repository.get_transaction_id()[source]
Repository.list(limit=None, marker=None)[source]
Repository.load_key()[source]
Repository.open(path, exclusive, lock_wait=None, lock=True)[source]
Repository.open_index(transaction_id)[source]
Repository.preload(ids)[source]

Preload objects (only applies to remote repositories)

Repository.prepare_txn(transaction_id, do_cleanup=True)[source]
Repository.put(id, data, wait=True)[source]
Repository.replay_segments(index_transaction_id, segments_transaction_id)[source]
Repository.rollback()[source]
Repository.save_config(path, config)[source]
Repository.save_key(keydata)[source]
Repository.write_index()[source]
class borg.lrucache.LRUCache(capacity, dispose)[source]
clear()[source]
items()[source]
exception borg.remote.ConnectionClosed[source]

Connection closed by remote host

exception borg.remote.ConnectionClosedWithHint[source]

Connection closed by remote host. {}

exception borg.remote.InvalidRPCMethod[source]

RPC method {} is not valid

exception borg.remote.PathNotAllowed[source]

Repository path not allowed

class borg.remote.RemoteRepository(location, create=False, lock_wait=None, lock=True, args=None)[source]
exception RPCError(name)[source]
RemoteRepository.borg_cmd(args, testing)[source]

return a borg serve command line

RemoteRepository.break_lock()[source]
RemoteRepository.call(cmd, *args, **kw)[source]
RemoteRepository.call_many(cmd, calls, wait=True, is_preloaded=False)[source]
RemoteRepository.check(repair=False, save_space=False)[source]
RemoteRepository.close()[source]
RemoteRepository.commit(save_space=False)[source]
RemoteRepository.delete(id_, wait=True)[source]
RemoteRepository.destroy()[source]
RemoteRepository.extra_test_args = []
RemoteRepository.get(id_)[source]
RemoteRepository.get_many(ids, is_preloaded=False)[source]
RemoteRepository.list(limit=None, marker=None)[source]
RemoteRepository.load_key()[source]
RemoteRepository.preload(ids)[source]
RemoteRepository.put(id_, data, wait=True)[source]
RemoteRepository.rollback(*args)[source]
RemoteRepository.save_key(keydata)[source]
RemoteRepository.ssh_cmd(location)[source]

return a ssh command line that can be prefixed to a borg command line

class borg.remote.RepositoryCache(repository)[source]

A caching Repository wrapper

Caches Repository GET operations using a local temporary Repository.

close()[source]
get_many(keys)[source]
class borg.remote.RepositoryNoCache(repository)[source]

A not caching Repository wrapper, passes through to repository.

Just to have same API (including the context manager) as RepositoryCache.

close()[source]
get(key)[source]
get_many(keys)[source]
class borg.remote.RepositoryServer(restrict_to_paths)[source]
negotiate(versions)[source]
open(path, create=False, lock_wait=None, lock=True)[source]
rpc_methods = ('__len__', 'check', 'commit', 'delete', 'destroy', 'get', 'list', 'negotiate', 'open', 'put', 'rollback', 'save_key', 'load_key', 'break_lock')
serve()[source]
borg.remote.cache_if_remote(repository)[source]

Compute hashtable sizes with nices properties - prime sizes (for small to medium sizes) - 2 prime-factor sizes (for big sizes) - fast growth for small sizes - slow growth for big sizes

Note:
this is just a tool for developers. within borgbackup, it is just used to generate hash_sizes definition for _hashindex.c.
class borg.hash_sizes.Policy

Policy(upto, grow)

grow

Alias for field number 1

upto

Alias for field number 0

borg.hash_sizes.eratosthenes()[source]

Yields the sequence of prime numbers via the Sieve of Eratosthenes.

borg.hash_sizes.find_bigger_prime(gen, i)[source]
borg.hash_sizes.get_grow_factor(size)[source]
borg.hash_sizes.main()[source]
borg.hash_sizes.two_prime_factors(pfix=65537)[source]

Yields numbers with 2 prime factors pfix and p.

A basic extended attributes (xattr) implementation for Linux and MacOS X

borg.xattr.get_all(path, follow_symlinks=True)[source]
borg.xattr.getxattr(path, name, *, follow_symlinks=True)[source]
borg.xattr.is_enabled(path=None)[source]

Determine if xattr is enabled on the filesystem

borg.xattr.listxattr(path, *, follow_symlinks=True)[source]
borg.xattr.setxattr(path, name, value, *, follow_symlinks=True)[source]
borg.helpers.ChunkerParams(s)[source]
borg.helpers.CompressionSpec(s)[source]
exception borg.helpers.Error[source]

Error base class

exit_code = 2
get_message()[source]
traceback = False
exception borg.helpers.ErrorWithTraceback[source]

like Error, but show a traceback also

traceback = True
exception borg.helpers.ExtensionModuleError[source]

The Borg binary extension modules do not seem to be properly installed

class borg.helpers.FnmatchPattern(pattern)[source]

Shell glob patterns to exclude. A trailing slash means to exclude the contents of a directory, but not the directory itself.

PREFIX = 'fm'
exception borg.helpers.IntegrityError[source]

Data integrity error

class borg.helpers.Location(text='')[source]

Object representing a repository / archive location

archive = None
canonical_path()[source]
env_re = re.compile('(?:::(?P<archive>[^/]+)?)?$')
file_re = re.compile('(?P<proto>file)://(?P<path>[^:]+)(?:::(?P<archive>[^/]+))?$')
host = None
parse(text)[source]
path = None
port = None
preformat_text(text)[source]

Format repository and archive path with common tags

proto = None
scp_re = re.compile('((?:(?P<user>[^@]+)@)?(?P<host>[^:/]+):)?(?P<path>[^:]+)(?:::(?P<archive>[^/]+))?$')
ssh_re = re.compile('(?P<proto>ssh)://(?:(?P<user>[^@]+)@)?(?P<host>[^:/#]+)(?::(?P<port>\\d+))?(?P<path>[^:]+)(?:::(?P<archive>[^/]+))?$')
to_key_filename()[source]
user = None
class borg.helpers.Manifest(key, repository)[source]
MANIFEST_ID = b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
list_archive_infos(sort_by=None, reverse=False)[source]
classmethod load(repository, key=None)[source]
write()[source]
class borg.helpers.PathPrefixPattern(pattern)[source]

Literal files or directories listed on the command line for some operations (e.g. extract, but not create). If a directory is specified, all paths that start with that path match as well. A trailing slash makes no difference.

PREFIX = 'pp'
class borg.helpers.PatternBase(pattern)[source]

Shared logic for inclusion/exclusion patterns.

PREFIX = NotImplemented
match(path)[source]
class borg.helpers.PatternMatcher(fallback=None)[source]
add(patterns, value)[source]

Add list of patterns to internal list. The given value is returned from the match function when one of the given patterns matches.

match(path)[source]
class borg.helpers.ProgressIndicatorEndless(step=10, file=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='ANSI_X3.4-1968'>)[source]
finish()[source]
output(triggered)[source]
progress()[source]
show()[source]
class borg.helpers.ProgressIndicatorPercent(total, step=5, start=0, same_line=False, msg='%3.0f%%', file=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='ANSI_X3.4-1968'>)[source]
finish()[source]
output(percent)[source]
progress(current=None)[source]
show(current=None)[source]
class borg.helpers.RegexPattern(pattern)[source]

Regular expression to exclude.

PREFIX = 're'
class borg.helpers.ShellPattern(pattern)[source]

Shell glob patterns to exclude. A trailing slash means to exclude the contents of a directory, but not the directory itself.

PREFIX = 'sh'
class borg.helpers.StableDict[source]

A dict subclass with stable items() ordering

items()[source]
class borg.helpers.Statistics[source]
csize_fmt[source]
osize_fmt[source]
show_progress(item=None, final=False, stream=None, dt=None)[source]
summary = ' Original size Compressed size Deduplicated size\n{label:15} {stats.osize_fmt:>20s} {stats.csize_fmt:>20s} {stats.usize_fmt:>20s}'
update(size, csize, unique)[source]
usize_fmt[source]
borg.helpers.archivename_validator()[source]
borg.helpers.bigint_to_int(mtime)[source]

Convert bytearray to int

borg.helpers.check_extension_modules()[source]
borg.helpers.daemonize()[source]

Detach process from controlling terminal and run in background

borg.helpers.decode_dict(d, keys, encoding='utf-8', errors='surrogateescape')[source]
borg.helpers.dir_is_cachedir(path)[source]

Determines whether the specified path is a cache directory (and therefore should potentially be excluded from the backup) according to the CACHEDIR.TAG protocol (http://www.brynosaurus.com/cachedir/spec.html).

borg.helpers.dir_is_tagged(path, exclude_caches, exclude_if_present)[source]

Determines whether the specified path is excluded by being a cache directory or containing user-specified tag files. Returns a list of the paths of the tag files (either CACHEDIR.TAG or the matching user-specified files).

borg.helpers.format_archive(archive)[source]
borg.helpers.format_file_size(v, precision=2)[source]

Format file size into a human friendly format

borg.helpers.format_line(format, data)[source]
borg.helpers.format_time(t)[source]

use ISO-8601 date and time format

borg.helpers.format_timedelta(td)[source]

Format timedelta in a human friendly format

borg.helpers.get_cache_dir()[source]

Determine where to repository keys and cache

borg.helpers.get_keys_dir()[source]

Determine where to repository keys and cache

borg.helpers.gid2group(*args)[source]
borg.helpers.group2gid(*args)[source]
borg.helpers.int_to_bigint(value)[source]

Convert integers larger than 64 bits to bytearray

Smaller integers are left alone

borg.helpers.is_slow_msgpack()[source]
borg.helpers.load_excludes(fh)[source]

Load and parse exclude patterns from file object. Lines empty or starting with ‘#’ after stripping whitespace on both line ends are ignored.

borg.helpers.location_validator(archive=None)[source]
borg.helpers.log_multi(*msgs, level=20)[source]

log multiple lines of text, each line by a separate logging call for cosmetic reasons

each positional argument may be a single or multiple lines (separated by

) of text.

borg.helpers.make_path_safe(path)[source]

Make path safe by making it relative and local

borg.helpers.memoize(function)[source]
borg.helpers.normalized(func)[source]

Decorator for the Pattern match methods, returning a wrapper that normalizes OSX paths to match the normalized pattern on OSX, and returning the original method on other platforms

borg.helpers.parse_pattern(pattern, fallback=<class 'borg.helpers.FnmatchPattern'>)[source]

Read pattern from string and return an instance of the appropriate implementation class.

borg.helpers.parse_timestamp(timestamp)[source]

Parse a ISO 8601 timestamp string

borg.helpers.posix_acl_use_stored_uid_gid(acl)[source]

Replace the user/group field with the stored uid/gid

borg.helpers.prune_split(archives, pattern, n, skip=[])[source]
borg.helpers.prune_within(archives, within)[source]
borg.helpers.remove_surrogates(s, errors='replace')[source]

Replace surrogates generated by fsdecode with ‘?’

borg.helpers.safe_decode(s, coding='utf-8', errors='surrogateescape')[source]

decode bytes to str, with round-tripping “invalid” bytes

borg.helpers.safe_encode(s, coding='utf-8', errors='surrogateescape')[source]

encode str to bytes, with round-tripping “invalid” bytes

borg.helpers.safe_timestamp(item_timestamp_ns)[source]
borg.helpers.sizeof_fmt(num, suffix='B', units=None, power=None, sep='', precision=2)[source]
borg.helpers.sizeof_fmt_decimal(num, suffix='B', sep='', precision=2)[source]
borg.helpers.sizeof_fmt_iec(num, suffix='B', sep='', precision=2)[source]
borg.helpers.sysinfo()[source]
borg.helpers.timestamp(s)[source]

Convert a –timestamp=s argument to a datetime object

borg.helpers.to_localtime(ts)[source]

Convert datetime object from UTC to local time zone

borg.helpers.uid2user(*args)[source]
borg.helpers.update_excludes(args)[source]

Merge exclude patterns from files with those on command line.

borg.helpers.user2uid(*args)[source]
borg.helpers.yes(msg=None, false_msg=None, true_msg=None, default_msg=None, retry_msg=None, invalid_msg=None, env_msg=None, falsish=('No', 'NO', 'no', 'N', 'n', '0'), truish=('Yes', 'YES', 'yes', 'Y', 'y', '1'), defaultish=('Default', 'DEFAULT', 'default', 'D', 'd', ''), default=False, retry=True, env_var_override=None, ofile=None, input=<built-in function input>)[source]

Output <msg> (usually a question) and let user input an answer. Qualifies the answer according to falsish, truish and defaultish as True, False or <default>. If it didn’t qualify and retry_msg is None (no retries wanted), return the default [which defaults to False]. Otherwise let user retry answering until answer is qualified.

If env_var_override is given and this var is present in the environment, do not ask the user, but just use the env var contents as answer as if it was typed in. Otherwise read input from stdin and proceed as normal. If EOF is received instead an input or an invalid input without retry possibility, return default.

param msg:introducing message to output on ofile, no
is added [None]
param retry_msg:
 retry message to output on ofile, no
is added [None]
param false_msg:
 message to output before returning False [None]
param true_msg:message to output before returning True [None]
param default_msg:
 message to output before returning a <default> [None]
param invalid_msg:
 message to output after a invalid answer was given [None]
param env_msg:message to output when using input from env_var_override [None], needs to have 2 placeholders for answer and env var name, e.g.: “{} (from {})”
param falsish:sequence of answers qualifying as False
param truish:sequence of answers qualifying as True
param defaultish:
 sequence of answers qualifying as <default>
param default:default return value (defaultish answer was given or no-answer condition) [False]
param retry:if True and input is incorrect, retry. Otherwise return default. [True]
param env_var_override:
 environment variable name [None]
param ofile:output stream [sys.stderr]
param input:input function [input from builtins]
return:boolean answer value, True or False
class borg.cache.Cache(repository, key, manifest, path=None, sync=True, do_files=False, warn_if_unencrypted=True, lock_wait=None)[source]

Client Side cache

exception CacheInitAbortedError[source]

Cache initialization aborted

exception Cache.EncryptionMethodMismatch[source]

Repository encryption method changed since last access, refusing to continue

exception Cache.RepositoryAccessAborted[source]

Repository access aborted

exception Cache.RepositoryReplay[source]

Cache is newer than repository, refusing to continue

Cache.add_chunk(id, data, stats)[source]
Cache.begin_txn()[source]
static Cache.break_lock(repository, path=None)[source]
Cache.chunk_decref(id, stats)[source]
Cache.chunk_incref(id, stats)[source]
Cache.close()[source]
Cache.commit()[source]

Commit transaction

Cache.create()[source]

Create a new empty cache at self.path

Cache.destroy()[source]

destroy the cache at self.path

Cache.file_known_and_unchanged(path_hash, st)[source]
Cache.format_tuple()[source]
Cache.memorize_file(path_hash, st, ids)[source]
Cache.open(lock_wait=None)[source]
Cache.rollback()[source]

Roll back partial and aborted transactions

Cache.seen_chunk(id, size=None)[source]
Cache.sync()[source]

Re-synchronize chunks cache with repository.

Maintains a directory with known backup archive indexes, so it only needs to fetch infos from repo and build a chunk index once per backup archive. If out of sync, missing archive indexes get added, outdated indexes get removed and a new master chunks index is built by merging all archive indexes.

class borg.key.AESKeyBase(repository)[source]

Common base class shared by KeyfileKey and PassphraseKey

Chunks are encrypted using 256bit AES in Counter Mode (CTR)

Payload layout: TYPE(1) + HMAC(32) + NONCE(8) + CIPHERTEXT

To reduce payload size only 8 bytes of the 16 bytes nonce is saved in the payload, the first 8 bytes are always zeros. This does not affect security but limits the maximum repository capacity to only 295 exabytes!

PAYLOAD_OVERHEAD = 41
decrypt(id, data)[source]
encrypt(data)[source]
extract_nonce(payload)[source]
id_hash(data)[source]

Return HMAC hash using the “id” HMAC key

init_ciphers(enc_iv=b'')[source]
init_from_random_data(data)[source]
class borg.key.KeyBase(repository)[source]
TYPE = None
decrypt(id, data)[source]
encrypt(data)[source]
id_hash(data)[source]

Return HMAC hash using the “id” HMAC key

class borg.key.KeyfileKey(repository)[source]
FILE_ID = 'BORG_KEY'
TYPE = 0
find_key()[source]
get_new_target(args)[source]
load(target, passphrase)[source]
save(target, passphrase)[source]
class borg.key.KeyfileKeyBase(repository)[source]
change_passphrase()[source]
classmethod create(repository, args)[source]
decrypt_key_file(data, passphrase)[source]
classmethod detect(repository, manifest_data)[source]
encrypt_key_file(data, passphrase)[source]
find_key()[source]
get_new_target(args)[source]
load(target, passphrase)[source]
save(target, passphrase)[source]
exception borg.key.KeyfileNotFoundError[source]

No key file for repository {} found in {}.

class borg.key.Passphrase[source]
classmethod env_passphrase(default=None)[source]
classmethod getpass(prompt)[source]
kdf(salt, iterations, length)[source]
classmethod new(allow_empty=False)[source]
classmethod verification(passphrase)[source]
class borg.key.PassphraseKey(repository)[source]
TYPE = 1
change_passphrase()[source]
classmethod create(repository, args)[source]
classmethod detect(repository, manifest_data)[source]
init(repository, passphrase)[source]
iterations = 100000
exception borg.key.PasswordRetriesExceeded[source]

exceeded the maximum password retries

class borg.key.PlaintextKey(repository)[source]
TYPE = 2
chunk_seed = 0
classmethod create(repository, args)[source]
decrypt(id, data)[source]
classmethod detect(repository, manifest_data)[source]
encrypt(data)[source]
id_hash(data)[source]
class borg.key.RepoKey(repository)[source]
TYPE = 3
find_key()[source]
get_new_target(args)[source]
load(target, passphrase)[source]
save(target, passphrase)[source]
exception borg.key.RepoKeyNotFoundError[source]

No key entry found in the config of repository {}.

exception borg.key.UnsupportedPayloadError[source]

Unsupported payload type {}. A newer version is required to access this repository.

borg.key.key_creator(repository, args)[source]
borg.key.key_factory(repository, manifest_data)[source]

logging facilities

The way to use this is as follows:

  • each module declares its own logger, using:

    from .logger import create_logger logger = create_logger()

  • then each module uses logger.info/warning/debug/etc according to the level it believes is appropriate:

    logger.debug(‘debugging info for developers or power users’) logger.info(‘normal, informational output’) logger.warning(‘warn about a non-fatal error or sth else’) logger.error(‘a fatal error’)

    ... and so on. see the logging documentation for more information

  • console interaction happens on stderr, that includes interactive reporting functions like help, info and list

  • ...except input() is special, because we can’t control the stream it is using, unfortunately. we assume that it won’t clutter stdout, because interaction would be broken then anyways

  • what is output on INFO level is additionally controlled by commandline flags

borg.logger.create_logger(name=None)[source]

lazily create a Logger object with the proper path, which is returned by find_parent_module() by default, or is provided via the commandline

this is really a shortcut for:

logger = logging.getLogger(__name__)

we use it to avoid errors and provide a more standard API.

We must create the logger lazily, because this is usually called from module level (and thus executed at import time - BEFORE setup_logging() was called). By doing it lazily we can do the setup first, we just have to be careful not to call any logger methods before the setup_logging() call. If you try, you’ll get an exception.

borg.logger.find_parent_module()[source]

find the name of a the first module calling this module

if we cannot find it, we return the current module’s name (__name__) instead.

borg.logger.setup_logging(stream=None, conf_fname=None, env_var='BORG_LOGGING_CONF', level='info', is_serve=False)[source]

setup logging module according to the arguments provided

if conf_fname is given (or the config file name can be determined via the env_var, if given): load this logging configuration.

otherwise, set up a stream handler logger on stderr (by default, if no stream is provided).

if is_serve == True, we configure a special log format as expected by the borg client log message interceptor.

borg.platform_linux.acl_get()

Saves ACL Entries

If numeric_owner is True the user/group field is not preserved only uid/gid

borg.platform_linux.acl_set()

Restore ACL Entries

If numeric_owner is True the stored uid/gid is used instead of the user/group names

borg.platform_linux.acl_use_local_uid_gid()

Replace the user/group field with the local uid/gid if possible

class borg.hashindex.ChunkIndex
add()
iteritems()
merge()
summarize()
value_size = 12
class borg.hashindex.ChunkKeyIterator
class borg.hashindex.NSIndex
iteritems()
value_size = 8
class borg.hashindex.NSKeyIterator
class borg.compress.CNONE

none - no compression, just pass through data

ID = b'\x00\x00'
compress()
decompress()
name = 'none'
class borg.compress.Compressor

compresses using a compressor with given name and parameters decompresses everything we can handle (autodetect)

compress()
decompress()
class borg.compress.CompressorBase

base class for all (de)compression classes, also handles compression format auto detection and adding/stripping the ID header (which enable auto detection).

ID = b'\xff\xff'
compress()
decompress()
detect()
name = 'baseclass'
class borg.compress.LZ4

raw LZ4 compression / decompression (liblz4).

Features:
  • lz4 is super fast
  • wrapper releases CPython’s GIL to support multithreaded code
  • buffer given by caller, avoiding frequent reallocation and buffer duplication
  • uses safe lz4 methods that never go beyond the end of the output buffer
But beware:
  • this is not very generic, the given buffer MUST be large enough to handle all compression or decompression output (or it will fail).
  • you must not do method calls to the same LZ4 instance from different threads at the same time - create one LZ4 instance per thread!
ID = b'\x01\x00'
compress()
decompress()
name = 'lz4'
class borg.compress.LZMA

lzma compression / decompression

ID = b'\x02\x00'
compress()
decompress()
name = 'lzma'
class borg.compress.ZLIB

zlib compression / decompression (python stdlib)

ID = b'\x08\x00'
compress()
decompress()
classmethod detect()
name = 'zlib'
borg.compress.get_compressor()
class borg.chunker.Chunker
chunkify()

Cut a file into chunks.

Parameters:
  • fd – Python file object
  • fh – OS-level file handle (if available), defaults to -1 which means not to use OS-level fd.
borg.chunker.buzhash()
borg.chunker.buzhash_update()

A thin OpenSSL wrapper

This could be replaced by PyCrypto maybe?

class borg.crypto.AES

A thin wrapper around the OpenSSL EVP cipher API

decrypt()
encrypt()
iv
reset()
borg.crypto.bytes_to_int
borg.crypto.bytes_to_long
borg.crypto.long_to_bytes
borg.crypto.num_aes_blocks()

Return the number of AES blocks required to encrypt/decrypt length bytes of data. Note: this is only correct for modes without padding, like AES-CTR.