From WikiEducator
Jump to: navigation, search


  • Jim Tittsler <>
  • @jtittsler

Bad Things Happen

Backup Backup Backup - And Test Restores.jpg

ESO Hewlett Packard 2116 minicomputer.jpg



Mylar Punched tape.jpg



DECTape unit for the DEC PDP-11.jpg


DUMP (8)

     -0-9    Dump levels.  A level 0, full backup, guarantees the entire file
	     system is copied (but see also the	-h option below).  A level
	     number above 0, incremental backup, tells dump to copy all	files
	     new or modified since the last dump of any	lower level.  The
	     default level is 0.

-- FreeBSD man page


  • Andrew Tridgell and Paul Mackerras 1996
  • file synchronization and file transfer program
  • chunked scheme with MD5 hash and rolling checksum
    • Mark Adler's adler-32 checksum (used in zlib)


  • Mike Rubel
  • rsync & 'cp -al' (exploit hard links)
rm -rf backup.3
mv backup.2 backup.3
mv backup.1 backup.2
cp -al backup.0 backup.1
rsync -a --delete source_directory/  backup.0/


  • Ben Escoto
  • mirror
  • incremental *reverse* diffs


  • Ben Escoto
  • encrypted and signed archives
  • tar format containing
    • new files
    • incremental diffs
  • flexibility of backend (including WebDAV, S3, etc.)
  • Déjà Dup GUI

ZFS (and Btrfs)

  • zfs send
  • zfs receive


  • Jonas Borgström
  • deduplicating backup

Attic Features

  • space efficient variable length chunks
    • never before seen chunks are compressed and stored
    • file/directory names can change
    • small changes in big files - only a few new chunks
      • great for VM image
    • position of chunk in file

Attic Options

  • compression (zlib, 6)
  • optional 256 bit AES encryption
  • verification with HMAC-SHA256 checksums
  • off-site (most efficient if Attic is on remote host)
  • mountable as a filesystem

Borg Backup

  • fork of Attic
    • active development
    • many more "knobs"
  • Thomas Waldmann, Antoine Beaupré, et. al.
  • Python 3 (<10% C)
  • around 6000 LOC

Borg Design

  • FOSS in Python
  • append-only, transactions in K/V store
  • checkpoints every 5 minutes
  • msgpack (with constraints on unpacking)

Borg Compression

  • lz4 (very fast, low compression)
  • zlib (medium speed, medium compression)
  • lzma (low speed, high compression)

Borg Compression

  • ~2.5G, ~122K files, local backup
Comp Size Time
none 1.9G 2:50
lz4 993M 2:34
zlib,4 804M 3:22
zlib,9 790M 8:47
lzma,0 751M 6:24
lzma,4 735M 56:28
restic 2.1G 2:16

Borg Memory Use

  • trades memory use for speed

chunk_count ~= total_file_size / 2 ^ HASH_MASK_BITS

repo_index_usage = chunk_count*40

chunks_cache_usage = chunk_count*44

files_cache_usage = total_file_count*240 + chunk_count*80

mem_usage ~= repo_index_usage + chunks_cache_usage + files_cache_usage = chunk_count*164 + total_file_count*240

Borg Commands

  • borg init --encryption=repokey|keyfile PATH
  • borg create --compression lz4 PATH::archive SRCS
  • borg list PATH
  • borg list PATH::archive
  • borg check repo or archive
  • borg extract PATH::archive
  • borg delete PATH::archive
  • borg prune PATH retention rules
  • borg mount repo or archive MOUNTPOINT