User:JimTittsler/Backup
From WikiEducator
Jump to: navigation, search
Backup
- Jim Tittsler <jwt-tlug@OnJapan.net>
- @jtittsler
Bad Things Happen
DECtape
DUMP (8)
-0-9 Dump levels. A level 0, full backup, guarantees the entire file
system is copied (but see also the -h option below). A level
number above 0, incremental backup, tells dump to copy all files
new or modified since the last dump of any lower level. The
default level is 0.
-- FreeBSD man page
rsync
- Andrew Tridgell and Paul Mackerras 1996
- file synchronization and file transfer program
- chunked scheme with MD5 hash and rolling checksum
- Mark Adler's adler-32 checksum (used in zlib)
rsnapshot
- Mike Rubel
- rsync & 'cp -al' (exploit hard links)
rm -rf backup.3 mv backup.2 backup.3 mv backup.1 backup.2 cp -al backup.0 backup.1 rsync -a --delete source_directory/ backup.0/
rdiff-backup
- Ben Escoto
- mirror
- incremental *reverse* diffs
Duplicity
- Ben Escoto
- encrypted and signed archives
- tar format containing
- new files
- incremental diffs
- flexibility of backend (including WebDAV, S3, etc.)
- Déjà Dup GUI
ZFS (and Btrfs)
- zfs send
- zfs receive
Attic
- Jonas Borgström
- deduplicating backup
Attic Features
- space efficient variable length chunks
- never before seen chunks are compressed and stored
- file/directory names can change
- small changes in big files - only a few new chunks
- great for VM image
- position of chunk in file
Attic Options
- compression (zlib, 6)
- optional 256 bit AES encryption
- verification with HMAC-SHA256 checksums
- off-site (most efficient if Attic is on remote host)
- mountable as a filesystem
Borg Backup
- fork of Attic
- active development
- many more "knobs"
- Thomas Waldmann, Antoine Beaupré, et. al.
- Python 3 (<10% C)
- around 6000 LOC
Borg Design
- FOSS in Python
- append-only, transactions in K/V store
- checkpoints every 5 minutes
- msgpack (with constraints on unpacking)
Borg Compression
- lz4 (very fast, low compression)
- zlib (medium speed, medium compression)
- lzma (low speed, high compression)
Borg Compression
- ~2.5G, ~122K files, local backup
| Comp | Size | Time |
|---|---|---|
| none | 1.9G | 2:50 |
| lz4 | 993M | 2:34 |
| zlib,4 | 804M | 3:22 |
| zlib,9 | 790M | 8:47 |
| lzma,0 | 751M | 6:24 |
| lzma,4 | 735M | 56:28 |
| restic | 2.1G | 2:16 |
Borg Memory Use
- trades memory use for speed
chunk_count ~= total_file_size / 2 ^ HASH_MASK_BITS
repo_index_usage = chunk_count*40
chunks_cache_usage = chunk_count*44
files_cache_usage = total_file_count*240 + chunk_count*80
mem_usage ~= repo_index_usage + chunks_cache_usage + files_cache_usage = chunk_count*164 + total_file_count*240
Borg Commands
- borg init --encryption=repokey|keyfile PATH
- borg create --compression lz4 PATH::archive SRCS
- borg list PATH
- borg list PATH::archive
- borg check repo or archive
- borg extract PATH::archive
- borg delete PATH::archive
- borg prune PATH retention rules
- borg mount repo or archive MOUNTPOINT
