Syncing two backup drives, take two: ZFS

Sep 26, 2023

Following up on my previous setup of syncing external data backups, and my fears of bitrot gradually rising as my data becomes more fragmented, more at rest, and also more subject to frequently copy over sometimes non-ECC RAM, I’m moving to ZFS, which is nothing short of magic, on my external drives.

I’ve used ZFS via TrueNAS for years for “hot”, bulk storage data on spinning rust but it never occurred to me I could do it on my externals as well. And because my data on cold backups is very important, I’m opting to mirror two drives.

Setting up the main storage pool

  • zfs create -o encryption=on -o keylocation=prompt -o keyformat=passphrase main/encrypted

Check encryption and dataset creation:

  • zfs list -r main
  • zfs get encryption main/encrypted

To load keys for mounting

  • zfs load-key -r main/encrypted (check key status using zfs get keystatus main/encrypted)
  • zfs mount -a

Unmounting or unloading keys:

  • zfs unmount main/encrypted
  • zfs unload-key -r main/encrypted

Setting up the backup storage pool

TBD - need to check whether I can zfs send | zfs receive encrypted data in transit or need to decrypt first.

Backing up to the backup pool

  • zpool import the main and backup pools
  • Take a snapshot of the main pool using zfs snapshot main@newsnapshot -r (-r recursively creates snapshots of all datasets)
  • Find the most recent shapshot that both main and backup have using zfs list main backup -t snapshot; for example oldsnapshot is the common one
  • zfs send -R -I main@oldsnapshot main@newsnapshot | zfs receive backup to incrementally transfer between snapshot deltas
  • Delete any old snapshots (except the latest common ancestor) using zfs destroy -r main@nolongerneeded

Notes

  • Make sure you use ECC RAM or bad, bad things will happen.
  • zfs receive does not

References