Backup system that handles deltas for large files?

[microlinux]

Hi,

I’m managing a dozen servers for web and mail and a handful of web applications. Currently these installations are bare-metal (no virtualization) and follow the KISS principle. For backups, I’m running two separate servers with Rsnapshot.

I’m currently thinking about replacing these installation by a series of virtual machines. Now here’s the problem I’m facing. Let’s say one of these servers hosts a LAMP server installation with OwnCloud and something like 500 GB of data. This would result in one big VM image file under /var/lib/libvirt/images.

Now how would I handle incremental backups with this? Usually I do backups every night with Rsnapshot, but since Rsnapshot uses Rsync under the hood, only the delta of all these multiple files is transferred. But when I use virtualization, my whole VM is in one big file, and as far as I understand, I would have to retransfer the whole 500 GB or so everytime there is a change to the VM.

Now I wonder: is there some backup system that handles binary deltas, a bit like what the .drpm format is to .rpm? So even if I have one single 500 GB file, only the “altered bits” (if I may say so) get transferred in my daily backup?

Any suggestions ?

[iwalker]

Why don’t you allow the backup servers to connect over SSH to each of the VM’s and backup the data inside the VM? You can still use rsnapshot/rsync that way, and you are doing it at the file-level instead of attempting to do the whole VM disk. Incidently, attempting to do the VM image backup whilst it is turned on will not be considered a safe backup.

Technically you could snapshot the machine and back the snapshot up, but you are still going to have problems with this since to do what you mention requires a block-level type backup.

Far easier and simple would continue the rsync method, and give access to the VM over ssh to the backup machines and then backup /var/www or whatever is inside the VM.

[gerry666uk]

In general the two types of backup serve different purposes:

• The VM “snapshots” or “images” are so you can quickly restore a broken virtual machine (but without it’s “data”).
• The “data” backups, are things like 200 customer websites and databases, but that have nothing to do with the o/s or a virtual machine.

[bobar]

I knew I’d come across something like this before …

I think you might be able to do it with rdiff (see man rdiff)

From my reading of the man page, where the VM vol is /working/vm.qcow2

# save starting state ( vm 0 )
cp /working/vm.qcow2 /backup/vm-0.qcow2  
rdiff signature /working/vm.qcow2 /backup/vm-0.signature 

# backup 1 (vm 1)
rdiff delta /backup/vm-0.signature vm.qcow2 /backup/1.delta # delta = change between vm-0 and current vm
rdiff signature /working/vm.qcow2 /backup/vm-1.signature

# backup 2 (vm 2)
rdiff delta /backup/vm-1.signature vm.qcow2 /backup/2.delta # delta = change between vm-1 and current vm
rdiff signature /working/vm.qcow2 /backup/vm-2.signature

# recover
rdiff patch /backup/vm-0.qcow2 /backup/1.delta /recovered/vm-1.qcow2
rdiff patch /recovered/vm-1.qcow2 /backup/2.delta /recovered/vm-2.qcow2

I haven’t tested any of the above as to whether it works or not…
but here’s some output for an idea of times (Intel® Xeon® CPU E5-2620 v3 @ 2.40GHz, 32GB ) and file/delta sizes:

[root@devpc vmStorage]# ls -lashi fedora-server-base.qcow2
136 31G -rwxrwxrwx. 1 qemu qemu 31G Apr 23 09:51 fedora-server-base.qcow2
[root@devpc vmStorage]# time -p rdiff signature fedora-server-base.qcow2 sigtest
real 185.09
user 71.60
sys 7.73
[root@devpc vmStorage]# time -p rdiff delta sigtest fedora-server-base.qcow2 delta
real 185.27
user 74.38
sys 7.81

[root@devpc vmStorage]# ls -lashi 
...
144 1.4M -rw-r--r--.  1 root root 1.4M Apr 23 11:11 delta
136  31G -rwxrwxrwx.  1 qemu qemu  31G Apr 23 09:51 fedora-server-base.qcow2
143 6.3M -rw-r--r--.  1 root root 6.3M Apr 23 11:00 sigtest
...

There were no changes to the file so the 1.4M delta was a surprise - I have no idea how rdiff works its magic.

All that said, I use rdiff-backup which alas works the wrong way round for you / VMs in that it stores the latest as a full copy and deltas going back in time - very handy if you want to pull the last backup - its just a file copy.

I’m in a similar situation to yourself and I agree with @iwalker. I’m going to keep the same process for backups on VMs as I have on bare metal : the machine backs up to its own backup device (real or virtual) using rdiff-backup and then rsyncs that (the local backup) to a removable/offsite/rotated backup - a real disk where the VM can write via NFS/Samba or another virtual disk file (alas filesystem passthru not avail on RHEL).

EDIT
If you still want to go with backing up the VM as opposed to the files then an alternative to rdiff above would be to use a backed qcow2 image and back that up each day before commiting it to the main image. Along with the original main image you could recover to any of the backed files.

EDIT 2
(on backing up the (shutdown) VM virtual disk / volume as opposed to the files … and selling rdiff-backup )
Thinking about it some more… I suspect that rdiff-backup …
initially copies the whole file to backup copy A and then on subsequent backups …
1.copies the (“reverse”) delta between the current file and A to the backup and then
2.rsyncs the diffs between the current file and A to update A to be the same as the current file.
In which case rdiff-backup would only ever rsync the changes (albeit twice) to the backup and then the user, using rsync (or rdiff-backup) to revert, would only be rsyncing the file diffs back.

That would seem sensible and is my best guess at what happens.

If that is the case then rdiff-backup should do incremental backups and reverts just moving the diffs (twice for backups or pre-last restores).

To date I’ve only ever restored by directly rsyncing the top level/most recent backup to where I needed it as opposed to using rdiff-backup to do a restore (eg. of a previous backup).

The time to process (altered) large files (~30GB+) and work out the diffs is noticable though it hasn’t been a problem.

So you could just use rdiff-backup instead of rdiff (above) and get the generally preferable situation where you only have to restore incrementally as you go back in time.

That said, the simplest way to do incremental backups on virtual disks is probably using backed qcow2 files (shutdown, backup live qcow2 file, commit it to backing qcow2 file, restart).

EDIT 3 (DOH !)
Of course the rdiff approach involves transfering signatures too and depending on how it works/is configured etc. rdiff-backup may have to read the whole last-state backup file each time to do the calcs so, depending on specifics, data transfer costs might include significantly more than the deltas. Similar may apply to rsyncing a file-level backup. The qcow2s though only involve pushing/pulling the “changes” ( … E&OE / caveat emptor etc. )

EDIT 4 (DOH ! - there’s no need to store signatures with rdiff method)

# save starting state ( vm 0 )
cp /working/vm.qcow2 /backup/vm-0.qcow2  
rdiff signature /working/vm.qcow2 /working/vm-last-bu.signature 

# backup 1 (vm 1)
rdiff delta /working/vm-last-bu.signature /working/vm.qcow2 /backup/1.delta # delta = change between vm-0 and current vm
rdiff -f signature /working/vm.qcow2 /working/vm-last-bu.signature 

# backup 2 (vm 2)
rdiff delta /working/vm-last-bu.signature /working/vm.qcow2 /backup/2.delta # delta = change between vm-1 and current vm
rdiff -f signature /working/vm.qcow2 /working/vm-last-bu.signature 

# recover
cp /backup/vm-0.qcow2 /recovered/vm-0.qcow2
rdiff patch /recovered/vm-0.qcow2 /backup/1.delta /recovered/vm-1.qcow2
rdiff patch /recovered/vm-1.qcow2 /backup/2.delta /recovered/vm-2.qcow2

So, just the diffs. Again, not tested.

In the other direction / rdiff with reverse diffs…

# save current state ( vm )
cp /working/vm.qcow2 /backup/vm.qcow2  
cp /working/vm.qcow2 /working/vm-last-bu.qcow2  

# backup 1 (vm 1) 
rdiff -f signature /working/vm.qcow2 /working/vm.signature 
rdiff delta /working/vm.signature /working/vm-last-bu.qcow2 /backup/1.delta 
rsync /working/vm.qcow2 /backup/vm.qcow2
cp -f /working/vm.qcow2 /working/vm-last-bu.qcow2  

# backup 2 (vm 2)
rdiff -f signature /working/vm.qcow2 /working/vm.signature 
rdiff delta /working/vm.signature /working/vm-last-bu.qcow2 /backup/2.delta 
rsync /working/vm.qcow2 /backup/vm.qcow2
cp -f /working/vm.qcow2 /working/vm-last-bu.qcow2 

# recover
rsync /backup/vm.qcow2 /working/vm.qcow2 # last backup state

rdiff patch /working/vm.qcow2 /backup/2.delta /working/vm-2.qcow2 # 1 increment back
mv -f /working/vm-2.qcow2 /working/vm.qcow2

rdiff patch /working/vm.qcow2 /backup/1.delta /working/vm-1.qcow2 # 2 increment back
mv -f /working/vm-1.qcow2 /working/vm.qcow2

So, the (reverse) diffs AND rsyncing the “head” AND keeping a local copy of the file.
Again, not tested.

[sspencerwire]

Several of the respondents here have covered some of this: backup data, not the VM, etc. Depending on your needs, you can also use LXD containers in a production environment similar to what this document outlines:
https://docs.rockylinux.org/guides/containers/lxd_server/

This also outlines the process for keeping container images backed up along with the data. I used this in production and the process worked flawlessly. You can still use rsnapshot for the backup process of the data. I used this all the time to backup the data on the container images.

Just another option to consider.

[abi]

hi,

latest libvirt/qemu versions have a new feature that allows to create incremental backups of running
virtual machines. You can configure your virtual machines to support the new dirty bitmap (changed
block tracking) feature. Heres a backup utility that makes use of the new features and provides a
full featured backup/restore workflow for libvirt based domains:

GitHub - abbbi/virtnbdbackup: Backup utility for Libvirt / qemu / kvm...

It also saves the virtual machine configuration aswell as other settings that are required to
restore the virtual machines, and supports remote backup, too. Also comes with a small
utility that allows you to map the thin provisioned backups into a block device for single
file recovery.

[j6tqgf.s]

borgbackup - I am very happy with this one.
restic

If you are redesigning your infra, go one step further: containers. If you are not ready to embrace openshift, than look at podman pods. I’d go after backing up data only. Backing up infra, especially virtualized - that is wasting disk space.