Recommendations for system backup strategy

Rocky Linux General
1

It’s time for me to systematize my backup strategy for my primary personal workstation. I’d like some guidance from this community about a reasonable approach. I apologize if this is inappropriate for this forum, and if so then I invite recommendations for more appropriate sources.

I appreciate your attention!

My plan is to use AWS S3 for offsite storage. I have a reasonable internet connection with the following performance (according to speedtest):

Download: 971 Mbps
Upload: 87 Mbps

UPDATE: I just looked carefully at the upload rate, and realize that if my numbers are correct, it translates to 25 hours per TB – about a week for the entire ~6.5 TB of my system.

I anticipate using the aws cli tool for uploading and downloading. That tool offers both “sync” and “cp” operations to and from arbitrary s3 buckets.

This is for disaster recovery – I don’t anticipate needing to restore specific files or directories. I want to be able to, for example, recover my world after a failed dnf update.

Here are some specific questions I have:

  1. Does it makes sense to backup individual disks?
  2. What, if any, special attention is needed to backup and restore things like /boot, /boot/efi, [SWAP], and so on?
  3. Is it safe to assume that I do not need or want to backup [SWAP]?
  4. What is a reasonable backup frequency? How is this affected by the choice I make in item 1?

Here is the local storage I need to back up (according to lsblk):

Disks (all SSDs):
sda: 931.5 G
sdb: 1.8 T
sdc: 3.7 T

Here are the partitions and their types and mountpoints (where applicable) on sda.
sda1: ntfs, 450 M (“Recovery”)
sda2: vfat, 99 M, /boot/efi
sda3: 16 M (“Microsoft reserved partition”)
sda4: ntfs, 100.4 G (“Basic data partition”)
sda5: xfs, 1 G, /boot
sda6: LVM2_member, 829.6 G

UPDATE: I used blkid to establish that the several unmounted partitions of sda are leftover from the transition to Rocky Linux from the original state as delivered by the system supplier. As delivered, this was Windows 10 Pro system. I’ve added the types and labels to the above information.

Here are the lvm partitions and mountpoints for sda6:
r1-root: 70 G, /
r1-swap: 15.7 G, [SWAP]
r1-home: 743.9 G, /home

Here are the partitions and mountpoints for sdb and sdc:
sdb1: xfs, 1.8 T, /mnt/internalhd0
sdc1: xfs, 3.7 T, /mnt/internalhd1

2

But even if you made a backup of the whole o/s, how would you restore it from S3? You boot into bare metal, and then what?
If you’re using VMs you could back up the VM image and then restore it, but what if the host breaks?

3

I think I need just sda2 and sda5 to get the system back.

Your point is well-taken.

I have 3 external USB3.1 SSDs on-hand, 2 @ 2 TB and 1 @ 1 TB.

I think I need to backup sda to one of those, perhaps in a round-robin or tower-of-hanoi approach.

I’ve started a separate topic to address this.

4

OK this is my approach to backups. This is EASY and Down-and-Dirty – any idiot can do it… which includes yours truly.

  1. Get a 2-4 TB HDD of spinning rust – they are bloody CHEAP relatively speaking.
  2. Dedicate one full drive per OS. I usually use 1 TB sized disks. SSD’s at the low end and NVMe’s on the high end.
  3. Download a copy of BackupNinja
  4. Download a copy of KNOPPIX 9.1 and burn to a DVD… or a Thumb-drive in case you do not have a DVD optical drive.

MY Backup Procedure

I run two OS’s – a Primary and a Secondary. One is Rocky Linux 8.5 ( * ) and the other is openSUSE 15.3 Leap. Rocky Linux 8.5 ( * ) is on a single 1TB Corsair Force MP600 NVMe 4th Gen. drive; openSUSE 15.3 Leap is located on a second 1TB Corsair Force MP600 NVMe 4th Gen.drive.

  1. Right after a new point release comes out, and after a “settling time” when a ton of updates have been released, I make a FULL DISK BACKUP of the drive. To do this I use boot up my copy of KNOPPIX (everyone should have a copy of this really nifty utility IMHO), After becoming /root (su) I then mount my 2TB HDD [mount dev/sda1 /mnt => cd /mnt] I then issue a dd command [dd if=/dev/nvme0n1 conv=sync,noerror bs=64K | gzip -c > Name-of-OS_Name-of-Machine_Date-of-backup,dd.gz]

EXAMPLE: Current Backups

rl-8.5-2021_ocelot_nvme0n1_20220518.dd.gz
openSUSE-15.3_Leap_ocelot_nvme1n1_20220427.dd.gz

The Backups are both located on my 2TB HDD

I make a SECOND Full disk Backup RIGHT BEFORE A NEW POINT RELEASE COME OUT. In the event the rollover goes south, you can RESTORE your entire drive to the state it was before you did the rollover. You might have noticed that I referenced Rocky Linux 8.5 (*) this way. That is because the night before I rolled over 8.5 => 8.6 I did a FULL DISK BACKUP of 8.5 After I thought 8.6 was going to go smooth as silk, only to discover a full melt down. After an initial panic, I drew in my breath, brought up KNOPPIX 9.1, started GParted, deleted all the partitions and scrubbed the drive clean, and then RESTORED 8.5 FROM MY DISK BACKUP. It took about an hour to restore the entire 1TB disk back to 8.5.

So that is Part 1 of my Backup System: A full disk Backup right AFTER a new point release comes out; and right BEFORE a new point release is due out.

Part 2:: What to do BETWEEN the two Full Disk Backups. This is where BackupNinja comes in. This is a very EASY backup utility anyone can master. I have it set to automatically backup all my key partition on a weekly basis, but you can set it to backup whatever to want on any time scale you want. This too is set up to backup to my 2 TB HDD. Periodically I have to go in a purge some of these weekly backups otherwise my 2 TB drive would run out of room. I suppose you could dedicate say a 4TB drive just to your weekly backups, and keep your Full Disk Backups on a separate drive.

I suppose if you really want to be sure you probably could also upload it to the cloud. But that would would cost a lot of money…, plus might be SLOW. What might be cheaper would be to backup to two or more drives at the same time, thus if anything happens to you Primary Backup Drive, you have the other drives that have your backups on them.

The BIG question becomes how much is your data worth, and how much can you afford. In my case I have data on computers that are at least 15 years old. Somehow that data tends to “migrate” up the computer chain thus if one machine or drive croaks, the data can be found on one of my other computers. My Genealogical data is a purrrfect example that tends to migrate from machine to machine, as I keeps updating that data all the time. There is other data that does the same type of migration. That does not include my weekly backups using BackupNinja.

Now is this the PURRFECT System Backup Strategy?? Probably NOT – definitely NOT for a major business – but for someone who runs a small network of workstations who has a limited budget, this is rather CHEAP, EASY TO EXECUTE, and PROVIDES 2 points of failure: Your everyday drive that is in constant use, and is a dedicated drive to one OS; and your backup drive. If you splurged and went with a 2 TB SSD rather than a 2 TB HDD you could further reduce your chance for a catastrophic failure of a mechanical drive.

Any rate just some ideas that I use and has more than once saved me from certain disaster, the most recent was when I rolled over my ws to Rocky Linux 8.6. That did not work out so great, but because I had a BACKUP, it was a piece of cake to roll back to 8.5.

Hope this helps and gives you some ideas of your own.

D’ Cat

5

I spent ages getting to a reasonable solution and read a lot of misleading stuff that was ultimately a waste of time eg. Duplicity.

Make sure when looking at any given solution that the author has covered recovery - they often don’t and you will encounter problems when you try to use the backup.

The situation for a lot of pros is going to be very different to say a home worker/contractor - with a lot of machines / enterprise situation I might look at Bacula but for me that’s overkill.

PC :
sda - working drive - 1TB
sdb - local, always available backup - 1TB
sdc - usb caddy - 2TB (3.5 inch) rotated disks, physically taken offsite (boxed, padded, sililca gel)

I haven’t got an online backup at the moment precisely for the reason you mention - upload times. I will eventually get around to backing up work commits online as a part of git workflow but that’s small, frequent uploads.

For on premises backup I use :

rdiff-backup - backs sda up to sdb with reverse incremental deltas.
rsync - updates sdc to be the same as sdb
ReaR - Functionality to allow recreate system ( “Relax” there is prob something for advertising standards)
bash script to tie it all together.
Then I rotate disks in sdc

this is what I do:

Nightly cron job (pc is always on)runs a script.

  1. Run ReaR / update ReaR data / recovery disk image
  2. Create snapshots using LVM ( you need to have left free space in your PVs )
  3. rdiff-backup snapshots
  4. remove snapshots
  5. rsync

At least once do test a full recovery, make notes and keep the notes as part of your backup files.

word of WARNING:
If you’re doing backups with scripts always check the targets are mounted eg

if mountpoint -q ‘/extBackup’

otherwise if you forget to put a drive in the caddy you’ll get everything written to your source (/) device and end up with a full root directory / broken system.

6

You should put a link to the BackupNinja you’re talking about - I think there might be more than one / wasn’t clear when I went looking.

7

may also be of interest.

8

Aren’t copies usually on block, file, or dump level?

  • Low level approach, like dd, LVM snapshot, or snapshot of VM, copies blocks without caring what content is on them.
  • File level copy, often with rsync, simply copies files from one filesystem to another
  • Dumps are necessary from, say databases, in order to ensure that one gets consistent output – database content could change while the file that it is in is copied. Snapshot of VM is probably a dump too as the VM is frozen for the duration of snapshot?

You get more consistent results by “offline” than “online” backup. Boot another OS, like CloneZilla, Knoppix, or shut down the VM and then your OS to be copied is offline – safer to copy as a whole.
Online, from the OS that is being copied, access is more limited but you won’t have downtime.

There are clearly three categories of data: system files, configuration, and user data. System files ought to be possible to restore by reinstalling packages/system. You just need to know which packages to install. The list of packages is part of configuration. A copy of (system) configuration could be stored in format that can be redeployed from, as in “run Ansible playbook”. That is an alternative to the more traditional “restore from backup”. Only the user data is unique and definitely requires more traditional backup.

Most “backup tools” are frontends to dd or rsync, aren’t they?

9

rsync and rdiff I think for the most part…good points though about DB backups being their own thing / consistency etc…

I think unless you’re comfortable with scripting, fstab, crypttab, system startup process etc and know which parts of the system you need and why then regular shut down and Clonezilla backups are probably the easiest safest way for a system backup and then maybe an rdiff-backup for file recovery though you likely will have to play with that to get what you want…

10

You are right!! OK here is the link: Well that did not work, as it was tagged to a Linux Mag. review. That said the install was REALLY EASY in Rocky Linux:

dnf install backupninja

I just did it in jaguar.

To start backupninja type ninjahelper. That will bring up some thing about needing a dialog to run just type (yes) and you are put in the place where you can begin your configuration. This is almost a no-brainer it is very easy to configure. I am sure there are tutorials on how in configure etc. A lot of the configuration is done via menus . Very easy to modify and set up to YOUR liking.

Hope this helps.

D’ Cat

11

These suggestions are all extremely helpful and exactly what I had hoped for. I appreciate the attention of each of you.

My current contemplated approach is to:

  1. Use clonezilla to create a bootable recovery disk (SSD)
  2. Buy a 2TB internal SSD, remove all the current internal SSDs, and install the new SSD
  3. Clone from the backup to the new internal SSD and confirm that the round-trip works
  4. Physically replace the current sda drive
  5. Try the various suggestions to backup sdb and sdc to AWS S3
  6. Do at least one s3 download to exercise the round-trip.

Please let me know if you’d like me to update this topic as I proceed. I want to avoid churning the forum with material that isn’t useful.

12

Few thoughts on drives:

Not sure how often Clonezilla changes - I’ve only ever used it on a DVD (RW / could update multiple times but never had to) but these days I’d prob try it on a usb stick before I spent money on an SSD for it.

I’d use cheaper media (ie mechanical drives) for local backups instead of SSDs. If you want to, and you should want to, keep several rotated backups, then there’s a cost multiplier there.

SSDs have other issues and performance can drop off noticeably over time - but this will depend on how you use them - something called TRIM which you can read about here and main reason I still use mechanical drives as I haven’t got around to understanding how that fits in with LUKS.

If you decide to use mechanical drives as working drives: for 3.5" drives 2TB Seagate Baracudas use SMR whereas 1TB use CMR. I think all the WDs are SMR these days.

If working drives are SSDs prob negligible but if using mechanical drives then multiple drives can allow you to avoid io bottlenecks eg. by placing your system on one and your VMs on another. Depending what you’re doing multiple SSDs might make sense too but of course you have to take other bottlenecks into account too.

I wouldn’t generally be backing up a full system to the cloud unless that’s where the system was. But you do want a backup/backups offsite.

13

Yes please do keep us updated . I’m not sure WHY you want to use AWS S3 cloud services – it is going take time to upload plus there is the issue of cost [Shrug] . I’d like to know where your day-to-day drive that contains the working OS is located – I will assume that would be /dev/sda. after that… not sure what /dev/sdb and dev/sdc are used for.

I’d like to know how often you do backups and the type – full disk, if they are “snapshots”, or they are some other type, and how often you do these these backups – daily, weekly, etc. If daily then Bobar’s suggestion is good of having 2-7 HDD that you “rotate” through. makes sense; that said that sounds like a Giant PITA. BackupNinja can be set up to do backups automatically.

I like your idea of using a 2TB SSD for your backup media as mechanical spinning rust media is a mechanical drive that is subject to failure, while SSD’s are memory. [SHRUG] My 2 TB HDD is a Segate Barracuda – I don’t trust WD as I’ve had bad luck with them.

In the short term do try Backupninja – it’s a simple download and really easy to set up.

Regardless do keep us updated as to the final plan you stumble upon.

D’ Cat

14

SSDs are essentially multiple USB sticks used in parallel and if you’ve ever had a usb stick go: it just goes - phht! no warning, all gone, dead, bricked, recovery is not an option… no intermittent scratching noise nada…though to be fair there are different pros and cons on reliability.

1 Like
15

Regarding external SSD vs external mechanical drives for backup, at the moment I’m leaning towards external SSDs – based on scar tissue from 5 or so years ago.

At that time – and I know of no reason why this should change – an external HD was a vanilla HD wrapped in a skin with a power supply and SATA power and data interfaces. I’m not sure any of them had fans. I know from empirical data that large (more than 1TB) external hard drives run significantly hotter than smaller capacity devices.

My experience (5-10 years ago) was that 3 of 5 external HD units bought for backup failed within the first few backup cycles, apparently because of excess heat within the enclosure. I found that the external enclosures literally cooked the drives in my 70-80 deg F operating environment.

I’ve had no such issues with external SSDs.

I already have 3 suitably-sized external SSDs in hand, so cost is not an issue. I plan to use one for clonezilla (mostly because it’s large enough to put an easily-read white label on it) and then alternate the others (system_backup_a and system_backup_b) on some regular (weekly or monthly) basis.

I plan to migrate the personal data I care about (/home/tms) to a separately mounted volume that I can backup using more conventional means.

I plan to use my system backup primarily for convenience so that I’m not so dependent on my process for tracking what packages and versions I’ve installed on this system. I understand that all of this can be recreated from external sources, and that is my ultimate backup strategy if this fails for some reason.

So my clonezilla backup is primarily a convenience. Once in place, I’ll resume my efforts to get RL 8.6 loaded and working.

16

I’ve seen tales that some SSD lose their content, when they are some months without power. Spinners do not have that vulnerability. Therefore, unplugging external drive and leaving it on shelf sounds safer when that media is HDD or tape, rather than a SSD.

17

I hear you.

As I mentioned upthread, my issue with using an external spinner was that it baked in its enclosure, so that the disk failed during the second or third backup pass.

I tried removing the disk from the enclosure and sliding it into a hot-swap bay, and the disk wouldn’t even spin up enough to load the heads.

I have two SSDs on the shelf at the moment, so I’ll defer new hardware until one of them fails (hopefully during backup rather than restore … :slight_smile: ).