New backup concept needed because of hasty purchase ;-)

[Sayuuk]

Hey guys,

 

so… I did a thing.

 

TL;DR

I have for the past couple of years employed a very simple, but transparent and robust backup system.

My unraid box had multiple HDDs in it and I used external HDDs of the same capacity and simply copied them from their disk shares using Beyond Compare - this gave me a visual tool to see what I was backing up, deleting, adding and it meant that just like the array disks, I could take a single backup drive and access all the files on it.

 

Last week I was able to pick up a couple of „large“ SSDs for cheap though and jumped for it, thinking I’d just put them in the array instead of the spinners. Transferred all the data to them and only went online before adding the parity SSD to make sure that wouldn’t kill it… and then I realised TRIM wasn’t a thing in the array and parity would indeed fill up the drive, potentially killing it quickly.

 

Well, the SSDs have been used now so I’m uncomfortable returning them just because of my stupidity, so they’re going to get used…


Right now, they’re in the array without parity, which with my 1:1 backup is not great, but okay…

They are mostly holding persistent data, so will not see a lot of writes from now on, I’ve made sure to put all that data on the cache where it can get trimmed.

 

The officially better option would be to put them in a pool.

 

This is where my question begins:

How do you backup a single large pool/disk to multiple smaller disks?

Is there a GUI solution that keeps track of which part of the pool is backed up to which drive, meaning I can visually keep an overview of my backed up data?

In the past I had 1:1 backups, meaning the target drive was always the same size as the source, I didn’t have to split shares or remember/check which part of a share went where.

Shares could grow over time and it would still work.

When I attached the backup drive I had instant access to all the files on it without attaching a whole backup array/raid/pool.

 

I‘m a bit stumped and could have researched this issue in advance - or stuck to HDDs - but I simply didn’t anticipate this.

 

I‘m looking forward to your ideas!

 

Cheers

Stefan

[TimTheSettler]

Can you elaborate on the following?

• How many HDDs did you have in the array before?  What size of drives were they?  Where are they now?
• How many HDDs do you have as your backup drives?  What size of drives are these?
• How many SSDs did you buy?  What size of drives are these?
• How many drives can you physically fit into the case?
• How many SATA ports do you have (how many drives can you connect to the system)?
• How much data are the hard drives holding right now (in total - across the board)?

Edited August 10, 2023 by TimTheSettler

[Sayuuk]

9 hours ago, TimTheSettler said:

Can you elaborate on the following?

• How many HDDs did you have in the array before?  What size of drives were they?  Where are they now?
  • I had a 4TB Parity, 2x4TB and 1x3TB data, all 3.5“
    • they are on a shelve in my office
      • the 3TB is slowly dying, one of the reasons „a“ change was needed
• How many HDDs do you have as your backup drives?  What size of drives are these?
  • 3x4TB external 2.5“ drives
• How many SSDs did you buy?  What size of drives are these?
  • 4x4TB SSDs
• How many drives can you physically fit into the case?
  • 6x3.5“ drives or „more“ 2.5“ drives with creative mounting
• How many SATA ports do you have (how many drives can you connect to the system)?
  • 4 on the board + 2 via an expansion card, I could add some more though
• How much data are the hard drives holding right now (in total - across the board)?
  • about 10TB, so the old drives where almost maxed, but I’m not adding a lot of data these days

Hi Tim,

 

I‘ve crammed my answers into the quote above.

I had a reason for not adding all the info initially, because I’m curious if there is a simple solution for backing up any kind of larger array/pool without immediately having to create a full backup site with another array/pool - no matter if on ZFS or indeed the same OS

 

A simple way right now would obviously be to get a 16TB+ drive and simply copy everything onto it, but then if I add a drive I’d have to get a 20TB drive, then a 24TB drive - each time chucking the old one, because I really don’t need spinners anymore in my use case…

I could off course also run the SSDs as a pool AND the HDDs as the array and rsync the data over periodically, but I’d quickly run out of physical space in my current PC case with my current config.

 

Or I just keep it as is, the SSDs in the array, with simple access to the disk shares for backups and not worrying about TRIM so much, because most of the data is persistent and I value independent disk access and backup more than the redundancy a Parity drive would give me…

 

Dtill hoping for more insight on other options

 

Cheers

Stefan

 

P.S.:

I used to have a backup unraid server based on a thin client with all my backup drives attached, but it would periodically drop the USB drives and was in general not as solid a solution as simply attaching the drives 1 by 1, also decreasing the risk of damaging all the drives/data at once.

I still have the license, but don’t have the spares right now to build a full box cheaply.

I also store my backup drives in a „waterproof“ case in the basement, therefore following the 3-2-1 principle roughly.

A full second server would be a big hassle (the basement is 2 stories from my apartment, so no way for me to hook it up to Ethernet)

[TimTheSettler]

I'm not aware of any special app that will do what you want but here's what I would do if I were you.

 

Option 1

• Build a new, cheap machine for about $500 and buy two 4TB drives.
• Each machine will have 1x4TB parity and 3x4TB data.
• One machine will be your main server and the other will be a backup server.  The main server will use Vorta to back up all the files from that main server to the backup server.  When the backup is complete store the whole computer in the basement.
  • Possibly encrypt the drives in case the stored computer might be stolen.
  • Possibly get a small case so that the backup server has a small footprint.  Put the smaller drives in there.  Easier to store.
• Use the SSD drives as backup drives or split them up and put two in each machine as cache drives.

Option 2

• Buy an 18TB drive.  These are currently at a nice price.
• Connect it to the server as an unassigned device and copy all the data to there or use it as an archive for Vorta.  One big array -> to one big HDD.

Option 3

• Use the SSDs as your backup.
• Copy everything from your server (the SSDs) to your current backup drives (the 2.5" HDDs) so that you have a copy there.
• Put the old 3.5" drives back into the array and use them like you did before.  Leave the 3TB drive out (the one that is dying).  This means you will have 1x4TB parity and 2x4TB data.
• Copy all data from one of the backup drives (2.5") to the server.  Add that 2.5" drive (that you just copied) to the server.  You would now have 1x4TB parity and 3x4TB data.
  • Stop here or do the same with another 4TB drive.  Add it as a data drive or parity.  You would now have 1x4TB parity and 4x4TB data or 2x4TB parity and 3x4TB data.
  • Stop here or do the same with another 4TB drive.  Add it as a data drive or parity.  You would now have 1x4TB parity and 5x4TB data or 2x4TB parity and 4x4TB data.
• Copy the data onto the SSD drives in logical chunks (movies, pictures, etc.) so that they each grouping fits onto a drive.  This is what I used to do.