(SOLVED) - 3 tier unRAID...or other ideas welcome

[Cull2ArcaHeresy]

I have a r720XD running unRAID (2x12tb parity, 5x12tb + 3x6tb  array, 3 cache, 1 unassigned). A chunk of the storage is "vault/archive" type files/folders/projects, but also the stacks of external harddrives. I bought a 4246 with 24 x 4tb drives (and the cable and a 9202-16e) which should be delivered friday. When including everything shipping/tax/warranties/cable/lsi/das/drives, total was ~$2k but will make future upgrades easier (and still cheaper $/tb than 12tb exos). After preclearing the drives, I could add most of them to my array, but hoping to do better.

 

You know how you have cache-->array, I'm trying to concoct some way to do (cache-->array)-->vault/archive. So when I search for a file or [plenty of other examples] I just go into the share it would be in and it will show up functionally speaking. I get that unRAID does not have 3 layers. Thinking give the 24 x 4tb drives to a VM running an actual RAID config (figure out number later) and have it mounted on the unRAID side. This would require some custom "mover" script to create the links for files to show up on the share when they are in the VM array (and move them to the VM array). The more I think about this method the more it seems like it will crash and burn hard. Maybe the solution is to run a basic unRAID server using the max number of drives and just have the other 4 (or 2?) drives ready to hot swap in (they are used sas drives). I am open to other ideas too, worst case I have main storage and vault/archive storage (still run from the VM), and they are separate network entities.

Edited June 22, 2020 by Cull2ArcaHeresy
marking as solved

[sota]

Here's my question:

What the purpose of having a vault layer?

[Cull2ArcaHeresy]

3 hours ago, sota said:

Here's my question:

What the purpose of having a vault layer?

I'm adding 24 more bays, when adding the 8 in array and 2 parity would be 34. The new drives also have a (presumably) higher failure rate, seller told me to expect 1 a year to fail if running 24/7. With the higher failures, the disk rebuilds will put more stress on the original 8+2 drives. Vault layer would use all 24 bays and not have 4 free (think about if using a 60+ bay). Vault layer would also have the more likely to fail drives add extra stress only to the same set of drives for rebuilds. This is assuming that a rebuild is more stressful than the monthly parity check.

 

For ease of use it would make archiving a seamless integration similar to not knowing if a file is on cache or array when you access it (I know you can find out). Future version would have the archive section made of slower larger disks. Haven't messed with this yet, but guessing this is not compatible with unRAID https://github.com/45Drives/autotier

 

Maybe I chased this idea too far down the rabbit hole and the right move is to add 20 drives (28array+2parity) and then have 5 hot spares, in that config would go from 78tb + 2tb cache to 158tb + 2tb cache so still doubling space. Concerned about the rebuild stress still, but maybe that is me being paranoid. If just adding 20 to array, thinking would probably only add them in 2/3/4/5 at a time (as space is needed) to reduce power on hours thus reducing drive failure rate...but every time I add drives the parity would need to be rebuilt.

[sota]

Let me start with the last bit.

Parity does NOT need to be rebuilt when you add a drive; new drive are zero'ed out completely when added, so their addition does not affect parity.  Only when you start adding data to the new drive does it alter parity, and that's a normal process.  In terms of rebuild stress, add as many as you can in one shot to minimize that.

 

Keep in mind that power on hours are not necessarily the same as active spin time hours.  My array is approaching 5500 hours of power on time, but actual spin/data access time is I'd guesstimate 10% or less of that.  My array drives spend the majority of the day asleep, and I do the majority of my work on either the cache or dedicated Unassigned Devices.

 

If you're concerned about data loss, nothing will beat a cold storage backup.  I have a link to my process in my sig.

 

One thing I'd love to see is, unRAID copying large files being accessed, off the array and to the cache temporarily.  Example: we're watching the 2003 Battlestar Gallactica series right now, but only 1 show a night or so.  It'd be nice to have the entire series copied to the SSD cache, so the array isn't even touched afterwards.  Note I said copied, not moved.  My array is very much treated as giant WORM drive.

Edited June 17, 2020 by sota

[Cull2ArcaHeresy]

Not sure where I got that from then.

 

Guessing you are talking about "9 Power on hours" vs "240 Head flying hours" from disk attributes? Checking a few disks, parity ones having the highest numbers have a "240 Head flying hours" of ~22.5% of "9 Power on hours".

 

I'd love to have actual backups, but not there yet $. I'm sitting at 3.12tb free of 78tb (plus 1.10tb free of 2tb cache). Adding in other devices backups (need to do still) we can round that to a full 80. I want to move my 3 x 8tb and 1 x 6tb external drives onto the array, data then probably shuck too. I definitely ordered too many of those 4tb sas drives being I forgot the 4 external ones would be added (and the whole drive limit starting this 3 tier idea)      But that would put me at ~110tb of data (+ stacks of full 250g drives that need to be emptied), so the 100tb worth of disks ordered wouldn't even cover that ~110tb. Eventually I plan to get another server (low powered since just a backup) full of largee/slow/cheap drives, create a backup locally, then get it in a datacenter somewhere doing continuous backup (and the whole 2 locations backup rule). But also that would be more of a hot backup (pending upload speeds of ~9gigs/hr) vs your cold backup, altho wouldn't snapshot/timemachine style backup be the best of both (just maybe slower restore speeds)?

 

What do you mean WORM drive? If really write once read many, wouldn't your server be a "cold backup" categorically and would explain low active/read/write time (maybe 10% is normal, but my 22.5% seems low to me as well).

 

And yea, I've been contemplating making an intelligent plex plugin caching thing for unRAID to keep the next unwatched episodes in cache for the most recently watched X shows. Ideal it would watch your watch rate for each show and keep the number of episodes you watch a day cached. Either where X is set or it does as many shows as it can before it reaches whatever storage limit you set (or a combination). Reads from plex db and will cache the next episode (Z) as you are watching episode Y of some show (or maybe trak.tv api instead of plex db). For your case you could always add another plex library pointing at a cache only share and copy the series there manually. I have a second TV lib for 1 episode in each of 2 shows that plex wouldn't grab, I should investigate that again and delete the lib if fixed. Thinking about your use case it would make sense to add a selector to the plugin where you could tell it to keep a selection in cache. I haven't looked into how to make the mover move a file to cache, but thinking maybe touch -ac would do it. Does mover actually move files to cache? I thought it copied them to it and then if nothing was changed it deleted off cache and if something was changed it moved it back overwriting the file on array. Rereading the "Use cache" text, I had forgot that mover does not do any kind of intelligent thing like moving what you're working with to cache...so the plugin would have to move the file from array to cache, and keep an active read on the file so mover does not take the files away since they are in use. This might be a little more difficult. My use is because with many shows in plex, when I want to go back or forward more than 30 seconds it gets stuck at loading and I have to close and reopen it. I'm assuming it has something to do with reading from disk so cache would fix it.

 

If I am thinking this thru right, my correct (and simple) move steps would be to (after getting the hardware setup):

1. insert 16 drives and preclear them
2. add the 16 to array
3. move data off the 4 external drives
4. shuck drives and add insert them
5. preclear 4 shucked
6. add 4 shucked to array
7. in turns (only 4 bays open) preclear the other 9 4tb sas drives so they are ready to be used when a drive fails (almost sounds morbid 😛 )

[sota]

apparently either I don't have access to, or my drives don't report, attribute 240.  that's annoying.

 

[Cull2ArcaHeresy]

On 6/18/2020 at 6:15 AM, sota said:

apparently either I don't have access to, or my drives don't report, attribute 240.  that's annoying.

 

My drives are exos, plus a few barracuda. After getting my docker/VM issue fixed, I'll see what the hitachi drives ordered with DAS say (before and after preclearing). Thought is maybe consumer (or older?) drives do not have all the same metrics as enterprise ones. All current drives are SATA, the new ones are SAS, if that would make a diff.

[Cull2ArcaHeresy]

On 6/18/2020 at 6:15 AM, sota said:

apparently either I don't have access to, or my drives don't report, attribute 240.  that's annoying.

 

All my sata seagate drives (bare and shucked) have that attribute and large list of others, the sas hitachi drives have few attributes available and 240 aint one. Wonder if its a seagate only thing.

[sota]

none of my drives, WD/HGST, WD, Samsung, report anything above 200 for an attribute, and most only go to 199.