Hi all,

 

I've had unRAID for a couple years now and have been running with a relatively small amount of storage. 2x 3TB and 4x 4TB drives, 1 parity, 4 data, 1 "cache" pool (I stick my docker containers and vms here, and make backups to the array). I've been using btrfs for my data drives. It's been running on an old ASUS ROG Maximus V Formula with Intel i7 3770s and 16gb of DDR3. I upgraded my license to Plus during the Black Friday sale.

What started as a "I'm going to get an LSI HBA instead of continuing to use the onboard Intel and asmedia SATA ports" turned into a Dell PER730 8LFF with 2x E5-2680v4, 128GB ECC RDIMM (only 2133mhz for now), 5x 10TB HGST SAS3 2017 drives (homelabsales, $55 each and free shipping if purchasing 5 or more). Couldn't pass that up. I did get an LSI 9300-8i and 2x  1x SFF-9643 to 4x SFF-8482 SAS3 cables (the new drives are 12gb/s). But I didn't realize this R730 came with a Dell PERC H730. The LSI card has the SAS3008 chipset, the PERC has the SAS3108 chipset. I'll post a separate question about that.

 

So now I have my old unRAID server and the R730 is ready to put in service with the 5x 10TB drives installed. I recently screwed up my current array. I corrupted parity while a drive was missing. One of my drives went out and I got a replacement that started kicking back read/write errors about 16gb into the drive. Yes I should have run the full preclear, but I only partly ran it (aborted it after an hour as I was anxious to get my array back online). So aborted the rebuild and had to exchange the drive. I thought the drive I was replacing was 4TB but it was one of my 3TB drives. I replaced it with a 4TB. Since disk 4 in my array had nothing on it (it's a 3TB, as is my pool drive), I started up in safe mode, wiped the MBR/GPT of disk 4, assigned it to disk 3, only to realize that by trying to rebuild with the 4TB faulty drive, I could now only replace it with a 4TB drive. My dumb mistake (don't tell anyone that I work and have worked in high profile IT/infosec for 20something years) was clearing disk 4, as I now had 2 disks out of the array. I reassigned the 3TB to disk 4 and then put the replacement 4TB into disk 3. I tried a rebuild anyway, but most of the data (was mostly only urbackup files from my desktop which I can easily remake) was unreadable and I started getting all kinds of errors. Even trying to delete the files. Everything is fine with my disk 1 and disk 2, which is where almost all my data is stored. I was able to retrieve almost all the data from disk 3 by mounting it with option ro,rescue=ignoredatacsums. That's after I couldn't get anywhere with btrfs scrub and btrfs check. I hadn't bothered to create any checksums with the file integrity plugin yet. Anyway I'm not really worrying about that anymore at this point.

 

I want to build a new array from scratch on my new system. Since I have 5 identical drives with a sufficient amount of storage, I'm thinking I might try the zfs option. I've always liked btrfs but my setup is somewhat ideal for running zfs if I wanted.

 

My plan is to, once the new array is set up, copy all my data from the old array to the new one, then decommission that array, clear the drives, and use them as pool drives alongside the new array. I'm considering possibly putting the 4x 4TB drives into RAID 10 and pooling that, since I have a proper hardware raid solution. For pool drives, would this be acceptable? Or even two RAID 1s, and maybe RAID 0 on the 2x 3TB drives, still using those for my VMs and containers. I'm open to suggestions on how to use those 6 drives for pools.

 

My old unRAID machine only has 1 1gbE NIC. I have a 2x 1gbE NIC, I could possibly set up LAGG on my switch for the new and the old server, to help migrate my data faster.

 

Getting to the meat of my question: my license is assigned to my old server. No unRAID boot drive yet for my new one. Would the proper way to do this be to start a new trial with my new server, do the migration, then transfer my license to the new server? Part of the problem is that I want to use that same USB drive to boot with on my new server. It's a 128gb SSD Flash drive. I'm not 100% how I would do this. Use another USB stick to set up the new system, migrate the files, wipe the old flash drive, back up the flash drive on the new server, then restore it to the SSD Flash drive attached to the new system? Or can I just transfer it directly by having both flash drives attached to the new server? I'll fdisk the SSD flash drive first and just clean and create a new partition table first. Will a trial license be sufficient for the day or two I spend migrating my data?

 

thanks in advance.

11 hours ago, technotic said:

I'm considering possibly putting the 4x 4TB drives into RAID 10 and pooling that, since I have a proper hardware raid solution. For pool drives, would this be acceptable?

You can but would recommend using Unraid pools instead of the hardware RAID, you can create a zfs mirror for example.

 

11 hours ago, technotic said:

My old unRAID machine only has 1 1gbE NIC. I have a 2x 1gbE NIC, I could possibly set up LAGG on my switch for the new and the old server, to help migrate my data faster.

Note that a LAGG won't help with a single transfer, only if you plan to make simultaneous transfers.

 

11 hours ago, technotic said:

Getting to the meat of my question:

 

You can create a new trial flash drive for the new server, once the data is copied, backup /config from old flash drive and recreate it, then copy the complete /config folder from the trial flash driver except for the key and finally restore the old key.

 

 

P.S.:

11 hours ago, technotic said:

I want to build a new array from scratch on my new system. Since I have 5 identical drives with a sufficient amount of storage, I'm thinking I might try the zfs option. I've always liked btrfs but my setup is somewhat ideal for running zfs if I wanted.

If you mean zfs on the array note that currently there's a bug affecting write speed, I hope it will be fixed soon, but something to be aware of, this bug does not affect zfs pools, only zfs on the array.

On 12/24/2023 at 1:38 AM, JorgeB said:

You can but would recommend using Unraid pools instead of the hardware RAID, you can create a zfs mirror for example.

I will go this route then, thanks. I'm just not sure whether to trust the ZFS implementation right now. You did mention it's working fine with pools tho.

 

On 12/24/2023 at 1:38 AM, JorgeB said:

If you mean zfs on the array note that currently there's a bug affecting write speed, I hope it will be fixed soon, but something to be aware of, this bug does not affect zfs pools, only zfs on the array.

 

I happen to like btrfs and have for a long time, so I guess I should go ahead and use this for my array? 

 

On 12/24/2023 at 1:38 AM, JorgeB said:

You can create a new trial flash drive for the new server, once the data is copied, backup /config from old flash drive and recreate it, then copy the complete /config folder from the trial flash driver except for the key and finally restore the old key.

 

Will do this, thanks. I'm actually wanting to install unRAID onto the internal vFlash that is preloaded with esxi. I decommissioned the vFlash and just got done upgrading my iDRAC8 to the latest 2.85.85.85 (which was a pain, because I had to do an incremental upgrade in the middle, and had to boot from the support live image to do that.

 

The only problem is that I need to ISO image it to load it onto vFlash. I'll probably just create an ISO from the USB drive, since that should set it up correctly. Anyway thanks for all your help.

2 hours ago, technotic said:

I'm just not sure whether to trust the ZFS implementation right now.

There's nothing special about zfs in Unraid when used with pools, it's the same implantation as any other OS.

 

2 hours ago, technotic said:

I happen to like btrfs and have for a long time, so I guess I should go ahead and use this for my array? 

Valid option, it's what I use myself.

 

 

 

Thanks again @JorgeB! What I ended up doing is since my parity was already trashed on the old array/old server, i created a new flash config. i dd'd it over to my 16gb IDSDM module slot 1, which is Dell's Internal Dual SD Module. seems that as soon as i did that, my trial key showed blacklisted. I'm guessing the dell SD card doesn't use unique CID. i considered buying an SD card with an editable CID, but i'll consider that later.

 

*Note: one cool feature of the IDSDM is that you can use mirror mode and choose the mirror target, so i could use 2 editable CID SD cards, one to boot from and mirror to the other. during POST, the system compares the selected primary slot to the mirror. if the primary slot is different, it assumes the primary slot has been changed or is corrupted, and boots slot 2 instead. basically would give me a hot spare boot config. might require minimal interaction on my part to update the CID of slot 2 to the CID from slot 1, but its a fully offline operation and would always have my current config without having to schedule local backups. 

 

Anyway, I went ahead and made a backup of my old flash config and my new flash config, for safety. i stuck both drives in my laptop, copied my Plus.key file to my desktop, reformatted the old flash drive's partition, copied everything from the new flash drive to the empty fat32 partition on my old flash drive, deleted the Trial.key, pasted my Plus.key, re-ran the make-bootable.bat for good measure, stuck the flash flash drive in my new server's internal USB slot, and booted up. License was recognized valid and i went to work on the data transfer.

 

i used the unassigned devices plugins and i think? the dynamix file browser plugin (or maybe it was advanced copy) and formatted and mounted one of the 10TB drives to btrfs. then i put my 3 data drives from the old array into caddies and loaded them into the bays. since i got those 5 10TB drives for my new array, and my chassis has 8 3.5" bays in the front, this worked out perfect. for now. I formatted one of the 10TB drives with btrfs and mounted it under unassigned devices. then i mounted the 3 drives from my old array, did some storage cleanup (deleted a couple TB of urbackup images and such. with the plugin installed i was able to browse each drive and select directories. one at a time, i selected everything i wanted to keep and told it to copy those files to the mounted btrfs partition on the 10TB. had some issues initially but finally got it sorted and the minimal 1.3TB of data was happily on one of my new array data drives.

 

i unmounted the old array drives and used fdisk to delete the partition off each one. created a new pool of 3 disks. set it up to use zfs raidz with compression on, autotrim off. i started up the array with only the pool populated, then formatted the pool. a few seconds later i had a happy 8TB usable space 3-drive raidz pool up and running. stopped the array again to finish up.

 

then i created a new config, preserving the pool. configured the same settings again on the pool, stuck one of the 10TB drives on parity, slotted the 10TB i now had all my data on, formatted the remaining 3 10TB drives with btrfs, and added them to the data slots on the array. with everything looking good, i started the array up and everything seems good! its currently building the parity on the array which should only take about 11 hours total i think.

 

QUESTION: once I get the parity build finished, should I open up the file integrity plugin, create a valid checksum and set up a schedule for it to create regular checksums? it would be an added redundancy for the array data drives, right? how often would you suggest i set the schedule to build a checksum table? how frequently should i export it? i've never used this particular feature. is it unreasonable to even use this? i don't want to keep a massive checksum table on my array, and it would sorta defeat the purpose anyway.

 

planning to schedule array scrubbing once a month, maybe weekly for the zfs pool. i'm open to input and suggestions.

 

the pool is going to be used for all my libvirt/vm files, docker containers and data volumes, and just things that'll be getting lots of writes and load up array with my read-focused data. i intend to redeploy urbackup server for keeping my desktop and laptop backed up regularly. i like setting up weekly full image backups of my windows c partitions. i'm dozing off while writing this so i'll post this as is.