[6.3.3] Parity & Data Drive and Filesystem Upgrade Help

[AnotherITGuy]

Hi all,

 

I'm a longtime happy unRAID free version user (since v4.x I think) with a growing family and growing home data backup needs, and I decided it was time to purchase a Basic license and upgrade the aging disks in my array while adding some cache drives.  I have already upgraded my array to unRAID 6.3 stable, and replaced an old USB flash drive with a more current unit. 

 

I have studied multiple wiki articles and forum posts but I'm still not sure what the best path is for my multi-stage upgrade in order to minimize downtime, maximize speed of data transfer, and preserve data integrity.  Let me start with this:

 

Goals:

1) replace all data and parity drives in array with new, larger drives

2) use XFS for all new data drives instead of ReiserFS

3) add one additional data drive

4) add a pool of two cache SSD's

 

Use Case: Home file server supporting 6 users data backup needs (via rsync/syncthing) and media streaming to Kodi frontends on home 1 Gigabit wired network

 

Existing Configuration:

Software: unRAID OS v 6.3.3 stable with these apps installed: CA Preclear Disks, CA Unassigned Devices, unBALANCE

CPU Hardware: 6-bay, 6-SATA port 2U rack server with Quad-Core AMD Opteron 1385 on SuperMicro H8SMi with 8GB DDR2 ECC RAM (Passmark=3442)

Drives:

2TB Parity Drive

2TB Data Drive (Data 1 - ReiserFS)

2TB Data Drive (Data 2 - ReiserFS)

 

Desired New Configuration:

CPU Hardware/Software: no change

Drives:

6TB Parity Drive

6TB Data Drive (Data 1 - XFS)

6TB Data Drive (Data 2 - XFS)

6TB Data Drive (Data 3 - XFS)

240GB Cache SSD Pool Drive 1

240GB Cache SSD Pool Drive 2

 

Steps taken: I have precleared all new data drives (WD Red 6TB) and all passed with flying colors. 

 

Tentative plan: I had thought to format these drives as XFS, copy data on old Drive 1 to new Drive 1, copy data on old Drive 2 to new Drive 2, remove all old drives, insert all new drives, and rebuild parity.  This would have ensured that if anything went wrong, I still had three old drives with all data and parity intact.  Once new config is up and running with valid parity, I would repurpose old drives in other family computer systems.  But I can't seem to figure out how to do this in a way UnRAID allows given parity disk changes and filesystem changes.

 

Can someone please advise on best procedure here?

 

Many thanks!

 

 

[trurl]

3 hours ago, AnotherITGuy said:

CA Preclear Disks, CA Unassigned Devices

These are plugins which are installable by Community Applications, just as many things are, but in general only those plugins specifically labelled CA are part of the Community Applications suite. Various other plugins and dockers have various other authors and support threads. The author of Community Applications does not officially support these others (though he is very helpful in general).

 

3 hours ago, AnotherITGuy said:

Tentative plan: I had thought to format these drives as XFS, copy data on old Drive 1 to new Drive 1, copy data on old Drive 2 to new Drive 2, remove all old drives, insert all new drives, and rebuild parity.

This won't work as stated since you need to let unRAID format the new drives in the array. So inserting new drives after copying then removing the old drives will not work. You could:

 

replace parity with larger disk, resync

add new data drives, format as XFS

copy old drives contents to new drives

remove old drives

New Config with only new drives

resync parity.

 

Of course, this would require having enough bays/ports to have old and new drives installed at the same time.

 

Do you have backups of any irreplaceable or important files?

 

Assuming you do have backups, I think I would just start with a new array with only new disks formatted XFS, and then mount the old disks one at a time using Unassigned Devices, and copy their contents to the new array. If you wait until after all the copying was done to install parity it would make the writes faster.

 

[AnotherITGuy]

2 minutes ago, trurl said:

Do you have backups of any irreplaceable or important files?

 

Assuming you do have backups, I think I would just start with a new array with only new disks formatted XFS, and then mount the old disks one at a time using Unassigned Devices, and copy their contents to the new array. If you wait until after all the copying was done to install parity it would make the writes faster.

 

 

Thanks for your response trurl!  I do indeed have backups of essential files, and I very much like this strategy for simplicity and speed.  My current array has User Shares though, and I was hoping to preserve those.  Would I be able to preserve those using this method?

 

 

[trurl]

6 minutes ago, AnotherITGuy said:

 

Thanks for your response trurl!  I do indeed have backups of essential files, and I very much like this strategy for simplicity and speed.  My current array has User Shares though, and I was hoping to preserve those.  Would I be able to preserve those using this method?

User shares are simply the aggregate of top level folders on cache and array with the same name. So if you have a share named "media", you will have top level folder(s) named "media" on cache or array disk(s). This is the way user shares has always worked as long as I've been using unRAID (since 4.7).

 

If you create a user share, unRAID creates top level folder(s) on cache or array disks (named for the share) as needed according to that user shares settings.

 

Conversely, if you already have or create top level folder(s) on cache or array disks, they are automatically user shares.

 

If you don't make settings for a user share it will have default settings.

 

So, if you copy the top level folders from the old disks to the new disks, then you will have those user shares. And the settings for those user shares are already stored on flash in config/shares, so you can even keep their settings if they are appropriate to your new storage configuration.

[AnotherITGuy]

Fantastic, thanks again.

 

Since I only have 6 ports/bays and my new configuration as described above would have 6 devices including the two new cache drives, I'm unsure of which drives should remain uninstalled temporarily so that the old data drives can be mounted using Unassigned Devices.  I suppose leaving the cache drives out for now makes sense?

 

And so, does this look like the right procedure based on above:

 

1) Stop old array, power down

2) Remove old parity drive, replace with new larger parity drive (1 device)

3) Install 3 new data drives and retain 2 existing data drives (5 devices)

4) Start unRAID system

5) Go to Tools -> New Config and reset the array (Retain Current Config set to None?)

6) Assign new parity and 3 new data drives to array slots (leave old data drives unassigned)

7) Start new array, mount old data drives 1 and 2

8- Use (unBALANCE?/rsync via command line?) to copy data from old data drive 1 to new data drive 1

9) Use (unBALANCE?/rsync via command line?) to copy data from old data drive 2 to new data drive 2

10) Stop array, power down

11) Remove old data drives, install cache drive SSD's x 2

12) Start unRAID system, start array, verify proper settings

13) Sync parity

14) Enjoy a beverage

 

Does this look right?

 

Many thanks again.

[trurl]

Been a while since I used unBalance, I don't know if it supports Unassigned Devices or not.

 

In addition to rsync, another possibility would be to use mc (Midnight Commander).

 

I usually use mc for everday assorted file management, and rsync for scripting things I will reuse.

[AnotherITGuy]

Got it.  Looks like unBalance does not yet support Unassigned Devices.  So, I'm using Midnight Commander in a local session to make sure things go smoothly, and am copying top-level folders from the old drive mounted via Unassigned devices as /mnt/disks/[NameOfOldDisk] to /mnt/disk1 which is a new drive in the array.  Will repeat with second old drive to another new drive, as suggested, and will then remove old drives permanently, install cache drives, start the array, and build parity.

 

Thanks again!