Replace 5 smaller drives with 3 larger

[solidus28]

I currently have 12 drives in my array, 5 of which are 8TB drives.  I have dual 16TB parity drives already up and running.  I have 3 new 16TB drives that I want to use to replace the 5 8TB and leave an additional 8TB free.  What's the best way to go about this?

 

Do I need to add 1 16TB at a time and do something like this?  

Stop array
Add the 3 16TB
New config, assign the 3 new drives, unassign both parity, start array

rsync -avX --progress /mnt/disk1/ /mnt/disk2 /mnt/disk10 (disk1 and disk2 being 2 of the 8TB, disk 10 being a new 16TB)

let that process complete

rsync -avX --progress /mnt/disk3/ /mnt/disk4 /mnt/disk11 (disk3 and disk4 being 2 of the 8TB, disk 11 being a new 16TB)

let that process complete

rsync -avX --progress /mnt/disk5/ /mnt/disk12 (disk5 being the final 8TB, disk 12 being a new 16TB)

 

I realize this is not the "safe" approach, but that would require 3 parity rebuilds, right?  I do have a full backup.

Is there another way to fire off all 3 of those commands simultaneously or is that a bad idea?  Any other/better approach?

 

Thanks in advance!

 

[trurl]

Replace and rebuild the 3 disks with the most contents, one at a time. Or you can do 2 at once since you have dual parity.

Copy 2 remaining disks to those new ones.

New Config without those last 2 and rebuild parity.

 

Don't start this unless you are sure everything is working well as it is. Post diagnostics if you want someone to take a look.

[solidus28]

Thanks for the feedback!  Attaching diagnostics.  FYI, all 5 of the 8TBs are roughly 90% utilized right now.

 

If I'm following your guidance, would this be the suggested procedure:

 

Stop Array
Unassign 2 8TB drives
Install and assign 2 16TB drives
Start array and let rebuild.

Once complete, stop array again
Unassign a 3rd 8TB drive
Install and assing 1 16TB drive
Start array and let rebuild

Once complete
rsync -avX --progress /mnt/disk4/ /mnt/disk10 (disk4 being one of the final 2 of the 8TB, disk 10 being one of the previously upgraded 16TB)

rsync -avX --progress /mnt/disk5/ /mnt/disk11 (disk5 being the final of the 8TB, disk 11 being another of the previously upgraded 16TB)

Stop array, unassign last 2 8TB.  Start array and rebuild a final time.

unraidserver1-diagnostics-20220630-0931.zip

[trurl]

Your diagnostics shows more disks than just those mentioned. Since you have so many disks, I didn't look at SMART reports for them. Do any have SMART warnings on the Dashboard page?

 

Similar to what you posted, but no need to unassign anything, just replace or remove them. You need to New Config to get rid of those last 2, and you can reassign any disks however you want when you New Config if you want to close the gaps left by those last 2 removed. Be sure you don't assign any data disk to any parity slot when you New Config.

 

I assume you will power down to install new disks. Disable autostart in Disk Settings until finished.

 

Shutdown, install 2x16TB in place of 2x8TB, reboot

Assign 2x16TB in place of 2x8TB, start array and let rebuild complete

Shutdown, install 1x16TB in place of 1x8TB, reboot

Assign 1x16TB in place of 1x8TB, start array and let rebuild complete

Copy remaining 2x8TB to those new disks using rsync or whatever

Shutdown, remove remaining 2x8TB, reboot

New Config without those (now missing) disks, start array to rebuild parity.

 

 

[trurl]

Unrelated, but your system share has files on disk1

[trurl]

Double check all connections, including other disks. Bad connections is usually the reason for problems when replacing disks.

[solidus28]

3 minutes ago, trurl said:

Your diagnostics shows more disks than just those mentioned. Since you have so many disks, I didn't look at SMART reports for them. Do any have SMART warnings on the Dashboard page?

 

Similar to what you posted, but no need to unassign anything, just replace or remove them. You need to New Config to get rid of those last 2, and you can reassign any disks however you want when you New Config if you want to close the gaps left by those last 2 removed. Be sure you don't assign any data disk to any parity slot when you New Config.

 

I assume you will power down to install new disks. Disable autostart in Disk Settings until finished.

 

Shutdown, install 2x16TB in place of 2x8TB, reboot

Assign 2x16TB in place of 2x8TB, start array and let rebuild complete

Shutdown, install 1x16TB in place of 1x8TB, reboot

Assign 1x16TB in place of 1x8TB, start array and let rebuild complete

Copy remaining 2x8TB to those new disks using rsync or whatever

Shutdown, remove remaining 2x8TB, reboot

New Config without those (now missing) disks, start array to rebuild parity.

 

 

 

Is the "Errors" column where I'd see SMART errors?  Looks clean.

[trurl]

1 minute ago, solidus28 said:

Is the "Errors" column where I'd see SMART errors? 

No

5 minutes ago, trurl said:

Do any have SMART warnings on the Dashboard page?

 

[solidus28]

3 minutes ago, trurl said:

Unrelated, but your system share has files on disk1

Oh.  Is that something I need to look at changing/correcting?  Anything I need to worry about going through this process?  disk1 is one of the 8TBs that's going away.

[solidus28]

Just now, trurl said:

No

 

You did say "Dashboard", didn't you Looks good:

 

[trurl]

Typically you want appdata, domains, system shares all on fast pool (cache) with nothing on the array, so Docker/VM performance are not affected by slower parity writes, and so array disks can spin down since these files are always open. Those shares are all on cache except for some system files on disk1.

 

You can examine the files in system share on cache and disk1. Move them to cache, or if they are duplicates, decide which to keep and delete the other. Nothing can move or delete open files so you will have to disable Docker and VM Manager in Settings.

 

[solidus28]

7 minutes ago, trurl said:

Typically you want appdata, domains, system shares all on fast pool (cache) with nothing on the array, so Docker/VM performance are not affected by slower parity writes, and so array disks can spin down since these files are always open. Those shares are all on cache except for some system files on disk1.

 

You can examine the files in system share on cache and disk1. Move them to cache, or if they are duplicates, decide which to keep and delete the other. Nothing can move or delete open files so you will have to disable Docker and VM Manager in Settings.

 

Thanks.  It looks like disk1 contains docker.img and libvirt.img.  When I browse to system on the cache drive, while those directories both exist, they are both empty, no .img files.  Is that something I simply can mv to /mnt/cache/system?  I see Docker has the path /mnt/user/system, will that still see the file when I move it or do I need to update another setting somewhere with the updated location?

 

Thanks for your help.

[trurl]

2 minutes ago, solidus28 said:

disk1 contains docker.img and libvirt.img

Probably you enabled Docker and VM Manager before installing cache so they got created on the array.

 

3 minutes ago, solidus28 said:

mv to /mnt/cache/system?

Or you can get mover to move them since the share is cache-prefer, except

14 minutes ago, trurl said:

Nothing can move or delete open files so you will have to disable Docker and VM Manager in Settings.

 

4 minutes ago, solidus28 said:

path /mnt/user/system, will that still see the file when I move it

yes

[solidus28]

6 minutes ago, trurl said:

Probably you enabled Docker and VM Manager before installing cache so they got created on the array.

 

Or you can get mover to move them since the share is cache-prefer, except

 

yes

Thanks again!  Mover did the trick after stopping Docker and VM Manager.  No more system directory on disk1!

 

I'm running a Long Generic Test in SeaTools on my 16TBs as a pre-check before I throw them in the server.  Hoping to get started on the 8TB>16TB moves tomorrow!

 

Second part of this process is going to be using these 5 8TB (assuming they all test successfully after removing them) to replace eight 4TB drives in my backup server, so I'll follow a similar process over there with the caveat that I only have 1 parity drive over there, so will have to repeat the process 5 times, then copy the remaining 3 4TB drives before removing them.

 

Thanks for your help!  I feel more comfortable moving forward