shrink array follow up questions

[wildwolf]

Good morning. The past couple days I followed the shrink array docs here https://wiki.unraid.net/Shrink_array (clear drive then remove drive method), to use unbalance to move contents from 1 drive (disk 7 of 11 with 2 parity drives) and scatter it to a few of the other drives. I have a lot of array restructuring* I want to do, and this was a 1st attempt to see if I have the process down. I think I was mostly successful, with some user/novice errors, and I'd like to question what to do to see if I can make the next iterations better. * = I have 2 parity drives & 9 data drives and I would like to reduce my 9 data drives to 6 so I can utilize only 1 SAS card instead of the 2 I'm currently using. I had (in order of Parity --> Data1-9), 

p1 8TB

p2 8

d1 5

d2 5

d3 5

d4 5

d5 5

d6 5

d7 5

d8 8

d9 8.

I removed disk #7, as it was the least full. My misunderstanding/problems (self-inflicted) started after I completed step 16. I know it was recommended to do a parity check. I started one, and decided instead I wanted to remove the drive, so I stopped array, powered down, and removed it. After it started back up, the configuration showed my drive 7 as unassigned - should I have re-ordered assignments when I did Tools then New Config? I started to do a parity check then, but noticed it was stating that drive 7 was emulated. Didn't think that was what I needed to see - since I didn't want that drive in the array at all? I also noticed I hadn't yet changed all my shares to use "all drives" as the instructions had asked to remove the 'removal drive' from all shares on step 1. 

So, in a panic, I selected Tools then New Config again, reset the drive assignments back to how they were (no #7 after removal), and forgot to check parity is valid, so now my parity is rebuilding for p1 & p2 drives. I'm assuming (since all my data still appears intact, that I should be fine, I'll just have disks d1 - d6, then d8 & d9 in my array when complete.

I've got a few more drives to do this with, and then upgrade to some larger drives (lots of WAIT time coming up again).
I guess my questions are now:
1. When finishing step 14 & 15, do I reassign the drives to the new slots then?
2. To get the array using sequential # slots again, after this parity rebuild is complete, can I just stop array and rearrange assignments to ignore the #7 no drive slot?

3. When in the process would I modify all the shares again to use any missing drive I removed from shares?

4. Step 4 of instructions have us turn on reconstruct write, when do we change that back (and I didn't capture what it was, like an idiot, so what do we change it back to?

 

I have 4 new 14TB drives I plan to incorporate into this mix. I will eventually be replacing p1 & p2 with 14TB drives, re-using those old parity drives as new data drives replacing 2 of the remaining 4 5TB drives, and repeating the shrink array process by removing the remaining 3x 5TB drives. The array, when I'm done (hours, and hours, and HOURS of data moves later...), should like like:
p1 14TB
p2 14TB

d1 14TB

d2 14TB

d3 8TB

d4 8TB

d5 8TB

d6 8TB
(eventually shrinking from 9 data drives to 6, but increasing array size from 51TB to 60TB).

I've got about 11 hours before the parity rebuild is complete. I'm sure I will take the slow road to get there, but as long as I keep the data intact (and hopefully with less errors like where I am currently rebuilding parity), I'll be fine with the lost time.

 

Edited November 7, 2021 by wildwolf
Added 4th question on reconstruct write.

[JonathanM]

Why not just build parity to the 2 new 14TB with all your current data drives,

then rebuild data1 and data2 with the other 14TB drives,

rebuild data3 and data4 with the old 8TB parity drives,

and all that would remain would be copying the content of the remaining 3 5TB drives into the free space of the 14TB drives,

set a new config and rebuild parity in the final layout?

 

Seems like much less work than what you currently have laid out, and all the removed drives would still have copies of the data for backup instead of wiping it out.

[trurl]

Also note that you have to rebuild parity2 anyway if you change the drive slots of any data disks you are keeping, whether to reorder or just close gaps. I usually just recommend forgetting about the clear then remove method.

 

As suggested, rebuild as much as possible and then remove all at once with parity rebuild.

[wildwolf]

49 minutes ago, JonathanM said:

Why not just build parity to the 2 new 14TB with all your current data drives,

then rebuild data1 and data2 with the other 14TB drives,

rebuild data3 and data4 with the old 8TB parity drives,

and all that would remain would be copying the content of the remaining 3 5TB drives into the free space of the 14TB drives,

set a new config and rebuild parity in the final layout?

 

Seems like much less work than what you currently have laid out, and all the removed drives would still have copies of the data for backup instead of wiping it out.

So, the recommended path is what?
1. Just pull both 8TB parity drives, stick in 2x 14TB drives label as parity & rebuild parity (do I need to do anything in tools new/config to do so?)
2. Once parity rebuilt, replace 2 data drives with 2 x 14TB drives, rebuild from parity? Should I do them 1 at a time? What sequence/steps should be followed to ensure no data loss?
3. Once both are done, repeat using the 2x 8TB to replace 2 5TB.
4. Move data from the final 3 to the other drives.
5. Once done, set new config again and rebuild parity 1 last time with the down to 8 drives?

I guess I'm not familiar/am unsure about how to just pull a drive & drop in a bigger one & still maintain all the data. Let me go find some more documentation.
Thanks for suggestions.

I'd still be interested in answers to my original 1 - 4 questions to see when/where in the process steps are changed/corrected back to 'normal' process.

[trurl]

1. Replacing parity disks is just like any other replacement so New Config isn't needed as long as you don't make any other assignment changes
2. Since you have dual parity you can replace 2 data disks at once. Might be a little safer to do one at a time, but as long as you are confident in the health of all disks, and you are careful with connections, should be OK to do 2 at a time

3,4,5 OK

 

40 minutes ago, wildwolf said:

pull a drive & drop in a bigger one & still maintain all the data.

Replacing disks is the whole point of parity. Parity doesn't know or care about your files, it is all just bits. Rebuild writes every bit of the entire disk so it is in sync with parity. The result is all of the bits that were on the original disk are duplicated on the replacement disk, and the rest of the replacement disk is also made in sync with parity (really just zeros to get in sync after you get beyond the original disks capacity). And none of that extra capacity is part of the filesystem yet so the result is exactly what you had before except more free space.

[wildwolf]

Thanks again, Jonathan & trurl. This has indeed speed up the process. 
I can't seem to find the "yes I'm sure" checkbox anywhere, but so far, everything seems to be working smoothly.

[JonathanM]

43 minutes ago, wildwolf said:

I can't seem to find the "yes I'm sure" checkbox anywhere

And I can't seem to find where that was mentioned in this thread. What are you referring to?

[wildwolf]

Thanks again, Jonathan & trurl. This has indeed speed up the process. 
I can't seem to find the "yes I'm sure" checkbox anywhere, but so far, everything seems to be working smoothly.

 

Sorry - that's in the "Replacing a Data Drive" unRAID wiki: https://wiki.unraid.net/Replacing_a_Data_Drive

I have noticed a small discrepancy in some (very little, a key word or two) of the text in the wikis, but made my way through it.

I am almost done, and I'm sure someone more experienced might have done all this faster. I do have more questions, though.
Currently, I'm sitting here:
P1 14TB
P2 14TB
D1 14TB
D2 14TB
D3 8TB
D4 8TB
D5 (removed)
D6 (removed)
D7 (removed)
D8 8TB
D9 8TB

I have 2x SAS9211 cards. 
Card 1 has 2x SFF8087-4SATA cables. I can't look right now, but I believe:
1st cable has 4 drives (1 of which is parity)
1nd cable has 2 drives 

Card 2 has 1x SFF8087-4SATA cable.
1st (only) cable has 1 drive (I think this is the other parity drive).

I know for a fact that I can trace my cables to drives, look at my serial numbers, and determine which 2 drives are the 2x 14TB (and which is #1 and which is #2).

I'd like to remove card 2 from the system. Hook up all 8 drives to card #1.
I think (someone tell me if I'm wrong?) it would be smart to have
1st cable - probably 1st drive parity, other 3 data drives
2nd cable - probably 1st drive parity, other 3 data drives

Is there a safe/easy way to do this? I understand I'll have to rebuild parity drive #2 from this thread: https://forums.unraid.net/topic/54221-reorder-disks/

Is it as simple as disconnecting everything, removing my 2nd card, attaching drives as I've indicated to the 8 slots that are left (in the order I choose), identify both the parity drives, and the other 6 as the data drives, and start array?
 

 

 

Edited November 19, 2021 by wildwolf

[JonathanM]

Physically rearranging disks doesn't require any configuration changes or rebuilding of parity. Unraid tracks disks by serial number, it doesn't care which cable they connect to.

 

Rearranging logical slots in the management GUI is what you are referencing in the reorder disks thread.

 

Totally different and unrelated concepts in Unraid.

 

[trurl]

10 hours ago, wildwolf said:

I'll have to rebuild parity drive #2

You mean because you want to "close the gap" left from removing 5,6,7?

[wildwolf]

2 hours ago, trurl said:

You mean because you want to "close the gap" left from removing 5,6,7?

Correct, and I want to reduce from using 2 SAS cards down to 1 SAS card.

Also, when I 'cleared' the last 2 5TB drives, the script just stopped working - on both instances. (I ran and tried separately). I went ahead and removed the drives and changed configuration each time because I had already moved the data off each drive using unbalance.  I assumed it failed, and that my parity, even though it said valid wasn't valid.

After I got the 2nd one out, I ran another parity check to be sure. It just finished:
Last check completed on Fri 19 Nov 2021 11:39:53 AM EST (today)
Finding 760363348 errors Duration: 1 day, 12 hours, 51 minutes, 44 seconds. Average speed: 105.5 MB/se

I assume because I was right, I removed and it didn't finish/update parity along the way when I tried to clear. However, I do appear to still have same size used array, all my larger drives in place, and I just finished the parity check so it all should be good now.

I just need to consolidate the cards/cables down to 8 lines of a single SAS card now.
 

[trurl]

2 minutes ago, wildwolf said:

After I got the 2nd one out, I ran another parity check to be sure. It just finished:
Last check completed on Fri 19 Nov 2021 11:39:53 AM EST (today)
Finding 760363348 errors

Was that a correcting parity check?

 

Doesn't really matter since it is simpler to just go ahead and rebuild both parity when you New Config to close that gap.

[wildwolf]

39 minutes ago, trurl said:

Was that a correcting parity check?

 

Doesn't really matter since it is simpler to just go ahead and rebuild both parity when you New Config to close that gap.

Yes, I'm pretty sure, but have no way to know?

So, as long as I rearrange SAS cards, put drives back in the same order (1-2 parity in serial # order, 3-8 in serial # order) in new slots 1-8, I should be good?

Added screenshot of parity checks - this is the only time I've ever received errors on a parity check, so my assumption is they are 'correcting parity check' errors.

 

Edited November 19, 2021 by wildwolf
providing screenshot of parity checks

[trurl]

1 hour ago, wildwolf said:

no way to know?

If you manually run a parity check there is a checkbox you have to check to make it correct parity errors, so you should know whether or not you checked the box. All that history is showing is that it found a large number of sync errors and doesn't tell you whether or not they were corrected.

 

Post new diagnostics if you haven't rebooted and we can see from syslog whether or not it was correcting, but as I said

2 hours ago, trurl said:

Doesn't really matter since it is simpler to just go ahead and rebuild both parity when you New Config to close that gap.

even though technically only parity2 would need rebuilding to close the gap (assuming parity1 is valid which is in doubt).

[wildwolf]

4 hours ago, trurl said:

If you manually run a parity check there is a checkbox you have to check to make it correct parity errors, so you should know whether or not you checked the box. All that history is showing is that it found a large number of sync errors and doesn't tell you whether or not they were corrected.

 

Post new diagnostics if you haven't rebooted and we can see from syslog whether or not it was correcting, but as I said

even though technically only parity2 would need rebuilding to close the gap (assuming parity1 is valid which is in doubt).

Yes, it was checked, so I assume it was correcting parity errors, and not some other type of errors.

[wildwolf]

12 hours ago, JonathanM said:

Physically rearranging disks doesn't require any configuration changes or rebuilding of parity. Unraid tracks disks by serial number, it doesn't care which cable they connect to.

 

Rearranging logical slots in the management GUI is what you are referencing in the reorder disks thread.

 

Totally different and unrelated concepts in Unraid.

 

This is where I'm confused then. I want to accomplish both. I want to physically rearrange the 8 disks that remain, by swapping cables & such around so they are all on 1 controller card, instead of 2. Plus, I also hope to "close the gap" in the GUI layout.

Or, will physically rearranging things (thus eliminating the 3 empty slots by eliminating all the extra slots from that 2nd card when it's removed and consolidating down to 1 card) automatically cause the drives to show up in all the right places, as long as I have parity  & parity 2 identified as the correct 2 drives? Are you/others stating I/anybody could (in my view below) move disk 9 to the cable connector that is disk 5 (different controller card), and it'd still boot up with all the data intact, without having to run new config or do anything else?

[JonathanM]

To change disk numbers in the Unraid array layout, you must use the new config function and build parity2 fresh.

 

To physically move drives between controllers requires no changes in the Unraid software, and the drives will stay in the same disk slots in the Unraid array layout. They will not change places in Unraid.

 

Physical location and disk number assignment are two totally separate concepts in Unraid, they do not interact. At all.

13 hours ago, JonathanM said:

Physically rearranging disks doesn't require any configuration changes or rebuilding of parity. Unraid tracks disks by serial number, it doesn't care which cable they connect to.

 

Rearranging logical slots in the management GUI is what you are referencing in the reorder disks thread.

 

Totally different and unrelated concepts in Unraid.

 

[wildwolf]

3 hours ago, JonathanM said:

To change disk numbers in the Unraid array layout, you must use the new config function and build parity2 fresh.

 

To physically move drives between controllers requires no changes in the Unraid software, and the drives will stay in the same disk slots in the Unraid array layout. They will not change places in Unraid.

 

Physical location and disk number assignment are two totally separate concepts in Unraid, they do not interact. At all.

 

I think I finally understand. I can remove my other card, rearrange physical disks as I want. But my setup will still remain:

P1
P2
D1
D2
D3

D4
D8
D9

until I do new config, which will then require rebuilding new parity 2 to 'close the gap' in the GUI to:
P1
p2
D1

D2
D3
D4
D5
D6
Is that correct?

Edited November 20, 2021 by wildwolf
truri was correct

[trurl]

3 minutes ago, wildwolf said:

do new config, which will then require rebuilding new parity to 'close the gap' in the GUI to:
P1
p2
D1

D2
D3
D4
D8
D9

That is what you had before new config. Did you really mean to say this?

P1

P2

D1

D2

D3

D4

D5

D6

[wildwolf]

3 hours ago, trurl said:

That is what you had before new config. Did you really mean to say this?

P1

P2

D1

D2

D3

D4

D5

D6

Yes, corrected, thank you.

[trurl]

10 hours ago, wildwolf said:

Is that correct?

yes. Parity2 will have to be rebuilt since you have changed slots.

 

If you New Config and start the array with those assignments, it will rebuild both parity.

 

If you New Config without parity2, you can check the box saying parity1 is already valid, then start the array, stop the array, assign parity2, start the array to begin parity2 rebuild.

[wildwolf]

Thanks for all the help guys. Card removed, down to 1 card, 2 breakout cables, and all is looking a little better to me.
Now, off the wall question. My card (9211) has 2 connected cables with 2 drives connected each. Is there any benefit to having 1 parity drive on each connected breakout cable? Either performance or parity/data safety? Or no real world difference, just slap the drives where it's most convenient/easier to maintain/know where your drives are?

When I originally built, I put 1 parity per cable, thinking it might be safer. However, I now realize that may not be true and wanted to ask if it matters.
The system has 8 drives mounted vertically (Fractal Define R5), might be easier to know which drive is which if I put Parity at top, Parity 2 below that, Data 1 3rd, Data 2 4th, and on down the line. But is there any real detriment to doing that? Or would it be better to still separate parity drives per card's cable bundle?

[trurl]

6 minutes ago, wildwolf said:

know which drive is which

Label each drive with the last 4 characters of the serial number. It is the serial numbers you need to depend on when trying to decide which disk is which, because that is how Unraid is going to decide which disk is assigned to which slot. Many drives already have serial number on a small label on the end of the drive opposite the connections.

[wildwolf]

Just now, trurl said:

Label each drive with the last 4 characters of the serial number. It is the serial numbers you need to depend on when trying to decide which disk is which, because that is how Unraid is going to decide which disk is assigned to which slot. Many drives already have serial number on a small label on the end of the drive opposite the connections.

They're already labeled by manufacturer (WD drives are awesome for this!). Just wondering if there's any protection or performance benefit to separate the parity 1 per cable, or if not, I'll just slap them into a slot to match what the new config to be - labeled top to bottom 2 parity and 6 data.

[trurl]

Might be some slight safety benefit to having parity on different connections.

 

Even though the webUI lists parity2 just below parity, internally Unraid refers to parity as disk0 and parity2 as disk29. So you could think of parity2 as the last array disk.