Replacing several 4TB Drive with 8TB drives on large array with double parity

[FrozenGamer]

My plan is to do this as per the unraid 6 manual - 

Replace a Single Disk with a Bigger One 
This is the case where you are replacing a single small disk with a bigger one:  
1. Stop the array.  2. Power down the unit.  3. Replace smaller disk with new bigger disk.  4. Power up the unit.  5. Start the array

 

I also plan on preclearing the new 8tb to be safe.

 

This is a large array - i have

2 8tb parity

5 4tb

7 5tb

8 6tb

2 8tb data

 

Ultimately, i would like to get rid of all the 4tb drives or at least 4 of them, just to have a amount of disks.

 

Any thoughts? or advice.  My main concern is a power outage or failure of a drive during rebuild.  Even considering getting another license for unraid and starting a fresh 8tb only array, eventually pulling the 4 8tbs (2 parity and 2 data) out 

 

Would a rebuild of 4tb onto an 8 take just as long as rebuilding an 8tb parity drive does?  This takes a long time, really long time.  If i remember correctly it has to write zeros the whole time.

 

 

 

 

 

 

 

[trurl]

52 minutes ago, FrozenGamer said:

Would a rebuild of 4tb onto an 8 take just as long as rebuilding an 8tb parity drive does?  This takes a long time, really long time.  If i remember correctly it has to write zeros the whole time.

Since it is only rebuilding 4TB then all it has to do to the other 4TB is write zeros to it, but it does have to write the whole disk so it will agree with parity.

 

54 minutes ago, FrozenGamer said:

i would like to get rid of all the 4tb drives

I expect you already know this but you can only replace them one at a time, or since you have dual parity, you can replace 2 at a time. If you keep the original disks with their data intact, and all of the disks and connections are good there should be little risk with doing 2 at once. Be sure to double check all connections anytime you are messing about inside the case.

[FrozenGamer]

I am finally replacing 2 of my 4tb drives with 8tb drives and it really slows down at times (is this normal?)

 

"Total size:8 TB

Elapsed time:3 days, 15 hours, 51 minutes

Current position:3.64 TB (45.4 %)

Estimated speed:772.7 KB/sec

Estimated finish:"

 

 

 

If i have something like a power outage or a crash during this 6 to 8 day rebuild of both disks would i be able to use my old 4tb drives safely? or if it made it past the 4tb in the rebuild would my data be safe as it is since there are only zeros left to write (and they should already be zeros from preclear?) ( i have disabled mover so content on array should remain the same)

Are parity calculations going to be done for the last 4tb writing zeros, or just write zeros?  As far as i can tell there are is no change to parity happening during the data rebuild.

 

 At other times it peaks out at 20-23 MB/S and i anticipate the actual time to complete everything will be 6 to 8 days.  I am leaving all my dockers running - Plex media server, sonarr radarr, sabnzbd, nzbhyrdra2.  Should that have much effect on speed?

 

This is all kind of scary, i have 3 more 4tb drives that i would like to replace with 8's though i would not mind reducing the array size a little in the process, and eventually the 5's  to be replaced by 8's.

 

 

 

Thanks in advance for answering any questions.  

 

 

[itimpi]

There is something wrong - the rebuild should be running much faster than that.   It might be worth providing your diagnostics ZIP file (obtained via Tools->Diagnostics) to see if anyone can spot a reason for the performance you are seeing.

[FrozenGamer]

57 minutes ago, itimpi said:

There is something wrong - the rebuild should be running much faster than that.   It might be worth providing your diagnostics ZIP file (obtained via Tools->Diagnostics) to see if anyone can spot a reason for the performance you are seeing.

See attached diagnostics - i saw a PIO reference in here which i remember to mean shit disk transfer from windows years ago tower-diagnostics-20190120-1020.ziprs ago. But only that reference once.

Jan 16 16:49:16 Tower kernel: ata4.00: READ LOG DMA EXT failed, trying PIO

Edited January 20, 2019 by FrozenGamer

[klipp01]

If your worried about a power outage get a UPS. You should have one anyway as unclean shutdowns are no fun.

Sent from my Pixel 2 XL using Tapatalk

[trurl]

19 hours ago, FrozenGamer said:

See attached diagnostics

Nothing obvious in syslog.

 

Most of your disks are completely full but that shouldn't affect rebuild since the parity calculation doesn't know anything about that.

19 hours ago, FrozenGamer said:

Jan 16 16:49:16 Tower kernel: ata4.00: READ LOG DMA EXT failed, trying PIO

Not sure what that is about but ata4 is your cache drive so shouldn't affect the rebuild either.

Jan 16 16:49:16 Tower kernel: ata4.00: ATA-9: Crucial_CT960M500SSD1,         1330094769B4, MU03, max UDMA/133

You have such a large number of disks that instead of going through SMART for each I will just ask you.

 

Are any of your disks showing warning SMART indicators on the Dashboard page? Do you have anything in the Errors column in Main?

[FrozenGamer]

2 hours ago, trurl said:

 

You have such a large number of disks that instead of going through SMART for each I will just ask you.

 

Are any of your disks showing warning SMART indicators on the Dashboard page? Do you have anything in the Errors column in Main?

No Errors, No all smart indicators on dashboard are green thumbs up.

[Harro]

I just went through the replacement process of both parity drives from 8TB to 10TB, one at a time  (pic with black lines on bottom). I then used the 2 8TB (old parity Drives) in array, replacing 4TB drives, Replaced both 4TB drives at once (in pic with red line). I then replaced a 8TB drive in array with a 10TB drive (Green line top).

 Just thought I would give you an idea of times.

[FrozenGamer]

2 minutes ago, Harro said:

I just went through the replacement process of both parity drives from 8TB to 10TB, one at a time  (pic with black lines on bottom). I then used the 2 8TB (old parity Drives) in array, replacing 4TB drives, Replaced both 4TB drives at once (in pic with red line). I then replaced a 8TB drive in array with a 10TB drive (Green line top).

 Just thought I would give you an idea of times.

How many drives in your array and what type of communication with PC, backplate,caddy,pcie card?  Maybe i need to upgrade my equipment (all though i am not sure there is an affordable 20-30 drive solution?

 

[Harro]

Just now, FrozenGamer said:

How many drives in your array and what type of communication with PC, backplate,caddy,pcie card?  Maybe i need to upgrade my equipment (all though i am not sure there is an affordable 20-30 drive solution?

 

In my signature.

Unraid-6.6.6

MB: Asrock H77-M, Intel i5 [email protected], Norco SS-500 Hot Swap Rack module x3, HP H220 Controller X2, Kingston 240 DDR3 Sdram 16GB memory, Cooler Master 590 case, 14 data drives, 2 parity drives, 114 TB storage and still growing.

[trurl]

1 hour ago, FrozenGamer said:

How many drives in your array and what type of communication with PC, backplate,caddy,pcie card?  Maybe i need to upgrade my equipment (all though i am not sure there is an affordable 20-30 drive solution?

It might be useful to know what your situation is. Ideally, the number of disks won't affect parity syncs, etc., and it would primarily depend on the size and speed of parity. But in the real-world there can be controller bottlenecks for example.

 

How long does your 8TB parity check take? You can go to Main - Array Operations and click the History button for the answer.

[FrozenGamer]

57 minutes ago, trurl said:

 

 

How long does your 8TB parity check take? You can go to Main - Array Operations and click the History button for the answer.

Average is about 1 day and 4 hours.  A little over 6 days for changing parity drives to 8tb.  

 

It might be useful to know what your situation is. Ideally, the number of disks won't affect parity syncs, etc., and it would primarily depend on the size and speed of parity. But in the real-world there can be controller bottlenecks for example.

1 hour ago, trurl said:

It might be useful to know what your situation is. Ideally, the number of disks won't affect parity syncs, etc., and it would primarily depend on the size and speed of parity. But in the real-world there can be controller bottlenecks for example.

I have 2 of these 16 bay racks https://www.servethehome.com/sgi-rackable-3u-16-bay-se3016-sas-expander-chassis-forum-deal/ 

hooked up to LSI SAS HBA 9207-8e 6GBs 8-Port - one cable each directly to the card, not daisy chained.  The processor/computer its hooked up to is a gigbayte z97x-sli-cf mobo, i5 4590 @3.3ghz, 16gb Ram.

 

I know that there are some speed limitations on the expander (PCIE 1), but i would assume that shouldn't be causing the intermittent slowdowns in the parity rebuild? ie is it normal for an unraid server to have severe slowdowns for periods of time during the rebuild of data?

[JorgeB]

34 minutes ago, FrozenGamer said:

ie is it normal for an unraid server to have severe slowdowns for periods of time during the rebuild of data?

No, if I had to guess my number one suspect would be one the disks, there have been previous issues with other users and those Seagate disks, seems some of them can be duds and perform very poorly, you could try rebuilding one a a time a see if it's one of them.

[Zonediver]

If the array is in use during rebuild (streaming over Plex) it can slowdown the rebuild process for 2-3 hours...

In my case from 10h 29min to 12h 45min

Edited January 21, 2019 by Zonediver

[FrozenGamer]

On 1/21/2019 at 6:30 AM, trurl said:

 

 

 

Are any of your disks showing warning SMART indicators on the Dashboard page? Do you have anything in the Errors column in Main?

Ok this is really weird.  I got 37 read errors on disk 18 which is toshiba 5tb. And it is increasing - 1797 - started 15 minutes ago.  But I am almost to 6tb in disk rebuild now..   There is no temperature on disk 18 as well.  Perhaps it has shut down?

 

Since i have 2 8tb parity drives and i have replaced 2 4tbs with 8tb and am rebuilding them, if i were to pull the shut down 5tb out of hot swap and reinsert it would i jeopardize my data? I would assume it would.  This is really confusing.  I am rebuilding zeros from 4.01tb to 8tb on the 2 new drives, so the rebuild is technically done? and they are all zeros because i precleared them prior right?   I have attached another diagnostics that is a few minutes old.  the problem starts at 10:05 1/22/19.  Also attached a screenshot if that helps to show what is happening.

 

I have 16tb of parity drives (2x8) but if i had 3 drives fail, say a 4tb and 2 5's i would lose data right?  It would seem that normally i would have lost data, if it weren't for the fact that all this is happening past the points on drives that have data? ie if i am at 6tb on a 4tb rebuild to an 8 is my array safe with drives that are currently being rebuilt with zeros on the last half?

tower-diagnostics-20190122-1047.zip

Edited January 22, 2019 by FrozenGamer
Attach Proper diagnostics

[JorgeB]

Looks like disk 18 dropped offline, but diags are before the errors.

[FrozenGamer]

5 minutes ago, johnnie.black said:

Looks like disk 18 dropped offline, but diags are before the errors.

Diagnostics are Fixed in post above - thanks!

Edited January 22, 2019 by FrozenGamer

[FrozenGamer]

Also i just noticed that the 5tb is showing up in unassigned devices, i think i may have updated unassigned devices shortly before the problem, which could have possibly caused it to go offline out of the array?  And it appears that data is missing from the array in its current state - a tv show which was there is no longer there when i go to the array on the network.  This was likely written to disk 18 just prior to the disk rebuild.  Perhaps the increasing errors are from Dynamix Cache Directories refreshing every so often.

Edited January 22, 2019 by FrozenGamer
additional info.

[trurl]

Disk18 appearing in Unassigned Devices is likely just a result of it disappearing and then coming back. The errors are the result of disk18 dropping, since the rebuild is attempting to read it and every other disk.

 

1 hour ago, FrozenGamer said:

all this is happening past the points on drives that have data? ie if i am at 6tb on a 4tb rebuild to an 8 is my array safe with drives that are currently being rebuilt with zeros on the last half?

If that is the case then I think you should be good as far as the rebuilt data is concerned, and a parity check after the rebuild completes would be in order anyway. But the rebuilt disk is going to need filesystem repair since it will be back where it was before you repaired it in another system. That was another thread with another user.

 

I wonder if you will really be able to complete the rebuild like this. And if you stop to fix the problem it will want to start the rebuild over. Perhaps a New Config would be the way out of this place and on to fixing the other things.

 

Let's wait and see if there are any other opinions.

 

 

[FrozenGamer]

19 minutes ago, trurl said:

But the rebuilt disk is going to need filesystem repair since it will be back where it was before you repaired it in another system.

Not sure i understand the last comment. The 2 disks that are rebuilt are well past the sectors that contained data being that i replaced a 4tb with an 8tb right with no errors in the system until almost 6tb?  i didn't repair the disks in another system? Trust that you are right, just don't understand. - about 1 day 12 hours to complete rebuild.

[trurl]

1 minute ago, FrozenGamer said:

Not sure i understand the last comment. The 2 disks that are rebuilt are well past the sectors that contained data being that i replaced a 4tb with an 8tb right with no errors in the system until almost 6tb?  i didn't repair the disks in another system? Trust that you are right, just don't understand. - about 1 day 12 hours to complete rebuild.

Sorry, I had your thread confused with another. I will strike that part.

[JorgeB]

1 hour ago, FrozenGamer said:

The 2 disks that are rebuilt are well past the sectors that contained data being that i replaced a 4tb with an 8tb right with no errors in the system until almost 6tb?

You should post diags to confirm rebuilds were not affected, since the ones you posted are before the errors.

[FrozenGamer]

Prettty sure I fixed that diagnostics posted a few hours ago?   Should have the errors.  I’m at Crossfit gym and will be home to double  check in few hours.  Thanks for you help!

Edited January 22, 2019 by FrozenGamer

[JorgeB]

33 minutes ago, FrozenGamer said:

Prettty sure I fixed that diagnostics posted a few hours ago?

Yeah, sorry about that, some times the forum "eats" post from the unread list.

 

Disk dropped offline and came back online with a different identifier, but it didn't affect the rebuild, it wasn't disabled because you already got 2 invalid disks, it will comeback online after a reboot, take the opportunity to check/replace cables.