[SOLVED] Safe to Reboot unRAID During Data Drive Rebuild?

[AgentXXL]

I've got another thread here where I'm trying to resolve slow parity rebuild/sync speeds - estimated 6 days to do a 16TB data disk rebuild from parity. It looks like my LSI HBA is being throttled to PCIe x1 even though it's in a x8 slot. The message I see in the syslog says this:

 

Oct 27 02:54:54 AnimNAS kernel: pci 0000:03:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x4 link at 0000:00:1c.0 (capable of 63.008 Gb/s with 8 GT/s x8 link)

 

I want to try 2 things to see if I can resolve this:

 

1) Isolate the LSI HBA in its own IOMMU group - currently it shares a group with 3 other items, one in specific that the syslog mentioned is limiting the PCIe link speed.

2) Move the LSI HBA to another x8 slot on the motherboard.

 

Both of these will require me to power down/reboot the system. The data drive rebuild has been running for 3 days and estimates another 3 days to go. I just want to know if the rebuild state will continue from where it left off before the reboot or if I'll have to start the rebuild from scratch? I know that preclears can be resumed after reboot so I'm hoping a data drive rebuild from parity also can be resumed.

 

Note that the 16TB drive was replacing an empty 10TB data drive. In the 3 days that the rebuild has been running, 2TB of data have been copied onto the drive, even though other drives still had plenty of free space. Not something I'm too concerned about but is there a setting in unRAID to prevent data copies to a rebuilding drive? I could have gone into each of my shares and added the disk to the 'Excluded Disks' section I suppose...

 

TIA!

 

 

Edited October 31, 2020 by AgentXXL

[Vr2Io]

Seems you plug the HBA to ICH PCIe, no IOH ( 5520 Tylersburg ) PCIe for HBA ?

BTW, platform was Gen2 max , so LSI SAS2308 won't perform better then SAS2008.

 

Edited October 31, 2020 by Vr2Io

[AgentXXL]

24 minutes ago, Vr2Io said:

Seems you plug the HBA to ICH PCIe, no IOH ( 5520 Tylersburg ) PCIe for HBA ?

BTW, platform was Gen2 max , so LSI SAS2308 won't perform better then SAS2008.

There are only 2 PCIe x8 slots on the Supermicro x8DTN+. From the block diagram in the manual it shows that the one I'm plugged into should be connected via the 5520. That's why I want to shutdown/reboot - to move it to the other slot to see if that improves the link speed. I still would prefer to know if the shutdown/reboot will mean restarting the data drive rebuild from scratch or if a clean shutdown will let the system continue the rebuild from where it left off.

[Vr2Io]

1 minute ago, AgentXXL said:

shutdown/reboot will mean restarting the data drive rebuild from scratch

Yes

[AgentXXL]

1 minute ago, Vr2Io said:

Yes

So if my performance is unlikely to improve, I'd rather just leave the rebuild running for the remaining 3+ days it estimates before completion. I'll have to try the other slot after the rebuild completes.

 

Certain things have improved by replacing the older Dell H310 with the 9217-8i. Namely I notice that the boot is faster as it takes less time for the newer SAS2308 chip to recognize and pass-through the drives. While still using the Dell, I also saw some unusual errors when trying to rebuild my parity after upgrading to the 16TB disks. I decided that the $50 CAD for the 9217-8i was worth a try so I ordered it and ran my unRAID without the parity drives until it arrived. It came in quickly and my 1st attempt to build the new dual parity set worked well with no errors reported. It was just incredibly slow, taking almost 6 days to complete.

 

[Vr2Io]

22 minutes ago, AgentXXL said:

So if my performance is unlikely to improve

No improve were base on both HBA in 5GT/s x8 state, but you currently running at about ~25% ( 2.5GT/s x4 vs 5GT/s x8 ).

So even rebuild from scratch, still expect complete in less then 3 days. ( I would expect ~1.5days to finish )

Edited October 31, 2020 by Vr2Io

[AgentXXL]

Just now, Vr2Io said:

No improve were base on both HBA in normal non-blocking state, but you currently running at about ~25% ( 2.5GT/s x4 vs 5GT/s x8 ).

So even rebuild from scratch, still expect complete less then 3 days.

But that's only if I can get the card to negotiate the x8 link speed. If moving to the other slot or isolating in its own IOMMU group doesn't work, I'd hate to waste the 3 days worth of rebuild time and have to wait another 6 days.

 

This likely won't be an issue when I convert the CSE-847 to a DAS and use a new outboard PC with a new HBA that has external connnections. I'm looking at either building a Threadripper or Epyc based solution, but still need to save up the cash.

 

Thanks for the feedback!

[Vr2Io]

5 minutes ago, AgentXXL said:

But that's only if I can get the card to negotiate the x8 link speed. If moving to the other slot or isolating in its own IOMMU group doesn't work, I'd hate to waste the 3 days worth of rebuild time and have to wait another 6 days.

Yes ( IOMMU group won't be the issue ), real issue was PCIe bandwidth provide for HBA, so you need make decision. 

 

8 minutes ago, AgentXXL said:

I'm looking at either building a Threadripper or Epyc based solution

Seems cheap 1st gen TR not available in market anymore and cooler choice for TR also limited in 4U case. BTW I will build a X299 platform with i7-9800X in 3U for Unraid in coming days, mainboard was EVGA X299 FTWK, it really cheap in Amazon.

[AgentXXL]

1 minute ago, Vr2Io said:

Yes ( IOMMU group won't be the issue ), real issue was PCIe bandwidth provide for HBA, so you need make decision. 

 

Seems cheap 1st gen TR not available in market anymore and cooler choice for TR also limited in 4U case. BTW I will build a X299 platform with i7-9800X in 3U for Unraid in coming days, mainboard was EVGA X299 FTWK, it really cheap in Amazon.

I'm looking at the TR or Epyc as I want to consolidate my VMs that currently run on VirtualBox and VMWare on another system. An i7 just won't cut it for the CPU and RAM resources I want to allocate to the VMs. And as I'm tired of using older hardware, I'm willing to keep saving my cash so I can go with the new TR or Epyc generation coming in 2021.

 

I'm also still keeping a dual Xeon platform on the drawing board. While AMD certainly has my interest right now, I've always been an Intel guy and usually quite happy with them. But my needs have changed, as it has for many people impacted by working from home during the pandemic.

 

My backup unRAID on a 9 year old i7-980 hexacore is still useable and actually gets better I/O performance than my media unRAID. At least until I get this link speed issue resolved. Good luck with your build!