Hey Guys...

 

I found myself in a pickle.  

 

I run unraid on an old HP tower server with 7 drives in my array with 1 parity.  4x drives via a SATA, and 3 drives in 2 identical USB 2-bay enclosures via USB 3.0.  

 

I recently discovered that one of my drives in one of the USB enclosures was marked as bad due to the enclosure supposedly going on the fritze.  I power cycled all the devices and everything seemed to come back and I proceeded to "remove" and "re-add" the bad drive and rebuild the array.

 

During the rebuild, I believe either the enclosure disconnected again which resulted in a 2nd drive reporting a bunch of read errors at 91.9%.

 

I stopped the rebuild, tried power cycling the enclosures and tried the rebuild again and it works but died the same place at 91.9%... 

 

So this is where I'm at right now.   I'm still suspicous that the enclosure is flakey but I'm also worried the USB controller on the HP's motherboard is also unstable (it's had issues detecting the unraid USB stick during boot requiring a full power down).  

 

Any advice on next steps?   I've got 2 issues I need addressed...  

 

1. Rebuilding the array : I assume I can't really do anything until I verify the drive throwing the read errors is OK
2. Scanning/verifying the integrity of the read error drive : I'm moving the drive with the read errors into *another* separate enclosure to do the rebuild.  Is there a way to scan the drive outside of unraid on windows?

 

I'm also assuming, the original drive that was marked as bad is a lost cause and there's no way to recover anything from that drive..   Is there a way to recover anything on the read error drive?  

 

Thanks in advanced!

Edited December 9, 20241 yr by machineglow

Just tried using a different 4-bay USB enclosure and moved the 3 drives to that...

 

They show up in unraid but under a different label and when I try to slot them into now 3 empty slots in the array, unraid says the drives are "wrong".  I assume because the identifier is different, it won't allow you to add those drives back into the slot.  

 

So I guess I have to finish the rebuild before I try to move drives around to isolate the enclosure?  

 

 

Edited December 9, 20241 yr by machineglow

So I'm reusing the 2x 2bay enclosures and isolated the drive that needs to be rebuilt on it's own and moved the drive with the read errors to the other enclosure.

 

Everything is detecting ok and the rebuild is even faster (150mb/s vs. 70-100mb/s).  the enclosure with the rebuilding drive is only doing writes now so hopefully it won't overheat or cause issues during the rebuild....    Crossing fingers!

Hi all,

 

bad news....  so even after shuffling the drives around in the enclosures, the same drive is crapping out with read errors at basically the same spot (91.9%) while rebuilding the array..

 

Am I SoL?  

 

here's the drive logs that just keep repeating this every 3 minutes:

 

Dec 10 02:25:41 datastore kernel: sd 0:0:0:1: [sdb] tag#0 timing out command, waited 180s
Dec 10 02:25:41 datastore kernel: sd 0:0:0:1: [sdb] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=184s
Dec 10 02:25:41 datastore kernel: sd 0:0:0:1: [sdb] tag#0 Sense Key : 0x2 [current] 
Dec 10 02:25:41 datastore kernel: sd 0:0:0:1: [sdb] tag#0 ASC=0x4 ASCQ=0x1 
Dec 10 02:25:41 datastore kernel: sd 0:0:0:1: [sdb] tag#0 CDB: opcode=0x88 88 00 00 00 00 01 ab e8 a9 d0 00 00 08 00 00 00
Dec 10 02:25:41 datastore kernel: I/O error, dev sdb, sector 7179119056 op 0x0:(READ) flags 0x0 phys_seg 256 prio class 2

 

Trying to pull the diagnostic logs now but it's taking forever stuck on a ssd drive pool usage....

 

Is anything recoverable at this point?  are there any diagnosis steps or recovery steps I can do on the drive throwing read errors?  

 

Here's my logs that finally finished downloading...  

 

would appreciate any advice!  Am I SoL with 2 bad drives in a 1 parity array?   

 

what are my options now?   

 

Thanks in advance!

 

 

Datastore Diagnostics Dec 10 2024.zip

Dec 10 00:38:10 datastore kernel: usb 5-1: reset SuperSpeed USB device number 2 using xhci_hcd

 

Looks like USB problems, both devices are dropping at the same time, note that we don't recommend USB for array or pool devices, this being one of the common issues.

Ah,  thank you for finding that....  

 

Soooo...  in one of my attempts to remove the usb enclosures out of the equation, i moved the 3 drives into a 4-bay eSata enclosure.   Unraid found the devices but gave them different labels and complained that they were "wrong" when I tried to put them into the array slots.   Is there a way to override that warning?

 

Alternatively, I am trying to source another server with more sata ports but I wonder if I'll just get the same error above as all the drives will likely get detected with new labels...  

 

Or is there a way to revive the usb devices and resume the array rebuild?  

 

thanks,

Edited December 10, 20241 yr by machineglow

9 minutes ago, machineglow said:

Alternatively, I am trying to source another server with more sata ports but I wonder if I'll just get the same error above as all the drives will likely get detected with new labels...  

Anything SATA should pass the original labels through, some USB enclosures tend to replace them with their own, and may cause other issues like not passing the entire drive's contents. 

 

So avoid USB, real SATA should work fine.

thank you for confirming that.   Are there any tips on prepping the unraid install for migrating to a new server?  I may have to skip a gpu but will also need to move an LSI raid card + 4x SSDs that's attached to them.  I assume i should also turn off autostart on any of my VMs and Containers.    

 

Any licensing issues as well?

 

Thanks,

License is tied to the USB drive, so no issues there. Anything that's SAS or SATA should transfer straight, anything that was previously on USB may or not depending on the USB enclosures used. 

Edited December 10, 20241 yr by Kilrah

In the event that the drives from the usb enclosures change names on the new server, then is there a way to relabel them? Also, would you recommend turning off the server to prevent any more changes to the array?

 

Thanks,

 

Edit:  got the harddrives spun up in another server and indeed, the labels are A-OK.    starting another rebuild!  crossing fingers!

Edited December 10, 20241 yr by machineglow

drive rebuilt successfully!

 

Thank you @JorgeB @Kilrah for all the help!

 

Guess it's time to junk the old HP and figure out a new NAS build.