March 30, 20251 yr Hello! I had a disk go red marked disabled, I'm not 100% sure if this was after a reboot or by itself during operation. Either way, I span the array down and replaced the disk by pulling it out, putting in a pre-cleared disk of the same capacity (and make) and assigning it to the same disk slot out of an abundance of caution (logs seemed to hint at a filesystem error, but I decided not to take any chances), and started up the array for data rebuild. This made a SECOND disk also get disabled, which prompted me to make this post. I have two parity disks and 11 data disks total. Hardware is a supermicro x9dri-ln4+ motherboard running a single intel xeon e5-2697v3 (the second socket got some traces destroyed by yours truly when attempting to mount a third party cooler with too long screws, whoops). I also have a GTX 1070 attached for accelerating GPU workloads, spesifically ML stuff on Immich. So all three PCIe slots for CPU 1 are populated. See diagnostics data attached. All disks apart from my two SSD cache drives are attached to the system via a SATA/SAS backplane, again via two DELL H200 in IT-mode. Disks of odd and even numbers are attached to separate h200's respectively (i.e. all odd numbers are on one H200, and all even numbers on another). I say this because disks 5, 7 and 9 have had issues, although disks 1, 3 and 11 are fine (for now). I also had disk 9 throw some errors during the initial rebuild (so at one point I had 3 disks be in varying degrees of not-okay), but after a reboot this went away. Seems this was due to SMART throwing Unraid off with a change in reallocated sectors, although I personally think this is within acceptable limits. Most of the harddrives are old DELL Constellation 2TB drives, about 8 years old or more, bought used. I marked disk 9 and 5 SMART-errors as Acknowledged. I figure as long as reallocated sectors is not climbing fast or consistently, the disk is safe to use. This is the state of my Main-page as of right now. Disk 7 is what prompted this change in the first place, it seemed to be in much the same state as disk 5 is now. I still have the original disk 7, the one currently in the array is the pre-cleared backup in need of data rebuild. I have cancelled the data rebuild for now. What should my next steps be? I'm keeping the array off/in maintenance mode until I'm sure of what to do next. I do have another 2TB pre-cleared backup drive, but I figured replacing two disks at the same time is either inadvisable or not supported, in the same vein where you're not allowed to i.e. remove AND add a drive simultaneously to the array. EDIT: I've also attached the diagnostics from during the data rebuild, as I rebooted after cancelling it the first time. As mentioned above, this was because I got scary read errors from Disk 9, which thankfully seemed to be transient or due to Unraid marking the disk as not safe because of reallocated sectors changing. The current diagnostics is the one timestamped 1139, and the one which includes the data rebuild is timestamped 1100. thedatadungeon-diagnostics-20250330-1139.zip thedatadungeon-diagnostics-20250330-1100.zip Edited March 30, 20251 yr by woothie 1. Added diagnostics from before reboot into maintenance. 2. Added more info about hardware (gpu and PCIe distribution)
March 30, 20251 yr Community Expert Solution There are issues with both disks, looks more like a power/connection issue, check/replace cables for both disks 5 and 7 and post new diags after array start.
March 30, 20251 yr Author Gotcha, thanks for the very quick response. I've reseated cables and the H200's, seems like I still get the same status. Disk 5 is still red, disk 7 seems fine but since it's new it's now rebuilding. I could try to replace disk 7 with the original disk 7 which (probably) has all the data on it still, but I assume that will just leave me with two reds as it did fail to begin with. As I started the array it once again started rebuilding disk 7. See attached diagnostics. I'll leave the array rebuild until advised otherwise. thedatadungeon-diagnostics-20250330-1229.zip
March 30, 20251 yr Community Expert 59 minutes ago, woothie said: seems like I still get the same status. That's expected, but no disk errors so far, you can stop the rebuild and rebuild disk5 together, assuming the emulated disk contents look correct: https://docs.unraid.net/unraid-os/manual/storage-management#rebuilding-a-drive-onto-itself
March 30, 20251 yr Author 26 minutes ago, JorgeB said: That's expected, but no disk errors so far, you can stop the rebuild and rebuild disk5 together, assuming the emulated disk contents look correct: https://docs.unraid.net/unraid-os/manual/storage-management#rebuilding-a-drive-onto-itself That seems to be the way to go, yes! I now have disk 5 and 7 rebuilding, I'll update with how it goes. Thanks for the help so far!
March 30, 20251 yr Author Well darn, it failed. I checked in on it after a few hours, turns out it had gone on pause after about 2 percent without me realizing. Seems there's a problem with disk 5, but I can't quite figure out exactly what. I can see in the disk info screen there are io issues, but exactly what is beyond my understanding. Any ideas? Attached is diagnostics. Worst comes to worst I have another 2tb precleared disk on standby thedatadungeon-diagnostics-20250330-1902.zip Edited March 30, 20251 yr by woothie
March 30, 20251 yr Community Expert Disk5 is a bit old and has a pending sector. Still not a lot of reallocated yet, and rebuild might reallocate that pending sector. Has been about 6 months since it successfully completed extended self-test though. The I/O errors look like connection problems to me. Disk7 similar age and time since last extended self-test, but it seemed to be rebuilding OK.
March 30, 20251 yr Author Noted, I'll do another check with the wiring. They're all connected via a backplane so the only actual wiring is power to the backplane via Molex and the data cables to the H200. Would be odd if it were power though, surely we'd see issues with other disks on the same line, at the very least. That's disks 5, 7, 9 and 11.
March 30, 20251 yr Community Expert 3 hours ago, woothie said: Would be odd if it were power When reading a file in the array, only the disk containing the file is involved. When writing a file to the array, the file is only written to one disk, and parity is updated. So only parity and the disk being written is involved, except as noted below*. When rebuilding, checking parity, or if reconstruct write (*Turbo Mode) is enabled, then all array disks are involved. So, power enough for some might not be power enough for all.
March 31, 20251 yr Author Turns out it (probably) was power, yeah. Reseated the Molex cables going into the backplane and added some extra force when putting them back in, rebuilt just fine. Thanks a bunch guys! What in the diags pointed you to cable/power as the likely culprit?
March 31, 20251 yr Community Expert 11 minutes ago, woothie said: What in the diags pointed you to cable/power as the likely culprit? for example Mar 30 13:00:01 TheDataDungeon kernel: sd 1:0:2:0: [sdf] tag#2181 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=3s Mar 30 13:00:01 TheDataDungeon kernel: sd 1:0:2:0: [sdf] tag#2181 Sense Key : 0x2 [current] Mar 30 13:00:01 TheDataDungeon kernel: sd 1:0:2:0: [sdf] tag#2181 ASC=0x4 ASCQ=0x0 Mar 30 13:00:01 TheDataDungeon kernel: sd 1:0:2:0: [sdf] tag#2181 CDB: opcode=0x28 28 00 1b 50 7a c0 00 04 00 00 Mar 30 13:00:01 TheDataDungeon kernel: I/O error, dev sdf, sector 458259136 op 0x0:(READ) flags 0x0 phys_seg 128 prio class 0 22 hours ago, trurl said: The I/O errors look like connection problems to me. What I really mean is it can't communicate with the drive(s) for some reason but the drive is probably OK. Could be cable, controller, power...
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.