I got a notification stating that one of the data drives had a read error and was disabled. Not even 30 seconds had passed before I got another read failure and drive disabling, this time the parity drive.

 

Server has been running fine since 2023, so the only hypotheses I have right now are:

 

- Recent upgrade to stable v7

- Dusted out server a couple of days ago, perhaps one of the connections got knocked loose over time?

- Or I was extremely unlucky and the drives happened to both fail.

 

What should I do here? Making matters complicated, I currently am overseas so only have remote access/assistance with family members. Is it safe to try a reboot and see if the drives come back? Thanks.

 

Diagnostics attached.

dipper-diagnostics-20250112-1023.zip

I've had some time to sit down with a laptop and properly go through the WebUI. Seems like the Parity device is back and detected -- while it is still marked as disabled I can see the SMART attributes. None of the values are different from the last time I checked, in particular the UDMA CRC error is at 0, so I'm guessing at least the connections are okay.

 

What is more worrying is disk 3 not showing up at all. On the SMART tab it just says: "Smartctl open device/dev/sde failed". I don't know if it's because the device hasn't come back yet so the kernel hasn't assigned/re-assigned it to a drive name, or if it's actually the drive not reporting SMART.

 

Looking through the drive logs, it seems like the two drives failed in a different way. Now I'm really lost and don't know what to do. At least I have two parity drives so I haven't lost data so far, but I'd like to know what's the safest way to proceed. Can I get away with reseating drives and checking cable connections, or is one or both drives faulty and I should replace them ASAP?

Bump; would appreciate any pointers on next steps as I'd like to get the array out of the degraded state as soon as possible to prevent data loss.

There's no SMART for disk3, and the initial error is logged as a disk problem, check/replace cables for both disks and post new diags after array start

Thanks for the reply @JorgeB. I was able to reseat all connections and the server now shows SMART for all drives. Here are the new diags:

 

dipper-diagnostics-20250113-0522.zip

 

Am I clear to start up the array and begin the rebuild? Should I disable Docker/SMB to minimize writes to the array during the rebuild?

 

Thanks again!

9 minutes ago, JorgeB said:

new diags after array start

please, to make sure the emulated disk is mounting before rebuilding on top

@JorgeB my bad, I thought array start would instantly kick off the rebuild process. Here are new diags after starting the array:

 

dipper-diagnostics-20250113-0532.zip

SMART looks OK, and the emulated disk3 is mounting, assuming contents look correct, you can rebuild on top, and re-sync parity at the same time:

 

https://docs.unraid.net/unraid-os/manual/storage-management#rebuilding-a-drive-onto-itself

Great, thank you so much!

@JorgeB Unfortunately I spoke too soon

I followed the rebuild instructions, but as soon as the sync started disk 3 dropped offline again with the exact same issue where SMART stopped reporting. The sync is currently in a paused state because of this.

 

Is that drive toast? I can order up a replacement for it, but in the meantime am wondering whether I should resume the sync so that the parity drive is rebuilt.

 

Attached diagnostics:

 

dipper-diagnostics-20250113-0559.zip

Parity just dropped offline again as well. Attaching new diagnostics:

dipper-diagnostics-20250113-0630.zip

Did you just check or replaced the cables? Also, do those disks share something like a power splitter?

@JorgeB I just made sure that the cables were firmly connected while reseating them.

 

I'm using the Jonsbo N1 case that has five drive bays and a backplane that has two Molex power ports and 5 SATA connections. 1 SATA cable goes direct to MB, while the other 4 are connected via 1 SAS-to-SATA right-angle cable that is connected to an LSI RAID card in HBA mode. Unfortunately I do not know which cable goes to which HDD, but I think it's safe to guess that both failing HDDs are to the HBA card.

 

The SAS-to-SATA cable was replaced 6 months ago with a right-angle one because I found that the straight one fails after a while because of the tight bend that the case necessitates. Even though it no longer has a bend, I'm now wondering if either the cabling or the HBA card itself is dying. Cable itself is from AliExpress since that's the only marketplace where I could find a right-angle cable.

 

Right now I'm planning to order another SAS cable, and test the drives on another machine with something like a SMART long to make sure the drives themselves aren't faulty. Please let me know if this is a good idea (and any recommendations on better sources for good quality SAS cables...)

 

You can try swapping cables between the disabled disks and the other two and retry, and see where the issues follow.

@JorgeB the issue is the array is already in a degraded state, and the errors don't seem to show up until UnRAID tries to read/write to them. I'd much rather not risk more drives getting disabled.

I've just ordered up a new cable that should arrive in a couple of days. I'll post an update once the cable is swapped and the array starts to rebuild.

25 minutes ago, ericswpark said:

 the issue is the array is already in a degraded state, and the errors don't seem to show up until UnRAID tries to read/write to them.

Unraid won't disable more drives than the installed redundancy can handle, in your case 2, if there are more write errors, it will just abort the rebuild.

That's good to know, thank you. So if I swap the drives around in the bay and get similar errors on the other two drives then unRAID will just stop in a safe manner? I'm planning to use maintenance mode so that no further read/writes of data occur while I troubleshoot.

 

The main concern is having four out of five drives go out of sync, since it would be quite the headache to verify that the data is all there and correct. I do have a backup of the entire server if the array completely fails, but that would take even longer since it has to restore over Internet.

7 minutes ago, ericswpark said:

So if I swap the drives around in the bay and get similar errors on the other two drives then unRAID will just stop in a safe manner?

If there are write errors yes, read errors will just be skipped, so still good to monitor during the rebuild and cancel if there are read errors, since the rebuilt disk would be missing info.

I had a chance today to go over swapping drive slots around and believe that I've isolated the issue down to the drives. The errors followed data drive #3 and parity drive #1 shortly thereafter. I've ordered up new replacements for both. Thank you @JorgeB for the assistance. Very strange that two drives just went like that, but I'm grateful that I went with dual parity now.

Quick question regarding the rebuild process. When I get the new drives I suppose I will have to do a parity swap procedure as all the drives that I had were 8 TB, and I'm replacing a parity drive and a data drive that were both 8 TB. For parity it's a larger drive so that's fine, but I assume I'd have to move parity 2 to data 3, assign the new 16 TB drive to parity 2 and 1, and start the array. Then UnRAID will offer to copy the parity data over, zero the remaining bits, then reconstruct data 3. Is my understanding correct?

 

Alternatively, I could assign parity 1, leave data 3 blank, rebuild, then do the parity swap procedure (parity 2 to data 3 slot, new drive in parity 2, copy parity data, rebuild data 3). This does mean I will rebuild twice, but it seems less riskier (?) since I can have another drive fail during the second rebuild if I'm understanding it correctly. Which approach should I go for?

 

Both replacement drives are 16 TB.

1 hour ago, ericswpark said:

but I assume I'd have to move parity 2 to data 3, assign the new 16 TB drive to parity 2 and 1, and start the array. Then UnRAID will offer to copy the parity data over, zero the remaining bits, then reconstruct data 3. Is my understanding correct?

That is correct, Unraid will copy the parity data to the new parity2 disk first, and then rebuild disk3 and resync the new parity drive at the same time.

1 minute ago, JorgeB said:

That is correct, Unraid will copy the parity data to the new parity2 disk first, and then rebuild disk3 and resync the new parity drive at the same time.

 

Thanks, but are two rebuilds necessary (or recommended) as outlined in the alternate approach I laid out? Or will it cause unnecessary strain and therefore fine to just rebuild in one go?

There will only be one rebuild for disk3, the other operation is just the old parity copy, it will read the new disk 3 and write the new parity2.

So I finally got the drives in today and connected them, but I'm _still_ somehow getting the same disk I/O errors on the brand new drives. I'm wondering now if the LSI card that I'm using has died. I'm using a LSI 9240-8i SAS card flashed into IT mode that I bought off of eBay in 2022. Should I try getting a replacement card?