How's this for a comedy of errors. My server mainly runs Plex. I have 6 data drives and 2 parity drives. In the last couple of months, one of the parity drives would fail during the parity check. I'd reseat the cables and rebuild the drive, and it's been fine until another parity check when it fails again. The second parity drive just failed as well in the latest parity check, so I shut the server down.

 

I finally got 2 new drives to replace the parity drives (just bought a house, so funds were tied up in that). One drive showed up, the other is on its way. Preclear was done on the first drive, so I added it to the array to rebuild parity. Almost immediately, one of the data drives was disabled because it had over 388k errors. Along with all the other data drives, this one never had any issues in the past when parity check would run. I shut the server down again to make sure I didn't have a loose cable or anything when I was in there replacing the parity drive. Everything was good, so I booted it back up.

 

When I started the array again, the same data drive gave me the same exact number of errors. I let parity rebuild for a bit, but the server hung up when it got to around 17% of the rebuild. Then the dashboard of the GUI stopped responding. No errors were showing in any logs. I could see the logins that I had from my desktop and laptop being logged, but nothing else. Shut down and checked every connection in the case (reseated the RAM and LSI card, power and data cables for the drives were all good still). Booted up and it hung up on the rebuild at about 0.4%.

 

I forget the exact error I got this time, but looking it up everyone is suggesting to run memtest, which is running now. It started about 12 hours ago and I was going to let it run for another 12 or so, unless I should let it go longer.

 

So, I'm trying to rebuild parity, but one of the data drives keeps coming up with errors. If that data drive is really bad and parity can't be rebuilt from the other drives, is that drive lost? It's "just" TV shows and movies so it's not the worst thing in the world apart from my time to get it all back on there. If there is some other hardware issue, would that data drive come back up on its own once I resolve that? Or would it try to rebuild from the parity drive, the one that won't build properly because one of the data drives keeps giving me errors?

 

If anyone wants to see logs, I don't have them now. I can get some after memtest is done, but that will be a bit. These were just my overnight thoughts on the topic, I wanted to see what people thought of my predicament.

Post the diags when you can, after the read errors happen.

Memtest ran for over 24 hours, 6 passes completed, no issues. Not the RAM.

 

Here are the diagnostics. In the smart folder, the drive that immediately fails is the one with the SN of ZTN0ASB1, and the new parity drive is the one with the SN of ZYD1PG7P.

 

If I try to assign drive ZTN0ASB1 to data drive 1, the "start" button for the array gets greyed out and it says "Too many wrong and/or missing disks!" next to the start button.

 

Unfortunately I didn't really have time to try anything else other than get the diagnostics tonight. I won't be able to try anything you might come up with until after work tomorrow, around 4 Phoenix time.

omega-diagnostics-20250109-2106.zip

You currently have 3 invalid disks, both parity drives and disk1, so it's not possible to rebuild any disk, is old disk1 still available and untouched?

Disk 1 was never an issue until I was trying to rebuild parity a couple of days ago. As soon as the parity rebuild started, it immediately failed.

 

So it sounds like there’s no chance of rebuilding that drive. Do I lose the entire array or just whatever is on that disk?

You would only lose the data for that disk, but is old disk1 no longer available, or it is always the currently assigned disk? If the latter, you can do a new config and it will mount the old data, assuming it's still there.

It’s still available. Still in the server all hooked up. That would be amazing if that could fix my screw up. Is there a guide I can follow to try this out when I get home tonight?

Go to Tools - New config - Keep all assignments - Aplly

 

The check all assignments, re-assign disk1, start array and post new diags.

Sweet, nice and easy. I’ll try it this afternoon when I get home, thanks

2 question on this new config tool. Do I want to preserve any assignments? And how much should I worry about this message in the tool:

 

"DO NOT USE THIS UTILITY THINKING IT WILL REBUILD A FAILED DRIVE - it will have the opposite effect of making it impossible to rebuild an existing failed drive - you have been warned!"

46 minutes ago, FearlessUser said:

Do I want to preserve any assignments?

 

12 hours ago, JorgeB said:

Keep all assignments

 

46 minutes ago, FearlessUser said:

DO NOT USE THIS UTILITY THINKING IT WILL REBUILD A FAILED DRIVE

You are not trying to rebuild anything. You just want it to accept the existing drives just as they are.

Geez I can’t believe I missed that. Long week. Thanks

Finally got around to doing this. When I started a new config, disk 1 was fine, but then disk 2 gave me 1061 errors and went offline shortly after starting the array up. These drives can't be failing in sequence like this, then be fine after adding them back to the array. I replaced these SAS to SATA cables I got from Amazon 6 years ago when I built the server. The parity drives were in different spots in the server, but these 2 data drives that were failing were on the same cable. Once it was replaced, I booted up and tried another new config, and so far so good.

 

At this point the rebuild has been going for about 20 minutes, much longer than when the drives would almost immediately throw errors and fail. The rebuild says it should take around 30 hours, and all data drives are still showing active with no errors..

 

On the old cable, 3 of the 4 cables that split out to the drives were bent to a crease. Fingers crossed that's what caused the issues.

Edited January 11, 20251 yr by FearlessUser

Rebuild took a few hours longer than estimated and completed overnight. I’m back in business. Thanks @JorgeB for the help on this, super appreciated!