Hardware before any changes: mini-itx mobo with 4 sata ports and a pcie card that adds 4 additional sata ports, 3x 12TB drives, 3x 4TB drives, 2x 500GB SSDs for redundant cache

The saga:

• unraid reports SDB (a 12TB drive) as not working, array unprotected

• I ordered a replacement drive + 3 more new 12 TB drives to swap out all of my old 4TB drives

• For the first replacement, I replaced the 'bad' 12TB drive, I used the unassigned devices plugin to run the full pre-check on the new drive as the documentation suggested, this took ~3.5 days, then I rebuilt the array to the new, tested good drive, all good. I let everything stay like this for ~3 days just to ensure everything was working as normal and to catch up on some file transfers that had been pending

• In the meantime of this waiting period, out of curiosity, I took the 'bad' 12TB drive and stuck it into my normal windows computer, reformatted it, and have been using it as a video editing scratch disk for ~2 weeks as of writing this, so I don't think there's anything wrong with the original culprit 'bad' drive

• I then proceeded to follow the documentation for swapping out the 3x 4TB drives with 3x 12TB new drives, all of which went smoothly and I was left with a functional 60TB array

• Fast forward around 10 days, and now unraid is reporting that SDB is bad again, the same slot, but with the new drive that I ran the full precheck on

• Troubleshooting steps taken so far: I changed out the SATA cable, unraid reports it's still bad. I unplugged one of my SSD cache drives and moved its' power connector and SATA cable over to the SDB 'bad' drive, unraid reports it's still bad. The hard drive makes a weird pinging noise and it takes unraid a very long time to get to where I can access the web gui in this state. Unsure of what else to do, I unplugged SDB completely and re-connected the SSD cache drive, and started the array because I need to be able to use my NAS but am unsure of what to do from here.

With the expanded capacity of the overall array, I can just live with having one less drive in my array for now, but I don't know how to re-build the array without that drive and can't immediately find anything on this in the documentation. Is it possible to just tell the array to rebuild with the remaining 5 drives onto the available space on those drives?

What should I do next on the hardware troubleshooting side of things? I'm really lost at this point and worried about data loss. I have some spare old computer hardware I could probably cobble together to see if there's something wrong with the non-hard-drive hardware of the system, but I'm not sure how unraid handles swapping everything around like that. I've read that it 'should just work' but it seems like a very large risk to take if it doesn't just work.

Please help!tower-diagnostics-20251115-1100.zip

Diags doesn't show any drive getting disabled; if it happens again, post new ones before rebooting.

I will try to be civil in my response here, but please understand I am very frustrated and worried about my data.

I wish I had pulled diagnostics at every stage of this process, but I did not. As I said in my post, I had the 'bad' drive unplugged because plugging it in causes the machine to take 4 minutes to boot to the normal 1 minute, and the 'bad' drive makes terrible sounds while it sits there waiting. During this period the unraid web gui just displays 500 server error.

Attached are updated diagnostics with the 'bad' drive plugged in, as well as the audio of the drive noises while it sits there 'booting'. The export diagnostics also took a long time, hanging at the SDB drive where it continued making much louder bad noises.

You also didn't answer any of my questions...even if the diagnostics say the new drive is 'bad', my post explains why I don't know that I believe that to be the case because of the weird behavior and the old drive that it reported bad in that slot is in fact totally fine and now in use in my windows machine.

tower-diagnostics-20251116-0810.zip

unraid bad.aac

18 hours ago, albrittbrat91 said:

I wish I had pulled diagnostics at every stage of this process, but I did not.

They are not needed at every stage, just after the disk got disabled, or we can't see what happened.

The drive currently showing up as sdb is not even giving a valid SMART report. If you have already tried to use different cables with that drive, both power and SATA, and the issue remains, it suggests a bad drive, or a power problem, but it may be worth replacing them or swapping them again to see where the issue follows.

Forgot to mention, that drive noise is typically a bad drive or bad power to the drive; if it works in another PC, it suggests the latter.

Thanks for the reply. I would still really appreciate an answer to my questions in my original post though.

"With the expanded capacity of the overall array, I can just live with having one less drive in my array for now, but I don't know how to re-build the array without that drive and can't immediately find anything on this in the documentation. Is it possible to just tell the array to rebuild with the remaining 5 drives onto the available space on those drives?"
ie go from 60TB array of 6 drives to 48TB array of 5 drives since only ~30TB is in use

6 hours ago, albrittbrat91 said:

but I don't know how to re-build the array without that drive

Tools - New Config

Note that any data on the removed drive will not be kept.

I am really starting to panic and lose my mind now.

I tried to use the unraid web GUI to move files from the failed disk 2 to disk 3, this ran most of the way and then threw up a bunch of warnings and errors on some random audio files.

Going back to the main page, the drives now report more errors on them, mostly on disk 3.

I rebooted the system using the reboot from the web UI, and now disk 3 (I think) is making the same aweful noises that disk 2 was making, and unraid took like 10 minutes to boot this time.

It might be worth noting that after unplugging disk 2, unraid automatically re-assigned disk 3 to be SDB, which is what disk 2 was before that caused the initial failure (but that drive works fine in my windows machine) and then reported the replacement brand new pre-checked drive as failed as well...I'm losing my mind here.

Unraid has no way to automatically rebuild the array with a drive missing, you have to do new config and that loses all the data. So I was trying to move the data off of it first, and now I have more errors. What do I do from here?

If I move the hard drives over to a different hardware platform (different motherboard, CPU, RAM, power supply), will unraid work and remember the array from the current hardware?

Prior to all this today I was thinking there was a hardware problem, but at this point I'm really thinking it makes more sense for it to be a software problem, because it has now affected three separate hard drives, two of which are brand new, none of which I am convinced are actually 'bad' drives, and they ALL are the SDB assigned drive. Is there a way to manually assign hard drives to different letters in unraid to test this theory? Can you look into this?

PLEASE HELP!?!?!tower-diagnostics-20251118-1543.zip

Disk3 is not providing a SMART report, and if it's making noises, it could be a bad power issuer, are you using any kind of power splitters? It may also be worth retesting with a different PSU.

Okay, after several hours of troubleshooting, I believe it is that the pci-e addin card that I have has gone bad.

I disassembled my NAS, put it on a table, re-connected everything data-wise how it was in the case, and switched in a different known-good power supply with much more wattage overhead than the original one. I got the same terrible beeping noises and unraid just wouldn't boot, sat there displaying 500 server error for over 15 minutes with the hard drive beeping and clicking away.

I then unplugged disk 2 from data and power, and booted it again, now disk 3 starts making those noises, though not quite as badly, and unraid does eventually get past the 500 server error page after ~10 minutes.

I then unplugged disk 3, booted it again, and now disk 4 starts making those noises, same behavior as disk 3.

What's the common denominator? They're all plugged into the addin card.

I unplugged all of the sata data connectors from the 4 hard drives plugged into the addin card, viola, no terrible hard drive noises, the drives plugged only into power all spin up without issue, unraid boots in ~1 minute as it should, just obviously reporting most of the array drives missing.

So I have a replacement (different) pci-e addin card on the way from Amazon and should be here Sunday. I really hope that swapping that out fixes all these weird issues. I also really, really hope that the addin card didn't somehow permanently damage disk 2 by making it sit there and beep and click for probably cumulatively close to 2 hours over this whole process of trying to figure out what's going on.

Anyway, I will report back Sunday/Monday if the mail actually runs on time. Fingers crossed.

Thanks for the help!

Just to give some updates on this
-All four drives that I got from Server Part Deals went bad. Pulling them all out of the NAS and using a USB dock on my windows machine, running SMART diagnostics, two drives just wouldn't register/show up at all, and the other two had tons of bad sectors and other SMART report errors, so I returned all those drives, still pending refund because they are still in transit
-Unraid somehow kept the array up and running despite the main page clearly saying that 2 drives had failed in the 1 parity disk setup, and I was able to copy a lot of our data off of the NAS, but we definitely lost data. It took me 3 days of babysitting getting our clould backed up data off of backblaze becuause of our slow internet and their obnoxiously aggressive auto-logout behavior that cancels any in-progress downloads (I know they recommend command line, but I couldn't figure it out). After all that time and effort and getting duplicacy on Windows to decrypt the archive....it hadn't run in 3 years because our license expired and it didn't auto-renew, or email me, or anything. It's ultimately on me for not checking, but that super, duper sucks. So we definitely lost data...how much data and what specific data I don't think we'll ever actually know, but I know that at least some crucial financial data got lost.
-I ordered four different drives and went with Newegg since they could get them to me the soonest, even though it was $50 more per drive vs Amazon (but 3-5 weeks faster delivery). The drives from newegg showed up in a partially destroyed box with no foam, no padding between the drives, just four drives in ESD bags with a little bit of the paper filler stuff in the box. Despite this, all four drives passed the unraid pre-check process and are currently being 'built' into a new config array
-The first set of 4 drives also showed no problems until ~10 days into being part of the array, so I'm not holding my breath on the new ones working until it's been at least a month or two. Once the parity builds (I set it up with 2x parity drives this time), I plan on starting to copy data back to the array from my windows machine and we'll just have to go from there.

Definitely a lot of lessons learned in this process. 3-2-1 backup is the golden rule for a reason, going forward we're going to have 2 parity drives in our array, we are going to build a whole second NAS to be a hot spare that will also have two parity drives in it, and I am going to set calendar reminders to check that our backups are actually running to backblaze every month going forward.

Is there any easy, straight-forward way to setup two unraid servers to sync with each other?

You can use rsync for that, for example, I believe there are a few scripts available on the forum to do that.