Hello, anyone who happens to click on this and takes the time to read and help me out! Any thoughts or advice is very much appreciated.

 

So, my situation in short: I've had an unraid server chugging along mostly fine for a few years now. I think I had to replace a drive here or there, but it's mostly been smooth sailing. I use my server pretty much exclusively for media storage and plex.

 

Recently (about a month back or so), I started having some issues with some drives becoming disabled and having to rebuild. I figured it was another failing drive and replaced it accordingly, but the issue popped up again a week or two later with the replaced drive and I started wondering if this was actually a sata cable issue (technically a 8087 split cable).

 

TBH, this issue struck at a time where I was thinking of down-sizing my server anyways. (Kind of an aside, but when I first built it, I "future proofed" with a DAS and had some wild idea in my head that I would eventually have 30 drives chugging along... Well, it's a few years later, and I've done a complete 180 on that line of thought. Never ended up using all the space I gave myself, moving has made me appreciate more minimal considerations, and facing server issues makes me appreciate the idea of minimizing the parts that can fail at any given time.)

 

I switched the cable attached to the drive that had been disabled then and seemed to strike gold because the drive I swapped cables to was struck by the disabled drive issue next. So I thought I was on the right track and swapped out the 8087 split cable with another one I had on hand... and that seems to have broken everything.

 

Before the 8087 split cable swap, I had 2 disabled drives being emulated. Afterwards, I think I have 3 (it's hard to tell because the Unraid UI isn't marking it as such now, but I think I saw the error pop up previously before a restart -- specifically, disk 5, 14 (marked as disabled/emulated), and disk 1 (?). Further, the array doesn't seem to want to start anymore. I turned the server off for a few days after I salvaged my most important files and was awaiting a drive I ordered to hopefully try salvaging some other files (arrived today)... but now I'm finding that the server is stuck on "Starting..." (I tried manually restarting, but it's stuck on the same thing).

I attached the diagnostics in case anyone can take a look and tell me if I'm missing something obvious. I would like to be able to get my server going again so I can do some salvage. My end goal however is to format everything, pull out the older shucked drives (4-6 drives, I believe) and start a new fresh array -- but before that, I want to make sure I'm not missing some glaring hardware issue that I need to address first??

Thank you again!!

starryskynet-diagnostics-20250508-1624.zip

Edited May 8, 20251 yr by Pluvichor

Disk5 disabled and disk14 is invalid, disk1 is OK but there are read errors, check/replace cables for that disk and post new diags.

I'll try that and report back.

Before that though, I'm experiencing some pretty strange behavior from Unraid. I tried starting the server just now and one of my parity drives was listed as a smaller size than is correct (wrong -- and the correct size and drive serial listed below it). I tried initiating a restart through the UI and the server hasn't come back online 8+ minutes later. I'll do a manual restart shortly, but I've never encountered this kind of delayed response from unraid before.

 

EDIT: Server UI popped back up shortly after posting this update. Now the parity drive is missing entirely instead of being reported wrong size. I'll try changing cables around now.

Edited May 9, 20251 yr by Pluvichor

I swapped the sata cables of parity2 and disk1 to their neighboring disks (parity1 and disk2 respectively) and now all four disks are missing from the UI window. Attaching the diagnostics. I'll be tinkering with the cables to see if they can come back by reinserting/replacing/etc...

starryskynet-diagnostics-20250509-0126.zip

You won't be able to start until all disk are present, check also power cables, especially if using splitters.

I unplugged/replugged all the sata cables to the mother board and returned the previously swapped cables to disk1 and parity2 and it no longer showed the four missing drives -- just the disabled disk5 and disk14 as before. (First diagnostics.)

 

THEN, I tried starting the array and when I did that, the four disks (parity1, parity2, disk1, and disk2) disappeared from the UI list. (Second diagnostics.)

 

They are incidentally daisy chained to one splitter so I'll try replacing that splitter (if I have a spare one somewhere around), or moving around power cables.

EDIT: I hadn't looked taken a closer look at the power cabling in a good long while. 5 of the disks are daisy chained -- including one of the disabled ones, disk5. I believe I had an issue a year or two back with daisy chaining too many disks causing an issue so I'm going to split that up into two and three chains now, see if that helps.

starryskynet-diagnostics-20250509-0143.zip starryskynet-diagnostics-20250509-0150.zip

Edited May 9, 20251 yr by Pluvichor

14 minutes ago, Pluvichor said:

They are incidentally daisy chained to one splitter so I'll try replacing that splitter

If using a SATA->SATA splitter you not want to split it more than 2 ways, whereas with Molex->SATA splitters then splitting it 4-ways is normally fine.   

Err, slight hiccup. I added a new splitter to the PSU and connected parity1 and disk1 to it. The big issue is that now, the server doesn't start when I try powering on.

 

Quote

If using a SATA->SATA splitter you not want to split it more than 2 ways, whereas with Molex->SATA splitters then splitting it 4-ways is normally fine.   

I had no idea! I'll take a look at how the rest of the disks are split, but I'm pretty sure they're mostly daisy chained via sata with one or two molex connections only. I'll see what I can do about that now...

Tinkering with the PSU has me wondering if my initial problems (disabled disks) were power-related all along. My problems began to really kick off a month ago when I shrunk my array and moved data disks from the DAS into the main NAS. Maybe the DAS was off-setting some of the power issues the main NAS was straining under due to daisy chaining too many disks? I'm tempted to get the DAS going again and plug some disks back into it to see if that helps.

EDIT: Yep, not good. My daisy chains look like this: 3, 4, 4, 3, 2 (these last two split from the 5 which reduced it to 3). I'm going to try a different splitter just in case (I believe I used one which did not come with the PSU -- just grabbed it from a pile, but I checked my PSU box and it came loaded with more).

ALSO, sorry for my ignorance, by molex to sata, do you mean that the daisy chain begins with a connection to the PSU, right? I panicked at first because I wasn't sure what a SATA-> SATA splitter was and thought all my connections were that, but googling images is telling me different things.

 

EDIT2: Replacing the splitter now lets the server power on. Had no idea you couldn't just use whichever, but I compared the heads that plug into the PSU, and the "missing" prong were in different places. Will report back with an update shortly...
 

Edited May 9, 20251 yr by Pluvichor

I'm reporting my updates live so excuse the strange tense of this post.

First, Unraid UI booted up fine and is reporting no missing disks, only disk5 and disk14 with their disabled status (this will be first diagnostic attached).

I hit START, and it's taking longer to start up than usual but after about a minute~, it starts up and begins a data rebuild. That's progress -- but now a very big issue: Disk1, Disk5, and Disk14 are showing this error message: "Unmountable: wrong or no file system." (this will be the second diagnostic attached).

Any advice for how to proceed from here? I'm salvaging the files I can access in the meantime, but should I data rebuild Disk 14 and Disk5, and then try a new configuration to see if the data on Disk1 is fine? Don't know if that makes sense, I'm working off the idea that my issues were power related all along and the drives are actually fine, but maybe I'm missing something obvious?

 

EDIT: Figured out how to try repairing file system corruption in maintenance mode. Did it (aside from the disks above, Disk2 apparently also had some corruption to fix), and that seems to have fixed my unmountable issues. Will try re-building the disabled drives now.

 

 

starryskynet-diagnostics-20250509-0255.zip starryskynet-diagnostics-20250509-0303.zip

Edited May 9, 20251 yr by Pluvichor

No disk errors so far, the filesystem issues could be the result of all the previous errors, check filesystem on all thread disks.

Just a quick update because I'm still going to have two days or so of disk re-building, but I think my major issues have been solved. The answer does seem to have been the power/splitters causing my disks to randomly not be recognized, corrupting file systems (I think when I was restarting when unraid was trying to rebuild). I'll hold off until the rebuild is complete to mark as solved, but thank you everyone that helped!!

A minor issue I noted in the meantime, my dockers don't seem to be working properly. When I go to the Docker tab, I just see a yellow exclamation and "Docker Service failed to start." It's not super important just now, but any pointer about how to fix that after rebuilding would be appreciated. Thank you.

25 minutes ago, Pluvichor said:

but any pointer about how to fix that after rebuilding would be appreciated.

Docker image is corrupt, recreate it:

 

https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file
Then:
https://docs.unraid.net/unraid-os/manual/docker-management/#re-installing-docker-applications
Also see below if you have any custom docker networks:
https://docs.unraid.net/unraid-os/manual/docker-management/#docker-custom-networks