beatmurph

OK, I followed the steps and everything worked great. Thanks again, yet another lesson learned

Hello, So I did as you asked and parity successfully rebuilt without issue. After that completed I set out to swap out the disabled Disk 11: Stopped the array Removed the current drive from the Drive 11 slot Added the previously issue free Parity drive to the Drive 11 slot The system started a process - "data rebuild" I believe? After completing that process Drive 11 is listed as "Unmountable: wrong or no file system" despite the file system showing as xfs. Also when I hover over the green dot next to Drive 11 I get the message "Normal operation, device is active". Did I do something wrong in trying to replace Disk 11? Diagnostics attached if needed. calculon-diagnostics-20250717-1003.zip Thanks again

calculon-diagnostics-20250713-1135.zipThank you. I opened the case to double checked all the connections and there was definitely one drive that didn't seem well connected (Disk 11 I'm pretty sure). I wasn't able to feel an issue with the HBA connection, but regardless everything seems to be proceeding properly now. The parity-sync is currently at 8% without a single read error so I think that may have (fingers crossed, knock on wood) fixed the issue. I have included the current logs if you're curious, but for now I think we're good. Thanks again.

Hello, I have a 2 Parity set up on my server. Close to 2 weeks ago I had two disks get the infamous red X's at the same time. I re-added one of them in it's original spot and it seemed to rebuild without issue (I'm not sure exactly which disk that was) and it has remained green for about a week. The second disk (Disk 11) didn't succeed so I ordered a new drive, and have been emulating it since. The replacement drive is bigger than the current parity drives, so the plan was to do some shuffling of drives. The drives arrived so today (about 6 hours ago maybe?) I did the following: Shut down the server and added the new drive, then rebooted Stopped the array and swapped out the newly added drive with Parity 1 Started the array so it would rebuild Parity 1 I just noticed two things: Parity sync is still stuck at .5% I also noticed Disk 2 is not showing a temperature and has a growing list of read errors I checked the logs and there are in fact tons of Disk 2 read errors So now I have the following: Disk 11 seems to be bad after failing the first attempt to rebuild Disk 2 (which I now suspect is the one I "successfully" rebuilt) seems to actually be bad despite showing green Parity 1 rebuild seems to not be stalled Have a screwed my parity and lost tons of my data? It seems disk 2 is actually bad along with disk 11 and I compounded the problem by attempting to rebuild Parity 1 . . . I would love your advice. Parity Rebuild is still running but I downloaded the logs and attached them here. Thank you calculon-diagnostics-20250712-1836.zip

Thank you Turl & JorgeB. I think I have gotten to a good place. I did some research on this topic and found a thread indicating that while it seemed this would be safe there was also a threat of corruption as you mention. I wasn't sure if it was best to take that chance or the risk of another drive unmounting during a rebuild, but ultimately decided to go with a rebuild of the 2 unmountable disks. That rebuild just finished about 30 min ago. I also added some disks I did preclear on over the weekend. It seems that worked, as they are showing as available space on the array instantly, except for one of them. I'm not sure why, but that one is in the process of being cleared now *shrug*. Again, I can't thank you both enough for this help. Every time I go through one of if these fiascos I learn some new stuff and my server comes out a bit more stable. I can't say it's ever fun in the moment, but your support really helped ease the process.

OK, so I pulled out the box with all the odds and ends from my server build and found enough cables to separate the drive power down to 4 drives per power cable. The server has been rebooted and all the drives are recognized. The 2 drives that were disabled Sunday night both returned extended SMART scans with no issues. Since I've not made any changes to server content and had the array and docker off most of the time, I'm curious if there is a way to reintegrate those drives without either rebuilding them or parity? Or is that unavoidable?

OK, so it seems I have one SATA cable coming out of my modular 750 gold power supply that ends in with a 3-way SATA power daisy chain. I then added a 4-way SATA power splitter (which I had modded to a 5-way) to each one of those. So it seems for the last 6 years that I've been running between 10 and 15 drives off of a single SATA cable . . .

calculon-smart-20250128-0714 (1).zip Thanks again. I'll be taking the server apart in a bit to investigate what's going on in there. Attached are the SMART reports. Both indicated they passed without error, though I don't know how to distinguish the normal from the scary in these reports so I always appreciate an educated set of eyes giving another look.

Thank you Turl, I'll investigate these things. This was my first server build (second anything) and it was a bit overwhelming at the time, especially with the mix of new and used parts. I wont be able to say with certainty what I did, but now that you have me thinking about it, I'm thinking I do have a 4 SATA power splitter cable that I added an addition connector to so one cable would hit all 5 drives in the bay. I actually also did a cable mod to make them all line up perfectly with each drive as well as remove the 3.3v power. Now that the there are more drives than ever in the server it's very likely I've overloaded it. I have 4 drives that have nearly completed preclear. When they that finishes I should be able to open the server and take a look to be sure. Thanks again.

So I woke up this morning and now I have two more red X disabled drives. I immediately stopped the array, then mounted the drives as unassigned to check that the data contents were still there and they are. I then began extended smart scans of the 2 drives. I'd estimate prior to my recent issue that I've had red Xs approximately 5 times since I built my server in 2019. I have no idea if that would be considered a normal problem rate, but given the 5 instances I've had in the last 2 weeks I believe I'm having some serious hardware issue based on the many times I've read about it here. My issue is that I don't know where to begin. Is it the power supply, the cables for the supply, the PCI SAS breakout board, or the cables for that? Is there anyway to troubleshoot where I could truly know? Is it best to just replace it all and hope everything is fine, or is there systematic sensible approach? Can a power supply be tested? At any rate. I'm sorry, this update is more me expressing some frustration over the situation as I felt I just finally got my server operational again this weekend, only to wake up on a Monday morning on the verge of suffering ANOTHER catastrophic loss. I know the hardware issue is likely outside the scope of this forum, so I don't expect you to help with that if you think that may be my actual issue, but if you can offer any advice or guidance I definitely welcome it. I'm also of course willing to make a new post if that's needed. Thank you again for all your help so far, as well as any help you'd be able to offer here. calculon-diagnostics-20250127-0947.zip

Diagnostics as requested. calculon-diagnostics-20250124-1024.zip

Yes and no. I did stop Docker in settings before beginning the Mover; however, there was something I needed to access after the first 6 hours, so I did restart the Docker. I didn't want to but I had no idea how long it was going to go at that point. Mover stopped, Downloads share updated, Docked disabled, rebooted, and diagnostics attached as requested. calculon-diagnostics-20250123-1152.zip

The mover is still going after 14 hours. I'm pretty sure it's never taken more than a few hours in the past. Does this sound like a crazy amount of time, or within reason? Is it possible there is a problem? I don't know if it's relevant, but typically the largest files that have been moved in the past are media files from the download share (cache) that are moved to shares that reside on the array. Since the drive failures started a few weeks ago, I haven't been doing any downloading. I've really only been running dockers on occasion to access what is currently on the server. A have attached current diagnostics. I also noticed these took much longer than normal to complete. 2-3 min vs 20-30 seconds. To be clear Mover is still running. calculon-diagnostics-20250123-1033.zip

Haha, more than fair. Thank you for all the help. I will work on this and get back to you.

I don't recall exactly, but I believe since Saturday or Sunday (~96hrs). Since you've mentioned it, I do recall thinking my RAM usage was high (I think 50%?) Glances has 2 host paths: /mnt & /var/run/docker.sock Plex Media Server has a AppData Config Path: /mnt/cache/appdata/Plex-Media-Server/ Plex also has a Host Path 3: /mnt/user but no host path 1 or 2??

Latest diagnostics attached calculon-diagnostics-20250122-1911.zip

The dockers were "working" as in they would load and I could do some actions. The reason I noticed the cache drive was missing is because I had began trying to download some of the missing files from my lost drive and I was receiving an error regarding not being able to create the necessary path. Ironically, one of my dockers wont load now that I've added the drive back in. Is it possible there is a conflict with the appdata between the backups I've been using the last few days and the appdata that would be on the newly restored cache drive?

All Dockers have host path of /mnt/cache... The only special thing I can think of that I've ever done with RAM is that I followed an IBRACORP guide a long time ago for transcoding media using RAM

No, I can't explain that one. I only discovered the issue after sending over the diagnostics. The drives were in the server, but not assigned to the array. Is that possibly the reason?

OK, I discovered that I didn't have a cache drive set up. I didn't think of it while re configuring the array, and it only just occurred to me now while looking into this issue. I added the pool back in and now the settings match and the errors are gone. That was silly. I'm sorry to waste your time on that one.

Diagnostics attached calculon-diagnostics-20250122-1804.zip

It seems you're correct. I'm seeing that also. In the Shares tab under the Storage column both appdata and system shares seem to indicate they are to be moved from Array to Cache with the arrow pointing that direction; however, when I go into each share's settings the Primary and Secondary storage for both are set to "Array" with the mover action set to "Array to Array". Would you say that they should be set to cache for at least the Primary storage? I do have a downloads folder that is listed under the Storage column of the Shares tab as "Cache", but when I go into the share setting Primary storage is set to "Array" and Secondary set to "None". I don't know if I'm reading these wrong, but I'm surprised as the settings don't seem to match what's indicated under the Shares tab

I do have the backup plugin set up and used it successfully (after some pains) for restoring appdata during this rebuild. I'm not sure about the system share, but the only files I see in there are docker, which I believe the backup plugin also covers since I used it to rebuild from templates?

Now that I think about it, these are both set to move from Array to Cache. Would that be the reason?

While I wait for the results of the extended SMART test, one other issue I'm seeing now that the rebuild is complete is that 2 of my shares have a warning (orange triangle: "some or all files unprotected"). These are the appdata and system shares. I'm not sure what to post for troubleshooting, but they do both have the same warning in the SMB Security Settings: Security is set to Public with the accompanying warning "Windows may require a valid User to be defined even for Public shares. See Help.". Is it possible this is why they are showing as possibly unprotected?