January 22, 20251 yr And your syslog has a lot of this Jan 22 17:16:54 Calculon shfs: share cache full which would be another symptom of trying to write to non-existent cache
January 22, 20251 yr Further examination of those previous diagnostics shows you did not have a mounted cache. Do any of your containers work?
January 23, 20251 yr Author 13 minutes ago, trurl said: Do any of your containers have host paths that refer to /mnt/cache... instead of /mnt/user... ? All Dockers have host path of /mnt/cache... The only special thing I can think of that I've ever done with RAM is that I followed an IBRACORP guide a long time ago for transcoding media using RAM
January 23, 20251 yr 3 minutes ago, beatmurph said: All Dockers have host path of /mnt/cache... If your dockers were started without cache, then they would have created the path /mnt/cache in rootfs (RAM). So anything they did was filling rootfs and will be gone when you reboot like anything else in RAM.
January 23, 20251 yr 12 minutes ago, trurl said: Post new diagnostics so I can see what changed after you added cache back in.
January 23, 20251 yr Author 5 minutes ago, trurl said: Do any of your containers work? The dockers were "working" as in they would load and I could do some actions. The reason I noticed the cache drive was missing is because I had began trying to download some of the missing files from my lost drive and I was receiving an error regarding not being able to create the necessary path. Ironically, one of my dockers wont load now that I've added the drive back in. Is it possible there is a conflict with the appdata between the backups I've been using the last few days and the appdata that would be on the newly restored cache drive?
January 23, 20251 yr The diagnostics you posted on jan 12 had a cache drive. How long do you think you were running without cache?
January 23, 20251 yr 13 minutes ago, beatmurph said: All Dockers have host path of /mnt/cache... All host paths? Including /mnt/cache/appdata?
January 23, 20251 yr Author 2 minutes ago, trurl said: How long do you think you were running without cache? I don't recall exactly, but I believe since Saturday or Sunday (~96hrs). Since you've mentioned it, I do recall thinking my RAM usage was high (I think 50%?) 4 minutes ago, trurl said: All host paths? Including /mnt/cache/appdata? Glances has 2 host paths: /mnt & /var/run/docker.sock Plex Media Server has a AppData Config Path: /mnt/cache/appdata/Plex-Media-Server/ Plex also has a Host Path 3: /mnt/user but no host path 1 or 2??
January 23, 20251 yr You may have to restore your appdata backup. First, let's try to get appdata and system shares files all back on cache where they belong. Disable Docker in Settings, go to Main - Array Operation, click Move, wait for it to complete. Then post new diagnostics
January 23, 20251 yr Author Just now, trurl said: I'm going to eat, be back later. Haha, more than fair. Thank you for all the help. I will work on this and get back to you.
January 23, 20251 yr Author The mover is still going after 14 hours. I'm pretty sure it's never taken more than a few hours in the past. Does this sound like a crazy amount of time, or within reason? Is it possible there is a problem? I don't know if it's relevant, but typically the largest files that have been moved in the past are media files from the download share (cache) that are moved to shares that reside on the array. Since the drive failures started a few weeks ago, I haven't been doing any downloading. I've really only been running dockers on occasion to access what is currently on the server. A have attached current diagnostics. I also noticed these took much longer than normal to complete. 2-3 min vs 20-30 seconds. To be clear Mover is still running. calculon-diagnostics-20250123-1033.zip
January 23, 20251 yr appdata still has files that weren't moved off disk5. Possibly these are duplicates. Mover won't overwrite files, so if something exists in both places they won't be moved. Or, maybe even more likely 15 hours ago, trurl said: Disable Docker in Settings Looks like you skipped this important detail. 12 minutes ago, beatmurph said: typically the largest files that have been moved in the past are media files from the download share (cache) that are moved to shares that reside on the array. That share is one I was going to get to later. You currently don't have that share configured to be moved at all. It has files on the array, but new downloads will go to cache and stay there the way it's currently configured. system share is all on cache where it belongs and is configured to be so that's all good. No other shares have files on cache or are configured to be moved to cache, so it doesn't look like there is anything left for mover to do. 19 minutes ago, beatmurph said: To be clear Mover is still running. Jan 23 02:00:02 Calculon crond[1897]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null Looks like it exited. Usually status 0 means normal exit, so probably some error. Next steps: Fix Downloads share so it will be moved to the array. Disable Docker in Settings. Reboot and post new diagnostics.
January 23, 20251 yr Author 21 minutes ago, trurl said: Looks like you skipped this important detail. Yes and no. I did stop Docker in settings before beginning the Mover; however, there was something I needed to access after the first 6 hours, so I did restart the Docker. I didn't want to but I had no idea how long it was going to go at that point. Mover stopped, Downloads share updated, Docked disabled, rebooted, and diagnostics attached as requested. calculon-diagnostics-20250123-1152.zip
January 24, 20251 yr Run mover again, it should move Downloads and TV from cache to array. And hopefully it will move appdata from disk5 to cache. Post new diagnostics after mover completes.
January 27, 20251 yr Author So I woke up this morning and now I have two more red X disabled drives. I immediately stopped the array, then mounted the drives as unassigned to check that the data contents were still there and they are. I then began extended smart scans of the 2 drives. I'd estimate prior to my recent issue that I've had red Xs approximately 5 times since I built my server in 2019. I have no idea if that would be considered a normal problem rate, but given the 5 instances I've had in the last 2 weeks I believe I'm having some serious hardware issue based on the many times I've read about it here. My issue is that I don't know where to begin. Is it the power supply, the cables for the supply, the PCI SAS breakout board, or the cables for that? Is there anyway to troubleshoot where I could truly know? Is it best to just replace it all and hope everything is fine, or is there systematic sensible approach? Can a power supply be tested? At any rate. I'm sorry, this update is more me expressing some frustration over the situation as I felt I just finally got my server operational again this weekend, only to wake up on a Monday morning on the verge of suffering ANOTHER catastrophic loss. I know the hardware issue is likely outside the scope of this forum, so I don't expect you to help with that if you think that may be my actual issue, but if you can offer any advice or guidance I definitely welcome it. I'm also of course willing to make a new post if that's needed. Thank you again for all your help so far, as well as any help you'd be able to offer here. calculon-diagnostics-20250127-0947.zip
January 27, 20251 yr How are these drives powered? Any power splitters? Ideally no more than 4 drives per PSU cable. Don't bundle data cables to make things "neat". Make sure there is some slack in the cables. In addition to knowing whether the physical disk contents are OK, since you have dual parity, it should be able to emulate both missing disks and you could check the contents of the emulated disks as well. If you decide to rebuild instead of New Config the disks back in, the contents of the emulated disks will be the contents of the rebuilt disks. SMART reports for both look OK. 31 minutes ago, beatmurph said: extended smart scans of the 2 drives. Let us know how that goes
January 27, 20251 yr Author Thank you Turl, I'll investigate these things. This was my first server build (second anything) and it was a bit overwhelming at the time, especially with the mix of new and used parts. I wont be able to say with certainty what I did, but now that you have me thinking about it, I'm thinking I do have a 4 SATA power splitter cable that I added an addition connector to so one cable would hit all 5 drives in the bay. I actually also did a cable mod to make them all line up perfectly with each drive as well as remove the 3.3v power. Now that the there are more drives than ever in the server it's very likely I've overloaded it. I have 4 drives that have nearly completed preclear. When they that finishes I should be able to open the server and take a look to be sure. Thanks again.
January 27, 20251 yr Also, Molex - SATA power splitter is more reliable than SATA - SATA power splitter.
January 28, 20251 yr Author calculon-smart-20250128-0714 (1).zip Thanks again. I'll be taking the server apart in a bit to investigate what's going on in there. Attached are the SMART reports. Both indicated they passed without error, though I don't know how to distinguish the normal from the scary in these reports so I always appreciate an educated set of eyes giving another look.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.