I have previously started a thread on a similar issue that I never really got resolved. I thought I narrowed it down to mergerfs, and I completely removed mergerfs from my setup and I've still to this day never had more than 2 days uptime, and every single time it is doing "unclean" shutdowns are the array isn't able to stop as /mnt becomes completely unresponsive. I can't ls the directory, I can't access /mnt via file manager. Everything comes to a complete standstill.

 

This means my Plex server, and all docker containers crash/freeze although I can still access the IP:PORT of each service, and I can access the UNRAID GUI, I can't access anything that relies on the disks.

 

As you can imagine, this is frustrating, it also means that my disks have a high risk of being damaged due to the sheer amount of unclean shut downs. 

 

I can't do a diagnostics after the crash, because the entire UNRAID won't let the diagnostics write to the disk.

 

Here are my Docker settings

 

 

I use Rclone to mount my Google Drive files, and I use three commands to achieve this with one of them merging the array and the google drive file structure into one directory. These commands are as follows;

 

screen -S googlereal -X stuff $'rclone mount --config=/mnt/nvme/plexified/mounts/rclone/rclone.conf --allow-non-empty --allow-other --vfs-fast-fingerprint --vfs-read-chunk-size 128M --vfs-cache-max-age 8706h --vfs-cache-mode full --cache-dir /mnt/ssd/plexified/mounts/googlecache/ --vfs-cache-max-size 650G --dir-cache-time 8760h --vfs-fast-fingerprint --vfs-refresh --vfs-read-ahead 128M --drive-impersonate [email protected] googledecrypted: /mnt/ssd/plexified/mounts/google/'

screen -S moviesreal -X stuff $'rclone --config=/mnt/nvme/plexified/mounts/rclone/rclone.conf mount --allow-other --allow-non-empty moviesunion: /mnt/ssd/plexified/moviesource/Movies/'

screen -S unionreal -X stuff $'rclone --config=/mnt/nvme/plexified/mounts/rclone/rclone.conf mount --allow-other --allow-non-empty newunionize: /mnt/ssd/plexified/secret/'

 

I have also attached diagnostics from when the system is running perfectly, although I am not sure these will be much use. How can I capture logs when the server freezes if I can't run diagnostics?

 

I am having a hard time troubleshooting this issue. Is there anything obvious that stands out as a probably cause? Any help is greatly appreciated.

Here are diagnostics from the fresh booted system.

subzero-diagnostics-20250228-2043.zip

See if you can get the syslog when that happens:

cp /var/log/syslog /boot/syslog.txt

also outputs from

df -h

and

free -h

 

On 3/1/2025 at 7:44 AM, JorgeB said:

See if you can get the syslog when that happens:

cp /var/log/syslog /boot/syslog.txt

also outputs from

df -h

and

free -h

 

 

It happened again, I had the logs mirror on the flash drive and here they are.

 

syslog.bin syslog-previous.bin

Edited March 2, 20251 yr by thatja

Assuming it was in this period, there's nothing logged:

 

Mar  2 02:18:35 SubZero kernel: docker0: port 3(vethfe98e30) entered blocking state
Mar  2 02:18:35 SubZero kernel: docker0: port 3(vethfe98e30) entered forwarding state
Mar  2 04:30:01 SubZero root: Checking for patches for OS version 7.0.1
Mar  2 04:30:01 SubZero root: No patches found
Mar  2 12:20:07 SubZero shutdown[143696]: shutting down for system reboot
Mar  2 12:20:07 SubZero init: Switching to runlevel: 6
Mar  2 12:20:07 SubZero rc.6: Running shutdown script /etc/rc.d/rc.6:
Mar  2 12:20:07 SubZero init: Trying to re-exec init
Mar  2 12:20:07 SubZero rc.6: Saving system time to the hardware clock (UTC).
Mar  2 12:20:07 SubZero rc.6: Creating system time correction file /etc/adjtime.
Mar  2 12:20:07 SubZero rc.6: /sbin/hwclock --utc --systohc
Mar  2 12:20:09 SubZero rc.local_shutdown: Waiting up to 90 seconds for graceful shutdown...

 

Did you save the outputs asked?

Also it looks like /mnt became unresponsive at 04:30 as that's when logging stops completely until I issue the shutdown command.

 

I forgot to run those commands you mentioned before I will do that next time it happens

On 3/1/2025 at 7:44 AM, JorgeB said:

See if you can get the syslog when that happens:

cp /var/log/syslog /boot/syslog.txt

also outputs from

df -h

and

free -h

 

 

df -h just hangs on the SSH terminal.

 

 

free -h

 

Edited March 6, 20251 yr by thatja

I managed to copy the syslog off the flash drive though, here it is.

 

 

syslog.txt

If you can still access the CLI, try:

 

df -h /

and if this works, try

df -h /mnt/user

and see if that one doesn't

16 hours ago, JorgeB said:

If you can still access the CLI, try:

 

df -h /

and if this works, try

df -h /mnt/user

and see if that one doesn't

 

df -h / shows (image below) but df -h /mnt/user hangs

 

Recommend trying a couple of other things, go to Settings - Global share settings and set the Number of fuse File Descriptors to the max, and also enable this:

https://docs.unraid.net/unraid-os/release-notes/7.0.0/#excessive-flash-drive-activity-slows-the-system-down

On 3/7/2025 at 10:19 AM, JorgeB said:

Recommend trying a couple of other things, go to Settings - Global share settings and set the Number of fuse File Descriptors to the max, and also enable this:

https://docs.unraid.net/unraid-os/release-notes/7.0.0/#excessive-flash-drive-activity-slows-the-system-down

 

I've done both of these just now.

 

I'll report back in a couple days/when it next crashes.

On 3/7/2025 at 10:19 AM, JorgeB said:

Recommend trying a couple of other things, go to Settings - Global share settings and set the Number of fuse File Descriptors to the max, and also enable this:

https://docs.unraid.net/unraid-os/release-notes/7.0.0/#excessive-flash-drive-activity-slows-the-system-down

 

After 3 days of uptime, same issue has occurred.

 

Cannot ls /mnt or access it via file manager, ssh etc. UNRAID UI is still completely accessible.

 

Edited March 12, 20251 yr by thatja

I have no idea if this us significant or not.

 

I'm afraid that without anything logged I'm out of ideas, you would need to try and find which app/service is causing the issue by an elimination method

12 hours ago, JorgeB said:

I'm afraid that without anything logged I'm out of ideas, you would need to try and find which app/service is causing the issue by an elimination method

 

Could it be hardware?

Could still be hardware related, but looks more like a software issue to me.

Having the exact same issue, looking at your previous thread my diagnostics even freeze on the exact same line as yours does..

I've found similar threads, but this is the only one I have seen with almost the EXACT same symptoms as mine:

 

df -h hangs, cant access anything on /mnt/, docker containers stuck and cant be stopped/removed/killed (shows execution error in webui), etc.

WebUI also shows multiple cores pinned at 100%, but htop doesn't show that much usage. It's driving me insane.

Syslog also shows no relevant info or errors. The only way to make the server usable again is to hard reset it, which obviously isn't good to be doing every day at this point.

 

Mine can hang at (seemingly) random, but for me it also happens when I start loading lots of content on one of my docker containers, LANraragi.

I can easily recreate the issue without having to wait on random chance by spam opening galleries on there for about a minute until LANraragi's webui stops serving me images, which leads to /mnt/ hanging and becoming inaccessible.

 

I've replaced my docker img file and reinstalled all my containers from scratch, but the problem persists. All my permissions look fine. 

What docker containers are you running?

 

Anyways, I am at my wit's end with this issue.

Currently running a memtest to see if it's RAM related, and my next step is running extended SMART tests on all my drives if I run out of other, less time-consuming ideas.

I'd attach diagnostics now, but like I said earlier it keeps freezing on a certain line, the "sed" command that ends with /shares/appdata.cfg 2>/dev/null

 

I'll grab diagnostics from a clean bootup after I finish running memtest, and also check the flash drive for any partially completed diagnostics.

Edited April 20, 20251 yr by bensored

3 hours ago, bensored said:

I can easily recreate the issue without having to wait on random chance by spam opening galleries on there for about a minute until LANraragi's webui stops serving me images, which leads to /mnt/ hanging and becoming inaccessible.

Can you post in detail exactly what to do to see if I can replicate?