carr

Wish I could, but the only docker container that's running is a live mail server. It can be down for hours, maybe a day, but not for weeks at a time. Even if disabling docker did turn out to solve the problem, that would just mean I'd have to switch from Unraid to something else. NAS services are not as critical (yet), except as backing for that container. Any thoughts on whether that older VirtIO iso existing on the system could be related? Does it affect anything other than the (disabled) VM? Is there any preparation I can do before the next crash hits? Any monitoring/logging i can set up? Is there anything I can do while the next round is in effect, and I still have a working shell? (See list of working/hanging commands tried in my initial post.) Any thoughts on how to test for hardware problems, or any other next steps? Is there any reason to think upgrading to 6.12.15 or 7.0 might help? I wanted to figure out the root cause here before tackling that, unless there are reasons to believe it would help.

Also, attached complete diagnostics pulled after the last restart, per suggestion from @itimpi can-diagnostics-20250125-1309 (after reboot).zip

Great, thanks! Two syslog files attached, timeline explanation below. 1) Jan 10 14:43:37: Startup after last time this happened Jan 10 14:44:28: Startup completed, normal operation resumes Jan 16 14:38:50: Error spam from nginx, probably a stale browser tab left open for the Web UI Jan 16 -> Jan 24: Various usage Jan 24 21:24:14: Last syslog entry (from normal activity) before unresponsiveness Jan 25 04:30:00: Last proof of life (email delivered), unresponsiveness starts some time between now and 10:00 Jan 25 10:00:00: Unresponsiveness discovered Jan 25 11:22:21: First syslog entry after unresponsiveness, troubleshooting starts, various logins and Web UI errors Jan 25 12:08:52: Syslog extracted 2) Jan 25 13:07:24: Startup after giving up and running shutdown -rn Jan 25 13:08:20: Startup completed, normal operation resumes 1-syslog-before-and-during-last-downtime.txt 2-syslog-after-reboot.txt

I have the full contents of /var/log/syslog both during the recent problem, and the full contents of it after the subsequent reboot. Will enabling the server and waiting for the next crash produce anything else?

Everything related to the array (shares and container access to it) is already down at that point, @JorgeB There has not been any user interaction or system load around the time things stop working. Typically I'll realize there's a problem, and login to the Web UI to investigate. Hitting the VM tab will then additionally take down the Web UI for a while. (Only on that hostname though. It's still responsive by browsing to the IP instead of the hostname, for example.)

Every couple of weeks, my Unraid box becomes partially unresponsive. SYSTEM Unraid 6.12.14 i7-7700K / 32gb ram Disk 1: 16tb Disk 2: 16tb Parity: 16tb SERVICES 1 array with a bunch of shares 1 active docker container 3 disabled docker containers 1 disabled Windows VM PLUGINS Dynamix System Temperature User Scripts SYMPTOMS IN UNRESPONSIVE STATE The Web UI works, until i open the VM tab. This hangs the web session for a long time. (10-30mins) The Web UI is still accessible if i connect using a different hostname/IP, until visiting the VM tab again. SSH'ing in works fine, I have a functional shell. No shares can be accessed over smb. Attempting to stop the array through Web UI -> Main -> Stop array sends a POST that hangs for 3 minutes and ends with an nginx 504 timeout. The running container (Docker mailserver / DMS) responds, provides logs, and can be SSH'ed to, but spams errors apparently related to interactions with mounted files (mail storage, certs..) on the host. Attempting to stop the container has no effect through the Web UI. Attempting to stop through SSH: docker stop bd330867953d "Error response from daemon: cannot stop container: bd330867953d: tried to kill container, but did not receive an exit event" I can run smart self-tests on all disks (1, 2, parity) through the webUI. They all complete without errors. Parity checks and syncs have never had any errors. The following commands will hang indefinitely (although ctrl-c works): df lsof ls /mnt ls /mnt/user All emcmd commands shutdown -r (hangs on "Trying to re-exec init") The following commands work fine: ls / ls /var ls /usr ls /run ls /mnt/disk1 ls /mnt/disk2 shutdown -rn top and htop works. No processes are consuming a noticeable amount of CPU. free -h works. Memory usage is low and there's a lot free. LOGS/DIAGNOSTICS /var/log/syslog has no entries in the hours before or after the problem starts. The last entry before is me doing things the previous day, and the first one after is me logging in. The only errors after logging in, is the nginx errors when trying to open the VM tab: Jan 25 11:32:15 can nginx: 2025/01/25 11:32:15 [error] 8658#8658: *4494722 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.1.92, server: , request: "GET /VMs HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "can", referrer: "http://can/Docker" Jan 25 11:33:43 can nginx: 2025/01/25 11:33:43 [error] 8658#8658: *4494893 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.1.92, server: , request: "GET /Dashboard HTTP/1.1", subrequest: "/auth-request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "can", referrer: "http://can/VMs" Jan 25 11:33:43 can nginx: 2025/01/25 11:33:43 [error] 8658#8658: *4494893 auth request unexpected status: 504 while sending to client, client: 192.168.1.92, server: , request: "GET /Dashboard HTTP/1.1", host: "can", referrer: "http://can/VMs" I can generate partial diagnostics, but it hangs on commands like lsof and df, so many files in the output are empty or missing. /var/log/messages is empty. dmesg has nothing in the hours before the problem starts. What could be happening here? It seems like the disks are accessible, but if i'm interpreting the evidence above correctly, everything that touches the array fails? (Why the VM tab though?) If i shutdown -rn, the box comes back up happy as ever, and Unraid starts a full parity check (without errors). The box then runs for another week or two, and this happens again. The only thing i can think of that I've customized related to the filesystem, is downgrading the VirtIO version to get an old VM to run. /mnt/user/isos contains virtio-win-0.1.266-1.iso. This VM hasn't been running, though. Does this affect anything?