Every couple of weeks, my Unraid box becomes partially unresponsive.

 

SYSTEM
Unraid 6.12.14
i7-7700K / 32gb ram
Disk 1: 16tb
Disk 2: 16tb
Parity: 16tb

SERVICES

1 array with a bunch of shares
1 active docker container
3 disabled docker containers
1 disabled Windows VM

PLUGINS

Dynamix System Temperature
User Scripts

 

SYMPTOMS IN UNRESPONSIVE STATE

 

The Web UI works, until i open the VM tab. This hangs the web session for a long time. (10-30mins)
The Web UI is still accessible if i connect using a different hostname/IP, until visiting the VM tab again.

 

SSH'ing in works fine, I have a functional shell.

 

No shares can be accessed over smb.

Attempting to stop the array through Web UI -> Main -> Stop array sends a POST that hangs for 3 minutes and ends with an nginx 504 timeout.

 

The running container (Docker mailserver / DMS) responds, provides logs, and can be SSH'ed to, but spams errors apparently related to interactions with mounted files (mail storage, certs..) on the host.
Attempting to stop the container has no effect through the Web UI.
Attempting to stop through SSH: docker stop bd330867953d
"Error response from daemon: cannot stop container: bd330867953d: tried to kill container, but did not receive an exit event"

 

I can run smart self-tests on all disks (1, 2, parity) through the webUI. They all complete without errors.
Parity checks and syncs have never had any errors.

 

The following commands will hang indefinitely (although ctrl-c works):
df
lsof
ls /mnt
ls /mnt/user
All emcmd commands
shutdown -r (hangs on "Trying to re-exec init")

 

The following commands work fine:
ls /
ls /var
ls /usr
ls /run
ls /mnt/disk1
ls /mnt/disk2
shutdown -rn

top and htop works. No processes are consuming a noticeable amount of CPU.
free -h works. Memory usage is low and there's a lot free.

 

LOGS/DIAGNOSTICS

 

/var/log/syslog has no entries in the hours before or after the problem starts.

The last entry before is me doing things the previous day, and the first one after is me logging in.

 

The only errors after logging in, is the nginx errors when trying to open the VM tab:

Jan 25 11:32:15 can nginx: 2025/01/25 11:32:15 [error] 8658#8658: *4494722 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.1.92, server: , request: "GET /VMs HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "can", referrer: "http://can/Docker"
Jan 25 11:33:43 can nginx: 2025/01/25 11:33:43 [error] 8658#8658: *4494893 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.1.92, server: , request: "GET /Dashboard HTTP/1.1", subrequest: "/auth-request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "can", referrer: "http://can/VMs"
Jan 25 11:33:43 can nginx: 2025/01/25 11:33:43 [error] 8658#8658: *4494893 auth request unexpected status: 504 while sending to client, client: 192.168.1.92, server: , request: "GET /Dashboard HTTP/1.1", host: "can", referrer: "http://can/VMs"

I can generate partial diagnostics, but it hangs on commands like lsof and df, so many files in the output are empty or missing.
/var/log/messages is empty.
dmesg has nothing in the hours before the problem starts.

 

 

What could be happening here? 
It seems like the disks are accessible, but if i'm interpreting the evidence above correctly, everything that touches the array fails? (Why the VM tab though?)
If i shutdown -rn, the box comes back up happy as ever, and Unraid starts a full parity check (without errors).
The box then runs for another week or two, and this happens again.

 

The only thing i can think of that I've customized related to the filesystem, is downgrading the VirtIO version to get an old VM to run.
/mnt/user/isos contains virtio-win-0.1.266-1.iso.
This VM hasn't been running, though. Does this affect anything?

9 hours ago, carr said:

The Web UI works, until i open the VM tab.

Just to confirm, it everything working normally until you open the VM tab, or it's just not responding correctly and opening the VM tab makes it worse?

Everything related to the array (shares and container access to it) is already down at that point, @JorgeB
There has not been any user interaction or system load around the time things stop working.

 

Typically I'll realize there's a problem, and login to the Web UI to investigate.

Hitting the VM tab will then additionally take down the Web UI for a while. (Only on that hostname though. It's still responsive by browsing to the IP instead of the hostname, for example.)

 

 

OK, then enable the syslog server and post that after you notice the initial issues.

I would suggest posting your system's diagnostics zip file so we can check if there are any configuration issues

1 hour ago, JorgeB said:

OK, then enable the syslog server and post that after you notice the initial issues.

I have the full contents of /var/log/syslog both during the recent problem, and the full contents of it after the subsequent reboot.
Will enabling the server and waiting for the next crash produce anything else?

You can post those.

Great, thanks!
Two syslog files attached, timeline explanation below.
 

1) 
Jan 10 14:43:37: Startup after last time this happened
Jan 10 14:44:28: Startup completed, normal operation resumes
Jan 16 14:38:50: Error spam from nginx, probably a stale browser tab left open for the Web UI
Jan 16 -> Jan 24: Various usage
Jan 24 21:24:14: Last syslog entry (from normal activity) before unresponsiveness
Jan 25 04:30:00: Last proof of life (email delivered), unresponsiveness starts some time between now and 10:00
Jan 25 10:00:00: Unresponsiveness discovered
Jan 25 11:22:21: First syslog entry after unresponsiveness, troubleshooting starts, various logins and Web UI errors
Jan 25 12:08:52: Syslog extracted
 

2)
Jan 25 13:07:24: Startup after giving up and running shutdown -rn
Jan 25 13:08:20: Startup completed, normal operation resumes

1-syslog-before-and-during-last-downtime.txt 2-syslog-after-reboot.txt

Also, attached complete diagnostics pulled after the last restart, per suggestion from @itimpi

can-diagnostics-20250125-1309 (after reboot).zip

Unfortunately, there's nothing relevant logged, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one, including the individual docker containers.

16 minutes ago, JorgeB said:

Unfortunately, there's nothing relevant logged, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one, including the individual docker containers.

Wish I could, but the only docker container that's running is a live mail server. It can be down for hours, maybe a day, but not for weeks at a time. Even if disabling docker did turn out to solve the problem, that would just mean I'd have to switch from Unraid to something else.

NAS services are not as critical (yet), except as backing for that container.

 

Any thoughts on whether that older VirtIO iso existing on the system could be related? Does it affect anything other than the (disabled) VM?

Is there any preparation I can do before the next crash hits? Any monitoring/logging i can set up?

Is there anything I can do while the next round is in effect, and I still have a working shell? (See list of working/hanging commands tried in my initial post.)

Any thoughts on how to test for hardware problems, or any other next steps?

 

Is there any reason to think upgrading to 6.12.15 or 7.0 might help? I wanted to figure out the root cause here before tackling that, unless there are reasons to believe it would help.

Edited January 26, 20251 yr by carr
Missed an important question

11 hours ago, carr said:

Any thoughts on whether that older VirtIO iso existing on the system could be related?

That should not be a problem.