Recently my server has developed a very strange problem, where it stops responding almost entirely. Please note, this is not a crash and it is not fully locked up. This started happening infrequently, but now happens every night. 

1) The server stops responding to SSH or WebUI connections.

2) The server does respond to ICMP and scans do show open TCP ports, though no service banners are returned.

3) There is no seg fault and nothing contained in the syslog (attached).

4) If I physically connect to the console (USB keyboard and VGA) I get a login prompt, however it times out (message returns that login attempt timed out) when I attempt to login. I can access alternate TTYs this way, but they all respond the same.

5) (Anecdotal) It seems to happen around the same time every day (between 3 and 4AM).

6) If I initiate a reboot from the console (CTRL-ALT-DEL) it switches to INIT 6 and starts the shutdown process. It tries to gracefully shutdown, then initiates forced shutdown, and then hangs.

 

• This is a relatively new chassis (Mobo/CPU), with RAM and drives taken from a stable system that had been running for months, and has been working smoothly for weeks before anything started happening.
• I upgraded from 6.12.6 to the latest, but that did not change anything. 
• I am using IPVLAN and Intel NICs so this is not the MACVLAN or Realtek chip issues.
• The lack of any error messages/significant log entries and it continuing to process interrupts has me leaning away from a hardware issue. 

 

Attaching syslogs. I attempted to SSH and the server would not accept the password and then stopped responding to anything from the network. Feb 21, I know stopped functioning around 3:20AM because of the SSH. Feb 22, I do not know exactly when it failed as it was not working when I work up in the morning. Please note that none of the interactions I attempted (either time) from the console show up in the syslog, nor do any of the init statements from the attempted reboot.

syslog-Feb-21.txt syslog-Feb-22.txt

Edited February 22, 20242 yr by ssb201
grammar

Without anything logged not much to go on, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one. 

5 minutes ago, JorgeB said:

Without anything logged not much to go on, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one. 

Yeah,. That may be my best bet. I had a similar problem with just the logins timing out (WebUI worked but SSH would not) a few months ago and found that it was an Active Directory issue (as soon as I turned off Samba everything was fine). That was a problem with the designated FSMO on my Windows servers. 

 

This is definitely something stranger. I hate to lose all server functionality (barely ever use it as an actual NAS), but I will try running without any dockers today and if it locks up again, go to safe mode and kill both dockers and VMs.

I was able to do a little more digging:

1) The system seemed to run just fine without plugins or the array running.

2) The system seemed to run just fine with plugins and the array running but no VMs or docker containers.

3) The system seemed to run just fine with plugins and the array and a single VM running but no docker containers.

4) I selected three docker containers to run along with the single VM (OrganizrV2, Nginx, and Plex) and the system (sometime over night) had the issue again. This time I had an SSH connection already established, so I did not run into the timeout trying to login.

 

I was able to run top (very slowly) and grabbed this:

top - 10:10:31 up 4 days,  6:08,  1 user,  load average: 139.34, 137.18, 136.81
Tasks: 621 total,   6 running, 593 sleeping,   0 stopped,  22 zombie
%Cpu(s):  0.0 us, 13.3 sy,  0.0 ni, 16.7 id, 69.8 wa,  0.0 hi,  0.2 si,  0.0 st
MiB Mem :  96299.8 total,    576.1 free,  95157.7 used,    566.1 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.    103.6 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  241 root      20   0       0      0      0 R 100.0   0.0 363:18.89 kswapd0
19866 root      20   0       0      0      0 R  37.7   0.0  11:33.42 kworker/u40:4+loop3
21057 root      20   0       0      0      0 R  15.5   0.0   4:18.59 kworker/u40:7+loop3
12497 root      20   0   47.9g  44.2g   6532 D  14.2  47.0   2323:15 qemu-system-x86
20737 root      20   0       0      0      0 R  14.2   0.0   8:47.72 kworker/u40:16+loop3
 3859 root      20   0   10576   7004   5748 D   7.0   0.0   0:12.33 sshd
13950 nobody    20   0  257492  36512      4 S   5.7   0.0  14:40.36 Plex Media Serv
 6287 root      20   0  755916  19604      0 S   4.4   0.0  21:09.10 containerd
13673 root      20   0  722964  10460      0 S   3.5   0.0  14:00.71 containerd0:0+loop1
21067 root      20   0       0      0      0 I   3.1   0.0   2:48.72 kworker/u40:8-btrfs-endio-meta
21308 root      20   0       0      0      0 I   3.1   0.0   0:52.46 kworker/u40:10-btrfs-endio
20239 root      20   0       0      0      0 I   2.8   0.0  10:10.98 kworker/u40:9-btrfs-endio
20954 root      20   0       0      0      0 I   2.8   0.0   3:37.87 kworker/u40:17-btrfs-endio
21023 root      20   0       0      0      0 I   2.8   0.0   2:07.98 kworker/u40:5-btrfs-endio-meta
21145 root      20   0       0      0      0 I   2.8   0.0   2:17.69 kworker/u40:2-btrfs-endio
17850 root      20   0  722708  10676      0 S   2.6   0.0  12:06.54 containerd-shim
21112 root      20   0       0      0      0 I   2.6   0.0   2:02.28 kworker/u40:1-btrfs-endio-meta
13673 root      20   0  722964  10460      0 S   2.0   0.0  14:00.60 containerd-shim
16045 root      20   0  722964  11644      0 S   2.0   0.0  11:02.11 containerd-shim
  630 root      20   0       0      0      0 D   1.7   0.0   6:12.87 usb-storage
17002 root      20   0  690924  41360    428 S   1.4   0.0  37:57.79 shfs
 1669 ntp       20   0   74592   3040   2304 D   1.1   0.0   5:18.07 ntpd
 6214 root      20   0 4354540  47096      0 S   1.1   0.0   7:20.32 dockerd

 

The load count is extraordinarily high and I am not sure what kswapd is doing to consume all that CPU. The system has plenty of RAM so there should be no virtual memory swapping of any magnitude.

 

This seems very similar to the issue here: 

 but (referring to the final comment) I am not using ZFS. 

 

And essentially identical to the issue here:

 

Edited February 29, 20242 yr by ssb201

15 minutes ago, ssb201 said:

OrganizrV2, Nginx, and Plex

 

You can try to see if it's one from these 3 container causing the issue, i.e., try with one at a time, I would start with Plex

The best I can tell, it seems that Plex is having an issue during content scanning as a nightly task and soaks up all the memory. Not sure what the hell is going on, but by adding a hard cap on the memory assigned to the container, I just get the OOM killer reaping the container rather than my entire box locking up. There really is no good reason for a non-privileged docker container to be able to effectively lock up the host operating system.