call trace // smbd: page allocation stalls

zoggy · March 22, 2018

should you flag call traces/etc as RED in unraid gui syslog viewer?

D:\data>grep "tainted" *
syslog-20180127-144626.txt:Jan 13 13:15:53 husky kernel: CPU: 0 PID: 13101 Comm: smbd Not tainted 4.14.13-unRAID #1
syslog-20180314-100310.txt:Mar  1 20:16:07 husky kernel: CPU: 3 PID: 4535 Comm: smbd Not tainted 4.14.16-unRAID #1
syslog-20180314-100310.txt:Mar  1 20:16:07 husky kernel: CPU: 0 PID: 2671 Comm: emhttpd Not tainted 4.14.16-unRAID #1
syslog-20180322-020712.txt:Mar 20 14:48:03 husky kernel: CPU: 2 PID: 22438 Comm: sh Not tainted 4.14.26-unRAID #1
syslog-20180322-020712.txt:Mar 20 14:48:03 husky kernel: CPU: 1 PID: 22437 Comm: sh Not tainted 4.14.26-unRAID #1
syslog-20180322-020712.txt:Mar 21 12:25:10 husky kernel: CPU: 2 PID: 29874 Comm: smbd Not tainted 4.14.26-unRAID #1
syslog-20180322-020712.txt:Mar 21 12:25:10 husky kernel: CPU: 3 PID: 18627 Comm: util.rb:23 Not tainted 4.14.26-unRAID #1
syslog-20180322-020712.txt:Mar 22 01:41:20 husky kernel: CPU: 0 PID: 18627 Comm: util.rb:23 Not tainted 4.14.26-unRAID #1
syslog-20180322-020712.txt:Mar 22 01:41:20 husky kernel: CPU: 1 PID: 6085 Comm: smbd Not tainted 4.14.26-unRAID #1
syslog-20180322-020712.txt:Mar 22 01:41:20 husky kernel: CPU: 3 PID: 7084 Comm: emhttpd Not tainted 4.14.26-unRAID #1

limetech · March 22, 2018

This appears to be a low memory situation. Do you think that can be the case? Next time attach diagnostics.zip instead of pasting a huge section of your system log, since the diags provide a better picture of the state of your server.

zoggy · March 23, 2018

I would have attached diagnostics but it didnt run due to the server being hung.. theres sensitive info thats in the actual log thats not anonymized which is why I didnt just share that.. I can send it privately if theres an email/way to do that.

Now about the concern for low memory, I dont really run much on the box except for unraid and a few dockers.

I currently have 3 dockers on the box, kodi-headless, netdata, gitlab-ce (added a few days ago).

Usually only kodi-headless is active. I turn netdata on from time to time when I wanted to check stats for the box.

root@husky:/var/log/nginx# free -m
              total        used        free      shared  buff/cache   available
Mem:           3832        2972         199         471         660          79
Swap:             0           0           0
root@husky:/var/log/nginx# top
top - 21:08:44 up 18:46,  1 user,  load average: 0.16, 0.18, 0.61
Tasks: 318 total,   1 running, 202 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.9 us,  2.8 sy,  0.6 ni, 94.3 id,  1.3 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  3924796 total,   215740 free,  3046880 used,   662176 buff/cache
KiB Swap:        0 total,        0 free,        0 used.    86156 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                
20196 998       20   0  931772 549124      0 S   1.3 14.0  11:17.45 bundle                                                                                                                                                                 
 6745 root      22   2  184796  23620  18224 S   1.0  0.6  11:09.37 php                                                                                                                                                                    
  644 root      20   0       0      0      0 S   0.7  0.0   5:25.73 kswapd0                                                                                                                                                                
 6593 nobody    20   0 1537964  29752      0 S   0.7  0.8   7:17.89 kodi.bin                                                                                                                                                               
 6887 root      20   0  294812   3924   2880 S   0.7  0.1   5:01.87 emhttpd

Edited March 23, 2018 by zoggy

zoggy · March 23, 2018

Just tried to stop the array.. its hung on stopping services.

t@husky:~# docker images
REPOSITORY                  TAG                 IMAGE ID            CREATED             SIZE
titpetric/netdata           latest              ac55dddb1e58        3 days ago          260MB
titpetric/netdata           <none>              6b5e6cde4d6f        4 days ago          260MB
linuxserver/kodi-headless   Krypton             8af360887842        6 days ago          425MB
gitlab/gitlab-ce            latest              9098c9b6eab0        6 days ago          1.46GB

i tried stopping the dockers by doing 'docker stop <container>' but the gitlab one wont stop... I'm guess thats the culprit right now

ok, as soon as I killed the gitlab-ce docker, the 'hung' stopping of the array finished.

root@husky:~#  ps -aux | grep docker
root     15649  0.0  0.0   3624  1644 ?        Sl   04:14   0:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 9443 -container-ip 172.17.0.3 -container-port 9443
root     15660  0.0  0.0   3624  1644 ?        Sl   04:14   0:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 9080 -container-ip 172.17.0.3 -container-port 9080
root     15672  0.0  0.0   3624  1640 ?        Sl   04:14   0:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 9022 -container-ip 172.17.0.3 -container-port 9022
root     15678  0.0  0.1  12228  4988 ?        Sl   04:14   0:05 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/7d7e689d5cab550ae2ecbd04ca30a8222f8f2432f291cf08b268b3e3cf1899c2 -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /usr/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc
root     16892  0.1  1.2 425656 49848 ?        Sl   02:26   1:53 /usr/bin/dockerd -p /var/run/dockerd.pid --storage-driver=btrfs --storage-driver=btrfs
root     16906  0.1  0.4 513612 19272 ?        Ssl  02:26   1:30 docker-containerd --config /var/run/docker/containerd/containerd.toml
root     22326  0.0  0.0   9824  2680 ?        S    21:19   0:00 sh -c set -o pipefail ; /etc/rc.d/rc.docker stop |& logger
root     22327  0.0  0.0  10072  3152 ?        S    21:19   0:00 /bin/sh /etc/rc.d/rc.docker stop
root     22342  0.0  0.4  23356 15908 ?        Sl   21:19   0:00 docker stop --time=5 GitLab-CE
root     27603  0.0  0.0   9812  2096 pts/0    S+   21:45   0:00 grep docker
root@husky:~#  kill -9 15678

Edited March 23, 2018 by zoggy

zoggy · March 23, 2018

While I have not seen any call traces this time, but the same symptoms are happening. The app page wont load, cant stop array.

husky-diagnostics-20180322-2125.zip

limetech · March 23, 2018

15 hours ago, zoggy said:

While I have not seen any call traces this time, but the same symptoms are happening. The app page wont load, cant stop array.

husky-diagnostics-20180322-2125.zip

Thanks for the diags. You are very close to running out of memory I think:

              total        used        free      shared  buff/cache   available
Mem:           3.7G        2.9G        228M        471M        622M         89M
Swap:            0B          0B          0B
Total:         3.7G        2.9G        228M

available = 89M out of 4G installed

zoggy · March 23, 2018

Digging into the memory issue it looks like gitlab-ce itself is just a memory hog (requires 2gb alone), I've removed this container for now. As I've only installed this docker recently and havent used it.. so not a big deal. But this doesnt solve the call traces issue that has been happening before this docker.

with gitlab-ce removed, per the unraid ui, memory used looks to sit at 32% now.

~# free -m
              total        used        free      shared  buff/cache   available
Mem:           3832         346         213         453        3273        2607
Swap:             0           0           0

Edited March 23, 2018 by zoggy

limetech · March 23, 2018

1 minute ago, zoggy said:

But this doesnt solve the call traces issue that has been happening before this docker.

I'm suffering now from information overload. I cannot go back and analyze old syslogs dating back a couple months and several releases.

We identified one possible issue having to do with low memory. Let's run like this for a while and if you start seeing issues again, without having changed anything else, then please open a new report. For now I will close this one. Not saying an issue still doesn't exist, instead just for the sake of being able to come back to it and understand the context of the issue.

limetech · March 23, 2018

Changed Status to Closed

call trace // smbd: page allocation stalls

User Feedback

Recommended Comments

zoggy 37

Link to comment

limetech 3,331

Link to comment

zoggy 37

Link to comment

zoggy 37

Link to comment

zoggy 37

Link to comment

limetech 3,331

Link to comment

zoggy 37

Link to comment

limetech 3,331

Link to comment

limetech 3,331

Link to comment

Join the conversation