March 30, 20188 yr I like to know if any of the generated files have no content, can you do ls -l /tower-diagnostics-20180330-0848/system
March 30, 20188 yr 1 minute ago, bonienl said: I like to know if any of the generated files have no content, can you do ls -l /tower-diagnostics-20180330-0848/system there you go: root@Tower:~# ls -l /tower-diagnostics-20180330-0848/system total 192 -rw-rw-rw- 1 root root 0 Mar 30 08:48 df.txt -rw-rw-rw- 1 root root 1341 Mar 30 08:48 lscpu.txt -rw-rw-rw- 1 root root 2701 Mar 30 08:48 lsmod.txt -rw-rw-rw- 1 root root 4950 Mar 30 08:48 lsof.txt -rw-rw-rw- 1 root root 17032 Mar 30 08:48 lspci.txt -rw-rw-rw- 1 root root 1854 Mar 30 08:48 lsscsi.txt -rw-rw-rw- 1 root root 698 Mar 30 08:48 lsusb.txt -rw-rw-rw- 1 root root 252 Mar 30 08:48 memory.txt -rw-rw-rw- 1 root root 52961 Mar 30 08:48 ps.txt -rw-rw-rw- 1 root root 40168 Mar 30 08:48 top.txt -rw-rw-rw- 1 root root 51030 Mar 30 08:48 vars.txt
March 30, 20188 yr For giggles, this should scroll through a ton of stuff du /var/lib/docker and end back at the terminal. Does it stop anywhere? Don't need the whole million lines from it. Just where it stops.
March 30, 20188 yr 2 minutes ago, Squid said: For giggles, this should scroll through a ton of stuff du /var/lib/docker and end back at the terminal. Does it stop anywhere? Don't need the whole million lines from it. Just where it stops. These are the last few lines before it stops: 0 /var/lib/docker/unraid/ca.backup2.datastore 0 /var/lib/docker/unraid 0 /var/lib/docker/containerd/daemon/io.containerd.content.v1.content/ingest 0 /var/lib/docker/containerd/daemon/io.containerd.content.v1.content 0 /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs/active 0 /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs/view 0 /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs/snapshots 0 /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs 0 /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots 0 /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs 648 /var/lib/docker/containerd/daemon/io.containerd.metadata.v1.bolt 0 /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/734d4b9da9e20a4b96828270d6c3f6d7ffc10e3dcf3c8554247673e76b344dd4 0 /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/19647dee6de0be1ca75840af5a88b29aea3aa6799b455ee1995094b51f83a2f0 0 /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/60ddf788b24a00021d4034e37ec760f34467a1c9179437836e43704ce1e445cc 0 /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/9f4716af81f9e00f5e94c53ff602c56902e88c05fbd6f22e9e2296c8e43ff7dd 0 /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/a3ecdae744a46beea66d9caec718aec884a4b0793f67f5e5fe17a83f1318fa08 0 /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/dcf1c167e8731cfdeb87f2081e66eff6ec1812f9a5ee3d4c15f3caae0f39906d 0 /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/68e3ad2825e29640a8d090fe0fd3090a3353d3f66cee11ceb042b7fddad8f7ef 0 /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/43349a50c0d6c50d4987fc88996c481cedfce67c2532fe1b44be8d237b4bb380 0 /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/7e44a412169eee513eafd0523eed2eabf216ce7d578640174a3ccb211b9e7cfa 0 /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8294edae393ac797db55b6060f621bc1f9efade4645888c2e4c10f6ddf8bdb93 0 /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby 0 /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux 648 /var/lib/docker/containerd/daemon 648 /var/lib/docker/containerd 0 /var/lib/docker/tmp 0 /var/lib/docker/runtimes 39355848 /var/lib/docker root@Tower:~#
March 30, 20188 yr 1 minute ago, Squid said: Threw one theory out how about df -t tmpfs There you go: root@Tower:~# df -t tmpfs Filesystem 1K-blocks Used Available Use% Mounted on tmpfs 32768 252 32516 1% /run tmpfs 16461524 0 16461524 0% /dev/shm cgroup_root 8192 0 8192 0% /sys/fs/cgroup tmpfs 131072 828 130244 1% /var/log shm 65536 0 65536 0% /var/lib/docker/containers/29665f5836321c13963824e87a964e6f041a7654c5d5a56d6878fc35bf8e7760/shm shm 65536 0 65536 0% /var/lib/docker/containers/9f4716af81f9e00f5e94c53ff602c56902e88c05fbd6f22e9e2296c8e43ff7dd/shm shm 65536 0 65536 0% /var/lib/docker/containers/034adf9c55653bb881968ca8d44355d9bdcb91702daabe289ca5fd0ed82a64d3/shm shm 65536 4 65532 1% /var/lib/docker/containers/a3ecdae744a46beea66d9caec718aec884a4b0793f67f5e5fe17a83f1318fa08/shm shm 65536 4 65532 1% /var/lib/docker/containers/dcf1c167e8731cfdeb87f2081e66eff6ec1812f9a5ee3d4c15f3caae0f39906d/shm shm 65536 8 65528 1% /var/lib/docker/containers/68e3ad2825e29640a8d090fe0fd3090a3353d3f66cee11ceb042b7fddad8f7ef/shm shm 65536 0 65536 0% /var/lib/docker/containers/1a2df515f75397057758aad73897792dc94411ac3aa98b7ae0bfe343fb70c366/shm shm 65536 0 65536 0% /var/lib/docker/containers/43349a50c0d6c50d4987fc88996c481cedfce67c2532fe1b44be8d237b4bb380/shm shm 65536 4 65532 1% /var/lib/docker/containers/7e44a412169eee513eafd0523eed2eabf216ce7d578640174a3ccb211b9e7cfa/shm shm 65536 0 65536 0% /var/lib/docker/containers/8294edae393ac797db55b6060f621bc1f9efade4645888c2e4c10f6ddf8bdb93/shm
March 30, 20188 yr Just now, bonienl said: Can you also post the syslog file (/var/log/syslog). Thanks for helping out Sure, i've attached the syslog file because its too much to quote. syslog
March 30, 20188 yr 13 minutes ago, Uncledome said: i've attached the syslog file Does any or your containers have a folder mapping to the remote share you set up with unassigned devices?
March 30, 20188 yr 1 minute ago, bonienl said: Does any or your containers have a folder mapping to the remote share you set up with unassigned devices? Yea, multiple should have a mapping in their config. From the top of my head those are: Plex, Sabnzbd, Sonarr, Radarr, JDownloader, Deluge and maybe a few others (all my media is stored on the remote share)
March 30, 20188 yr Just curios. The command below is likely to show one or more processes that are in state "D" (8th column of ps output). Quote ps -aux If you find any process with state "D" - what is the content of Quote ls -l /proc/<pid>/fd/ where <pid> is the PID (second column of ps output) of the process in state D? Regarding remote shares - they can sometimes hang badly. Sometimes possible to solve by doing Quote unmount -l <share> but that doesn't always solve the problem - often a reboot will be required.
March 30, 20188 yr 3 minutes ago, pwm said: Just curios. The command below is likely to show one or more processes that are in state "D" (8th column of ps output). If you find any process with state "D" - what is the content of where <pid> is the PID (second column of ps output) of the process in state D? Regarding remote shares - they can sometimes hang badly. Sometimes possible to solve by doing but that doesn't always solve the problem - often a reboot will be required. There are multiple PIDs with status D, Tried the ls -l with one of them: root@Tower:~# ls -l /proc/26457/fd/ total 0 lrwx------ 1 root root 64 Mar 30 11:47 0 -> socket:[811239] l-wx------ 1 root root 64 Mar 30 11:47 1 -> pipe:[857897] lrwx------ 1 root root 64 Mar 30 11:47 2 -> /dev/null lrwx------ 1 root root 64 Mar 30 11:47 3 -> socket:[863915] root@Tower:~# As for the remote mapping, I actually never had any issues but normally (before unraid which I am currently in trial) I used an ubuntu server for my docker (hosted on a XenServer Free Edition Server) where I mapped the remote share in fstab and passed it through to the containers. First time having those issues in my life .
March 30, 20188 yr Could you try again with unRAID 6.5.1-rc2 installed? In the syslog this is reported: 0 08:38:06 Tower kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000038 Mar 30 08:38:06 Tower kernel: IP: tcp_push+0x4e/0xee After this a call trace is recorded and you get the famous "connection time out" message, you mentioned earlier. From this point on communication hangs. The above should be resolved in rc2 which has the kernel patch for the TCP bug.
March 30, 20188 yr I had to reboot a Debian machine (not unRAID) just some days ago because it lost state of one share from a Windows machine i rebooted. I could have continued to use the Debian machine but I really needed to be able to mount the same share again and any new mount attempt would hang with process state 'D' which means uninterruptible wait for I/O. It's quite likely that you have one guilty process that is waiting for SMB data from the other machine on a now dead TCP connection. And since it's in a critical section within in the kernel, any other process that wants access to the same resource will also get stuck. One trick I missed testing is to fake TCP/IP data to try to force close TCP connections that has huge timeout times. But that isn't an easy path to take without proper tools - normal users aren't intended to be able to generate and inject arbitrary TCP packets so it isn't supported by any normally available tools.
March 30, 20188 yr 2 minutes ago, bonienl said: Could you try again with unRAID 6.5.1-rc2 installed? In the syslog this is reported: 0 08:38:06 Tower kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000038 Mar 30 08:38:06 Tower kernel: IP: tcp_push+0x4e/0xee After this a call trace is recorded and you get the famous "connection time out" message, you mentioned earlier. From this point on communication hangs. The above should be resolved in rc2 which has the kernel patch for the TCP bug. Sure, I'll forcefully shutdown the server now and will upgrade to 6.5.1-rc2. Should the server hang again, I'll let you guys know.
March 30, 20188 yr 6.5.1-rc3 is just released. It reverts Docker version back to the one used in unRAID 6.4.1. Give this a try.
March 30, 20188 yr 8 minutes ago, bonienl said: 6.5.1-rc3 is just released. It reverts Docker version back to the one used in unRAID 6.4.1. Give this a try. Just updated to rc3, currently rebooting. Will let you know if the problem persists.
April 3, 20188 yr Had a few issues after updating to rc3 (unraid was telling me partition issues and so on) but this was fixed with swapping usb to another port (had it on usb3 port, now usb2). Since then there were no other issues, docker is running smoothly until now.
April 14, 20188 yr I am on 6.5.0 and have this issue. Culprit appears to be pihole docker container. Is there a solution on how to force stop as normal 'docker stop <id>' is not working. I would prefer not to have to do an unclean system reboot. Edit: After going back and reading previous posts, I found my answer: # pkill -f docker # /etc/rc.d/rc.docker start Edited April 14, 20188 yr by noski
Archived
This topic is now archived and is closed to further replies.