[SOLVED] Cannot unmount /mnt/cache to Stop array


Recommended Posts

I tried to Stop my array and it's currently still stuck on `Retry unmounting disk share(s)...` for the last 30min. Some diagnostics from command line below (I cannot access diagnostics from GUI anymore).

 

Prior to this, I noticed one of my dockers was having weird issues... seemingly stopped after I killed it, but kept being listed as running in `docker ps`. I was using `docker exec` to execute some commands in that container and I think some processes got stuck in the container.

root@Tower:/# tail -n 5 /var/log/syslog
Apr 28 14:11:36 Tower emhttpd: Unmounting disks...
Apr 28 14:11:36 Tower emhttpd: shcmd (43474): umount /mnt/cache
Apr 28 14:11:36 Tower root: umount: /mnt/cache: target is busy.
Apr 28 14:11:36 Tower emhttpd: shcmd (43474): exit status: 32
Apr 28 14:11:36 Tower emhttpd: Retry unmounting disk share(s)...
root@Tower:/# lsof /mnt/cache
root@Tower:/# fuser -mv /mnt/cache
                     USER        PID ACCESS COMMAND
/mnt/cache:          root     kernel mount /mnt/cache

 

Edited by golli53
Link to comment

I think the docker image on /mnt/cache that's mounted on /dev/loop2 is preventing the unmount. I killed a zombie container process accessing /dev/loop2, but still cannot detach /dev/loop2 and still stuck trying to unmount.

 

Tried everything here: https://stackoverflow.com/questions/5881134/cannot-delete-device-dev-loop0

 

root@Tower:/# losetup
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                           DIO LOG-SEC
/dev/loop1         0      0         1  1 /boot/bzfirmware                      0     512
/dev/loop2         0      0         1  0 /mnt/cache/system/docker/docker.img   0     512
/dev/loop0         0      0         1  1 /boot/bzmodules                       0     512
root@Tower:/# lsof /dev/loop2
COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
container 15050 root    4u  FIFO   0,82      0t0 2917 /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8ea313440eef7c42a99526240f16a5438cf23beb769630a6ede14276aebe8ca5/shim.stdout.log
container 15050 root    7u  FIFO   0,82      0t0 2917 /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8ea313440eef7c42a99526240f16a5438cf23beb769630a6ede14276aebe8ca5/shim.stdout.log
container 15050 root    8u  FIFO   0,82      0t0 2918 /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8ea313440eef7c42a99526240f16a5438cf23beb769630a6ede14276aebe8ca5/shim.stderr.log
container 15050 root    9u  FIFO   0,82      0t0 2918 /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8ea313440eef7c42a99526240f16a5438cf23beb769630a6ede14276aebe8ca5/shim.stderr.log
root@Tower:/# kill 15050
root@Tower:/# lsof /dev/loop2
root@Tower:/# losetup -d /dev/loop2  # fails silently
root@Tower:/# echo $?
0
root@Tower:/# losetup
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                           DIO LOG-SEC
/dev/loop1         0      0         1  1 /boot/bzfirmware                      0     512
/dev/loop2         0      0         1  0 /mnt/cache/system/docker/docker.img   0     512
/dev/loop0         0      0         1  1 /boot/bzmodules                       0     512
root@Tower:/# lsof | grep loop2
loop2     12310                       root  cwd       DIR                0,2      440          2 /
loop2     12310                       root  rtd       DIR                0,2      440          2 /
loop2     12310                       root  txt   unknown                                        /proc/12310/exe
root@Tower:/# kill -9 12310  # not sure what this is, but killing it fails
root@Tower:/# lsof | grep loop2
loop2     12310                       root  cwd       DIR                0,2      440          2 /
loop2     12310                       root  rtd       DIR                0,2      440          2 /
loop2     12310                       root  txt   unknown                                        /proc/12310/exe
root@Tower:/# modprobe -r loop && modprobe loop  # try to reload the module, but it's builtin
modprobe: FATAL: Module loop is builtin.

 

Edited by golli53
added to log
  • Like 1
Link to comment
  • JorgeB changed the title to [SOLVED] Cannot unmount /mnt/cache to Stop array
  • 9 months later...
  • 1 month later...
  • 7 months later...

You forgot

1.a. Stop VM's.

1.b. Close SSH sessions if any

 

All of these items are theoretically handled by the stock Unraid shutdown, but it's not always foolproof, and it's much easier to work through an orderly shutdown than to deal with a hang and possible unclean shutdown.

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.