December 3, 20169 yr Hi All, I have a situation that is occurring, which is preventing me from being able to stop the array or even successfully issue a shutdown command from the CLI. I did find one article from 2013 referencing an issue with RC13, but I haven't found any discussion regarding newer releases. I'm running 6.2.4 with 14 drives + parity + boot; 12GB RAM; 2 - Xeon 1.??GHz. The system has been stable until this last week when I removed a handful of unused Docker containers and was preparing to shrink the array following 'Clear Drive Then Remove Drive' method from the following article: https://lime-technology.com/wiki/index.php/Shrink_array I (of course) wouldn't expect any changes to have led to this behavior, but I wanted to present the 'what's changed?' answers up front. Dynamix WebUI continues to report 'Spinning up all drives...' however the device lists reports all drives were spun up before initiating the stop command. Also of note, the WebUI is no longer responsive and will not allow new sessions to be established. From the CLI, docker images reports the docker daemon is not currently running (which it was before initiating the array stop) and ps eaf 6 serial TTYs listening, my active pts session, and the ps eaf query, so there don't appear to be any active handles open to mounted disks. The command "fuser -mv /mnt/disk* /mnt/user/*" does return a number of processes items pasted below due to the length of snip. Syslog output shows the following errors recurring: Dec 2 23:44:04 CENTRAL emhttp: Retry unmounting user share(s)... Dec 2 23:44:09 CENTRAL emhttp: shcmd (12033): set -o pipefail ; umount /mnt/user |& logger Dec 2 23:44:09 CENTRAL root: umount: /mnt/user: target is busy Dec 2 23:44:09 CENTRAL root: (In some cases useful info about processes that Dec 2 23:44:09 CENTRAL root: use the device is found by lsof( or fuser(1).) Dec 2 23:44:09 CENTRAL emhttp: err: shcmd: shcmd (12033): exit status: 32 Dec 2 23:44:09 CENTRAL emhttp: shcmd (12034): rmdir /mnt/user |& logger Dec 2 23:44:09 CENTRAL root: rmdir: failed to remove '/mnt/user': Device or resource busy Dec 2 23:44:09 CENTRAL emhttp: shcmd (12035): rm -f /boot/config/plugins/dynamix/mover.cron Dec 2 23:44:09 CENTRAL emhttp: shcmd (12036): /usr/local/sbin/update_cron &> /dev/null Dec 2 23:44:09 CENTRAL emhttp: Retry unmounting user share(s)...
December 3, 20169 yr Author As I was finalizing the debug information for the above post I was able to solve the issue and considered not posting. On the other hand, I've read far too many threads where the author solves their own problem and abandons the thread. So, just in case someone finds themselves in a similar situation in the future, the resolution to this particular problem was to change directory in my active bash session out of the /mnt/user path...
September 11, 20205 yr For anyone else running into this issue (I found this post from google) Thanks for the tip. I also used # Watch the log to see which disk is preventing the array from stopping tail -f /var/log/syslog In my case, it was disk1 Sep 11 09:22:04 unraid root: umount: /mnt/disk1: target is busy. <-------------------- Sep 11 09:22:04 unraid emhttpd: shcmd (59235): exit status: 32 Sep 11 09:22:04 unraid emhttpd: Retry unmounting disk share(s)... Sep 11 09:22:09 unraid emhttpd: Unmounting disks... Sep 11 09:22:09 unraid emhttpd: shcmd (59236): umount /mnt/disk1 Using @cookwjc's output, I used lsof to find the offender root@unraid:/mnt# lsof | grep disk1 bash 24735 root cwd DIR 9,1 4096 133 /mnt/disk1/backups/QNAP/qpkg I then killed the PID kill 24735 # or kill -9 24735 if it's safe and it's not responding The array was then able to finish stopping. It would be nice if unRAID would be able to detect these types of issues. Thanks for the tip @cookwjc! Edited September 11, 20205 yr by MrLinux
June 19, 20215 yr as this is the first site google spits out with the search "unraid array won't stop" i will try to append my problem. above mentioned solution does not always work: root@D1:/var/log# tail -f syslog Jun 20 01:15:09 Dose4000 emhttpd: Unmounting disks... Jun 20 01:15:09 Dose4000 emhttpd: shcmd (4331): umount /mnt/disk1 Jun 20 01:15:09 Dose4000 root: umount: /mnt/disk1: target is busy. Jun 20 01:15:09 Dose4000 emhttpd: shcmd (4331): exit status: 32 Jun 20 01:15:09 Dose4000 emhttpd: Retry unmounting disk share(s)... Jun 20 01:15:14 Dose4000 emhttpd: Unmounting disks... Jun 20 01:15:14 Dose4000 emhttpd: shcmd (4332): umount /mnt/disk1 Jun 20 01:15:14 Dose4000 root: umount: /mnt/disk1: target is busy. Jun 20 01:15:14 Dose4000 emhttpd: shcmd (4332): exit status: 32 Jun 20 01:15:14 Dose4000 emhttpd: Retry unmounting disk share(s)... ^C root@D1:/var/log# lsof | grep disk1 root@D1:/var/log# so... what now?
February 4, 20224 yr In my case, syslog showed disk2 and disk3 are busy. And it's because I have SSH tunnel opened and cd'ed to disk2 and disk3. Once I terminated the SSH, the arrays can be stopped immediately.
February 4, 20224 yr There is the Dynamix Stop Shell plugin which you can install which should catch this stuff and terminate the shells accordingly.
October 17, 20223 yr I just now recently had this same issue as I was troubleshooting some dockers and then I realized I still had my FTP connection open. So if anyone else runs into this make sure you have closed your FTP/SMB connections
April 15, 20233 yr On 12/2/2016 at 11:14 PM, cookwjc said: As I was finalizing the debug information for the above post I was able to solve the issue and considered not posting. On the other hand, I've read far too many threads where the author solves their own problem and abandons the thread. So, just in case someone finds themselves in a similar situation in the future, the resolution to this particular problem was to change directory in my active bash session out of the /mnt/user path... Thanks! That helped a lot! I wish more people did this!
November 25, 20232 yr On 9/11/2020 at 9:30 AM, MrLinux said: For anyone else running into this issue (I found this post from google) Thanks for the tip. I also used # Watch the log to see which disk is preventing the array from stopping tail -f /var/log/syslog In my case, it was disk1 Sep 11 09:22:04 unraid root: umount: /mnt/disk1: target is busy. <-------------------- Sep 11 09:22:04 unraid emhttpd: shcmd (59235): exit status: 32 Sep 11 09:22:04 unraid emhttpd: Retry unmounting disk share(s)... Sep 11 09:22:09 unraid emhttpd: Unmounting disks... Sep 11 09:22:09 unraid emhttpd: shcmd (59236): umount /mnt/disk1 Using @cookwjc's output, I used lsof to find the offender root@unraid:/mnt# lsof | grep disk1 bash 24735 root cwd DIR 9,1 4096 133 /mnt/disk1/backups/QNAP/qpkg I then killed the PID kill 24735 # or kill -9 24735 if it's safe and it's not responding The array was then able to finish stopping. It would be nice if unRAID would be able to detect these types of issues. Thanks for the tip @cookwjc! Thank you for the solution! But I wonder, why can't unraid do this itself?
October 4, 20241 yr Same issue for me guys. I had an SSH terminal open which just was sitting idle at the path /mnt/user/. This prevented the shutdown of the array, see syslog: Oct 4 16:09:33 MediaServer root: rmdir: failed to remove '/mnt/user': Device or resource busy Oct 4 16:09:33 MediaServer emhttpd: shcmd (1939176): exit status: 1 Oct 4 16:09:33 MediaServer emhttpd: shcmd (1939178): rm -f /boot/config/plugins/dynamix/mover.cron Oct 4 16:09:33 MediaServer emhttpd: shcmd (1939179): /usr/local/sbin/update_cron Oct 4 16:09:33 MediaServer emhttpd: Retry unmounting user share(s)... Oct 4 16:09:38 MediaServer emhttpd: shcmd (1939180): /usr/sbin/zfs unmount -a Oct 4 16:09:39 MediaServer emhttpd: shcmd (1939181): umount /mnt/user Oct 4 16:09:39 MediaServer root: umount: /mnt/user: target is busy. Oct 4 16:09:39 MediaServer emhttpd: shcmd (1939181): exit status: 32 Oct 4 16:09:39 MediaServer emhttpd: shcmd (1939182): rmdir /mnt/user Oct 4 16:09:39 MediaServer root: rmdir: failed to remove '/mnt/user': Device or resource busy Oct 4 16:09:39 MediaServer emhttpd: shcmd (1939182): exit status: 1 Oct 4 16:09:39 MediaServer emhttpd: shcmd (1939184): rm -f /boot/config/plugins/dynamix/mover.cron Oct 4 16:09:39 MediaServer emhttpd: shcmd (1939185): /usr/local/sbin/update_cron Oct 4 16:09:39 MediaServer emhttpd: Retry unmounting user share(s)... Oct 4 16:09:44 MediaServer emhttpd: shcmd (1939186): /usr/sbin/zfs unmount -a Oct 4 16:09:44 MediaServer emhttpd: shcmd (1939187): umount /mnt/user Oct 4 16:09:44 MediaServer root: umount: /mnt/user: target is busy. Oct 4 16:09:44 MediaServer emhttpd: shcmd (1939187): exit status: 32 Oct 4 16:09:44 MediaServer emhttpd: shcmd (1939188): rmdir /mnt/user ... So the solution was just to exit the ssh session! @limetech would be nice if after 3x unmount retry a force dialog would pop up!?
October 4, 20241 yr Community Expert 33 minutes ago, pOpYRaid said: So the solution was just to exit the ssh session! You might want to install the Dynamix Stop Shell plugin to automate this.
October 4, 20241 yr 15 minutes ago, itimpi said: You might want to install the Dynamix Stop Shell plugin to automate this. thx a lot, installed it Just my thought's: That should be an default "feature" (maybe with an popup). I don't want to know how many of us users, was just pressing the power button or do an force reboot, because of this. Edited October 4, 20241 yr by pOpYRaid
October 4, 20241 yr Community Expert Just now, pOpYRaid said: Just my thought's: That should be an default "feature" I agree, and maybe one day it will be. The plugin is a good stopgap..
October 9, 20241 yr ran into this problem while trying to shrink my array using the SI video with zeroing the drive.. getting this reoccurring so the array wont stop Oct 8 20:59:07 FatJoe emhttpd: shcmd (315265): umount /mnt/disk8 Oct 8 20:59:07 FatJoe root: umount: /mnt/disk8: not mounted. Oct 8 20:59:07 FatJoe emhttpd: shcmd (315265): exit status: 32 Oct 8 20:59:07 FatJoe emhttpd: shcmd (315266): umount /mnt/disk10 Oct 8 20:59:07 FatJoe root: umount: /mnt/disk10: not mounted. Oct 8 20:59:07 FatJoe emhttpd: shcmd (315266): exit status: 32 Oct 8 20:59:07 FatJoe emhttpd: Retry unmounting disk share(s)... Oct 8 20:59:12 FatJoe emhttpd: Unmounting disks... Oct 8 20:59:12 FatJoe emhttpd: shcmd (315267): umount /mnt/disk8 Oct 8 20:59:12 FatJoe root: umount: /mnt/disk8: not mounted. Oct 8 20:59:12 FatJoe emhttpd: shcmd (315267): exit status: 32 Oct 8 20:59:12 FatJoe emhttpd: shcmd (315268): umount /mnt/disk10 Oct 8 20:59:12 FatJoe root: umount: /mnt/disk10: not mounted. Oct 8 20:59:12 FatJoe emhttpd: shcmd (315268): exit status: 32 Oct 8 20:59:12 FatJoe emhttpd: Retry unmounting disk share(s)... Oct 8 20:59:17 FatJoe emhttpd: Unmounting disks... tried running the 2nd command but it wont work root@FatJoe:~# /mnt# lsof | grep disk10 bash: /mnt#: No such file or directory root@FatJoe:~# lsof | grep disk10 root@FatJoe:/mnt# lsof | grep disk10 root@FatJoe:/mnt# 2nd command not coming back with anything Edit2: fix the loop from SI comments Edited October 9, 20241 yr by ToXIc
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.