cookwjc Posted December 3, 2016 Share Posted December 3, 2016 Hi All, I have a situation that is occurring, which is preventing me from being able to stop the array or even successfully issue a shutdown command from the CLI. I did find one article from 2013 referencing an issue with RC13, but I haven't found any discussion regarding newer releases. I'm running 6.2.4 with 14 drives + parity + boot; 12GB RAM; 2 - Xeon 1.??GHz. The system has been stable until this last week when I removed a handful of unused Docker containers and was preparing to shrink the array following 'Clear Drive Then Remove Drive' method from the following article: https://lime-technology.com/wiki/index.php/Shrink_array I (of course) wouldn't expect any changes to have led to this behavior, but I wanted to present the 'what's changed?' answers up front. Dynamix WebUI continues to report 'Spinning up all drives...' however the device lists reports all drives were spun up before initiating the stop command. Also of note, the WebUI is no longer responsive and will not allow new sessions to be established. From the CLI, docker images reports the docker daemon is not currently running (which it was before initiating the array stop) and ps eaf 6 serial TTYs listening, my active pts session, and the ps eaf query, so there don't appear to be any active handles open to mounted disks. The command "fuser -mv /mnt/disk* /mnt/user/*" does return a number of processes items pasted below due to the length of snip. Syslog output shows the following errors recurring: Dec 2 23:44:04 CENTRAL emhttp: Retry unmounting user share(s)... Dec 2 23:44:09 CENTRAL emhttp: shcmd (12033): set -o pipefail ; umount /mnt/user |& logger Dec 2 23:44:09 CENTRAL root: umount: /mnt/user: target is busy Dec 2 23:44:09 CENTRAL root: (In some cases useful info about processes that Dec 2 23:44:09 CENTRAL root: use the device is found by lsof( or fuser(1).) Dec 2 23:44:09 CENTRAL emhttp: err: shcmd: shcmd (12033): exit status: 32 Dec 2 23:44:09 CENTRAL emhttp: shcmd (12034): rmdir /mnt/user |& logger Dec 2 23:44:09 CENTRAL root: rmdir: failed to remove '/mnt/user': Device or resource busy Dec 2 23:44:09 CENTRAL emhttp: shcmd (12035): rm -f /boot/config/plugins/dynamix/mover.cron Dec 2 23:44:09 CENTRAL emhttp: shcmd (12036): /usr/local/sbin/update_cron &> /dev/null Dec 2 23:44:09 CENTRAL emhttp: Retry unmounting user share(s)... Quote Link to comment
cookwjc Posted December 3, 2016 Author Share Posted December 3, 2016 As I was finalizing the debug information for the above post I was able to solve the issue and considered not posting. On the other hand, I've read far too many threads where the author solves their own problem and abandons the thread. So, just in case someone finds themselves in a similar situation in the future, the resolution to this particular problem was to change directory in my active bash session out of the /mnt/user path... 7 Quote Link to comment
MrLinux Posted September 11, 2020 Share Posted September 11, 2020 (edited) For anyone else running into this issue (I found this post from google) Thanks for the tip. I also used # Watch the log to see which disk is preventing the array from stopping tail -f /var/log/syslog In my case, it was disk1 Sep 11 09:22:04 unraid root: umount: /mnt/disk1: target is busy. <-------------------- Sep 11 09:22:04 unraid emhttpd: shcmd (59235): exit status: 32 Sep 11 09:22:04 unraid emhttpd: Retry unmounting disk share(s)... Sep 11 09:22:09 unraid emhttpd: Unmounting disks... Sep 11 09:22:09 unraid emhttpd: shcmd (59236): umount /mnt/disk1 Using @cookwjc's output, I used lsof to find the offender root@unraid:/mnt# lsof | grep disk1 bash 24735 root cwd DIR 9,1 4096 133 /mnt/disk1/backups/QNAP/qpkg I then killed the PID kill 24735 # or kill -9 24735 if it's safe and it's not responding The array was then able to finish stopping. It would be nice if unRAID would be able to detect these types of issues. Thanks for the tip @cookwjc! Edited September 11, 2020 by MrLinux 3 Quote Link to comment
Knutowskie Posted June 19, 2021 Share Posted June 19, 2021 as this is the first site google spits out with the search "unraid array won't stop" i will try to append my problem. above mentioned solution does not always work: root@D1:/var/log# tail -f syslog Jun 20 01:15:09 Dose4000 emhttpd: Unmounting disks... Jun 20 01:15:09 Dose4000 emhttpd: shcmd (4331): umount /mnt/disk1 Jun 20 01:15:09 Dose4000 root: umount: /mnt/disk1: target is busy. Jun 20 01:15:09 Dose4000 emhttpd: shcmd (4331): exit status: 32 Jun 20 01:15:09 Dose4000 emhttpd: Retry unmounting disk share(s)... Jun 20 01:15:14 Dose4000 emhttpd: Unmounting disks... Jun 20 01:15:14 Dose4000 emhttpd: shcmd (4332): umount /mnt/disk1 Jun 20 01:15:14 Dose4000 root: umount: /mnt/disk1: target is busy. Jun 20 01:15:14 Dose4000 emhttpd: shcmd (4332): exit status: 32 Jun 20 01:15:14 Dose4000 emhttpd: Retry unmounting disk share(s)... ^C root@D1:/var/log# lsof | grep disk1 root@D1:/var/log# so... what now? Quote Link to comment
Futanada Posted February 4, 2022 Share Posted February 4, 2022 In my case, syslog showed disk2 and disk3 are busy. And it's because I have SSH tunnel opened and cd'ed to disk2 and disk3. Once I terminated the SSH, the arrays can be stopped immediately. 1 Quote Link to comment
Squid Posted February 4, 2022 Share Posted February 4, 2022 There is the Dynamix Stop Shell plugin which you can install which should catch this stuff and terminate the shells accordingly. Quote Link to comment
Triumph Cinema Posted October 17, 2022 Share Posted October 17, 2022 I just now recently had this same issue as I was troubleshooting some dockers and then I realized I still had my FTP connection open. So if anyone else runs into this make sure you have closed your FTP/SMB connections Quote Link to comment
supacupa Posted April 15, 2023 Share Posted April 15, 2023 On 12/2/2016 at 11:14 PM, cookwjc said: As I was finalizing the debug information for the above post I was able to solve the issue and considered not posting. On the other hand, I've read far too many threads where the author solves their own problem and abandons the thread. So, just in case someone finds themselves in a similar situation in the future, the resolution to this particular problem was to change directory in my active bash session out of the /mnt/user path... Thanks! That helped a lot! I wish more people did this! Quote Link to comment
cherrybullet Posted November 25, 2023 Share Posted November 25, 2023 On 9/11/2020 at 9:30 AM, MrLinux said: For anyone else running into this issue (I found this post from google) Thanks for the tip. I also used # Watch the log to see which disk is preventing the array from stopping tail -f /var/log/syslog In my case, it was disk1 Sep 11 09:22:04 unraid root: umount: /mnt/disk1: target is busy. <-------------------- Sep 11 09:22:04 unraid emhttpd: shcmd (59235): exit status: 32 Sep 11 09:22:04 unraid emhttpd: Retry unmounting disk share(s)... Sep 11 09:22:09 unraid emhttpd: Unmounting disks... Sep 11 09:22:09 unraid emhttpd: shcmd (59236): umount /mnt/disk1 Using @cookwjc's output, I used lsof to find the offender root@unraid:/mnt# lsof | grep disk1 bash 24735 root cwd DIR 9,1 4096 133 /mnt/disk1/backups/QNAP/qpkg I then killed the PID kill 24735 # or kill -9 24735 if it's safe and it's not responding The array was then able to finish stopping. It would be nice if unRAID would be able to detect these types of issues. Thanks for the tip @cookwjc! Thank you for the solution! But I wonder, why can't unraid do this itself? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.