Array Won't Stop [SOLVED]

[cookwjc]

Hi All,

I have a situation that is occurring, which is preventing me from being able to stop the array or even successfully issue a shutdown command from the CLI. 

I did find one article from 2013 referencing an issue with RC13, but I haven't found any discussion regarding newer releases.

 

I'm running 6.2.4 with 14 drives + parity + boot; 12GB RAM; 2 - Xeon 1.??GHz.  The system has been stable until this last week when I removed a handful of unused Docker containers and was preparing to shrink the array following 'Clear Drive Then Remove Drive' method from the following article:

https://lime-technology.com/wiki/index.php/Shrink_array

 

I (of course) wouldn't expect any changes to have led to this behavior, but I wanted to present the 'what's changed?' answers up front.

 

Dynamix WebUI continues to report 'Spinning up all drives...' however the device lists reports all drives were spun up before initiating the stop command.  Also of note, the WebUI is no longer responsive and will not allow new sessions to be established.

 

From the CLI, docker images reports the docker daemon is not currently running (which it was before initiating the array stop) and ps eaf 6 serial TTYs listening, my active pts session, and the ps eaf query, so there don't appear to be any active handles open to mounted disks. The command "fuser -mv /mnt/disk* /mnt/user/*" does return a number of processes items pasted below due to the length of snip.

 

 

Syslog output shows the following errors recurring:

Dec  2 23:44:04 CENTRAL emhttp: Retry unmounting user share(s)...
Dec  2 23:44:09 CENTRAL emhttp: shcmd (12033): set -o pipefail ; umount /mnt/user |& logger
Dec  2 23:44:09 CENTRAL root: umount: /mnt/user: target is busy
Dec  2 23:44:09 CENTRAL root:         (In some cases useful info about processes that
Dec  2 23:44:09 CENTRAL root:          use the device is found by lsof( or fuser(1).)
Dec  2 23:44:09 CENTRAL emhttp: err: shcmd: shcmd (12033): exit status: 32
Dec  2 23:44:09 CENTRAL emhttp: shcmd (12034): rmdir /mnt/user |& logger
Dec  2 23:44:09 CENTRAL root: rmdir: failed to remove '/mnt/user': Device or resource busy
Dec  2 23:44:09 CENTRAL emhttp: shcmd (12035): rm -f /boot/config/plugins/dynamix/mover.cron
Dec  2 23:44:09 CENTRAL emhttp: shcmd (12036): /usr/local/sbin/update_cron &> /dev/null
Dec  2 23:44:09 CENTRAL emhttp: Retry unmounting user share(s)...

 

[cookwjc]

As I was finalizing the debug information for the above post I was able to solve the issue and considered not posting.  On the other hand, I've read far too many threads where the author solves their own problem and abandons the thread.  So, just in case someone finds themselves in a similar situation in the future, the resolution to this particular problem was to change directory in my active bash session out of the /mnt/user path...

[MrLinux]

For anyone else running into this issue (I found this post from google)

 

Thanks for the tip. I also used

# Watch the log to see which disk is preventing the array from stopping
tail -f /var/log/syslog

In my case, it was disk1

Sep 11 09:22:04 unraid root: umount: /mnt/disk1: target is busy. <--------------------
Sep 11 09:22:04 unraid emhttpd: shcmd (59235): exit status: 32
Sep 11 09:22:04 unraid emhttpd: Retry unmounting disk share(s)...
Sep 11 09:22:09 unraid emhttpd: Unmounting disks...
Sep 11 09:22:09 unraid emhttpd: shcmd (59236): umount /mnt/disk1

Using @cookwjc's output, I used lsof to find the offender

root@unraid:/mnt# lsof | grep disk1
bash      24735                   root  cwd       DIR                9,1     4096        133 /mnt/disk1/backups/QNAP/qpkg

I then killed the PID

kill 24735 
# or kill -9 24735 if it's safe and it's not responding

The array was then able to finish stopping. It would be nice if unRAID would be able to detect these types of issues.

 

Thanks for the tip @cookwjc!

Edited September 11, 2020 by MrLinux

[Knutowskie]

as this is the first site google spits out with the search "unraid array won't stop" i will try to append my problem. above mentioned solution does not always work:

 

root@D1:/var/log# tail -f syslog 
Jun 20 01:15:09 Dose4000 emhttpd: Unmounting disks...
Jun 20 01:15:09 Dose4000 emhttpd: shcmd (4331): umount /mnt/disk1
Jun 20 01:15:09 Dose4000 root: umount: /mnt/disk1: target is busy.
Jun 20 01:15:09 Dose4000 emhttpd: shcmd (4331): exit status: 32
Jun 20 01:15:09 Dose4000 emhttpd: Retry unmounting disk share(s)...
Jun 20 01:15:14 Dose4000 emhttpd: Unmounting disks...
Jun 20 01:15:14 Dose4000 emhttpd: shcmd (4332): umount /mnt/disk1
Jun 20 01:15:14 Dose4000 root: umount: /mnt/disk1: target is busy.
Jun 20 01:15:14 Dose4000 emhttpd: shcmd (4332): exit status: 32
Jun 20 01:15:14 Dose4000 emhttpd: Retry unmounting disk share(s)...
^C
root@D1:/var/log# lsof | grep disk1
root@D1:/var/log#

 

so... what now?

[Futanada]

In my case, syslog showed disk2 and disk3 are busy. And it's because I have SSH tunnel opened and cd'ed to disk2 and disk3. Once I terminated the SSH, the arrays can be stopped immediately.

[Squid]

There is the Dynamix Stop Shell plugin which you can install which should catch this stuff and terminate the shells accordingly.

[Triumph Cinema]

I just now recently had this same issue as I was troubleshooting some dockers and then I realized I still had my FTP connection open. So if anyone else runs into this make sure you have closed your FTP/SMB connections

[supacupa]

On 12/2/2016 at 11:14 PM, cookwjc said:

As I was finalizing the debug information for the above post I was able to solve the issue and considered not posting.  On the other hand, I've read far too many threads where the author solves their own problem and abandons the thread.  So, just in case someone finds themselves in a similar situation in the future, the resolution to this particular problem was to change directory in my active bash session out of the /mnt/user path...

Thanks! That helped a lot! I wish more people did this!

[cherrybullet]

On 9/11/2020 at 9:30 AM, MrLinux said:

For anyone else running into this issue (I found this post from google)

 

Thanks for the tip. I also used

# Watch the log to see which disk is preventing the array from stopping
tail -f /var/log/syslog

In my case, it was disk1

Sep 11 09:22:04 unraid root: umount: /mnt/disk1: target is busy. <--------------------
Sep 11 09:22:04 unraid emhttpd: shcmd (59235): exit status: 32
Sep 11 09:22:04 unraid emhttpd: Retry unmounting disk share(s)...
Sep 11 09:22:09 unraid emhttpd: Unmounting disks...
Sep 11 09:22:09 unraid emhttpd: shcmd (59236): umount /mnt/disk1

Using @cookwjc's output, I used lsof to find the offender

root@unraid:/mnt# lsof | grep disk1
bash      24735                   root  cwd       DIR                9,1     4096        133 /mnt/disk1/backups/QNAP/qpkg

I then killed the PID

kill 24735 
# or kill -9 24735 if it's safe and it's not responding

The array was then able to finish stopping. It would be nice if unRAID would be able to detect these types of issues.

 

Thanks for the tip @cookwjc!

Thank you for the solution!

 

But I wonder, why can't unraid do this itself?