fuser, lsof unresponsive

[kaiguy]

Hello!

Woke up this morning and tried to access the webgui, and sure enough, no luck. Shares can be accessed; I can ssh into the server...

 

I was running a hacked preclear_disk.sh in a screen session, and that too seemed to be hung... maybe because of my once-a-week mover job? So I went ahead and control-c'd the job and exited out of screen. Tried kicking off a shutdown command but nothing happened. Figuring there's something keeping the array from unmounting, I tried executing a fuser command (fuser -mv /mnt/disk* /mnt/user/*) but it just hangs. lsof, too. I've tried to kill whatever PIDs look like it may be holding it up, but that doesn't seem to be doing anything.

 

I've exhausted all of my knowledge and forum searching at this point. Not sure what else I can do to get a clean shutdown. Any suggestions?

[pwm]

Do you have any processes in the "D" state? That is a non-interruptible wait for I/O that in some situations can be quite problematic because such processes can't be killed.

[kaiguy]

1 hour ago, pwm said:

Do you have any processes in the "D" state? That is a non-interruptible wait for I/O that in some situations can be quite problematic because such processes can't be killed.

Yeah, quite a few. I guess a dirty shutdown it is. 

 

[pwm]

What processes do you see in "D" state?

 

Note that some of them may exit that state of themselves when whatever they are waiting for gets fulfilled or if they possibly just have some ridiculously long timeout specified. Some may wait for some networking, some for disk data etc. But the implementation of non-interruptible wait in the Linux kernel is one of the shameful "secrets" that can create lots of trouble for end users. Linux processes will only see signals (like "kill -9") after they return from the non-interruptible wait which is why it isn't possible to just kill them.