Open Files Plugin - Can help with troubleshooting why server won't shut down


dlandon

Recommended Posts

A technical question.... when I go to Main and click on button to Stop the array... it normally goes through the process of stopping the dockers and all the other stuff. When something is not going well, I get the "Retrying to..." (cant recall exact words) that just keeps cycling over.

 

If one is stuck in this loop, are we able to then click on the Open Files link and it will take us to the Open Files page with the ability to kill the open stuff? In the past, when I am in one of those loops, I cannot go navigate anywhere on the unRAID web ui.

 

I would rather ask this question than try to make my system go unto a loop to test.  :P

 

Thanks!

Link to comment

A technical question.... when I go to Main and click on button to Stop the array... it normally goes through the process of stopping the dockers and all the other stuff. When something is not going well, I get the "Retrying to..." (cant recall exact words) that just keeps cycling over.

 

If one is stuck in this loop, are we able to then click on the Open Files link and it will take us to the Open Files page with the ability to kill the open stuff? In the past, when I am in one of those loops, I cannot go navigate anywhere on the unRAID web ui.

 

I would rather ask this question than try to make my system go unto a loop to test.  :P

 

Thanks!

I've been wondering the same thing. I was getting a web page that said my system is rebooting when my shutdowns hung. Tried to test to see if I could get back to the webgui but now that the System Buttons plugin has been changed to default to Powerdown shutdown I can't get a hang. My suspicion is that I won't be able to see the webgui though.

Link to comment

A technical question.... when I go to Main and click on button to Stop the array... it normally goes through the process of stopping the dockers and all the other stuff. When something is not going well, I get the "Retrying to..." (cant recall exact words) that just keeps cycling over.

 

If one is stuck in this loop, are we able to then click on the Open Files link and it will take us to the Open Files page with the ability to kill the open stuff? In the past, when I am in one of those loops, I cannot go navigate anywhere on the unRAID web ui.

 

I would rather ask this question than try to make my system go unto a loop to test.  :P

 

Thanks!

I've been wondering the same thing. I was getting a web page that said my system is rebooting when my shutdowns hung. Tried to test to see if I could get back to the webgui but now that the System Buttons plugin has been changed to default to Powerdown shutdown I can't get a hang. My suspicion is that I won't be able to see the webgui though.

 

Why can't you have two browse sessions at the same time?  One on the files open webgui and the other on Main doing the shutdown.  Do the shutdown and then switch to the other browse session with files open, refresh the page, and get down to what is holding up the shutdown.

Link to comment

I've been wondering the same thing. I was getting a web page that said my system is rebooting when my shutdowns hung. Tried to test to see if I could get back to the webgui but now that the System Buttons plugin has been changed to default to Powerdown shutdown I can't get a hang. My suspicion is that I won't be able to see the webgui though.

 

Why can't you have two browse sessions at the same time?  One on the files open webgui and the other on Main doing the shutdown.  Do the shutdown and then switch to the other browse session with files open, refresh the page, and get down to what is holding up the shutdown.

 

Perhaps then, on the Main page, some instruction saying something like:

 

    "Before initiating shutdown, click on link below, and come back to this page to initiate shutdown sequence."

 

Clicking the link will open page in new Tab or window. And in the Open Files window, you have a refresh button that sends the actual command and not a web page refresh that may get stuck because of something happening with the array stop procedure. I know from past experience that the web ui is pretty unforgiving when stuck on these loops. 

 

This would kick tail if it works properly!

 

 

 

Link to comment

I installed the plg from the first version and couldnt find the option to view open files in the Tools section (using untabbed display)

 

after the update to display open files in main, i can click the 'open files' link which takes me to the detailed open files page but still nothing shows up in the tools section

 

i have tried the tabbed display setting but this does not affect it.

 

when i go to the tools tab at the menu bar, should i see a section for open files?

 

Screenshots attached.

Main.png.27fd8892d4c36e4f32242cf59b730f83.png

Open_Files.png.9aaeaa57ee020ab968b6252c3f53f144.png

Tools.png.006f1452553f3ec5d86be5c8827ada92.png

Link to comment

I did a few quick tests on my test server and all I came up with was a hung webgui.

 

So how would one use this? Is there a way to send the command lines NOT on the webui.... something that sends the command to the console some way. Or create a console script that lists running process and allows you to kill them.

Link to comment

No there is nothing below the about section, running 6.1.2

 

The plugin has an issue when there is no other plugin installed at 'Tools->System Information'.  It is not creating that grouping and therefore does not show up.  If another plugin is installed there such as dynamix active streams, it will show up.  I'm working on the solution.

Link to comment

I did a few quick tests on my test server and all I came up with was a hung webgui.

 

So how would one use this? Is there a way to send the command lines NOT on the webui.... something that sends the command to the console some way. Or create a console script that lists running process and allows you to kill them.

The intended use was to check for issues using this plugin BEFORE you try to shut down the array, and then once the plugin gives the all clear, do the shutdown using the webui. It was suggested that the plugin should intercept the shutdown attempt if it detected issues and allow you to intervene before passing through the shutdown command, but that was deemed too intrusive for a non-limetech supported plugin.
Link to comment

No there is nothing below the about section, running 6.1.2

 

The plugin has an issue when there is no other plugin installed at 'Tools->System Information'.  It is not creating that grouping and therefore does not show up.  If another plugin is installed there such as dynamix active streams, it will show up.  I'm working on the solution.

Yes there is nothing installed in that category

 

Thanks dlandon

Link to comment

No there is nothing below the about section, running 6.1.2

 

The plugin has an issue when there is no other plugin installed at 'Tools->System Information'.  It is not creating that grouping and therefore does not show up.  If another plugin is installed there such as dynamix active streams, it will show up.  I'm working on the solution.

At work right now, but its probably the same as when plugins had to include the utilities.page file.  If you look in the folder for active streams, you'll probably see another .page file called something like system utilizes that is basically empty.  Just copy that file into your folder. 

Link to comment

I did a few quick tests on my test server and all I came up with was a hung webgui.

 

So how would one use this? Is there a way to send the command lines NOT on the webui.... something that sends the command to the console some way. Or create a console script that lists running process and allows you to kill them.

 

I have come up with an idea that I think will make this plugin more useful in troubleshooting shutdown problems.  I have extracted a part of the powerdown plugin script that essentially stops all array processes and leaves the array on line.  This will put the system in the state of all processes on the array being stopped so you can use open files to see what processes are left with open files on the array.  You can Kill the processes on the array holding files open, or better yet troubleshoot why the processes is holding the files open and solve that problem so a normal webgui shutdown can occur.

 

This script is executed in the open files webgui by a button click and will terminate when done and will not hang waiting for a process to terminate.  There is some status of what is happening on the bottom of the screen similar to what happens when the webgui shutdown is used.

 

I don't intend for this to substitute for a normal shutdown.  It is a tool to be used to troubleshoot shutdown problems.

 

I'll work on it some more to be sure it works as advertised before releasing it.

 

I will probably remove the 'Open Files' tab on Main, because it is not really needed if this idea of mine works, and is a bit messy.  You'll be able to do everything you need from the open files webgui.

Link to comment

I did a few quick tests on my test server and all I came up with was a hung webgui.

 

So how would one use this? Is there a way to send the command lines NOT on the webui.... something that sends the command to the console some way. Or create a console script that lists running process and allows you to kill them.

The intended use was to check for issues using this plugin BEFORE you try to shut down the array, and then once the plugin gives the all clear, do the shutdown using the webui. It was suggested that the plugin should intercept the shutdown attempt if it detected issues and allow you to intervene before passing through the shutdown command, but that was deemed too intrusive for a non-limetech supported plugin.

I must be missing something. What exactly is the "All Clear"? All I see is a list of running processes and kill buttons.

Link to comment

No there is nothing below the about section, running 6.1.2

 

The plugin has an issue when there is no other plugin installed at 'Tools->System Information'.  It is not creating that grouping and therefore does not show up.  If another plugin is installed there such as dynamix active streams, it will show up.  I'm working on the solution.

At work right now, but its probably the same as when plugins had to include the utilities.page file.  If you look in the folder for active streams, you'll probably see another .page file called something like system utilizes that is basically empty.  Just copy that file into your folder.

 

Ok.  I'll take a look.

Link to comment

I did a few quick tests on my test server and all I came up with was a hung webgui.

 

So how would one use this? Is there a way to send the command lines NOT on the webui.... something that sends the command to the console some way. Or create a console script that lists running process and allows you to kill them.

The intended use was to check for issues using this plugin BEFORE you try to shut down the array, and then once the plugin gives the all clear, do the shutdown using the webui. It was suggested that the plugin should intercept the shutdown attempt if it detected issues and allow you to intervene before passing through the shutdown command, but that was deemed too intrusive for a non-limetech supported plugin.

I must be missing something. What exactly is the "All Clear"? All I see is a list of running processes and kill buttons.

 

Read the previous post that will make this process a lot easier and more foolproof.

Link to comment

Read the previous post that will make this process a lot easier and more foolproof.

Yes. Thanks. The posts were only a few seconds apart so I didn't see yours until after I had already posted.

 

When running the revised script, only hung processes will still be listed in the plugin right? And then the procedure would be to kill any processes still running and proceed with a reboot? Or does the script just exit back to a normally operating server when you click the "Done" button.

 

Or maybe I should save my question until you have actually completed and released it. :D

Link to comment

Read the previous post that will make this process a lot easier and more foolproof.

Yes. Thanks. The posts were only a few seconds apart so I didn't see yours until after I had already posted.

 

When running the revised script, only hung processes will still be listed in the plugin right? And then the procedure would be to kill any processes still running and proceed with a reboot? Or does the script just exit back to a normally operating server when you click the "Done" button.

 

Or maybe I should save my question until you have actually completed and released it. :D

 

What I am doing is getting the array in a state where all processes are stopped: Plugins, docker, VMs, samba, nfs, avahi, etc.  The array is not stopped.  It will stay this way until you go to the 'Main->Array Operation' and choose what to do there.  If you want to reboot then stop the array and reboot as normal.  Or you can re-start the array by stopping and then starting

Link to comment

What I am doing is getting the array in a state where all processes are stopped: Plugins, docker, VMs, samba, nfs, avahi, etc.  The array is not stopped.  It will stay this way until you go to the 'Main->Array Operation' and choose what to do there.  If you want to reboot then stop the array and reboot as normal.  Or you can re-start the array by stopping and then starting

 

Soooo, its going to be like a 2 step process...? Step 1 stops dockers, plugins, etc... Step 2 would be the scanning and ability to kill leftover running processes not caught in step 1.

 

Step 3 would be to stop the array and any other stock unRAID operation on Main...

 

Is it possible that step 1 or 2 will "kill" anything needed for the web ui?

 

If correct, this sounds good to me....  8)

 

Thanks for working on this... I have pulled a lot of hair from my head over the years with the shutdown/reboot process.

 

 

Link to comment

What I am doing is getting the array in a state where all processes are stopped: Plugins, docker, VMs, samba, nfs, avahi, etc.  The array is not stopped.  It will stay this way until you go to the 'Main->Array Operation' and choose what to do there.  If you want to reboot then stop the array and reboot as normal.  Or you can re-start the array by stopping and then starting

 

Soooo, its going to be like a 2 step process...? Step 1 stops dockers, plugins, etc... Step 2 would be the scanning and ability to kill leftover running processes not caught in step 1.

 

Step 3 would be to stop the array and any other stock unRAID operation on Main...

 

Is it possible that step 1 or 2 will "kill" anything needed for the web ui?

 

If correct, this sounds good to me....  8)

 

Thanks for working on this... I have pulled a lot of hair from my head over the years with the shutdown/reboot process.

 

It's exactly the same as clicking the stop button but the array is not stopped.  The webgui is alive and you can stop the array as normal and then start it back up to normal operation, reboot, or shutdown.  If you kill all the processes holding files open, you will get a clean shut down.  I believe this state is the same as the maintenance mode - array is started and drives are mounted.

 

So the array ends up in a state where all processes on the array are supposed to be stopped, but the array is not stopped.  If the array is stopped with processes still having open files on the array, the array is marked as in an unclean shutdown state and a parity check will happen on the next boot.

 

Because the array is not stopped, an abrupt loss of power or abrupt shutdown (pressing the reset button), will cause an unclean shutdown.  You will want to treat this state the exact same way as the array is running and drives are mounted - which they are.

 

I suspect this process is going to show some weaknesses in plugins not stopping properly.  This is the most common cause of the 'hangs' - a plugin fails to shutdown properly, or has spawned child processes that continue to run.

 

My hope is that this will help all of us by making the process of finding the shutdown issues easier for the all of us, especially the novice.  Geez, I have problems finding these kinds of issues and I think I know what I am doing.  (Although some might question that).

 

I don't intend for this to be the normal shutdown procedure.  I only offer it up for those situations where a clean shutdown is not occurring and one wants to find the problem.

Link to comment

I appreciate everyone's help with bringing this plugin online and making it into what I feel now is a valuable tool.  Some of the names of those who contributed I recognize as being some of the best on the LimeTech forum.  Thanks.

 

I knew the work bubbaQ did would be valuable, but wasn't exactly sure how to make the most of it.  I think now we have something that can work effectively.

 

I am working remotely and will put the final touches on the plugin when I get home before I release a new version this evening.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.