Jump to content
dlandon

Unassigned Devices - Managing Disk Drives and Remote Shares Outside of The Unraid Array

4546 posts in this topic Last Reply

Recommended Posts

For anyone having issues with UD appearing to hang, please extract and copy the attached lib.php file.

cp lib.php /usr/local/emhttp/plugins/unassigned.devices/include/

I've added timeouts to all shell_exec commands to terminate any hanging commands with logging if it takes too long.  You'll see a warning in the log.  Hopefully UD won't hang any more.

 

lib.zip

Share this post


Link to post
19 hours ago, dlandon said:

You mounted the seagate disk.  I don't see anything in the log,  but I see issues in the smart report:


SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)

7 Seek_Error_Rate         POSR--   084   060   045    -    256581222

195 Hardware_ECC_Recovered  -O-RC-   079   064   000    -    87027788

I am not an expert on disk drives, but this looks like a controller or cable issue.  Disk problems can cause UD to hang if the Linux commands used to check the disk hang.

Good catch, thanks....better brush up on my smart report skills. Would it matter if I pulled the diagnostics after I had rebooted and uninstalled -UD? I'm going to see if it'll hang again and get diagnostics downloaded before it becomes completely unresponsive. Thanks again!

Share this post


Link to post
1 hour ago, geonerdist said:

Good catch, thanks....better brush up on my smart report skills. Would it matter if I pulled the diagnostics after I had rebooted and uninstalled -UD? I'm going to see if it'll hang again and get diagnostics downloaded before it becomes completely unresponsive. Thanks again!

Read the prior post.  I'd like you to try that to see if that helps keep UD from hanging.  After you copy the file, work with UD a bit to see if it hangs.  Post diagnostics so I can see where things go wrong.

Share this post


Link to post

Hello @dlandon,

 

Happy Holidays one and all!

 

So, I just had a power outage and the UPS shutdown executed as expected. However, (and I neglected to address this way back when I discovered it, ugh) my Unassigned Devices (UD) do not disconnect during shutdown, if those devices are already off (power outage or not), and the primary halts shutting down.

 

I was able to complete the shutdown and reboot after restarting the other servers UD was connected to prior to power outage. My question is, and please bear with me as I am trying to learn, in an effort to stop this from happening in the future, do I open Putty and copy (cp) the ‘lib.php’ to the ‘/usr/local/emhttp/plugins/unassigned.devices/include/’ directory and restart the primary server?

 

I attached the Diag log and my setup is as follows:

I have a dual Mellanox 10Gb NIC in the primary server and single Mellanox 10Gb NIC in two other servers. There is an NFS connection through UD on the primary server to each of the other two servers. If the connections in UD, on the primary server, are not disconnected before the other two servers shutdown, the primary will not execute a clean shutdown and hangs until the other two servers are back up.

 

I am open to any directions, suggestions, solutions.

Thanks in advance for your patience and any assistance!

 

tower01-diagnostics-20191129-0019.zip

Share this post


Link to post
3 minutes ago, TechMed said:

My question is, and please bear with me as I am trying to learn, in an effort to stop this from happening in the future, do I open Putty and copy (cp) the ‘lib.php’ to the ‘/usr/local/emhttp/plugins/unassigned.devices/include/’ directory and restart the primary server?

Just copy it and that's all.  If you reboot, the file will be replaced by the UD plugin when the array is started.

 

The lib.php file is a test version I want tested by people that have issues with UD hanging.  I'll release it in an updated plugin once I get some feedback on if it is helping and not creating problems.

Share this post


Link to post

 

4 minutes ago, dlandon said:

The lib.php file is a test version I want tested by people that have issues with UD hanging.  I'll release it in an updated plugin once I get some feedback on if it is helping and not creating problems.

Got it!

 

I do not want to dirty the thread, but for the benefit of the student, do you know where can I research how to write a (in DOS parlance) ".bat" file that can be run before the actual 'shutdown' command is sent?

 

(and thanks for the quick reply!)

Share this post


Link to post
19 minutes ago, TechMed said:

Hello @dlandon,

 

Happy Holidays one and all!

 

So, I just had a power outage and the UPS shutdown executed as expected. However, (and I neglected to address this way back when I discovered it, ugh) my Unassigned Devices (UD) do not disconnect during shutdown, if those devices are already off (power outage or not), and the primary halts shutting down.

 

I was able to complete the shutdown and reboot after restarting the other servers UD was connected to prior to power outage. My question is, and please bear with me as I am trying to learn, in an effort to stop this from happening in the future, do I open Putty and copy (cp) the ‘lib.php’ to the ‘/usr/local/emhttp/plugins/unassigned.devices/include/’ directory and restart the primary server?

 

I attached the Diag log and my setup is as follows:

I have a dual Mellanox 10Gb NIC in the primary server and single Mellanox 10Gb NIC in two other servers. There is an NFS connection through UD on the primary server to each of the other two servers. If the connections in UD, on the primary server, are not disconnected before the other two servers shutdown, the primary will not execute a clean shutdown and hangs until the other two servers are back up.

 

I am open to any directions, suggestions, solutions.

Thanks in advance for your patience and any assistance!

 

tower01-diagnostics-20191129-0019.zip 121.61 kB · 1 download

When there is an unclean shutdown, a diagnostics fie is created if possible at /flash/logs/.  Please post it if there is one.

Share this post


Link to post
18 minutes ago, TechMed said:

Some log entries:

Nov 28 18:09:56 Tower01 unassigned.devices: Unmount cmd: /bin/umount -fl '10.69.69.240://mnt/user/black_hole' 2>&1

Nov 28 18:19:59 Tower01 kernel: nfs: server 10.69.69.240 not responding, still trying
### [PREVIOUS LINE REPEATED 1 TIMES] ###

Nov 28 19:00:20 Tower01 unassigned.devices: Successfully unmounted '10.69.69.240://mnt/user/black_hole'

It looks like the unount of the nfs share is stuck.  The unmount '-fl' parameter is supposed to force an unmount.  Looks like that is taking a long time.  The new timeout on the 'umount' should help.

 

 

Share this post


Link to post

Thanks much for taking a look at it.

 

If this helps at all, when I catch this happening I restart the server that is down and the shutdown proceeds as expected.

 

Thanks again for the info.

Share this post


Link to post
6 minutes ago, TechMed said:

If this helps at all, when I catch this happening I restart the server that is down and the shutdown proceeds as expected.

This is apparently because the unmount is able to complete.

Share this post


Link to post

That is the assumption I was running with; stumbled across it by shear luck! haha

 

Despite the occasional head scratching, I really appreciate the tool.

For me it makes transferring files between servers, regardless if it between floors or cities, much easier.

 

Thanks again for the assist.

Share this post


Link to post

I set up a Backup server sitting right next to my Primary server. From Primary, I used UD to SMB mount a share on Backup, and I'm using Duplicati to run the backups from Primary to Backup. Backup had been assigned a static IP address and all was good. In preparation for moving Backup off-site I switched it from static to DHCP (I'm not sure of networking at its new home-to-be) and now Primary cannot access it via the UD mount point. This makes some sense to me since its IP address changed (though I'd specified the UD mount by name, not IP) - I believe this is because the DNS cache has not yet updated to recognize Backup at its new IP address. 

 

Having done this yesterday evening, this morning (about 12 hours later) it still wasn't properly recognizing the server at its new IP. (in UD, I clicked the mount point to browse and all it showed me was "parent folder"). This morning I unmounted and remounted the share (it took quite a while to unmount, but the remount went very quickly), and now I can properly browse the directory structure via UD's mount point share. 

 

A couple of questions:

1) Is it normally necessary to unmount/remount when an IP address changes or should I normally just have to wait out the DNS cache update?

2) Was it something else totally that was the issue here?

3) Once the server goes to its remote location, it will be on a totally different subnet as I will be using the great ZeroTier plug in to get the two of them to talk to each other, I presume that going from a 192.168.* address to a 172.29.* address will not have any impact on UD's ability to map this, correct?

4) Is SMB the best option for having the 2 servers talk to each other, or would I be better off using an NFS mount? (If so, I'd probably keep the SMB mount in order to be able to browse via Windows, just because I can.)

 

Thanks (again, I hope) for picking this up and continuing to run with it! 

Share this post


Link to post

Hi @FreeMan,

 

First I am FAR from an expert, but coincidentally I am actually backing up systems right now.

There are folks here significantly more knowledgeable about UD than myself. That said I can speak to #3 & #4 from personal experience.

 

#3 - No problems. Have  read through the link in #4 and you should be golden!

 

#4 - I think most everyone will agree that a NFS connection is better.

       With respect to the connection, I HIGHLY recommend looking at this link as @tr0910 (and others) put in a great deal of work to make backups a breeze. I will not lie, it is a bit of work, but not only is it worth the work, it is a fast efficient and @#$%ed near fool-proof method to doing backups. I too started with Duplicati and bailed after the second multi-day backup; Rsync is WAY better.

 

Have fun! This is actually an enjoyable project!

Share this post


Link to post

A plugin update fails with the following lines:

plugin: updating: unassigned.devices.plg
plugin: downloading: "https://github.com/dlandon/unassigned.devices/raw/master/unassigned.devices-2019.11.29.tgz" ... done
plugin: bad file MD5: /boot/config/plugins/unassigned.devices/packages/unassigned.devices-2019.11.29.tgz

Somebody else having this issue?

Share this post


Link to post

I'm also having the same issues. If you want to fix it yourself right now, download the .plg to somewhere on your server, change the 12th line so that the md5 sum is:

 

8796a7b713b0d121385f1452eb7880b1

 

and then tell unRAID to install from that downloaded plg file. Otherwise, I expect it will be fixed shortly.

Share this post


Link to post
34 minutes ago, bamhm182 said:

I'm also having the same issues. If you want to fix it yourself right now, download the .plg to somewhere on your server, change the 12th line so that the md5 sum is:

 

8796a7b713b0d121385f1452eb7880b1

 

and then tell unRAID to install from that downloaded plg file. Otherwise, I expect it will be fixed shortly.

Where i find the .plg ? After Reboot  my Server the unassigend device / plugin its gone 😞 

 

PS: Find it ... Hope it will help

Edited by KinGSiZ3
Find it

Share this post


Link to post

You can run these commands:

cd $(mktemp -d)
wget https://raw.githubusercontent.com/dlandon/unassigned.devices/master/unassigned.devices.plg
sed -i s/fc6f30a2f824f5b56e367ed73edea275/8796a7b713b0d121385f1452eb7880b1/g unassigned.devices.plg

 

Then you can go to the Plugins tab > Install Plugin, and then paste the file location in there, for example: /tmp/tmp.tivBjv7aFZ/unassigned.devices.plg and click install.

 

Edited by bamhm182

Share this post


Link to post
4 minutes ago, bamhm182 said:

You can run these commands:


cd $(mktemp -d)
wget https://raw.githubusercontent.com/dlandon/unassigned.devices/master/unassigned.devices.plg
sed -i s/fc6f30a2f824f5b56e367ed73edea275/8796a7b713b0d121385f1452eb7880b1/g unassigned.devices.plg

 

Then you can go to the Plugins tab > Install Plugin, and then paste the file location in there, for example: /tmp/tmp.tivBjv7aFZ/unassigned.devices.plg and click install.

 

Really you are my daily Hero today ... THX for the help. All working now :-*

Edited by KinGSiZ3

Share this post


Link to post

No worries. dlandon is the real hero, I'm just a guy that needed something done and figured out how to get it done. I'm sure he'll fix it very soon. :)

Share this post


Link to post
15 minutes ago, bamhm182 said:

No worries. dlandon is the real hero, I'm just a guy that needed something done and figured out how to get it done. I'm sure he'll fix it very soon. :)

Fixed.  Update again.

Share this post


Link to post
27 minutes ago, KinGSiZ3 said:

Where i find the .plg ? After Reboot  my Server the unassigend device / plugin its gone 😞 

 

PS: Find it ... Hope it will help

Download it again from CA.

Share this post


Link to post

I was finally able to reproduce some of the hangs people have experienced.  I had to add code to handle some Linux commands that hang when a remote CIFS share server goes offline.  Some commands that didn't even have anything to do with the CIFS mounts.

 

All Linux commands now have time outs to prevent hangs.  You'll see error messages in the log when a command times out.  I may have to extend some time outs for different systems.

 

Anyone having the UD hang, please test and then give me a diagnostics regardless if you have issues or not.  I need to see if the time outs are appropriate for all cases.

Share this post


Link to post

Hi @dlandon,

 

Just double checking, you want the logs from '/flash/logs/' correct?

I will play with this over the weekend and post for you.

 

Thanks again for all you do!

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.