1 hour ago, geonerdist said:

Good catch, thanks....better brush up on my smart report skills. Would it matter if I pulled the diagnostics after I had rebooted and uninstalled -UD? I'm going to see if it'll hang again and get diagnostics downloaded before it becomes completely unresponsive. Thanks again!

Read the prior post.  I'd like you to try that to see if that helps keep UD from hanging.  After you copy the file, work with UD a bit to see if it hangs.  Post diagnostics so I can see where things go wrong.

Hello @dlandon,

 

Happy Holidays one and all!

 

So, I just had a power outage and the UPS shutdown executed as expected. However, (and I neglected to address this way back when I discovered it, ugh) my Unassigned Devices (UD) do not disconnect during shutdown, if those devices are already off (power outage or not), and the primary halts shutting down.

 

I was able to complete the shutdown and reboot after restarting the other servers UD was connected to prior to power outage. My question is, and please bear with me as I am trying to learn, in an effort to stop this from happening in the future, do I open Putty and copy (cp) the ‘lib.php’ to the ‘/usr/local/emhttp/plugins/unassigned.devices/include/’ directory and restart the primary server?

 

I attached the Diag log and my setup is as follows:

I have a dual Mellanox 10Gb NIC in the primary server and single Mellanox 10Gb NIC in two other servers. There is an NFS connection through UD on the primary server to each of the other two servers. If the connections in UD, on the primary server, are not disconnected before the other two servers shutdown, the primary will not execute a clean shutdown and hangs until the other two servers are back up.

 

I am open to any directions, suggestions, solutions.

Thanks in advance for your patience and any assistance!

 

tower01-diagnostics-20191129-0019.zip

3 minutes ago, TechMed said:

My question is, and please bear with me as I am trying to learn, in an effort to stop this from happening in the future, do I open Putty and copy (cp) the ‘lib.php’ to the ‘/usr/local/emhttp/plugins/unassigned.devices/include/’ directory and restart the primary server?

Just copy it and that's all.  If you reboot, the file will be replaced by the UD plugin when the array is started.

 

The lib.php file is a test version I want tested by people that have issues with UD hanging.  I'll release it in an updated plugin once I get some feedback on if it is helping and not creating problems.

 

4 minutes ago, dlandon said:

The lib.php file is a test version I want tested by people that have issues with UD hanging.  I'll release it in an updated plugin once I get some feedback on if it is helping and not creating problems.

Got it!

 

I do not want to dirty the thread, but for the benefit of the student, do you know where can I research how to write a (in DOS parlance) ".bat" file that can be run before the actual 'shutdown' command is sent?

 

(and thanks for the quick reply!)

19 minutes ago, TechMed said:

Hello @dlandon,

 

Happy Holidays one and all!

 

So, I just had a power outage and the UPS shutdown executed as expected. However, (and I neglected to address this way back when I discovered it, ugh) my Unassigned Devices (UD) do not disconnect during shutdown, if those devices are already off (power outage or not), and the primary halts shutting down.

 

I was able to complete the shutdown and reboot after restarting the other servers UD was connected to prior to power outage. My question is, and please bear with me as I am trying to learn, in an effort to stop this from happening in the future, do I open Putty and copy (cp) the ‘lib.php’ to the ‘/usr/local/emhttp/plugins/unassigned.devices/include/’ directory and restart the primary server?

 

I attached the Diag log and my setup is as follows:

I have a dual Mellanox 10Gb NIC in the primary server and single Mellanox 10Gb NIC in two other servers. There is an NFS connection through UD on the primary server to each of the other two servers. If the connections in UD, on the primary server, are not disconnected before the other two servers shutdown, the primary will not execute a clean shutdown and hangs until the other two servers are back up.

 

I am open to any directions, suggestions, solutions.

Thanks in advance for your patience and any assistance!

 

tower01-diagnostics-20191129-0019.zip 121.61 kB · 1 download

When there is an unclean shutdown, a diagnostics fie is created if possible at /flash/logs/.  Please post it if there is one.

please see attached...

 

tower01-diagnostics-20191128-1900.zip

18 minutes ago, TechMed said:

please see attached...

 

tower01-diagnostics-20191128-1900.zip 179.13 kB · 0 downloads

Some log entries:

Nov 28 18:09:56 Tower01 unassigned.devices: Unmount cmd: /bin/umount -fl '10.69.69.240://mnt/user/black_hole' 2>&1

Nov 28 18:19:59 Tower01 kernel: nfs: server 10.69.69.240 not responding, still trying
### [PREVIOUS LINE REPEATED 1 TIMES] ###

Nov 28 19:00:20 Tower01 unassigned.devices: Successfully unmounted '10.69.69.240://mnt/user/black_hole'

It looks like the unount of the nfs share is stuck.  The unmount '-fl' parameter is supposed to force an unmount.  Looks like that is taking a long time.  The new timeout on the 'umount' should help.

 

 

Thanks much for taking a look at it.

 

If this helps at all, when I catch this happening I restart the server that is down and the shutdown proceeds as expected.

 

Thanks again for the info.

6 minutes ago, TechMed said:

If this helps at all, when I catch this happening I restart the server that is down and the shutdown proceeds as expected.

This is apparently because the unmount is able to complete.

That is the assumption I was running with; stumbled across it by shear luck! haha

 

Despite the occasional head scratching, I really appreciate the tool.

For me it makes transferring files between servers, regardless if it between floors or cities, much easier.

 

Thanks again for the assist.

I set up a Backup server sitting right next to my Primary server. From Primary, I used UD to SMB mount a share on Backup, and I'm using Duplicati to run the backups from Primary to Backup. Backup had been assigned a static IP address and all was good. In preparation for moving Backup off-site I switched it from static to DHCP (I'm not sure of networking at its new home-to-be) and now Primary cannot access it via the UD mount point. This makes some sense to me since its IP address changed (though I'd specified the UD mount by name, not IP) - I believe this is because the DNS cache has not yet updated to recognize Backup at its new IP address. 

 

Having done this yesterday evening, this morning (about 12 hours later) it still wasn't properly recognizing the server at its new IP. (in UD, I clicked the mount point to browse and all it showed me was "parent folder"). This morning I unmounted and remounted the share (it took quite a while to unmount, but the remount went very quickly), and now I can properly browse the directory structure via UD's mount point share. 

 

A couple of questions:

1) Is it normally necessary to unmount/remount when an IP address changes or should I normally just have to wait out the DNS cache update?

2) Was it something else totally that was the issue here?

3) Once the server goes to its remote location, it will be on a totally different subnet as I will be using the great ZeroTier plug in to get the two of them to talk to each other, I presume that going from a 192.168.* address to a 172.29.* address will not have any impact on UD's ability to map this, correct?

4) Is SMB the best option for having the 2 servers talk to each other, or would I be better off using an NFS mount? (If so, I'd probably keep the SMB mount in order to be able to browse via Windows, just because I can.)

 

Thanks (again, I hope) for picking this up and continuing to run with it! 

Hi @FreeMan,

 

First I am FAR from an expert, but coincidentally I am actually backing up systems right now.

There are folks here significantly more knowledgeable about UD than myself. That said I can speak to #3 & #4 from personal experience.

 

#3 - No problems. Have  read through the link in #4 and you should be golden!

 

#4 - I think most everyone will agree that a NFS connection is better.

       With respect to the connection, I HIGHLY recommend looking at this link as @tr0910 (and others) put in a great deal of work to make backups a breeze. I will not lie, it is a bit of work, but not only is it worth the work, it is a fast efficient and @#$%ed near fool-proof method to doing backups. I too started with Duplicati and bailed after the second multi-day backup; Rsync is WAY better.

 

Have fun! This is actually an enjoyable project!

A plugin update fails with the following lines:

plugin: updating: unassigned.devices.plg
plugin: downloading: "https://github.com/dlandon/unassigned.devices/raw/master/unassigned.devices-2019.11.29.tgz" ... done
plugin: bad file MD5: /boot/config/plugins/unassigned.devices/packages/unassigned.devices-2019.11.29.tgz

Somebody else having this issue?

I'm also having the same issues. If you want to fix it yourself right now, download the .plg to somewhere on your server, change the 12th line so that the md5 sum is:

 

8796a7b713b0d121385f1452eb7880b1

 

and then tell unRAID to install from that downloaded plg file. Otherwise, I expect it will be fixed shortly.

34 minutes ago, bamhm182 said:

I'm also having the same issues. If you want to fix it yourself right now, download the .plg to somewhere on your server, change the 12th line so that the md5 sum is:

 

8796a7b713b0d121385f1452eb7880b1

 

and then tell unRAID to install from that downloaded plg file. Otherwise, I expect it will be fixed shortly.

Where i find the .plg ? After Reboot  my Server the unassigend device / plugin its gone 😞 

 

PS: Find it ... Hope it will help

Edited November 29, 20196 yr by KinGSiZ3
Find it

You can run these commands:

cd $(mktemp -d)
wget https://raw.githubusercontent.com/dlandon/unassigned.devices/master/unassigned.devices.plg
sed -i s/fc6f30a2f824f5b56e367ed73edea275/8796a7b713b0d121385f1452eb7880b1/g unassigned.devices.plg

 

Then you can go to the Plugins tab > Install Plugin, and then paste the file location in there, for example: /tmp/tmp.tivBjv7aFZ/unassigned.devices.plg and click install.

 

Edited November 29, 20196 yr by bamhm182

4 minutes ago, bamhm182 said:

You can run these commands:

cd $(mktemp -d)
wget https://raw.githubusercontent.com/dlandon/unassigned.devices/master/unassigned.devices.plg
sed -i s/fc6f30a2f824f5b56e367ed73edea275/8796a7b713b0d121385f1452eb7880b1/g unassigned.devices.plg

 

Then you can go to the Plugins tab > Install Plugin, and then paste the file location in there, for example: /tmp/tmp.tivBjv7aFZ/unassigned.devices.plg and click install.

 

Really you are my daily Hero today ... THX for the help. All working now

Edited November 29, 20196 yr by KinGSiZ3

No worries. dlandon is the real hero, I'm just a guy that needed something done and figured out how to get it done. I'm sure he'll fix it very soon.

15 minutes ago, bamhm182 said:

No worries. dlandon is the real hero, I'm just a guy that needed something done and figured out how to get it done. I'm sure he'll fix it very soon.

Fixed.  Update again.

27 minutes ago, KinGSiZ3 said:

Where i find the .plg ? After Reboot  my Server the unassigend device / plugin its gone 😞 

 

PS: Find it ... Hope it will help

Download it again from CA.

Works now. Thanks!

I was finally able to reproduce some of the hangs people have experienced.  I had to add code to handle some Linux commands that hang when a remote CIFS share server goes offline.  Some commands that didn't even have anything to do with the CIFS mounts.

 

All Linux commands now have time outs to prevent hangs.  You'll see error messages in the log when a command times out.  I may have to extend some time outs for different systems.

 

Anyone having the UD hang, please test and then give me a diagnostics regardless if you have issues or not.  I need to see if the time outs are appropriate for all cases.

Hi @dlandon,

 

Just double checking, you want the logs from '/flash/logs/' correct?

I will play with this over the weekend and post for you.

 

Thanks again for all you do!

Thanks, @TechMed. I've looked through that thread a couple of times. I've wanted the "Oh, carp. I just realized I made changes and now I need to go back to an older version" version of backup. But, in the last 10 years or so, I don't think I've ever gone back to recover an old version of a file, so maybe it's time to give rsync a 2nd look.  

6 minutes ago, TechMed said:

Hi @dlandon,

 

Just double checking, you want the logs from '/flash/logs/' correct?

I will play with this over the weekend and post for you.

 

Thanks again for all you do!

No.  Please post complete diagnostics.