Unassigned Devices - Managing Disk Drives and Remote Shares Outside of The Unraid Array


Recommended Posts

1.) Why is Unassigned Devices sending so many POST requests to "http://thoth/plugins/unassigned.devices/UnassignedDevices.php" while the MAIN tab is open? Regarding my network monitor it connects to the server every second:

1572667596_2020-11-1710_44_58.thumb.png.653cd4d07cb92cc733d9d3a7d7f5f5a0.png

 

POST content:

action=detect&csrf_token=blabla

Response:

{"reload":false,"diskinfo":0}

2.) And I have/had an additional problem. I start at the beginning.  Yesterday I decided to replace my SSD. So I changed all cached shares to "prefer" and executed the mover to move all data to the HDD. But the mover was extremely slow. I stopped the mover and enabled the mover logs. Result: New log entries popped up really slow and the mover tried to move files that were already moved?! Didn't make sense. And the even more strange thing was, that there was still action on the HDD. So I used inotifywait to check the traffic on the HDD and finally tried to use lsof to check which process was accessing this HDD and after 20 seconds (!) it returned this:

lsof
lsof: WARNING: can't stat() cifs file system /mnt/disks/DESKTOP-QQ89TOG_Desktop
      Output information may be incomplete.

After additional 20 seconds a WARNING for the next SMB share path appeared. I decided to stop the command and unmount all SMB shares. But it was not possible through the Dashboard. The logs returned that it took more than 10 seconds:

Nov 17 08:56:22 Thoth unassigned.devices: Removing Remote SMB/NFS share '//DESKTOP-/Pictures'...
Nov 17 08:56:22 Thoth unassigned.devices: Unmounting Remote SMB/NFS Share '//DESKTOP-/Pictures'...
Nov 17 08:56:22 Thoth unassigned.devices: Unmounting '//DESKTOP-/Pictures'...
Nov 17 08:56:22 Thoth unassigned.devices: Unmount cmd: /sbin/umount -t cifs '//DESKTOP-/Pictures' 2>&1
Nov 17 08:56:32 Thoth unassigned.devices: Error: shell_exec(/sbin/umount -t cifs '//DESKTOP-/Pictures' 2>&1) took longer than 10s!
Nov 17 08:56:32 Thoth unassigned.devices: Unmount of '//DESKTOP-/Pictures' failed. Error message: command timed out

Finally I unmounted manually ALL SMB shares by the WebTerminal which took each 30 to 120 seconds (!), until I was able to (fast) mount and unmount through the dashboard again. I think it had something to do with some mounted SMB shares that were offline (shutdown Windows PCs). That's how it looks now (before all SMB shares were mounted):

1943678678_2020-11-1710_43_28.png.a300150b60da1ef327299d86507bef56.png

 

Its only a guess, but I think this delay caused, that the mover has been executed twice (race condition) and so started to move files, that were already moved. Or the mover suffers from the huge timeout which is similar to what happend to me with the lsof command?!

 

P.S. After unmounting all the SMB shares the mover is fast as usual and does not return any errors. Even after I mounted the SMB shares that are online at the moment.

 

 

Conclusions:

- the timeout of 10 seconds should be raised or changeable through the UD settings

- something needs to be changed with the handling of mounted, but offline SMB shares. Maybe an auto unmount after x seconds and if the Auto Mount option has been selected it checks by interval (ping or similar) if it is able to remount the share.

 

Link to comment

You have a lot of remote mounted shares.  Is it really necessary to do things that way?

16 minutes ago, mgutt said:

the timeout of 10 seconds should be raised or changeable through the UD settings

What would this do?  If the remote server is not reachable, increasing the delay will only slow down things further.  The reason for the timeouts is to keep some commands like 'df' from completely hanging the Unraid server.  If it can't complete in 10 seconds, it probably won't happen at all.

18 minutes ago, mgutt said:

something needs to be changed with the handling of mounted, but offline SMB shares. Maybe an auto unmount after x seconds and if the Auto Mount option has been selected it checks by interval (ping or similar) if it is able to remount the share.

I've given this a lot of thought because of other users having the same issues.  Unmounting the remote share whenever the remote server is off-line will create more problems than it solves.  CIFS does not work as cleanly as you might think when it comes to unmounting and re-mounting a share when the remote server is off-line.

 

Conclusion:

- Keep the remote server online and available, or only mount the remote shares when actually needed.

Link to comment
15 minutes ago, dlandon said:

What would this do?  If the remote server is not reachable, increasing the delay will only slow down things further. 

I meant only the timeout of the unmount command, not the general timeout. As long the command is interrupted after 10 seconds its not possible to unmount through the Dashboard. And shouldn't the plugin allow the user to unmount something which was mounted through the plugin? 🤔

18 minutes ago, dlandon said:

Is it really necessary to do things that way?

Yes. I use the mounted shares to backup the client PCs. Works really nice and does not need any backup software.

20 minutes ago, dlandon said:

Unmounting the remote share whenever the remote server is off-line will create more problems than it solves

What are these problems? I mean what is different compared to a disconnected USB drive?

21 minutes ago, dlandon said:

only mount the remote shares when actually needed.

Of course I could use scripts only, but this is nothing I prefer and nothing I could easily explain to other people. I mean thats the reason why I'm using a Plugin. GUI power 😅

Link to comment
3 minutes ago, mgutt said:

I meant only the timeout of the unmount command, not the general timeout. As long the command is interrupted after 10 seconds its not possible to unmount through the Dashboard. And shouldn't the plugin allow the user to unmount something which was mounted through the plugin?

CIFS does not work the same as a hard drive unmount.  Unmounting a remote share that is off-line is messy and shouldn't be done.  The best bet is to get the remote server back on-line.  I'll take a look at some options.

5 minutes ago, mgutt said:

What are these problems? I mean what is different compared to a disconnected USB drive?

Unmounting a remote share that is off-line is not a good idea for several reasons.  CIFS will do a recovery once the remote server is back on line.  Unmounting a remote share when the remote server is off-line will pull the rug out from under any application using that share and is a really bad idea.  What would be the criteria for unmounting the share?  There are so many different servers I don't know where to even start on the time to wait to unmount, and whether or not the remote server would handle it properly.

Link to comment
27 minutes ago, mgutt said:

I meant only the timeout of the unmount command, not the general timeout. As long the command is interrupted after 10 seconds its not possible to unmount through the Dashboard. And shouldn't the plugin allow the user to unmount something which was mounted through the plugin?

The unmount of a CIFS mount is supposed to be forced when the server is off-line.  Looking into the code, I see a logic error that I will correct.  I'll spend a little time testing before including it in UD.

  • Thanks 1
Link to comment
14 hours ago, dlandon said:

Can you post a screen shot so we can see what you are talking about.

I'm not clear what the screen shot you are asking for would be. In my case when I have unassigned devices plugin installed, after a reboot, there is no unraid web page to display, and when I telnet in nginx isn't running, I can manually start nginx, and after a couple minutes unraid's web page comes up, and all appears normal, until the next reboot.

Link to comment
2 hours ago, bondoo0 said:

I'm not clear what the screen shot you are asking for would be. In my case when I have unassigned devices plugin installed, after a reboot, there is no unraid web page to display, and when I telnet in nginx isn't running, I can manually start nginx, and after a couple minutes unraid's web page comes up, and all appears normal, until the next reboot.

You said you could see Dockers.  What webpage does not show?  Just UD?  Do you get a blank screen?

Link to comment
12 minutes ago, dlandon said:

You said you could see Dockers.  What webpage does not show?  Just UD?  Do you get a blank screen?

Sorry, I should have been clear.  I get a timeout page going to the unraid home page (or other unraid pages such as Dashboard, UD, Main, etc).  However, I have dockers, such as Plex that publish their own web page outside of unraid, and those pages are available.

 

The only way to get the unraid pages to come back up is to login via SSH, and start nginx, which shows as not running on system boot when I have UD installed, and I believe is what provides the unraid web front end.

Link to comment
4 hours ago, bondoo0 said:

Sorry, I should have been clear.  I get a timeout page going to the unraid home page (or other unraid pages such as Dashboard, UD, Main, etc).  However, I have dockers, such as Plex that publish their own web page outside of unraid, and those pages are available.

 

The only way to get the unraid pages to come back up is to login via SSH, and start nginx, which shows as not running on system boot when I have UD installed, and I believe is what provides the unraid web front end.

Post a screen shot of the UD page when it is working.  Also post your diagnostics.

Link to comment
21 hours ago, dlandon said:

Unmounting a remote share that is off-line is not a good idea for several reasons.  CIFS will do a recovery once the remote server is back on line.

Can't confirm that. "mount -t cifs" returns nothing, although both SMB servers are online again (which is shown in the UD dashboard). "cat /etc/mtab" does not return shares, too.

 

21 hours ago, dlandon said:

Unmounting a remote share when the remote server is off-line will pull the rug out from under any application using that share and is a really bad idea.

Again: What is the difference compared to a USB drive (applications could rely on that, too)? Its up the the user to decide if he needs permanent connection. And the Unraid mover is a main unraid component which is in my opinion more important than a backup script which fails because of a unmounted server, which is not reachable anyway. Why not adding a new option?

460718735_2020-11-1808_33_26.png.322a5a06c169f4107481a0640ac43550.png

 

20 hours ago, dlandon said:

The unmount of a CIFS mount is supposed to be forced when the server is off-line.

Regarding my experience not needed. It takes longer, but it disconnects through the usual command, too.

 

21 hours ago, dlandon said:

What would be the criteria for unmounting the share?

 

That's how I solved it at the moment:

#!/bin/bash
# make script race condition safe
if [[ -d "/tmp/${0///}" ]] || ! mkdir "/tmp/${0///}"; then exit 1; fi; trap 'rmdir "/tmp/${0///}"' EXIT;
# auto unmount offline smb shares
while read line; do
    smb_server=$(echo $line | grep -oP '^//.*?/')
    smb_server=${smb_server:2:-1}
    smb_share=$(echo $line | grep -oP '^//.*? on ')
    smb_share=${smb_share:0:-4}
    timeout 1 ping -c 1 $smb_server
    if [[ $? -eq 124 ]]; then
        echo "Automatically unmount offline $smb_share"
        /sbin/umount -t cifs $smb_share
    fi
done < <(mount -t cifs)

Only auto-re-mount is missing as I'm searching for the config file of UD at the moment ;)

Edited by mgutt
Link to comment

When I make changes/enhancements to UD I have to take into consideration the application to the wider community.  It appears you have a unique need and have come up with a solution for your situation.  At this time, I don't feel it appropriate to add an automatic unmount/re-mount of remote shares.  We've discussed some of the technical issues.

 

I do continually look for better ways to handle CIFS shares.  CIFS mounts depend on a reliable network connection and when the remote server goes off-line, the Linux commands to query things like used and free space will hard hang.  'df' is the worst.  That's why the commands are timed out by UD.  Every remote server also seems to do things differently and it's quite a challenge to keep up with every server's quirks.  Some older NAS devices are still using SMBv1, which has known security issues.

Link to comment
20 hours ago, dlandon said:

Post a screen shot of the UD page when it is working.  Also post your diagnostics.

Here are my diagnostics and a screen shots.  I wasn't sure if you wanted the UD config page (I believe I left everything at the default) or the page showing the UD tab from main.  I have one drive that is used for offline backups, so it isn't typically connected, but is still shown/remembered.1602694758_2020-11-18(1).thumb.png.c341c9c09ce8647095b0de174f3d35e8.png

2020-11-18 (2).png

unraid-server1-diagnostics-20201116-1320.zip

Link to comment
2 hours ago, bondoo0 said:

Here are my diagnostics and a screen shots.

I see this in the log:

Nov 16 13:14:06 unraid-server1 unassigned.devices: Mounting 'Auto Mount' Remote Shares...
...
Nov 16 13:19:36 unraid-server1 unassigned.devices: Cannot 'Auto Mount' Remote Shares.  Network not available!

The startup check for the network to be available before auto mounting remote shares is taking over 5 minutes to determine the network is not available.  There is an issue on the network available script that causes this lengthy time, and because it is in an Unraid startup event, the Unraid GUI will not be available until it completes the network check.  I have a fix applied, but it is not released yet.

 

When you reboot Unraid, the GUI will eventually come up after the 5 minute delay caused by this issue.

Link to comment
24 minutes ago, dlandon said:

I see this in the log:


Nov 16 13:14:06 unraid-server1 unassigned.devices: Mounting 'Auto Mount' Remote Shares...
...
Nov 16 13:19:36 unraid-server1 unassigned.devices: Cannot 'Auto Mount' Remote Shares.  Network not available!

The startup check for the network to be available before auto mounting remote shares is taking over 5 minutes to determine the network is not available.  There is an issue on the network available script that causes this lengthy time, and because it is in an Unraid startup event, the Unraid GUI will not be available until it completes the network check.  I have a fix applied, but it is not released yet.

 

When you reboot Unraid, the GUI will eventually come up after the 5 minute delay caused by this issue.

Okay, that makes sense, thanks for looking into this for me, and thanks for all your work on this plugin (and also explains why when I manually restarted nginx I still had to wait 5 minutes or so for the GUI to come up).

Link to comment

@dlandon:

 

I rechecked the error code today, tried around with the command UD gives in the webterminal and concluded that there must be something wrong with the share.

 

Then i discovered that my Win machines could not connect to the share too today (checked it right at the beginning with UD, not after)

 

Checked into my openwrt Router and bumm, it lost the user account i build and used for the share. Dont know why, but it happened.

 

TL;DR: It works now and i am sorry wasting your time

 

Conclusion: Mount Error Code 13 or 95: Check your share at the outside server

Link to comment

So I have upgraded to Beta 35 and was transfering some files from a drive via krusader with the drive being mounted via unassigned devices. and that worked great. unmounted the device added more files to the drived from another pc, re added the drive to unraid and it shows up in unassigned devices, but when I click mount it tries and the just come back to the option to mount. 

 

I have tried this with several drives. the can connect the firast time, but once unmounted and removed I and then re added it cant be remounted. 

 

This is what the log shows when tryiong to remount a drive again

Nov 20 13:09:49 TOWER unassigned.devices: Adding disk '/dev/sdn1'...
Nov 20 13:09:49 TOWER unassigned.devices: Mount drive command: /sbin/mount -t xfs -o rw,noatime,nodiratime '/dev/sdn1' '/mnt/disks/WD_easystore_264D'
Nov 20 13:09:49 TOWER kernel: XFS (sdn1): Filesystem has duplicate UUID 4e86ed38-0af8-4794-9635-dd5554fa7f1a - can't mount
Nov 20 13:09:49 TOWER unassigned.devices: Mount of '/dev/sdn1' failed. Error message: mount: /mnt/disks/WD_easystore_264D: wrong fs type, bad option, bad superblock on /dev/sdn1, missing codepage or helper program, or other error.
Nov 20 13:09:49 TOWER unassigned.devices: Partition 'WD easystore_264D' could not be mounted.
Nov 20 13:09:49 TOWER unassigned.devices: Issue spin down timer for device '/dev/sdn'.

 

Link to comment
6 minutes ago, almulder said:

So I have upgraded to Beta 35 and was transfering some files from a drive via krusader with the drive being mounted via unassigned devices. and that worked great. unmounted the device added more files to the drived from another pc, re added the drive to unraid and it shows up in unassigned devices, but when I click mount it tries and the just come back to the option to mount. 

 

I have tried this with several drives. the can connect the firast time, but once unmounted and removed I and then re added it cant be remounted. 

 

This is what the log shows when tryiong to remount a drive again


Nov 20 13:09:49 TOWER unassigned.devices: Adding disk '/dev/sdn1'...
Nov 20 13:09:49 TOWER unassigned.devices: Mount drive command: /sbin/mount -t xfs -o rw,noatime,nodiratime '/dev/sdn1' '/mnt/disks/WD_easystore_264D'
Nov 20 13:09:49 TOWER kernel: XFS (sdn1): Filesystem has duplicate UUID 4e86ed38-0af8-4794-9635-dd5554fa7f1a - can't mount
Nov 20 13:09:49 TOWER unassigned.devices: Mount of '/dev/sdn1' failed. Error message: mount: /mnt/disks/WD_easystore_264D: wrong fs type, bad option, bad superblock on /dev/sdn1, missing codepage or helper program, or other error.
Nov 20 13:09:49 TOWER unassigned.devices: Partition 'WD easystore_264D' could not be mounted.
Nov 20 13:09:49 TOWER unassigned.devices: Issue spin down timer for device '/dev/sdn'.

 

Go to the UD settings and you can change the UUID for that drive so it is not a duplicate.

Link to comment

Ok I went into settings and clicked chnage UUID and got the following in the log. (UUID not changed). Tried removing drive, the added it back and then tried to chnage uuid.

 

Nov 20 13:22:15 TOWER unassigned.devices: Changing disk '/dev/sdn1' UUID. Result: ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_admin. If you are unable to mount the filesystem, then use the xfs_repair -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this.

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.