Unassigned Devices - Managing Disk Drives and Remote Shares Outside of The Unraid Array

[dlandon]

5 minutes ago, mackid1993 said:

Hey @dlandon my SMB shares from my VM were working great, but the problem I'm running to now is when shutting my server down the VM shuts down but the SMB shares don't seem to want to automatically unmount. Is there anything I can do, possibly a user script to unmount these?

They should be taken care of on a shutdown.  Post a diagnostics.

[DuzAwe]

32 minutes ago, dlandon said:

Is this mount there at /mnt/LongTerm?

there is indeed.

root@PiBox:/# ls /mnt/
dietpi_userdata  ftp_client  LongTerm  samba  secondary  store_path

there is also a /var/lib/rancher

root@PiBox:/# ls /var/lib/rancher
cross-seed-config  downloads  lost+found  qbit-config  qbit-manage-config  watchstate

 

[mackid1993]

9 minutes ago, dlandon said:

They should be take care of on a shutdown.  Post a diagnostics.

I'll update this post in a moment, looking at the syslog it seems to hang on:

Mar 27 11:21:12 Apollo unassigned.devices: Unmounting All Devices...
Mar 27 11:21:34 Apollo unassigned.devices: Remote server '10.0.0.2' port '445' is not open; server apears to be offline.

 

then after a while is says:

Mar 27 11:23:02 Apollo kernel: CIFS: VFS: \\10.0.0.2 has not responded in 180 seconds. Reconnecting...

 

Edit:

It seems to force unmount and then shutdown gracefully. It just delays shutdown. I set my unclean shutdown timeout really high to avoid dirty shutdowns.

 

Maybe I'll just add a userscript to umount -f on array stop.

Edited March 27 by mackid1993

[dlandon]

Just now, DuzAwe said:

there is indeed.

root@PiBox:/# ls /mnt/
dietpi_userdata  ftp_client  LongTerm  samba  secondary  store_path

there is also a /var/lib/rancher

root@PiBox:/# ls /var/lib/rancher
cross-seed-config  downloads  lost+found  qbit-config  qbit-manage-config  watchstate

 

I would remove and readd the remote shares and verify that UD can see the server (Search For Servers) and the NFS share (Load Shares).  Use the server name UD finds and don't enter the IP address.

[dlandon]

2 minutes ago, mackid1993 said:

Mar 27 11:21:34 Apollo unassigned.devices: Remote server '10.0.0.2' port '445' is not open; server apears to be offline.

So that is a Docker?

[mackid1993]

Just now, dlandon said:

So that is a Docker?

That is a Windows VM hosted on my Unraid server that powers down when the system shuts down but before the SMB share is unmounted.

[dlandon]

Just now, mackid1993 said:

That is a Windows VM hosted on my Unraid server that powers down when the system shuts down but before the SMB share is unmounted.

That's the problem.  UD needs to go ahead and unmount the share even if the server goes offline.  I'll research and see if I need to apply a fix.

[mackid1993]

Just now, dlandon said:

That's the problem.  UD needs to go ahead and unmount the share even if the server goes offline.  I'll research and see if I need to apply a fix.

Thanks so much! For now I made a user script to run on array stop, umount -f /path/to/mount.

[DuzAwe]

Finds and adds the share doesn't mount it and says the server is off line.

[mackid1993]

14 minutes ago, dlandon said:

That's the problem.  UD needs to go ahead and unmount the share even if the server goes offline.  I'll research and see if I need to apply a fix.

Unfortunately umount -f or umount -l at array stop doesn't seem to work as a workaround. It seems to be a bug with UD. Fortunately my timeout is so high that it eventually unmounts. It just takes 3 minutes or so.

[dlandon]

17 minutes ago, mackid1993 said:

Unfortunately umount -f or umount -l at array stop doesn't seem to work as a workaround. It seems to be a bug with UD. Fortunately my timeout is so high that it eventually unmounts. It just takes 3 minutes or so.

UD does a sync on the mount point before forcing the unmount.  It has a timeout of 10 seconds.  It that command times out, the remote share is force unmounted.  I suspect a race condition of some sort.  Post diagnostics so I can have a closer look.

[DuzAwe]

33 minutes ago, dlandon said:

I would remove and readd the remote shares and verify that UD can see the server (Search For Servers) and the NFS share (Load Shares).  Use the server name UD finds and don't enter the IP address.

I can get it to work on my Mint Laptop. But Unraid wont mount it.

[dlandon]

16 minutes ago, DuzAwe said:

Finds and adds the share doesn't mount it and says the server is off line.

The "List Shares" for NFS shares uses showmount to list the NFS shares found on the server.  That's why it finds them.

 

The server status (online) has changed from being a ping on the server to a check if the port is open.  Your server needs to use the standard port (2049) for NFS.  Check that your server is using that port.

 

The reason you're just now seeing this is you probably updated UD and the OS at the same time.  This changed occurred in UD some time ago.

[DuzAwe]

2 minutes ago, dlandon said:

The "List Shares" for NFS shares uses showmount to list the NFS shares found on the server.  That's why it finds them.

 

The server status (online) has changed from being a ping on the server to a check if the port is open.  Your server needs to use the standard port (2049) for NFS.  Check that your server is using that port.

 

The reason you're just now seeing this is you probably updated UD and the OS at the same time.  This changed occurred in UD some time ago.

root@PiBox:/# rpcinfo -p | grep nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100003    3   udp   2049  nfs
root@PiBox:/#

[mackid1993]

15 minutes ago, dlandon said:

UD does a sync on the mount point before forcing the unmount.  It has a timeout of 10 seconds.  It that command times out, the remote share is force unounted.  I suspect a race condition of some sort.  Post diagnostics so I can have a closer look.

I rebooted my server to repro the issue. My unlcean shutdown timeout is set to 420 seconds for issues just like this thankfully! Attached are my diagnostics zip.

Thanks!!

I guess UD doesn't expect the SMB share to be coming from within the house lol!

 

apollo-diagnostics-20240327-1208.zip

[dlandon]

4 minutes ago, DuzAwe said:

root@PiBox:/# rpcinfo -p | grep nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100003    3   udp   2049  nfs
root@PiBox:/#

Give me the output of this command:

timeout -s 5 1 bash -c "(echo >/dev/tcp/IP/2049)"

change IP to the server IP address.

[dlandon]

2 minutes ago, mackid1993 said:

I rebooted my server to repro the issue. My unlcean shutdown timeout is set to 420 seconds for issues just like this thankfully! Attached are my diagnostics zip.

Thanks!!

I guess UD doesn't expect the SMB share to be coming from within the house lol!

 

apollo-diagnostics-20240327-1208.zip 169.25 kB · 0 downloads

When the server is shutdown, a copy of the disgnostics is saved on the flash at /flash/logs/.  Please post that.  I need the shutdown sequence.

[DuzAwe]

3 minutes ago, dlandon said:

Give me the output of this command:

timeout -s 5 1 bash -c "(echo >/dev/tcp/IP/2049)"

change IP to the server IP address.

I get nothing from the unraid box

root@TheLibrary:~#  timeout -s 5 1 bash -c "(echo >/dev/tcp/192.168.0.40/2049)"  
root@TheLibrary:~# 

 

[dlandon]

Just now, DuzAwe said:

I get nothing from the unraid box

root@TheLibrary:~#  timeout -s 5 1 bash -c "(echo >/dev/tcp/192.168.0.40/2049)"  
root@TheLibrary:~# 

 

That indicates that the port is open.

 

Show a screen shot of the UD page showing the remote shares.

[mackid1993]

1 minute ago, dlandon said:

When the server is shutdown, a copy of the disgnostics is saved on the flash at /flash/logs/.  Please post that.  I need the shutdown sequence.

Is that file called syslog-previous?

[dlandon]

3 minutes ago, mackid1993 said:

Is that file called syslog-previous?

Check the date and time.  Probably.

[DuzAwe]

Is this what you need?

[mackid1993]

7 minutes ago, dlandon said:

Check the date and time.  Probably.

@dlandon This file has the shutdown sequence. It took about 4 minutes to shut down.

syslog-previous

[dlandon]

18 minutes ago, DuzAwe said:

Is this what you need?

The mount points you are trying to mount remotely are not Unraid shares and you are setting the export rules manually for those shares.  This is an Unraid share entry in the exports file:

"/mnt/user/isos" -fsid=105,async,no_subtree_check *(rw,sec=sys,insecure,anongid=100,anonuid=99,all_squash)

I suggest you take a look at this entry and compare to your entries and see if changes are needed.  I suspect a permissions issue that UD is having trouble with.  Check if 'all_squash' makes a difference.

 

As a test, share a normal Unraid share with NFS and try to mount that and see how it goes.

 

I fully understand that you can mount those with other systems.  The issue is that the UD NFS client is probably more restrictive about permissions.

[dlandon]

27 minutes ago, mackid1993 said:

@dlandon This file has the shutdown sequence. It took about 4 minutes to shut down.

syslog-previous 223.38 kB · 0 downloads

It's not UD.  The remote shares unmounted fine.  It's the dockers taking a long time to stop:

Mar 27 12:19:06 Apollo root: stopping dockerd ...
Mar 27 12:19:07 Apollo root: ... Waiting to die.
Mar 27 12:19:08 Apollo root: ... Waiting to die.
Mar 27 12:19:09 Apollo root: ... Waiting to die.
Mar 27 12:19:10 Apollo root: ... Waiting to die.
Mar 27 12:19:11 Apollo root: ... Waiting to die.
Mar 27 12:19:12 Apollo emhttpd: shcmd (94): umount --lazy /var/lib/docker
Mar 27 12:19:12 Apollo unassigned.devices: Unmounting All Devices...
Mar 27 12:21:02 Apollo sshd[1168]: Connection from 192.168.2.35 port 17590 on 192.168.2.40 port 22 rdomain ""
Mar 27 12:21:02 Apollo sshd[29211]: Connection closed by 192.168.2.35 port 17480
Mar 27 12:21:02 Apollo sshd[29211]: Close session: user root from 192.168.2.35 port 17480 id 0
Mar 27 12:21:02 Apollo sshd[29211]: pam_unix(sshd:session): session closed for user root
Mar 27 12:21:02 Apollo sshd[29211]: Transferred: sent 154472, received 4672 bytes
Mar 27 12:21:02 Apollo sshd[29211]: Closing connection to 192.168.2.35 port 17480
Mar 27 12:21:02 Apollo elogind-daemon[1528]: Removed session c4.
Mar 27 12:21:02 Apollo sshd[1170]: Connection from 192.168.2.35 port 17591 on 192.168.2.40 port 22 rdomain ""
Mar 27 12:21:02 Apollo sshd[1172]: Connection from 192.168.2.35 port 17592 on 192.168.2.40 port 22 rdomain ""
Mar 27 12:21:02 Apollo sshd[1168]: Postponed keyboard-interactive for root from 192.168.2.35 port 17590 ssh2 [preauth]
Mar 27 12:21:02 Apollo sshd[1168]: Postponed keyboard-interactive/pam for root from 192.168.2.35 port 17590 ssh2 [preauth]
Mar 27 12:21:02 Apollo sshd[1168]: Accepted keyboard-interactive/pam for root from 192.168.2.35 port 17590 ssh2
Mar 27 12:21:02 Apollo sshd[1168]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Mar 27 12:21:02 Apollo elogind-daemon[1528]: New session c5 of user root.
Mar 27 12:21:02 Apollo sshd[1170]: Postponed keyboard-interactive for root from 192.168.2.35 port 17591 ssh2 [preauth]
Mar 27 12:21:02 Apollo sshd[1170]: Postponed keyboard-interactive/pam for root from 192.168.2.35 port 17591 ssh2 [preauth]
Mar 27 12:21:02 Apollo sshd[1170]: Accepted keyboard-interactive/pam for root from 192.168.2.35 port 17591 ssh2
Mar 27 12:21:02 Apollo sshd[1170]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Mar 27 12:21:03 Apollo elogind-daemon[1528]: New session c6 of user root.
Mar 27 12:21:03 Apollo sshd[1172]: Postponed keyboard-interactive for root from 192.168.2.35 port 17592 ssh2 [preauth]
Mar 27 12:21:03 Apollo sshd[1172]: Postponed keyboard-interactive/pam for root from 192.168.2.35 port 17592 ssh2 [preauth]
Mar 27 12:21:03 Apollo sshd[1172]: Accepted keyboard-interactive/pam for root from 192.168.2.35 port 17592 ssh2
Mar 27 12:21:03 Apollo sshd[1172]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Mar 27 12:21:03 Apollo elogind-daemon[1528]: New session c7 of user root.
Mar 27 12:21:03 Apollo sshd[1168]: Starting session: subsystem 'sftp' for root from 192.168.2.35 port 17590 id 0
Mar 27 12:21:03 Apollo sshd[1170]: Starting session: subsystem 'sftp' for root from 192.168.2.35 port 17591 id 0
Mar 27 12:21:03 Apollo sshd[1172]: Starting session: subsystem 'sftp' for root from 192.168.2.35 port 17592 id 0
Mar 27 12:21:03 Apollo sshd[1172]: Connection closed by 192.168.2.35 port 17592
Mar 27 12:21:03 Apollo sshd[1172]: Close session: user root from 192.168.2.35 port 17592 id 0
Mar 27 12:21:03 Apollo sshd[1172]: pam_unix(sshd:session): session closed for user root
Mar 27 12:21:03 Apollo sshd[1172]: Transferred: sent 3656, received 3232 bytes
Mar 27 12:21:03 Apollo sshd[1172]: Closing connection to 192.168.2.35 port 17592
Mar 27 12:21:03 Apollo elogind-daemon[1528]: Removed session c7.
Mar 27 12:21:03 Apollo sshd[1170]: Connection closed by 192.168.2.35 port 17591
Mar 27 12:21:03 Apollo sshd[1170]: Close session: user root from 192.168.2.35 port 17591 id 0
Mar 27 12:21:03 Apollo sshd[1170]: pam_unix(sshd:session): session closed for user root
Mar 27 12:21:03 Apollo sshd[1170]: Transferred: sent 3656, received 3232 bytes
Mar 27 12:21:03 Apollo sshd[1170]: Closing connection to 192.168.2.35 port 17591
Mar 27 12:21:03 Apollo elogind-daemon[1528]: Removed session c6.
Mar 27 12:21:51 Apollo kernel: CIFS: VFS: \\10.0.0.2 has not responded in 180 seconds. Reconnecting...
Mar 27 12:21:51 Apollo kernel: CIFS: VFS: \\10.0.0.2\Dropbox Close cancelled mid failed rc:-11
Mar 27 12:21:51 Apollo kernel: CIFS: VFS: \\10.0.0.2\Dropbox Close cancelled mid failed rc:-11
Mar 27 12:21:51 Apollo kernel: CIFS: VFS: \\10.0.0.2\Dropbox Close cancelled mid failed rc:-11
Mar 27 12:22:22 Apollo unassigned.devices: Unmounting Remote SMB/NFS Share '//10.0.0.2/Dropbox'...
Mar 27 12:22:22 Apollo unassigned.devices: Unmount cmd: /sbin/umount -t cifs -f '/mnt/remotes/Dropbox' 2>&1
Mar 27 12:22:22 Apollo unassigned.devices: Successfully unmounted '//10.0.0.2/Dropbox'
Mar 27 12:22:43 Apollo unassigned.devices: Unmounting Remote SMB/NFS Share '//10.0.0.2/Dropbox/Pictures'...
Mar 27 12:22:43 Apollo unassigned.devices: Unmount cmd: /sbin/umount -t cifs -f '/mnt/remotes/Pictures' 2>&1
Mar 27 12:22:43 Apollo unassigned.devices: Successfully unmounted '//10.0.0.2/Dropbox/Pictures'

I think your shutdown timer is set properly to handle this.