Jump to content
dlandon

Unassigned Devices - Managing Disk Drives and Remote Shares Outside of The Unraid Array

3122 posts in this topic Last Reply

Recommended Posts

4 minutes ago, 1812 said:

You had previously said that I should try to make sure remote shares are online or unmount before they go offline. While that is possible for those on my LAN, it is not possible to ensure when they are connected via vpn or other tunnel to offsite shares, which is the case with the above message.

I have several remote mounted shares using a VPN and I don't have this issue.

 

Post diagnostics before the system hangs so I can look at your setup.

Share this post


Link to post
3 minutes ago, dlandon said:

I have several remote mounted shares using a VPN and I don't have this issue.

 

Post diagnostics before the system hangs so I can look at your setup.

thank you

brahms1-diagnostics-20190713-1145.zip

Share this post


Link to post
4 hours ago, dlandon said:

Setup the syslog server on your Unraid server to capture the log.  You can then post the log after a reboot so I can see what it shows when the situation happens.

done. this is on the same boot those the diags are from.

 

as far as timeline and how it lines up with the log:

 

shortly before 12:00:00 I stopped the array that was mounted via smb in unassigned devices without unmounting it. The loading/refreshing icon appeared and never went away in the UD section of the main tab. After a minute or two, the error at 12:00:21 appeared, which is similar to the nginx error previously reported. The loading/refreshing icon never went away, even after refreshing the page. This is an unrecoverable error and UD loses all functionality. Any services requiring mounting or un-mounting of disks to operate is also affected (backup services, etc)

 

I then initiated a power down via webgui at 12:01:03. The server sat in a hung state for 5 minutes with the last entry at 12:02:21 showing active pids still, which I believe are via UD as the server powers down cleanly if all smb mounts are unmounted before turning off. After 5 minutes I had to shut the server off by holding the power button as it was clear it was not going to recover.

 

 

syslog-192.168.1.253.log

Share this post


Link to post
4 hours ago, 1812 said:

done. this is on the same boot those the diags are from.

 

as far as timeline and how it lines up with the log:

 

shortly before 12:00:00 I stopped the array that was mounted via smb in unassigned devices without unmounting it. The loading/refreshing icon appeared and never went away in the UD section of the main tab. After a minute or two, the error at 12:00:21 appeared, which is similar to the nginx error previously reported. The loading/refreshing icon never went away, even after refreshing the page. This is an unrecoverable error and UD loses all functionality. Any services requiring mounting or un-mounting of disks to operate is also affected (backup services, etc)

 

I then initiated a power down via webgui at 12:01:03. The server sat in a hung state for 5 minutes with the last entry at 12:02:21 showing active pids still, which I believe are via UD as the server powers down cleanly if all smb mounts are unmounted before turning off. After 5 minutes I had to shut the server off by holding the power button as it was clear it was not going to recover.

 

 

syslog-192.168.1.253.log 7.65 kB · 1 download

You need to review your shutdown timers.  You have the disk shutdown time out and the VM shutdown time out both set to 60 seconds.  That's generally not enough time for Unraid to shutdown.  Read this post and see if you need to make any changes.

You are giving me snipets of logs that have little to no value.  Run your system until you have problems.  Then reboot and let Unraid time out on shutdown.  Don't force a shutdown.  Then post the diagnostics that Unraid collects at /flash/logs/.  If the diagnostics aren't collected, then post your syslog server log for the server.  Give me the full log, not just the last few entries.

Share this post


Link to post
4 hours ago, 1812 said:

done. this is on the same boot those the diags are from.

 

as far as timeline and how it lines up with the log:

 

shortly before 12:00:00 I stopped the array that was mounted via smb in unassigned devices without unmounting it. The loading/refreshing icon appeared and never went away in the UD section of the main tab. After a minute or two, the error at 12:00:21 appeared, which is similar to the nginx error previously reported. The loading/refreshing icon never went away, even after refreshing the page. This is an unrecoverable error and UD loses all functionality. Any services requiring mounting or un-mounting of disks to operate is also affected (backup services, etc)

 

I then initiated a power down via webgui at 12:01:03. The server sat in a hung state for 5 minutes with the last entry at 12:02:21 showing active pids still, which I believe are via UD as the server powers down cleanly if all smb mounts are unmounted before turning off. After 5 minutes I had to shut the server off by holding the power button as it was clear it was not going to recover.

 

 

syslog-192.168.1.253.log 7.65 kB · 1 download

Do you have dockers using the remote SMB shares?

Share this post


Link to post

I have not been able to find an answer to if this is a bug or by design. 

 

When I have an NFS mount configured for auto mount and it is not available at unRAID boot the device will remain unmounted. Once it becomes available the mount button will become green/orange and allow me to mount it. Should it not just auto mount the share when it sees it available?

 

I don't know if this is intended or a bug but it would be really nice if NFS shares would always auto connect when they become available. 

Share this post


Link to post
1 hour ago, _rogue said:

I have not been able to find an answer to if this is a bug or by design. 

 

When I have an NFS mount configured for auto mount and it is not available at unRAID boot the device will remain unmounted. Once it becomes available the mount button will become green/orange and allow me to mount it. Should it not just auto mount the share when it sees it available?

 

I don't know if this is intended or a bug but it would be really nice if NFS shares would always auto connect when they become available. 

Remote mounts are mounted when the array starts if they are set to auto mount.  UD does not attempt another mount if the remote mount comes on-line.

 

You could set up a User Script on a cron to auto mount remote mounts with the following script:

/usr/local/sbin/rc.unassigned mount auto

This will mount any devices that are not mounted if they are set to auto mount.  It doesn't affect any devices already mounted, but could add a lot of entries to the log.  It will also mount any disks set to auto mount that are not mounted.

Share this post


Link to post
47 minutes ago, dlandon said:

Remote mounts are mounted when the array starts if they are set to auto mount.  UD does not attempt another mount if the remote mount comes on-line.

 

You could set up a User Script on a cron to auto mount remote mounts with the following script:


/usr/local/sbin/rc.unassigned mount auto

This will mount any devices that are not mounted if they are set to auto mount.  It doesn't affect any devices already mounted, but could add a lot of entries to the log.  It will also mount any disks set to auto mount that are not mounted.

This is exactly what I needed! I can make a quick script that checks if the mount point exists and if it does not run this command.

 

Thank you!

Share this post


Link to post
2 hours ago, dlandon said:

You need to review your shutdown timers.  You have the disk shutdown time out and the VM shutdown time out both set to 60 seconds.  That's generally not enough time for Unraid to shutdown.  Read this post and see if you need to make any changes.

There were on whatever the default is. But I will adjust them accordingly per your post. As I mentioned before the system hang exceeds the default shutdown timers. I don't understand how extending it will help since it isn't Unraid that has to force the unclean shutdown but me because the system is hung. But I'm no expert and have changed to your recommended settings in that post.

 

2 hours ago, dlandon said:

You are giving me snipets of logs that have little to no value.  Run your system until you have problems.  Then reboot and let Unraid time out on shutdown.  Don't force a shutdown.  Then post the diagnostics that Unraid collects at /flash/logs/.  If the diagnostics aren't collected, then post your syslog server log for the server.  Give me the full log, not just the last few entries.

I gave you exactly what you asked for, which was everything from when the remote syslog was activated (which is why I mentioned it was a continuation of the full diags I previously posted. I activated the remote syslog, triggered the error, attempted to shut down the server (which hung for over five minutes which had to then be forced off) then pulled the syslog of the remote storage and gave it to you.  

 

Regardless, I will replicate the error again and give you the full syslog since the remote logging has been active since reboot.

 

2 hours ago, dlandon said:

Do you have dockers using the remote SMB shares?

yes, cloudberry. I'll try replicating the issue with this docker off to this time to eliminate that variable.

 

thank you for your patience with me on this. I'm trying to provide accurate information so the cause can be discovered!

Share this post


Link to post
18 minutes ago, 1812 said:

There were on whatever the default is. But I will adjust them accordingly per your post. As I mentioned before the system hang exceeds the default shutdown timers. I don't understand how extending it will help since it isn't Unraid that has to force the unclean shutdown but me because the system is hung. But I'm no expert and have changed to your recommended settings in that post.

Unraid should time out and force a shutdown once the timeout has been reached.  If Unraid doesn't force a shutdown, you have other issues.  This should happen no matter what the remote shares are doing.

 

 

22 minutes ago, 1812 said:

yes, cloudberry. I'll try replicating the issue with this docker off to this time to eliminate that variable.

Some have reported issues with dockers using remote mounted shares.  I'm thinking your docker may be hanging the system when the remote share goes off-line.

Share this post


Link to post
3 hours ago, dlandon said:

Give me the full log, not just the last few entries.

sorry to spam you with quotes, but bad news/good news

 

So evidently I didn't setup my remote syslog properly. Even though I told it to to keep 3 logs in rotation, it overwrote my latest error. But, I have new info that I believe is the problem.

 

It may be that before when UD was doing the "refreshing/processing" moving graphic I was impatient even though it was throwing nginx errors.  I just tested again and it eventually shows the remote share, still mounted but disconnected. I tired this with and without cloudberry running (the only docker that uses remote mounts) and that had no change.

 

Then I realized that my router/firewall is in a vm. So I turned off that vm and retested, and sure enough, the timeout wouldn't happen and the error would return. UD was unusable as previously reported. But upon starting up my firewall vm again, UD would return to normal usage and show the expected disconnected remote smb share, which would resume operation normally when it rejoined the network. 

 

So I tested this theory by disconnecting the remote smb share, then refreshed UD which appears to get hung at loading. I then shut down the vm. this kept UD in a perpetual loading state (it appeared with errors about the remote share being offline) I then shut the server down. But oddly enough, it actually shut down this time. I'm not sure if it had to do with the readjustments to the timeouts you suggested, but it was gracefully powered off after only 2 minutes.

 

So I believe this is what is happening: a remote share goes offline, UD appeared to hang and was unresponsive. I'd send the restart command to my server which would shutdown the vm, exacerbating the "hang" with UD not finding the share, then the shortened shutdown timer was trying to power down the server before UD could actually unmount the missing disk, which caused the shutdown hang.

 

 

I don't know if any of this makes sense, or if I'm just rambling. But either way, I believe the issue is resolved. Thank you again for your time and suggestions which appear to have corrected the problem!

 

 

Share this post


Link to post

I guess I understand all that.  Regardless, I'm glad you worked it out.

Share this post


Link to post
Posted (edited)
On 5/24/2019 at 12:56 AM, Marshalleq said:

OK, I guess I have to live with it.  It doesn't seem to impact anything, just looks horrible.

Did you ever get this squared... I am experiencing the same.  Passing through nvme controller...and booting winVM off of it.  I did this previously without issues.  Nuked the VM and started again, and this popped up.

 

edit:  actually I can no longer see the partitions.  

Edited by J.Nerdy

Share this post


Link to post

No I didn't, it's bloody awful isn't it.  I'm really not sure why it's happening either.  Did you happen to be experimenting both with passthrough PCI device via stubbing the drive and also via the XML code in the VM?  I seem to remember it started when I did one of those but not the other, though removing either / both didn't fix the issue.

Share this post


Link to post
16 minutes ago, Marshalleq said:

No I didn't, it's bloody awful isn't it.  I'm really not sure why it's happening either.  Did you happen to be experimenting both with passthrough PCI device via stubbing the drive and also via the XML code in the VM?  I seem to remember it started when I did one of those but not the other, though removing either / both didn't fix the issue.

Come to think of it... I did at one point have it stubbed.  However, then I copied my winVM to it, and partitioned the remaining drive in xfs to share some high throughput storage to the array.

 

It did not manifest this issue at the time.  However, after nuking the VM and passing through the controller again with a new VM.

 

The performance on the VM is spot on... my only issue is that though UD can see the drive, it can not see the partitions - so I can not format the unused space and share back to the array.

 

i might stub it out.  Reboot.  Then reboot and pass through.  See if that makes a difference.

 

Share this post


Link to post
On 7/7/2019 at 4:35 PM, devros said:

Just tried it on an identical 4T drive also hooked up directly to the motherboard and this time it did mount as a 4T mount

had to reboot, UD shows the device with the "MOUNT" option greyed out.  If I try and run an fsck, it just searches forever for a superblock.  Running out of options here.  UD has served me very well over the years, but this one has be stumped.

Share this post


Link to post
5 hours ago, devros said:

had to reboot, UD shows the device with the "MOUNT" option greyed out.  If I try and run an fsck, it just searches forever for a superblock.  Running out of options here.  UD has served me very well over the years, but this one has be stumped.

Have you enabled Destructive mode in the UD settings?    You need this to be allowed to format.

Share this post


Link to post
Posted (edited)

@dlandon

 

I am getting the following error in UD:

Warning: Illegal string offset 'DEVTYPE' in /usr/local/emhttp/plugins/unassigned.devices/include/lib.php on line 1347

This is for a passed through nvme drive.  The device is using a modified clover to boot a win10 vm.  I had this same configuration previously with no issues (UD listed device and partition tables without issue when VM powered down).

 

I am way out of my depth when it comes to functions and setting arrays in php. 

 

What I am wondering is since this is the getting partition info function... could have something to do with the partition map created in windows?

 

Here is the disk log info:

Jul 16 12:29:02 nerdyRAID kernel: nvme1n1: p1 p2 p3 p4
Jul 16 12:29:15 nerdyRAID emhttpd: Samsung_SSD_970_PRO_512GB_S463NF0K713728K (nvme1n1) 512 1000215216
Jul 16 13:11:56 nerdyRAID unassigned.devices: Adding disk '/dev/nvme1n1p3'...
Jul 16 13:11:56 nerdyRAID unassigned.devices: Mount drive command: /sbin/mount -t ntfs -o auto,async,noatime,nodiratime,nodev,nosuid,umask=000 '/dev/nvme1n1p3' '/mnt/disks/GAMES'
Jul 16 13:11:56 nerdyRAID ntfs-3g[1459]: Mounted /dev/nvme1n1p3 (Read-Write, label "GAMES", NTFS 3.1)
Jul 16 13:11:56 nerdyRAID ntfs-3g[1459]: Mount options: rw,nodiratime,nodev,nosuid,allow_other,nonempty,noatime,default_permissions,fsname=/dev/nvme1n1p3,blkdev,blksize=4096
Jul 16 13:11:56 nerdyRAID unassigned.devices: Successfully mounted '/dev/nvme1n1p3' on '/mnt/disks/GAMES'.
Jul 16 13:11:56 nerdyRAID unassigned.devices: Adding disk '/dev/nvme1n1p4'...
Jul 16 13:11:56 nerdyRAID unassigned.devices: No filesystem detected on '/dev/nvme1n1p4'.

I clearly see 2 of the 4 partitions:  the unformatted raw partition (that I want to use as a XFS partition for array when machine is not running and the primary partition created for storage of 'games' in NTFS)

 

I do not see the 'efi' partition  nor the windows [C:\] drive.  

 

Clearly this is some type of user error on my part... but just not sure how or why it got borked when previous steps worked flawlessly in past.

 

If it would help I could post 1347 - 1403 of lib.php and see if something got corrupted?  Any suggestions would be greatly appreciated.  

 

Thank you!

 

EDIT:  I did not mention, the VM works flawlessly.  It is simply I can not pull accurate partition map in UD.  Sorry.

Edited by J.Nerdy
derp

Share this post


Link to post

Hi, what should I do if I want to use an USB flash drive of 8GB just to store data as backup (assignment of disks to array, logs, ...). Yesterday I had to install everything new from scratch and realized I did not have the image of how my disks were assigned to each slot. 

Just want to use a spare USB to store data like this, plug it then back in my windows laptop and then access it..

I am not sure what format to use.

Rgds

 

Share this post


Link to post
52 minutes ago, luca2 said:

Hi, what should I do if I want to use an USB flash drive of 8GB just to store data as backup (assignment of disks to array, logs, ...). Yesterday I had to install everything new from scratch and realized I did not have the image of how my disks were assigned to each slot. 

Just want to use a spare USB to store data like this, plug it then back in my windows laptop and then access it..

I am not sure what format to use.

Rgds

 

Format it in Windows.  UD will then mount it for backups.

Share this post


Link to post

I like this plugin a lot, however one thing bothers me.

 

When a drive is mounted (not even shared), your plugin issues "lsof -- /mnt/disks/tempdrive" command every few seconds.

This causes remarkable CPU spikes (30% out of 400%) via process "php" and don´t let that drive ever spin down.

 

Would it be possible to have an option to disable "Open files" info or at least don´t issue this command when the Main dashboard is not actively displayed?

This little enhancement is much appreciated.

Share this post


Link to post
23 minutes ago, Forusim said:

I like this plugin a lot, however one thing bothers me.

 

When a drive is mounted (not even shared), your plugin issues "lsof -- /mnt/disks/tempdrive" command every few seconds.

This causes remarkable CPU spikes (30% out of 400%) via process "php" and don´t let that drive ever spin down.

 

Would it be possible to have an option to disable "Open files" info or at least don´t issue this command when the Main dashboard is not actively displayed?

This little enhancement is much appreciated.

Do you have the preclear plugin installed?

Share this post


Link to post
Posted (edited)

Yes I have.

 

Edit: After uninstallation the spikes are gone, thanks!  Don´t understand, why this plugin issue any command without any activity...

Edited by Forusim

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.