Unassigned Devices - Managing Disk Drives and Remote Shares Outside of The Unraid Array


dlandon

Recommended Posts

4 hours ago, dlandon said:

Setup the syslog server on your Unraid server to capture the log.  You can then post the log after a reboot so I can see what it shows when the situation happens.

done. this is on the same boot those the diags are from.

 

as far as timeline and how it lines up with the log:

 

shortly before 12:00:00 I stopped the array that was mounted via smb in unassigned devices without unmounting it. The loading/refreshing icon appeared and never went away in the UD section of the main tab. After a minute or two, the error at 12:00:21 appeared, which is similar to the nginx error previously reported. The loading/refreshing icon never went away, even after refreshing the page. This is an unrecoverable error and UD loses all functionality. Any services requiring mounting or un-mounting of disks to operate is also affected (backup services, etc)

 

I then initiated a power down via webgui at 12:01:03. The server sat in a hung state for 5 minutes with the last entry at 12:02:21 showing active pids still, which I believe are via UD as the server powers down cleanly if all smb mounts are unmounted before turning off. After 5 minutes I had to shut the server off by holding the power button as it was clear it was not going to recover.

 

 

syslog-192.168.1.253.log

Link to comment
4 hours ago, 1812 said:

done. this is on the same boot those the diags are from.

 

as far as timeline and how it lines up with the log:

 

shortly before 12:00:00 I stopped the array that was mounted via smb in unassigned devices without unmounting it. The loading/refreshing icon appeared and never went away in the UD section of the main tab. After a minute or two, the error at 12:00:21 appeared, which is similar to the nginx error previously reported. The loading/refreshing icon never went away, even after refreshing the page. This is an unrecoverable error and UD loses all functionality. Any services requiring mounting or un-mounting of disks to operate is also affected (backup services, etc)

 

I then initiated a power down via webgui at 12:01:03. The server sat in a hung state for 5 minutes with the last entry at 12:02:21 showing active pids still, which I believe are via UD as the server powers down cleanly if all smb mounts are unmounted before turning off. After 5 minutes I had to shut the server off by holding the power button as it was clear it was not going to recover.

 

 

syslog-192.168.1.253.log 7.65 kB · 1 download

You need to review your shutdown timers.  You have the disk shutdown time out and the VM shutdown time out both set to 60 seconds.  That's generally not enough time for Unraid to shutdown.  Read this post and see if you need to make any changes.

You are giving me snipets of logs that have little to no value.  Run your system until you have problems.  Then reboot and let Unraid time out on shutdown.  Don't force a shutdown.  Then post the diagnostics that Unraid collects at /flash/logs/.  If the diagnostics aren't collected, then post your syslog server log for the server.  Give me the full log, not just the last few entries.

Link to comment
4 hours ago, 1812 said:

done. this is on the same boot those the diags are from.

 

as far as timeline and how it lines up with the log:

 

shortly before 12:00:00 I stopped the array that was mounted via smb in unassigned devices without unmounting it. The loading/refreshing icon appeared and never went away in the UD section of the main tab. After a minute or two, the error at 12:00:21 appeared, which is similar to the nginx error previously reported. The loading/refreshing icon never went away, even after refreshing the page. This is an unrecoverable error and UD loses all functionality. Any services requiring mounting or un-mounting of disks to operate is also affected (backup services, etc)

 

I then initiated a power down via webgui at 12:01:03. The server sat in a hung state for 5 minutes with the last entry at 12:02:21 showing active pids still, which I believe are via UD as the server powers down cleanly if all smb mounts are unmounted before turning off. After 5 minutes I had to shut the server off by holding the power button as it was clear it was not going to recover.

 

 

syslog-192.168.1.253.log 7.65 kB · 1 download

Do you have dockers using the remote SMB shares?

Link to comment

I have not been able to find an answer to if this is a bug or by design. 

 

When I have an NFS mount configured for auto mount and it is not available at unRAID boot the device will remain unmounted. Once it becomes available the mount button will become green/orange and allow me to mount it. Should it not just auto mount the share when it sees it available?

 

I don't know if this is intended or a bug but it would be really nice if NFS shares would always auto connect when they become available. 

Link to comment
1 hour ago, _rogue said:

I have not been able to find an answer to if this is a bug or by design. 

 

When I have an NFS mount configured for auto mount and it is not available at unRAID boot the device will remain unmounted. Once it becomes available the mount button will become green/orange and allow me to mount it. Should it not just auto mount the share when it sees it available?

 

I don't know if this is intended or a bug but it would be really nice if NFS shares would always auto connect when they become available. 

Remote mounts are mounted when the array starts if they are set to auto mount.  UD does not attempt another mount if the remote mount comes on-line.

 

You could set up a User Script on a cron to auto mount remote mounts with the following script:

/usr/local/sbin/rc.unassigned mount auto

This will mount any devices that are not mounted if they are set to auto mount.  It doesn't affect any devices already mounted, but could add a lot of entries to the log.  It will also mount any disks set to auto mount that are not mounted.

Link to comment
47 minutes ago, dlandon said:

Remote mounts are mounted when the array starts if they are set to auto mount.  UD does not attempt another mount if the remote mount comes on-line.

 

You could set up a User Script on a cron to auto mount remote mounts with the following script:


/usr/local/sbin/rc.unassigned mount auto

This will mount any devices that are not mounted if they are set to auto mount.  It doesn't affect any devices already mounted, but could add a lot of entries to the log.  It will also mount any disks set to auto mount that are not mounted.

This is exactly what I needed! I can make a quick script that checks if the mount point exists and if it does not run this command.

 

Thank you!

Link to comment
2 hours ago, dlandon said:

You need to review your shutdown timers.  You have the disk shutdown time out and the VM shutdown time out both set to 60 seconds.  That's generally not enough time for Unraid to shutdown.  Read this post and see if you need to make any changes.

There were on whatever the default is. But I will adjust them accordingly per your post. As I mentioned before the system hang exceeds the default shutdown timers. I don't understand how extending it will help since it isn't Unraid that has to force the unclean shutdown but me because the system is hung. But I'm no expert and have changed to your recommended settings in that post.

 

2 hours ago, dlandon said:

You are giving me snipets of logs that have little to no value.  Run your system until you have problems.  Then reboot and let Unraid time out on shutdown.  Don't force a shutdown.  Then post the diagnostics that Unraid collects at /flash/logs/.  If the diagnostics aren't collected, then post your syslog server log for the server.  Give me the full log, not just the last few entries.

I gave you exactly what you asked for, which was everything from when the remote syslog was activated (which is why I mentioned it was a continuation of the full diags I previously posted. I activated the remote syslog, triggered the error, attempted to shut down the server (which hung for over five minutes which had to then be forced off) then pulled the syslog of the remote storage and gave it to you.  

 

Regardless, I will replicate the error again and give you the full syslog since the remote logging has been active since reboot.

 

2 hours ago, dlandon said:

Do you have dockers using the remote SMB shares?

yes, cloudberry. I'll try replicating the issue with this docker off to this time to eliminate that variable.

 

thank you for your patience with me on this. I'm trying to provide accurate information so the cause can be discovered!

Link to comment
18 minutes ago, 1812 said:

There were on whatever the default is. But I will adjust them accordingly per your post. As I mentioned before the system hang exceeds the default shutdown timers. I don't understand how extending it will help since it isn't Unraid that has to force the unclean shutdown but me because the system is hung. But I'm no expert and have changed to your recommended settings in that post.

Unraid should time out and force a shutdown once the timeout has been reached.  If Unraid doesn't force a shutdown, you have other issues.  This should happen no matter what the remote shares are doing.

 

 

22 minutes ago, 1812 said:

yes, cloudberry. I'll try replicating the issue with this docker off to this time to eliminate that variable.

Some have reported issues with dockers using remote mounted shares.  I'm thinking your docker may be hanging the system when the remote share goes off-line.

Link to comment
3 hours ago, dlandon said:

Give me the full log, not just the last few entries.

sorry to spam you with quotes, but bad news/good news

 

So evidently I didn't setup my remote syslog properly. Even though I told it to to keep 3 logs in rotation, it overwrote my latest error. But, I have new info that I believe is the problem.

 

It may be that before when UD was doing the "refreshing/processing" moving graphic I was impatient even though it was throwing nginx errors.  I just tested again and it eventually shows the remote share, still mounted but disconnected. I tired this with and without cloudberry running (the only docker that uses remote mounts) and that had no change.

 

Then I realized that my router/firewall is in a vm. So I turned off that vm and retested, and sure enough, the timeout wouldn't happen and the error would return. UD was unusable as previously reported. But upon starting up my firewall vm again, UD would return to normal usage and show the expected disconnected remote smb share, which would resume operation normally when it rejoined the network. 

 

So I tested this theory by disconnecting the remote smb share, then refreshed UD which appears to get hung at loading. I then shut down the vm. this kept UD in a perpetual loading state (it appeared with errors about the remote share being offline) I then shut the server down. But oddly enough, it actually shut down this time. I'm not sure if it had to do with the readjustments to the timeouts you suggested, but it was gracefully powered off after only 2 minutes.

 

So I believe this is what is happening: a remote share goes offline, UD appeared to hang and was unresponsive. I'd send the restart command to my server which would shutdown the vm, exacerbating the "hang" with UD not finding the share, then the shortened shutdown timer was trying to power down the server before UD could actually unmount the missing disk, which caused the shutdown hang.

 

 

I don't know if any of this makes sense, or if I'm just rambling. But either way, I believe the issue is resolved. Thank you again for your time and suggestions which appear to have corrected the problem!

 

 

Link to comment
On 5/24/2019 at 12:56 AM, Marshalleq said:

OK, I guess I have to live with it.  It doesn't seem to impact anything, just looks horrible.

Did you ever get this squared... I am experiencing the same.  Passing through nvme controller...and booting winVM off of it.  I did this previously without issues.  Nuked the VM and started again, and this popped up.

 

edit:  actually I can no longer see the partitions.  

Edited by J.Nerdy
Link to comment

No I didn't, it's bloody awful isn't it.  I'm really not sure why it's happening either.  Did you happen to be experimenting both with passthrough PCI device via stubbing the drive and also via the XML code in the VM?  I seem to remember it started when I did one of those but not the other, though removing either / both didn't fix the issue.

Link to comment
16 minutes ago, Marshalleq said:

No I didn't, it's bloody awful isn't it.  I'm really not sure why it's happening either.  Did you happen to be experimenting both with passthrough PCI device via stubbing the drive and also via the XML code in the VM?  I seem to remember it started when I did one of those but not the other, though removing either / both didn't fix the issue.

Come to think of it... I did at one point have it stubbed.  However, then I copied my winVM to it, and partitioned the remaining drive in xfs to share some high throughput storage to the array.

 

It did not manifest this issue at the time.  However, after nuking the VM and passing through the controller again with a new VM.

 

The performance on the VM is spot on... my only issue is that though UD can see the drive, it can not see the partitions - so I can not format the unused space and share back to the array.

 

i might stub it out.  Reboot.  Then reboot and pass through.  See if that makes a difference.

 

Link to comment
On 7/7/2019 at 4:35 PM, devros said:

Just tried it on an identical 4T drive also hooked up directly to the motherboard and this time it did mount as a 4T mount

had to reboot, UD shows the device with the "MOUNT" option greyed out.  If I try and run an fsck, it just searches forever for a superblock.  Running out of options here.  UD has served me very well over the years, but this one has be stumped.

Link to comment
5 hours ago, devros said:

had to reboot, UD shows the device with the "MOUNT" option greyed out.  If I try and run an fsck, it just searches forever for a superblock.  Running out of options here.  UD has served me very well over the years, but this one has be stumped.

Have you enabled Destructive mode in the UD settings?    You need this to be allowed to format.

Link to comment

@dlandon

 

I am getting the following error in UD:

Warning: Illegal string offset 'DEVTYPE' in /usr/local/emhttp/plugins/unassigned.devices/include/lib.php on line 1347

This is for a passed through nvme drive.  The device is using a modified clover to boot a win10 vm.  I had this same configuration previously with no issues (UD listed device and partition tables without issue when VM powered down).

 

I am way out of my depth when it comes to functions and setting arrays in php. 

 

What I am wondering is since this is the getting partition info function... could have something to do with the partition map created in windows?

 

Here is the disk log info:

Jul 16 12:29:02 nerdyRAID kernel: nvme1n1: p1 p2 p3 p4
Jul 16 12:29:15 nerdyRAID emhttpd: Samsung_SSD_970_PRO_512GB_S463NF0K713728K (nvme1n1) 512 1000215216
Jul 16 13:11:56 nerdyRAID unassigned.devices: Adding disk '/dev/nvme1n1p3'...
Jul 16 13:11:56 nerdyRAID unassigned.devices: Mount drive command: /sbin/mount -t ntfs -o auto,async,noatime,nodiratime,nodev,nosuid,umask=000 '/dev/nvme1n1p3' '/mnt/disks/GAMES'
Jul 16 13:11:56 nerdyRAID ntfs-3g[1459]: Mounted /dev/nvme1n1p3 (Read-Write, label "GAMES", NTFS 3.1)
Jul 16 13:11:56 nerdyRAID ntfs-3g[1459]: Mount options: rw,nodiratime,nodev,nosuid,allow_other,nonempty,noatime,default_permissions,fsname=/dev/nvme1n1p3,blkdev,blksize=4096
Jul 16 13:11:56 nerdyRAID unassigned.devices: Successfully mounted '/dev/nvme1n1p3' on '/mnt/disks/GAMES'.
Jul 16 13:11:56 nerdyRAID unassigned.devices: Adding disk '/dev/nvme1n1p4'...
Jul 16 13:11:56 nerdyRAID unassigned.devices: No filesystem detected on '/dev/nvme1n1p4'.

I clearly see 2 of the 4 partitions:  the unformatted raw partition (that I want to use as a XFS partition for array when machine is not running and the primary partition created for storage of 'games' in NTFS)

 

I do not see the 'efi' partition  nor the windows [C:\] drive.  

 

Clearly this is some type of user error on my part... but just not sure how or why it got borked when previous steps worked flawlessly in past.

 

If it would help I could post 1347 - 1403 of lib.php and see if something got corrupted?  Any suggestions would be greatly appreciated.  

 

Thank you!

 

EDIT:  I did not mention, the VM works flawlessly.  It is simply I can not pull accurate partition map in UD.  Sorry.

Edited by J.Nerdy
derp
Link to comment

Hi, what should I do if I want to use an USB flash drive of 8GB just to store data as backup (assignment of disks to array, logs, ...). Yesterday I had to install everything new from scratch and realized I did not have the image of how my disks were assigned to each slot. 

Just want to use a spare USB to store data like this, plug it then back in my windows laptop and then access it..

I am not sure what format to use.

Rgds

 

Link to comment
52 minutes ago, luca2 said:

Hi, what should I do if I want to use an USB flash drive of 8GB just to store data as backup (assignment of disks to array, logs, ...). Yesterday I had to install everything new from scratch and realized I did not have the image of how my disks were assigned to each slot. 

Just want to use a spare USB to store data like this, plug it then back in my windows laptop and then access it..

I am not sure what format to use.

Rgds

 

Format it in Windows.  UD will then mount it for backups.

Link to comment

I like this plugin a lot, however one thing bothers me.

 

When a drive is mounted (not even shared), your plugin issues "lsof -- /mnt/disks/tempdrive" command every few seconds.

This causes remarkable CPU spikes (30% out of 400%) via process "php" and don´t let that drive ever spin down.

 

Would it be possible to have an option to disable "Open files" info or at least don´t issue this command when the Main dashboard is not actively displayed?

This little enhancement is much appreciated.

Link to comment
23 minutes ago, Forusim said:

I like this plugin a lot, however one thing bothers me.

 

When a drive is mounted (not even shared), your plugin issues "lsof -- /mnt/disks/tempdrive" command every few seconds.

This causes remarkable CPU spikes (30% out of 400%) via process "php" and don´t let that drive ever spin down.

 

Would it be possible to have an option to disable "Open files" info or at least don´t issue this command when the Main dashboard is not actively displayed?

This little enhancement is much appreciated.

Do you have the preclear plugin installed?

Link to comment

I'm having issues with my UD no longer mounting my 2-disk BTRFS RAID1 pool ("ssd" = nvme0n1p1 set to automount + "ssd2" = nvme1n1p1). The 2nd disk [mounted to "ssd2"] of the pool (not set to automount) doesn't get luksOpen'ed and therefore is "missing," so the automount of the 1st disk fails.

Jul 22 12:33:02 Tower unassigned.devices: Mounting 'Auto Mount' Devices...
Jul 22 12:33:02 Tower unassigned.devices: Adding disk '/dev/mapper/ssd'...
Jul 22 12:33:03 Tower kernel: BTRFS: device fsid 4939602d-ea6f-4dd0-a535-252580b60aac devid 1 transid 11909816 /dev/dm-13
Jul 22 12:33:03 Tower unassigned.devices: Mount drive command: /sbin/mount  '/dev/mapper/ssd' '/mnt/disks/ssd'
Jul 22 12:33:03 Tower kernel: BTRFS info (device dm-13): disk space caching is enabled
Jul 22 12:33:03 Tower kernel: BTRFS info (device dm-13): has skinny extents
Jul 22 12:33:03 Tower kernel: BTRFS error (device dm-13): devid 2 uuid f66ef13b-8a93-45da-b73e-0d6a478729b2 is missing
Jul 22 12:33:03 Tower kernel: BTRFS error (device dm-13): failed to read chunk tree: -2
Jul 22 12:33:03 Tower kernel: BTRFS error (device dm-13): open_ctree failed
Jul 22 12:33:03 Tower emhttpd: Warning: Use of undefined constant luks - assumed 'luks' (this will throw an Error in a future version of PHP) in /usr/local/emhttp/plugins/unassigned.devices/include/lib.php on line 634
Jul 22 12:33:03 Tower unassigned.devices: Mount of '/dev/mapper/ssd' failed. Error message: mount: /mnt/disks/ssd: wrong fs type, bad option, bad superblock on /dev/mapper/ssd, missing codepage or helper program, or other error. 
Jul 22 12:33:03 Tower unassigned.devices: Partition 'Samsung_SSD_970_PRO_1TB_S462NF0M300357X' could not be mounted...
Jul 22 12:33:03 Tower unassigned.devices: Disk with serial 'Samsung_SSD_970_PRO_1TB_S462NF0M311892X', mountpoint 'ssd2' is not set to auto mount and will not be mounted...

After every array start, I have to execute luksOpen on "ssd2" and mount "ssd" manually (see below). Then after this, unmount/mount works again in the GUI.

root@Tower:~# /usr/sbin/cryptsetup luksOpen /dev/nvme1n1p1 ssd2 --allow-discards --key-file /root/keyfile
root@Tower:~# mkdir /mnt/disks/ssd
root@Tower:~# /sbin/mount '/dev/mapper/ssd' '/mnt/disks/ssd'

I think this happened after I renamed the mounts and physically changed slots, though it seemed to work after one restart, but then broke after the next restart. Have restarted a few times since and it never works anymore, even if I change the names.

 

If I switch the nvme1n1 to automount instead of nvme0n1, it still results in the same error.

tower-diagnostics-20190722-1649.zip

Edited by golli53
fixed manual command
Link to comment

Hi,

 

I have two HDD's in an external enclosure which I want to connect to my server to act as backups for certain shares. What filesystem would be best to format these as so Unraid can read and write to them but also I can connect them directly to my Mac and it can read and write to them.

Link to comment
  • trurl pinned this topic

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.