Unassigned Devices - Managing Disk Drives and Remote Shares Outside of The Unraid Array


Recommended Posts

I'm having an issue where Unassigned Devices will stop responding randomly. The UD section in my Main tab just gives me an unending dancing unRAID logo, I can't browse my USB connected drive, and any docker containers with folders mapped to that drive become unresponsive. It will also cause all of my CPU cores to appear to be pegged out in the Dashboard.

 

When it does this, I am unable to stop any docker containers or my array. I am unable to shut down my server cleanly and my only option is to do a hard reset.

 

If I try to do anything with UD when it does this, it will hose my web UI completely and I can't access anything until the server is hard reset. Any docker containers that I have not touched and do not have any mapped folders to my UD drive will continue to function. I can also still RDP into my Win10 VM. However, anything to do with browsing the server UI is not possible.

 

I thought I read about this being a known issue and that it was addressed in a recent update, but it has just happened to me again. I am at work right now and am afraid to touch anything on it in fear that the web UI will get hosed and I won't be able to do anything with it until I get home.

Edited by Phastor
Link to comment
2 hours ago, Phastor said:

I'm having an issue where Unassigned Devices will stop responding randomly. The UD section in my Main tab just gives me an unending dancing unRAID logo, I can't browse my USB connected drive, and any docker containers with folders mapped to that drive become unresponsive. It will also cause all of my CPU cores to appear to be pegged out in the Dashboard.

 

When it does this, I am unable to stop any docker containers or my array. I am unable to shut down my server cleanly and my only option is to do a hard reset.

 

If I try to do anything with UD when it does this, it will hose my web UI completely and I can't access anything until the server is hard reset. Any docker containers that I have not touched and do not have any mapped folders to my UD drive will continue to function. I can also still RDP into my Win10 VM. However, anything to do with browsing the server UI is not possible.

 

I thought I read about this being a known issue and that it was addressed in a recent update, but it has just happened to me again. I am at work right now and am afraid to touch anything on it in fear that the web UI will get hosed and I won't be able to do anything with it until I get home.

Can't help you without diagnostics.

Edited by dlandon
Link to comment
51 minutes ago, dlandon said:

Can't help you without diagnostics.

Just tried running diagnostics and that seems to have hosed my UI. This thing will go at the drop of a hat as soon as UD starts to do this. Will have to wait till I'm home to do a hard reset before I can look into it further.

Edited by Phastor
Link to comment

I'm having an issue installing UD+ it seems the path for nmap is wrong?

Am I correct in thinking you must have UD+ to mount remote shares?

plugin: installing: https://github.com/dlandon/unassigned.devices/raw/master/unassigned.devices-plus.plg
plugin: downloading https://github.com/dlandon/unassigned.devices/raw/master/unassigned.devices-plus.plg
plugin: downloading: https://github.com/dlandon/unassigned.devices/raw/master/unassigned.devices-plus.plg ... done
plugin: downloading: "https://github.com/dlandon/unassigned.devices/raw/master/packages/nmap-7.80-x86_64-3.txz" ... failed (Invalid URL / Server error response)
plugin: wget: "https://github.com/dlandon/unassigned.devices/raw/master/packages/nmap-7.80-x86_64-3.txz" download failure (Invalid URL / Server error response)

Updating Support Links

 

Link to comment
7 minutes ago, dlandon said:

That means UD thinks the remote server is off-line.  The remote server must respond to a ping to be considered on-line.

ah that will be why then, the remote share is behind a router. Thanks should be able to sort that with firewall rule. Thank you very much for your help 🙂

Link to comment
6 minutes ago, Phastor said:

After about an hour of the diag page appearing locked up, it gave me the zip file.

aivas-diagnostics-20200113-2202.zip 146.1 kB · 0 downloads

You have a USB device failing:

Jan 13 12:55:41 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
### [PREVIOUS LINE REPEATED 1 TIMES] ###
Jan 13 12:55:57 AIVAS php-fpm[7449]: [WARNING] [pool www] server reached max_children setting (50), consider raising it
Jan 13 12:56:12 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
### [PREVIOUS LINE REPEATED 56 TIMES] ###
Jan 13 13:14:25 AIVAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 13 13:14:40 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
### [PREVIOUS LINE REPEATED 3 TIMES] ###
Jan 13 13:16:11 AIVAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 13 13:16:27 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
### [PREVIOUS LINE REPEATED 18 TIMES] ###
Jan 13 13:43:05 AIVAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 13 13:44:06 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
### [PREVIOUS LINE REPEATED 1 TIMES] ###
Jan 13 13:45:08 AIVAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 13 13:46:09 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd

 

Link to comment
36 minutes ago, dlandon said:

You have a USB device failing:


Jan 13 12:55:41 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
### [PREVIOUS LINE REPEATED 1 TIMES] ###
Jan 13 12:55:57 AIVAS php-fpm[7449]: [WARNING] [pool www] server reached max_children setting (50), consider raising it
Jan 13 12:56:12 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
### [PREVIOUS LINE REPEATED 56 TIMES] ###
Jan 13 13:14:25 AIVAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 13 13:14:40 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
### [PREVIOUS LINE REPEATED 3 TIMES] ###
Jan 13 13:16:11 AIVAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 13 13:16:27 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
### [PREVIOUS LINE REPEATED 18 TIMES] ###
Jan 13 13:43:05 AIVAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 13 13:44:06 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
### [PREVIOUS LINE REPEATED 1 TIMES] ###
Jan 13 13:45:08 AIVAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 13 13:46:09 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd

 

This has happened with two different USB drives. I guess I'll need to scrounge up a third one and see. Or perhaps it could be a controller? I'm using a PCIe USB3 controller since my board does not have it onboard. Is it UD's interaction with the failed device that's causing the server-wide issues? 

Link to comment
4 minutes ago, Phastor said:

This has happened with two different USB drives. I guess I'll need to scrounge up a third one and see. Or perhaps it could be a controller? I'm using a PCIe USB3 controller since my board does not have it onboard. Is it UD's interaction with the failed device that's causing the server-wide issues? 

Sort of.  UD manages the mounting/unmounting.  The issues with the drive itself would cause your whole system to 'hang'.  If you go to the UD webpage, UD will struggle trying to get the status on the disks, so it will appear to 'hang' up also.

 

I'm not a USB expert, but the USB3 may be an issue.  Try USB2.  I've heard people having issues with USB3 on Unraid.  Not Unraid's issue though.  It seems to be Linux USB3 drivers.

Link to comment
1 hour ago, dlandon said:

Sort of.  UD manages the mounting/unmounting.  The issues with the drive itself would cause your whole system to 'hang'.  If you go to the UD webpage, UD will struggle trying to get the status on the disks, so it will appear to 'hang' up also.

 

I'm not a USB expert, but the USB3 may be an issue.  Try USB2.  I've heard people having issues with USB3 on Unraid.  Not Unraid's issue though.  It seems to be Linux USB3 drivers.

As much as that would hurt, I'll give USB2 a shot. The USB drive in question is my Duplicati backup target, which is already slow the way it is. I may just get myself a small NAS for my backups if that turns out to be the issue. It's just weird since this issue only surfaced a few months ago after about two years of unRAID/UD use.

 

Thanks for the help! I'll keep you posted with what happens.

Link to comment
4 minutes ago, Phastor said:

As much as that would hurt, I'll give USB2 a shot. The USB drive in question is my Duplicati backup target, which is already slow the way it is. I may just get myself a small NAS for my backups if that turns out to be the issue. It's just weird since this issue only surfaced a few months ago after about two years of unRAID/UD use.

 

Thanks for the help! I'll keep you posted with what happens.

See if you don't get better reliability on UBS2.  That can tell you a lot.

 

Are you using a USB Hub?

Link to comment

Not sure if reported yet.  When formatting ext4 says 64-bit filesystem support is not enabled:

Jan 13 20:32:06 Tower unassigned.devices: Format disk '/dev/nvme0n1' with 'ext4' filesystem result: mke2fs 1.45.0 (6-Mar-2019) 64-bit filesystem support is not enabled.  The larger fields afforded by this feature enable full-strength checksumming.  Pass -O 64bit to rectify. Discarding device blocks:      4096/250050816#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010  629555{.....}done 

 

Link to comment

I'm having issues with unmounting the drive from a user script. I haven't had any issues out of UD until after I upgraded to Unraid 6.8.1. The issue I'm having is after my backup script(rclone) finishes the drive fails to unmount with error "Error: shell_exec(/sbin/umount '/dev/sda1' 2>&1) took longer than 90s!" in the syslog. I will put a full timeline from the syslog down below. However the drive in UD shows it is still mounted but it doesn't show the correct usage... and when I try to unmount it I get even more errors in the logs. Any help would be appreciated... Side note I can remount it via command line and the drive is accessible again...

Jan 13 07:00:01 Peter unassigned.devices: Adding disk '/dev/sda1'...
Jan 13 07:00:01 Peter unassigned.devices: Mount drive command: /sbin/mount -t xfs -o rw,noatime,nodiratime '/dev/sda1' '/mnt/disks/Unraid-Array-Backup'
Jan 13 07:00:02 Peter kernel: XFS (sda1): Mounting V5 Filesystem
Jan 13 07:00:02 Peter kernel: XFS (sda1): Ending clean mount
Jan 13 07:00:03 Peter unassigned.devices: Successfully mounted '/dev/sda1' on '/mnt/disks/Unraid-Array-Backup'.
Jan 13 07:00:08 Peter sSMTP[25148]: Creating SSL connection to host
Jan 13 07:00:08 Peter sSMTP[25148]: SSL connection using TLS_AES_256_GCM_SHA384
Jan 13 07:00:10 Peter sSMTP[25148]: Sent mail for STRIPPED (221 2.0.0 Bye) uid=0 username=root outbytes=753
Jan 13 07:32:01 Peter kernel: md: sync done. time=27120sec
Jan 13 07:32:04 Peter kernel: md: recovery thread: exit status: 0
Jan 13 07:33:02 Peter sSMTP[26155]: Creating SSL connection to host
Jan 13 07:33:02 Peter sSMTP[26155]: SSL connection using TLS_AES_256_GCM_SHA384
Jan 13 07:33:04 Peter sSMTP[26155]: Sent mail for STRIPPED (221 2.0.0 Bye) uid=0 username=root outbytes=791
Jan 13 07:37:51 Peter sSMTP[30935]: Creating SSL connection to host
Jan 13 07:37:51 Peter sSMTP[30935]: SSL connection using TLS_AES_256_GCM_SHA384
Jan 13 07:37:53 Peter sSMTP[30935]: Sent mail for STRIPPED (221 2.0.0 Bye) uid=0 username=root outbytes=1145
Jan 13 07:37:53 Peter unassigned.devices: Unmounting disk 'WD easystore_25FB'...
Jan 13 07:37:53 Peter unassigned.devices: Unmounting '/dev/sda1'...
Jan 13 07:37:53 Peter unassigned.devices: Unmount cmd: /sbin/umount '/dev/sda1' 2>&1
Jan 13 07:39:50 Peter kernel: XFS (sda1): Unmounting Filesystem
Jan 13 07:39:50 Peter unassigned.devices: Error: shell_exec(/sbin/umount '/dev/sda1' 2>&1) took longer than 90s!
Jan 13 07:39:50 Peter unassigned.devices: Unmount of '/dev/sda1' failed. Error message: command timed out
Jan 13 07:39:51 Peter unassigned.devices: Since there aren't any open files, will force unmount.
Jan 13 07:39:51 Peter unassigned.devices: Unmounting '/dev/sda1'...
Jan 13 07:39:51 Peter unassigned.devices: Unmount cmd: /sbin/umount -fl '/dev/sda1' 2>&1
Jan 13 07:39:51 Peter unassigned.devices: Unmount of '/dev/sda1' failed. Error message: umount: /mnt/disks/Unraid-Array-Backup: not mounted. 
Jan 13 07:39:52 Peter unassigned.devices: Disk 'WD easystore_25FB' could not be unmounted.

User script backup script

#!/bin/bash
#backgroundOnly=true
#arrayStarted=true

/usr/local/emhttp/plugins/unassigned.devices/scripts/rc.unassigned mount /dev/sda
sleep 5

#check if drive mounted
if [ -d "/mnt/disks/Unraid-Array-Backup/" ]
then

/usr/local/emhttp/webGui/scripts/notify -s "`hostname` to Local Easy Store Backup started" -i "normal" -d "Sync started. `date`"

/usr/bin/rsync -avrtHh --delete /mnt/user/ /mnt/disks/Unraid-Array-Backup/ >> /mnt/user/Backups/Unraid/logs/Unraid-Array-Backup/backup.log

/usr/local/emhttp/webGui/scripts/notify -s "`hostname` to Local Easy Store Backup completed" -i "normal" -d "Sync completed. `date`" -m "`tail /mnt/user/Backups/Unraid/logs/Unraid-Array-Backup/backup.log`"

#unmount drive
/usr/local/emhttp/plugins/unassigned.devices/scripts/rc.unassigned umount /dev/sda

else
echo "Backup failed!"

/usr/local/emhttp/webGui/scripts/notify -s "`hostname` to Local Easy Store Backup FAILED" -i "alert" -d "Sync FAILED `date`" -m "`tail /mnt/user/Backups/Unraid/logs/Unraid-Array-Backup/backup.log`"
fi

 

Edited by Killar34
Link to comment
7 hours ago, Killar34 said:

I'm having issues with unmounting the drive from a user script. I haven't had any issues out of UD until after I upgraded to Unraid 6.8.1. The issue I'm having is after my backup script(rclone) finishes the drive fails to unmount with error "Error: shell_exec(/sbin/umount '/dev/sda1' 2>&1) took longer than 90s!" in the syslog. I will put a full timeline from the syslog down below. However the drive in UD shows it is still mounted but it doesn't show the correct usage... and when I try to unmount it I get even more errors in the logs. Any help would be appreciated... Side note I can remount it via command line and the drive is accessible again...

Time outs have been added to UD unmounts so the array doesn't hang when shutting down.  It appears that your rsync command was not finished writing to the disk so the unmount timed out.  Add the following command to your script before the unmount:

sync -f $MOUNTPOINT

This will flush all disk buffers before you unmount the disk.

In your case:

sync -f /mnt/disks/Unraid-Array-Backup/

 

Edited by dlandon
Link to comment
2 hours ago, dlandon said:

Time outs have been added to UD unmounts so the array doesn't hang when shutting down.  It appears that your rsync command was not finished writing to the disk so the unmount timed out.  Add the following command to your script before the unmount:


sync -f $MOUNTPOINT

This will flush all disk buffers before you unmount the disk.

In your case:


sync -f /mnt/disks/Unraid-Array-Backup/

 

OK thank you. I will give that a try and let you know.

Link to comment

Every since a few updates ago, I've been getting the following errors in my logs every now and then regarding my UD ssd. The SSD is about 2 years old with a lot of writes but I don't see any other issues with it, SMART reports are all ok. Should I look at replacing it? Thoughts?

 

Quote

Jan 14 11:51:11 TOWER unassigned.devices: Error: shell_exec(/usr/bin/lsof '/mnt/disks/evo' 2>/dev/null | /bin/sort -k8 | /bin/uniq -f7 | /bin/grep -c -e REG) took longer than 5s!

 

Link to comment
34 minutes ago, FalconX said:

Every since a few updates ago, I've been getting the following errors in my logs every now and then regarding my UD ssd. The SSD is about 2 years old with a lot of writes but I don't see any other issues with it, SMART reports are all ok. Should I look at replacing it? Thoughts?

It's probably not related to your SSD.  This is most times caused by remote mounts having issues causing problems with lsof.  Post diagnostics.

Link to comment
  • trurl pinned this topic

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.