Since updating to 2020.01.12 I am seeing large CPU spikes from lsof on my unassigned drives. goes up to 30% every few minutes.

 

nevermind, looks like I had the File Activity plugin running. note to self: turn it off when not needed,

Edited January 12, 20206 yr by dbinott

I will test this shortly, thanks.

There is a new version of UD available.  SMB share passwords are now encrypted when stored in the config file.  You will need to remove any SMB shares using passwords and re-create them.  They will not mount after the latest update until you re-create them.

I'm having an issue where Unassigned Devices will stop responding randomly. The UD section in my Main tab just gives me an unending dancing unRAID logo, I can't browse my USB connected drive, and any docker containers with folders mapped to that drive become unresponsive. It will also cause all of my CPU cores to appear to be pegged out in the Dashboard.

 

When it does this, I am unable to stop any docker containers or my array. I am unable to shut down my server cleanly and my only option is to do a hard reset.

 

If I try to do anything with UD when it does this, it will hose my web UI completely and I can't access anything until the server is hard reset. Any docker containers that I have not touched and do not have any mapped folders to my UD drive will continue to function. I can also still RDP into my Win10 VM. However, anything to do with browsing the server UI is not possible.

 

I thought I read about this being a known issue and that it was addressed in a recent update, but it has just happened to me again. I am at work right now and am afraid to touch anything on it in fear that the web UI will get hosed and I won't be able to do anything with it until I get home.

Edited January 13, 20206 yr by Phastor

2 hours ago, Phastor said:

I'm having an issue where Unassigned Devices will stop responding randomly. The UD section in my Main tab just gives me an unending dancing unRAID logo, I can't browse my USB connected drive, and any docker containers with folders mapped to that drive become unresponsive. It will also cause all of my CPU cores to appear to be pegged out in the Dashboard.

 

When it does this, I am unable to stop any docker containers or my array. I am unable to shut down my server cleanly and my only option is to do a hard reset.

 

If I try to do anything with UD when it does this, it will hose my web UI completely and I can't access anything until the server is hard reset. Any docker containers that I have not touched and do not have any mapped folders to my UD drive will continue to function. I can also still RDP into my Win10 VM. However, anything to do with browsing the server UI is not possible.

 

I thought I read about this being a known issue and that it was addressed in a recent update, but it has just happened to me again. I am at work right now and am afraid to touch anything on it in fear that the web UI will get hosed and I won't be able to do anything with it until I get home.

Can't help you without diagnostics.

Edited January 13, 20206 yr by dlandon

51 minutes ago, dlandon said:

Can't help you without diagnostics.

Just tried running diagnostics and that seems to have hosed my UI. This thing will go at the drop of a hat as soon as UD starts to do this. Will have to wait till I'm home to do a hard reset before I can look into it further.

Edited January 13, 20206 yr by Phastor

I'm having an issue installing UD+ it seems the path for nmap is wrong?

Am I correct in thinking you must have UD+ to mount remote shares?

plugin: installing: https://github.com/dlandon/unassigned.devices/raw/master/unassigned.devices-plus.plg
plugin: downloading https://github.com/dlandon/unassigned.devices/raw/master/unassigned.devices-plus.plg
plugin: downloading: https://github.com/dlandon/unassigned.devices/raw/master/unassigned.devices-plus.plg ... done
plugin: downloading: "https://github.com/dlandon/unassigned.devices/raw/master/packages/nmap-7.80-x86_64-3.txz" ... failed (Invalid URL / Server error response)
plugin: wget: "https://github.com/dlandon/unassigned.devices/raw/master/packages/nmap-7.80-x86_64-3.txz" download failure (Invalid URL / Server error response)

Updating Support Links

 

7 minutes ago, buzzinh said:

Am I correct in thinking you must have UD+ to mount remote shares?

No.  Give it a few minutes and update again.  I've removed the Nmap and libnl packages.

4 minutes ago, dlandon said:

No.  Give it a few minutes and update again.  I've removed the Nmap and libnl packages.

Thanks its installed now. However im unable to mount remote shares on either of my unraid boxes. Mount is greyed out. 

18 minutes ago, buzzinh said:

Thanks its installed now. However im unable to mount remote shares on either of my unraid boxes. Mount is greyed out. 

That means UD thinks the remote server is off-line.  The remote server must respond to a ping to be considered on-line.

7 minutes ago, dlandon said:

That means UD thinks the remote server is off-line.  The remote server must respond to a ping to be considered on-line.

ah that will be why then, the remote share is behind a router. Thanks should be able to sort that with firewall rule. Thank you very much for your help 🙂

2 hours ago, dlandon said:

Can't help you without diagnostics.

After about an hour of the diag page appearing locked up, it gave me the zip file.

aivas-diagnostics-20200113-2202.zip

6 minutes ago, Phastor said:

After about an hour of the diag page appearing locked up, it gave me the zip file.

aivas-diagnostics-20200113-2202.zip 146.1 kB · 0 downloads

You have a USB device failing:

Jan 13 12:55:41 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
### [PREVIOUS LINE REPEATED 1 TIMES] ###
Jan 13 12:55:57 AIVAS php-fpm[7449]: [WARNING] [pool www] server reached max_children setting (50), consider raising it
Jan 13 12:56:12 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
### [PREVIOUS LINE REPEATED 56 TIMES] ###
Jan 13 13:14:25 AIVAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 13 13:14:40 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
### [PREVIOUS LINE REPEATED 3 TIMES] ###
Jan 13 13:16:11 AIVAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 13 13:16:27 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
### [PREVIOUS LINE REPEATED 18 TIMES] ###
Jan 13 13:43:05 AIVAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 13 13:44:06 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
### [PREVIOUS LINE REPEATED 1 TIMES] ###
Jan 13 13:45:08 AIVAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 13 13:46:09 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd

 

36 minutes ago, dlandon said:

You have a USB device failing:

Jan 13 12:55:41 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
### [PREVIOUS LINE REPEATED 1 TIMES] ###
Jan 13 12:55:57 AIVAS php-fpm[7449]: [WARNING] [pool www] server reached max_children setting (50), consider raising it
Jan 13 12:56:12 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
### [PREVIOUS LINE REPEATED 56 TIMES] ###
Jan 13 13:14:25 AIVAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 13 13:14:40 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
### [PREVIOUS LINE REPEATED 3 TIMES] ###
Jan 13 13:16:11 AIVAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 13 13:16:27 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
### [PREVIOUS LINE REPEATED 18 TIMES] ###
Jan 13 13:43:05 AIVAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 13 13:44:06 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
### [PREVIOUS LINE REPEATED 1 TIMES] ###
Jan 13 13:45:08 AIVAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 13 13:46:09 AIVAS kernel: usb 10-4: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd

 

This has happened with two different USB drives. I guess I'll need to scrounge up a third one and see. Or perhaps it could be a controller? I'm using a PCIe USB3 controller since my board does not have it onboard. Is it UD's interaction with the failed device that's causing the server-wide issues? 

4 minutes ago, Phastor said:

This has happened with two different USB drives. I guess I'll need to scrounge up a third one and see. Or perhaps it could be a controller? I'm using a PCIe USB3 controller since my board does not have it onboard. Is it UD's interaction with the failed device that's causing the server-wide issues? 

Sort of.  UD manages the mounting/unmounting.  The issues with the drive itself would cause your whole system to 'hang'.  If you go to the UD webpage, UD will struggle trying to get the status on the disks, so it will appear to 'hang' up also.

 

I'm not a USB expert, but the USB3 may be an issue.  Try USB2.  I've heard people having issues with USB3 on Unraid.  Not Unraid's issue though.  It seems to be Linux USB3 drivers.

1 hour ago, dlandon said:

Sort of.  UD manages the mounting/unmounting.  The issues with the drive itself would cause your whole system to 'hang'.  If you go to the UD webpage, UD will struggle trying to get the status on the disks, so it will appear to 'hang' up also.

 

I'm not a USB expert, but the USB3 may be an issue.  Try USB2.  I've heard people having issues with USB3 on Unraid.  Not Unraid's issue though.  It seems to be Linux USB3 drivers.

As much as that would hurt, I'll give USB2 a shot. The USB drive in question is my Duplicati backup target, which is already slow the way it is. I may just get myself a small NAS for my backups if that turns out to be the issue. It's just weird since this issue only surfaced a few months ago after about two years of unRAID/UD use.

 

Thanks for the help! I'll keep you posted with what happens.

4 minutes ago, Phastor said:

As much as that would hurt, I'll give USB2 a shot. The USB drive in question is my Duplicati backup target, which is already slow the way it is. I may just get myself a small NAS for my backups if that turns out to be the issue. It's just weird since this issue only surfaced a few months ago after about two years of unRAID/UD use.

 

Thanks for the help! I'll keep you posted with what happens.

See if you don't get better reliability on UBS2.  That can tell you a lot.

 

Are you using a USB Hub?

4 minutes ago, dlandon said:

See if you don't get better reliability on UBS2.  That can tell you a lot.

 

Are you using a USB Hub?

Nah. I've only got two USB devices aside from the flash. I've got my UPS plugged into USB2 on the board and the drive in the PCIe adapter.

16 minutes ago, Phastor said:

the drive in the PCIe adapter.

Re-seat the PCIe adapter?

Not sure if reported yet.  When formatting ext4 says 64-bit filesystem support is not enabled:

Jan 13 20:32:06 Tower unassigned.devices: Format disk '/dev/nvme0n1' with 'ext4' filesystem result: mke2fs 1.45.0 (6-Mar-2019) 64-bit filesystem support is not enabled.  The larger fields afforded by this feature enable full-strength checksumming.  Pass -O 64bit to rectify. Discarding device blocks:      4096/250050816#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010  629555{.....}done 

 

I'm having issues with unmounting the drive from a user script. I haven't had any issues out of UD until after I upgraded to Unraid 6.8.1. The issue I'm having is after my backup script(rclone) finishes the drive fails to unmount with error "Error: shell_exec(/sbin/umount '/dev/sda1' 2>&1) took longer than 90s!" in the syslog. I will put a full timeline from the syslog down below. However the drive in UD shows it is still mounted but it doesn't show the correct usage... and when I try to unmount it I get even more errors in the logs. Any help would be appreciated... Side note I can remount it via command line and the drive is accessible again...

Jan 13 07:00:01 Peter unassigned.devices: Adding disk '/dev/sda1'...
Jan 13 07:00:01 Peter unassigned.devices: Mount drive command: /sbin/mount -t xfs -o rw,noatime,nodiratime '/dev/sda1' '/mnt/disks/Unraid-Array-Backup'
Jan 13 07:00:02 Peter kernel: XFS (sda1): Mounting V5 Filesystem
Jan 13 07:00:02 Peter kernel: XFS (sda1): Ending clean mount
Jan 13 07:00:03 Peter unassigned.devices: Successfully mounted '/dev/sda1' on '/mnt/disks/Unraid-Array-Backup'.
Jan 13 07:00:08 Peter sSMTP[25148]: Creating SSL connection to host
Jan 13 07:00:08 Peter sSMTP[25148]: SSL connection using TLS_AES_256_GCM_SHA384
Jan 13 07:00:10 Peter sSMTP[25148]: Sent mail for STRIPPED (221 2.0.0 Bye) uid=0 username=root outbytes=753
Jan 13 07:32:01 Peter kernel: md: sync done. time=27120sec
Jan 13 07:32:04 Peter kernel: md: recovery thread: exit status: 0
Jan 13 07:33:02 Peter sSMTP[26155]: Creating SSL connection to host
Jan 13 07:33:02 Peter sSMTP[26155]: SSL connection using TLS_AES_256_GCM_SHA384
Jan 13 07:33:04 Peter sSMTP[26155]: Sent mail for STRIPPED (221 2.0.0 Bye) uid=0 username=root outbytes=791
Jan 13 07:37:51 Peter sSMTP[30935]: Creating SSL connection to host
Jan 13 07:37:51 Peter sSMTP[30935]: SSL connection using TLS_AES_256_GCM_SHA384
Jan 13 07:37:53 Peter sSMTP[30935]: Sent mail for STRIPPED (221 2.0.0 Bye) uid=0 username=root outbytes=1145
Jan 13 07:37:53 Peter unassigned.devices: Unmounting disk 'WD easystore_25FB'...
Jan 13 07:37:53 Peter unassigned.devices: Unmounting '/dev/sda1'...
Jan 13 07:37:53 Peter unassigned.devices: Unmount cmd: /sbin/umount '/dev/sda1' 2>&1
Jan 13 07:39:50 Peter kernel: XFS (sda1): Unmounting Filesystem
Jan 13 07:39:50 Peter unassigned.devices: Error: shell_exec(/sbin/umount '/dev/sda1' 2>&1) took longer than 90s!
Jan 13 07:39:50 Peter unassigned.devices: Unmount of '/dev/sda1' failed. Error message: command timed out
Jan 13 07:39:51 Peter unassigned.devices: Since there aren't any open files, will force unmount.
Jan 13 07:39:51 Peter unassigned.devices: Unmounting '/dev/sda1'...
Jan 13 07:39:51 Peter unassigned.devices: Unmount cmd: /sbin/umount -fl '/dev/sda1' 2>&1
Jan 13 07:39:51 Peter unassigned.devices: Unmount of '/dev/sda1' failed. Error message: umount: /mnt/disks/Unraid-Array-Backup: not mounted. 
Jan 13 07:39:52 Peter unassigned.devices: Disk 'WD easystore_25FB' could not be unmounted.

User script backup script

#!/bin/bash
#backgroundOnly=true
#arrayStarted=true

/usr/local/emhttp/plugins/unassigned.devices/scripts/rc.unassigned mount /dev/sda
sleep 5

#check if drive mounted
if [ -d "/mnt/disks/Unraid-Array-Backup/" ]
then

/usr/local/emhttp/webGui/scripts/notify -s "`hostname` to Local Easy Store Backup started" -i "normal" -d "Sync started. `date`"

/usr/bin/rsync -avrtHh --delete /mnt/user/ /mnt/disks/Unraid-Array-Backup/ >> /mnt/user/Backups/Unraid/logs/Unraid-Array-Backup/backup.log

/usr/local/emhttp/webGui/scripts/notify -s "`hostname` to Local Easy Store Backup completed" -i "normal" -d "Sync completed. `date`" -m "`tail /mnt/user/Backups/Unraid/logs/Unraid-Array-Backup/backup.log`"

#unmount drive
/usr/local/emhttp/plugins/unassigned.devices/scripts/rc.unassigned umount /dev/sda

else
echo "Backup failed!"

/usr/local/emhttp/webGui/scripts/notify -s "`hostname` to Local Easy Store Backup FAILED" -i "alert" -d "Sync FAILED `date`" -m "`tail /mnt/user/Backups/Unraid/logs/Unraid-Array-Backup/backup.log`"
fi

 

Edited January 14, 20206 yr by Killar34

7 hours ago, Killar34 said:

I'm having issues with unmounting the drive from a user script. I haven't had any issues out of UD until after I upgraded to Unraid 6.8.1. The issue I'm having is after my backup script(rclone) finishes the drive fails to unmount with error "Error: shell_exec(/sbin/umount '/dev/sda1' 2>&1) took longer than 90s!" in the syslog. I will put a full timeline from the syslog down below. However the drive in UD shows it is still mounted but it doesn't show the correct usage... and when I try to unmount it I get even more errors in the logs. Any help would be appreciated... Side note I can remount it via command line and the drive is accessible again...

Time outs have been added to UD unmounts so the array doesn't hang when shutting down.  It appears that your rsync command was not finished writing to the disk so the unmount timed out.  Add the following command to your script before the unmount:

sync -f $MOUNTPOINT

This will flush all disk buffers before you unmount the disk.

In your case:

sync -f /mnt/disks/Unraid-Array-Backup/

 

Edited January 14, 20206 yr by dlandon

2 hours ago, dlandon said:

Time outs have been added to UD unmounts so the array doesn't hang when shutting down.  It appears that your rsync command was not finished writing to the disk so the unmount timed out.  Add the following command to your script before the unmount:

sync -f $MOUNTPOINT

This will flush all disk buffers before you unmount the disk.

In your case:

sync -f /mnt/disks/Unraid-Array-Backup/

 

OK thank you. I will give that a try and let you know.

Every since a few updates ago, I've been getting the following errors in my logs every now and then regarding my UD ssd. The SSD is about 2 years old with a lot of writes but I don't see any other issues with it, SMART reports are all ok. Should I look at replacing it? Thoughts?

 

Quote

Jan 14 11:51:11 TOWER unassigned.devices: Error: shell_exec(/usr/bin/lsof '/mnt/disks/evo' 2>/dev/null | /bin/sort -k8 | /bin/uniq -f7 | /bin/grep -c -e REG) took longer than 5s!

 

34 minutes ago, FalconX said:

Every since a few updates ago, I've been getting the following errors in my logs every now and then regarding my UD ssd. The SSD is about 2 years old with a lot of writes but I don't see any other issues with it, SMART reports are all ok. Should I look at replacing it? Thoughts?

It's probably not related to your SSD.  This is most times caused by remote mounts having issues causing problems with lsof.  Post diagnostics.