Unassigned Devices - Managing Disk Drives and Remote Shares Outside of The Unraid Array


Recommended Posts

As this question relates to both UD and Preclear plugins, I’ll post it in both support threads. I’m one of the users who hasn’t had any UD or preclear issues until recently. I am currently preclearing 4 x 10TB drives (USB attached) via the UD integration with the preclear plugin. I’m 60% through the zero stage on all 4 drives but have seen some unusual issues today.

 

The only change to unRAID prior to these issues was updating the UD plugin to 2019.12.08 earlier this morning. I have been deploying the various updates for UD and preclear as they’ve been released over the last 2 weeks and up until today things seemed normal. Earlier this afternoon I started seeing a LOT of sporadic buffering/disk I/O issues and the CPU performance on my unRAID system is showing 65 - 75% load  pretty consistently.

 

This is based on the unRAID Dashboard tab of the web GUI. TOP in a terminal session reports about 15% utilization.

The buffering/disk I/O issues are real as transfers to/from unRAID speed up and slow down at random. I’m also seeing major interruptions to streams from Plex. As I can’t determine why the web GUI shows high CPU load but TOP doesn’t I’m wondering if the last update(s) to UD/Preclear might have broken something?

 

I’m still running unRAID 6.7.2 and note that most of the updates I’m seeing are for 6.8 related issues. I could update to 6.8 RC9 (with NVidia extensions) but I’ve held off. As the preclear is 60% through stage 2 (zeroing), I’m concerned about the upgrade and reboot possibly wiping the preclear state and making me start it over. If I have to I’ll tough it out until the preclear finishes tomorrow on all 4 drives.

Any thoughts on whether the preclear status will be saved if I update? Is there a way I can manually stop the preclears and save the state for restarting after the upgrade?

 

I’ve tried shutting down all unnecessary plugins/Dockers/VMs and the unRAID web GUI is still showing 60%+ CPU load and I’m still seeing disk I/O issues. What could cause the discrepancy between TOP in a terminal session and the web GUI?

 

Thoughts? Suggestions? Diagnostics would be attached but 3 attempts to download them have taken over an hour during the collection stage…. another bug/related issue? Tried on 2 separate computers on 3 different browsers (Safari/Firefox on MacOS Mojave and Firefox/Chrome on Ubuntu 18.04.

 

Dale

Link to comment
3 hours ago, AgentXXL said:

As this question relates to both UD and Preclear plugins, I’ll post it in both support threads. I’m one of the users who hasn’t had any UD or preclear issues until recently. I am currently preclearing 4 x 10TB drives (USB attached) via the UD integration with the preclear plugin. I’m 60% through the zero stage on all 4 drives but have seen some unusual issues today.

 

The only change to unRAID prior to these issues was updating the UD plugin to 2019.12.08 earlier this morning. I have been deploying the various updates for UD and preclear as they’ve been released over the last 2 weeks and up until today things seemed normal. Earlier this afternoon I started seeing a LOT of sporadic buffering/disk I/O issues and the CPU performance on my unRAID system is showing 65 - 75% load  pretty consistently.

 

This is based on the unRAID Dashboard tab of the web GUI. TOP in a terminal session reports about 15% utilization.

The buffering/disk I/O issues are real as transfers to/from unRAID speed up and slow down at random. I’m also seeing major interruptions to streams from Plex. As I can’t determine why the web GUI shows high CPU load but TOP doesn’t I’m wondering if the last update(s) to UD/Preclear might have broken something?

 

I’m still running unRAID 6.7.2 and note that most of the updates I’m seeing are for 6.8 related issues. I could update to 6.8 RC9 (with NVidia extensions) but I’ve held off. As the preclear is 60% through stage 2 (zeroing), I’m concerned about the upgrade and reboot possibly wiping the preclear state and making me start it over. If I have to I’ll tough it out until the preclear finishes tomorrow on all 4 drives.

Any thoughts on whether the preclear status will be saved if I update? Is there a way I can manually stop the preclears and save the state for restarting after the upgrade?

 

I’ve tried shutting down all unnecessary plugins/Dockers/VMs and the unRAID web GUI is still showing 60%+ CPU load and I’m still seeing disk I/O issues. What could cause the discrepancy between TOP in a terminal session and the web GUI?

 

Thoughts? Suggestions? Diagnostics would be attached but 3 attempts to download them have taken over an hour during the collection stage…. another bug/related issue? Tried on 2 separate computers on 3 different browsers (Safari/Firefox on MacOS Mojave and Firefox/Chrome on Ubuntu 18.04.

 

Dale

I would wait until the preclear is finished Nd then upgrade.

 

As.i just finishes precleari g a 10tb and it take 4 days to run.

Link to comment
31 minutes ago, chris_netsmart said:

I would wait until the preclear is finished Nd then upgrade.

 

As.i just finishes precleari g a 10tb and it take 4 days to run.

I'm preclearing all 4 drives at the same time, but from the looks of things, all 4 will complete at around the 46 - 48 hr mark after starting, so late tomorrow afternoon. Each major stage (pre-read, zero, post-read) takes about the same amount of time and the 1st stage completed in 16 hrs for all 4 drives. The zero stage is 85% complete at 13hrs so also looks like 16hrs to complete. Add another 16 hrs for the post-read and that's 48 hrs.

 

4 days is twice the time that mine usually take so it could be related to the SATA controller you're using. Regardless, I'm going to wait it out as I'd rather not waste the time already spent doing the preclears. In the past I've been able to reboot unRAID and the preclear was resumeable where it left off. I'm just not sure I want to trust that it'll work the same after all the recent updates.

 

I'm actually curious if the excessive CPU usage reported by the web GUI (and the poor disk I/O I'm seeing) might be that I'm preclearing all 4 drives at the same time. I haven't seen preclear slow down the system in the past, but I've previously only done 2 drives simultaneously.

 

Regardless, I'll wait now unless I can find a way to save the state of the preclear and ensure I can resume after a reboot.

 

EDIT: I've attached the diagnostics that I was able to get by using the 'diagnostics' command from a terminal shell.

 

Dale

 

 

 

animnas-diagnostics-20191209-2301.zip

Edited by AgentXXL
Attaching diagnostics file
Link to comment
This motherboard has a different Asmedia controller.

But it's still likely the problem, I would expect the Asmedia controller used on that board to support trim, but since the board is quite old it might be using an earlier revision, you can easily test by connecting the SSD to one of the Intel ports, and issue is definitely not UD related so you should post on the general support forum instead if more help is needed.  

 

 

 

 

Link to comment
5 hours ago, AgentXXL said:

I'm preclearing all 4 drives at the same time, but from the looks of things, all 4 will complete at around the 46 - 48 hr mark after starting, so late tomorrow afternoon. Each major stage (pre-read, zero, post-read) takes about the same amount of time and the 1st stage completed in 16 hrs for all 4 drives. The zero stage is 85% complete at 13hrs so also looks like 16hrs to complete. Add another 16 hrs for the post-read and that's 48 hrs.

 

4 days is twice the time that mine usually take so it could be related to the SATA controller you're using. Regardless, I'm going to wait it out as I'd rather not waste the time already spent doing the preclears. In the past I've been able to reboot unRAID and the preclear was resumeable where it left off. I'm just not sure I want to trust that it'll work the same after all the recent updates.

 

I'm actually curious if the excessive CPU usage reported by the web GUI (and the poor disk I/O I'm seeing) might be that I'm preclearing all 4 drives at the same time. I haven't seen preclear slow down the system in the past, but I've previously only done 2 drives simultaneously.

 

Regardless, I'll wait now unless I can find a way to save the state of the preclear and ensure I can resume after a reboot.

 

EDIT: I've attached the diagnostics that I was able to get by using the 'diagnostics' command from a terminal shell.

 

Dale

 

 

 

animnas-diagnostics-20191209-2301.zip 238.28 kB · 0 downloads

I'm not sure what is causing your high cpu usage, but you are remote mounting shares and when the remote server goes off-line, there are disk commands that will take longer and possibly time out.

 

In your log:

Dec  9 23:00:59 AnimNAS preclear_disk_4A4548303948504E[22584]: smartctl exec_time: 3s
Dec  9 23:01:10 AnimNAS preclear_disk_4A454B543957555A[28410]: smartctl exec_time: 3s
Dec  9 23:01:29 AnimNAS preclear_disk_4A4548303948504E[22584]: smartctl exec_time: 8s
Dec  9 23:01:40 AnimNAS preclear_disk_4A4548303948504E[22584]: smartctl exec_time: 19s
Dec  9 23:01:50 AnimNAS preclear_disk_4A4548303948504E[22584]: smartctl exec_time: 29s
Dec  9 23:02:00 AnimNAS preclear_disk_4A454B543957555A[28410]: smartctl exec_time: 9s
Dec  9 23:02:10 AnimNAS preclear_disk_4A4548303948504E[22584]: smartctl exec_time: 1s
Dec  9 23:02:11 AnimNAS preclear_disk_4A454B543957555A[28410]: smartctl exec_time: 20s
Dec  9 23:02:21 AnimNAS preclear_disk_4A454B543957555A[28410]: smartctl exec_time: 30s
Dec  9 23:02:32 AnimNAS preclear_disk_4A454B544233585A[2893]: smartctl exec_time: 10s
Dec  9 23:02:42 AnimNAS preclear_disk_4A454B544233585A[2893]: smartctl exec_time: 20s
Dec  9 23:02:52 AnimNAS preclear_disk_4A454B544233585A[2893]: smartctl exec_time: 30s
Dec  9 23:02:59 AnimNAS preclear_disk_4A454B55324D5A5A[17666]: smartctl exec_time: 6s
Dec  9 23:03:09 AnimNAS preclear_disk_4A454B55324D5A5A[17666]: smartctl exec_time: 16s
Dec  9 23:03:19 AnimNAS preclear_disk_4A454B55324D5A5A[17666]: smartctl exec_time: 26s

I think your expectation of what your Unraid server can/will do is too high.  Mounting UD disks, remote shares, and preclearing 4 disks at once is a bit much.  If you insist on preclearing disks (I'm one of those that believes it is not necessary and a waste of time), do it on another machine or stop mounting UD disks and remote shares until the preclears are done.

Link to comment

I just updated from unRAID 6.7.2 to 6.8.0 and discovered there's a nifty Plugin File Install Errors tab! Unfortunately, UD is currently listed there.

 

I'm pretty sure that I was running the latest release of UD (whatever that one was). I'd had to power off the system due to a weird WebUI 504 error. When it came back up (yesterday morning), I got all sorts of notifications that plugins and dockers were updating (there had been about 3 days of notifications that updates were available, but none were updating - probably related to the general weirdness that had been going on) so I'm mostly certain UD was updated to a release as of yesterday.

 

This is what I'm seeing now:

1593001546_2019-12-1219_52_15-NAS_Plugins-Brave.thumb.png.64023c9150217fdf13e954434631d8e0.png

 

Do I just go to the CA Apps "store" and reinstall it? Is there some trouble shooting that I can/should do first?

 

Link to comment
11 minutes ago, FreeMan said:

I just updated from unRAID 6.7.2 to 6.8.0 and discovered there's a nifty Plugin File Install Errors tab! Unfortunately, UD is currently listed there.

 

I'm pretty sure that I was running the latest release of UD (whatever that one was). I'd had to power off the system due to a weird WebUI 504 error. When it came back up (yesterday morning), I got all sorts of notifications that plugins and dockers were updating (there had been about 3 days of notifications that updates were available, but none were updating - probably related to the general weirdness that had been going on) so I'm mostly certain UD was updated to a release as of yesterday.

 

This is what I'm seeing now:

1593001546_2019-12-1219_52_15-NAS_Plugins-Brave.thumb.png.64023c9150217fdf13e954434631d8e0.png

 

Do I just go to the CA Apps "store" and reinstall it? Is there some trouble shooting that I can/should do first?

 

Post diagnostics.  Yes, you can re-install UD from CA.

Link to comment
11 minutes ago, FreeMan said:

I just updated from unRAID 6.7.2 to 6.8.0 and discovered there's a nifty Plugin File Install Errors tab! Unfortunately, UD is currently listed there.

 

I'm pretty sure that I was running the latest release of UD (whatever that one was). I'd had to power off the system due to a weird WebUI 504 error. When it came back up (yesterday morning), I got all sorts of notifications that plugins and dockers were updating (there had been about 3 days of notifications that updates were available, but none were updating - probably related to the general weirdness that had been going on) so I'm mostly certain UD was updated to a release as of yesterday.

 

This is what I'm seeing now:

1593001546_2019-12-1219_52_15-NAS_Plugins-Brave.thumb.png.64023c9150217fdf13e954434631d8e0.png

 

Do I just go to the CA Apps "store" and reinstall it? Is there some trouble shooting that I can/should do first?

 

Post diagnostics.  Yes, you can re-install UD from CA.

Link to comment

Reinstalled without issue. My SMB share to my backup server is there (even thought the backup server isn't plugged in at the moment).

 

It's like magic...

 

Thanks for continuing to support this and all the hard work you put in! If only every bit of software worked as well and were supported as well as unRAID and all its plugins and dockers are this would be a better world.

Link to comment
2 hours ago, J.Nerdy said:

@dlandon

 

Syslog is getting crushed with CIFS SMB verification error.  (Network mount via UD).  This started with upgrade to 6.8

 

The network mount is credentialed as it is a device attached to a Mac on the network.

 

Diag Attached.

 

952032422_ScreenShot2019-12-13at9_09_47AM.png.dca0ae01341f1f764e14931834d8f5dd.png

 

nerdyraid-diagnostics-20191213-0848.zip 150.42 kB · 0 downloads

This error is from SAMBA.  Start by un-mounting the remote share and see if the errors stop.  If they do then we know where the error is coming from.  I would suggest you check your credentials and be sure not to have special characters kn the password if you can avoid it.  Stick with alpha and numeric.

Link to comment
1 hour ago, dlandon said:

This error is from SAMBA.  Start by un-mounting the remote share and see if the errors stop.  If they do then we know where the error is coming from.  I would suggest you check your credentials and be sure not to have special characters kn the password if you can avoid it.  Stick with alpha and numeric.

I tried un-mounting and the error persists.  

 

I am going to un-mount, turn-off auto mount and reboot server to see if that makes a difference.  The credentials DO have special characters.

 

EDIT:  rebooting and removing auto mount killed the error.  Creating user with alphanumeric and will circle back

 

EDIT 2:  Created new user with long alphanumeric chain - mounted, accessible and no more error spamming.  Thanks @dlandon

 

EDIT 3:  As soon as docker service started, it began spamming again.  I wonder why?

Edited by J.Nerdy
Link to comment

I managed to get UD to hang, tried downloading diagnostics before I rebooted but it would never download the zip file. Previously errors were found on my seagate external drive that seemed to be the culprit and I am just hoping we can confirm that the drive is indeed bad and I need to get rid of it. I started to do a full smart scan of that drive yesterday and let it run. This morning UD was hung up. So I think that's probably the issue but wanted to be sure. 

tower-diagnostics-20191213-1658.zip

Link to comment
4 hours ago, dlandon said:

This error is from SAMBA.  Start by un-mounting the remote share and see if the errors stop.  If they do then we know where the error is coming from.  I would suggest you check your credentials and be sure not to have special characters kn the password if you can avoid it.  Stick with alpha and numeric.

@dlandon so it spamming even worse now.  I think it is related to a docker.  When you refer to credentials, you mean the user credentials for samba mount, correct?

 

EDIT:  so I have narrowed it down to this:  whenever the docker containers (Plex | CP | qbit) access the data on the remote mounted array (a thunderbay mounted on a Mac mini) it starts to spam the log.  

 

I created another user "unRAID" and the passphrase is alphanumeric without any spaces and use that to mount via UD.

 

Is it possible that the samba.conf is still registering the previous credentials as well?  They remain valid.  Also, it only started after upgrading to 6.8, so is it the new smb2 protocol?

Edited by J.Nerdy
Link to comment
On 12/8/2019 at 5:31 PM, Iormangund said:

I have some unassigned drives set in a btrfs pool mounted as a share. Has been working perfectly until I applied the recent updates at which point the drives will no longer auto mount or manual mount through the ui.

 

This is the log error I get when attempting to mount with plugin:

 

Server kernel: BTRFS error (device sdj1): open_ctree failed
Server unassigned.devices: Error: shell_exec(/sbin/mount -t btrfs -o auto,async,noatime,nodiratime '/dev/sdj1' '/mnt/disks/btrfspool' 2>&1) took longer than 10s!
Server unassigned.devices: Mount of '/dev/sdj1' failed. Error message: command timed out
Server unassigned.devices: Partition 'DISK' could not be mounted...


Mounting works as normal when done through terminal using commands:

mkdir /mnt/disks/btrfspool

/sbin/mount -t btrfs -o auto,async,noatime,nodiratime '/dev/sdj1' '/mnt/disks/btrfspool

 

I assume this is due to the changes made around update "2019.11.29a" where timeout was added?

Is it possible to change the timeout or do a check for btrfs pools and extend the timeout so auto mount works again?

 

Is there a fix that I can manually apply to get it working again the same way as before until an update comes out?

Hello, i think i have the same problem, i can't automount anymore after the last update:

 

Dec 14 00:13:08 NasDan kernel: BTRFS error (device sdi1): open_ctree failed
Dec 14 00:13:08 NasDan unassigned.devices: Error: shell_exec(/sbin/mount -t btrfs -o auto,async,noatime,nodiratime '/dev/sdi1' '/mnt/disks/WDC_WD40PURX-64NZ6Y0_WD-WCC7K0RJTU9R' 2>&1) took longer than 20s!
Dec 14 00:13:08 NasDan unassigned.devices: Mount of '/dev/sdi1' failed. Error message: command timed out
Dec 14 00:13:08 NasDan unassigned.devices: Partition 'WDC_WD40PURX-64NZ6Y0_WD-WCC7K0RJTU9R' could not be mounted...
Dec 14 00:13:08 NasDan unassigned.devices: Running device script: 'WDC_WD40PURX-64NZ6Y0_WD-WCC7K0RJTU9R.sh' with action 'ERROR_MOUNT'.

 

running "mount -t btrfs /dev/sdi1 /mnt/ciao" take almost 21 seconds and double for umount.

 

 

 

Link to comment
9 hours ago, J.Nerdy said:

As soon as docker service started, it began spamming again.  I wonder why?

There is not a lot of information on this error, but what I did find is that the error occurs when the remote device is written to.  If your Dockers are writing to the remove mount, then that's when you'd see the errors.  It was suggested that the permissions on the remote source (your Mac) might have to be changed.  I don't know what needs changing.

 

I'm sure this started on 6.8 because there is a new version of SAMBA.

Link to comment
7 hours ago, geonerdist said:

I managed to get UD to hang, tried downloading diagnostics before I rebooted but it would never download the zip file.

Sometimes the command line works when the gui doesn't.  In a terminal session enter 'diangostics'.  The diagnostics will be written to /flash/logs/

7 hours ago, geonerdist said:

Previously errors were found on my seagate external drive that seemed to be the culprit and I am just hoping we can confirm that the drive is indeed bad and I need to get rid of it.

The disk has issues, but looks to be more of a cable/controller issue.

SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)

This shows the disk can do 6.0 Gb/s, but is only running at 3.0 Gb/s.  If the controller is capable of 6.0 Gb/s, then the higher speed should be used.  This might also indicate a cable problem.

 

I'm not a disk expert, but I don't think the Seagate disk is failing.

Link to comment
7 hours ago, J.Nerdy said:

Is it possible that the samba.conf is still registering the previous credentials as well?

No.  The credentials are set when the device is first mounted.  If they are incorrect, the CIFS mount should fail.

 

I have an idea here.  I don't know how the Mac implements SMB.  Is it possible that you have the credentials set wrong and the Mac share mounts regardless and then fails when the share is accessed?

 

Remove the remote share from UD and then set it up again.  Be sure the credentials are set correctly.

7 hours ago, J.Nerdy said:

Also, it only started after upgrading to 6.8, so is it the new smb2 protocol?

I thought the latest SAMBA only had security fixes, but it's possible other changes were made.

 

SMB2 is not a new protocol.

Link to comment
2 hours ago, dany-dm said:

Hello, i think i have the same problem, i can't automount anymore after the last update:

 


Dec 14 00:13:08 NasDan kernel: BTRFS error (device sdi1): open_ctree failed
Dec 14 00:13:08 NasDan unassigned.devices: Error: shell_exec(/sbin/mount -t btrfs -o auto,async,noatime,nodiratime '/dev/sdi1' '/mnt/disks/WDC_WD40PURX-64NZ6Y0_WD-WCC7K0RJTU9R' 2>&1) took longer than 20s!
Dec 14 00:13:08 NasDan unassigned.devices: Mount of '/dev/sdi1' failed. Error message: command timed out
Dec 14 00:13:08 NasDan unassigned.devices: Partition 'WDC_WD40PURX-64NZ6Y0_WD-WCC7K0RJTU9R' could not be mounted...
Dec 14 00:13:08 NasDan unassigned.devices: Running device script: 'WDC_WD40PURX-64NZ6Y0_WD-WCC7K0RJTU9R.sh' with action 'ERROR_MOUNT'.

 

running "mount -t btrfs /dev/sdi1 /mnt/ciao" take almost 21 seconds and double for umount.

Looks like more time out adjustments are needed.

  • Thanks 1
Link to comment
  • trurl pinned this topic

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.