Unassigned Devices - Managing Disk Drives and Remote Shares Outside of The Unraid Array

[Arrior]

SMB Shares will not load automatically at boot

Every time I reboot my unraid server my SMB shares do not mount. The boxes are greyed out. The only way I have figured out to be able to turn the boxes orange is to act like I'm going to add an additional SMB share. I then browse for the share. Once I browse for the share all the boxes turn orange and I can mount.

 

Thank you so much in advance for your help.

alpha-diagnostics-20221129-1115.zip

[dlandon]

4 hours ago, Arrior said:

SMB Shares will not load automatically at boot

Every time I reboot my unraid server my SMB shares do not mount. The boxes are greyed out. The only way I have figured out to be able to turn the boxes orange is to act like I'm going to add an additional SMB share. I then browse for the share. Once I browse for the share all the boxes turn orange and I can mount.

 

Thank you so much in advance for your help.

alpha-diagnostics-20221129-1115.zip 135.3 kB · 0 downloads

UD does a check for the network to be up and running before it tries to mount the remote shares.  What's happening is your network is coming up just fine, but the remote server is not responding to a ping at that point.  When a remote server does not respond to a ping, UD thinks it's off-line and grays out the "Mount" button.  If you were to wait for a bit, the "Mount" buttons should turn Orange.  The ping poll is every 15 seconds.

 

What is the remote server?  Another Unraid server?

 

I have an idea, but have no way to test to see if it would fix your situation.  I want you to try a few things to help me decide:

• Stop the array and then restart it and see if the remote shares auto mount.  If not, wait at the UD page for a few minutes and see if the "Mount" buttons turn Orange on their own.
• Reboot the server and wait on the UD page for a few minutes and see if the 'Mount" buttons turn Orange on their own.

[Arrior]

Thanks so much!  So, when I stopped and restarted the array the shares auto mounted.

 

When I reboot the server they stay greyed out.

 

Really appreciate the help!

[dlandon]

5 minutes ago, Arrior said:

When I reboot the server they stay greyed out.

After a few minutes do they turn Orange?

[Arrior]

Its been over 10 min and not turning orange. Thanks!

[dlandon]

2 hours ago, Arrior said:

Its been over 10 min and not turning orange. Thanks!

It's like the polling in the background has stopped.

 

What happens when you click on the double arrows in the upper right of the UD page after a reboot?  The "Mount" buttons should go gray for a short period and then turn Orange.

[Arrior]

21 minutes ago, dlandon said:

It's like the polling in the background has stopped.

 

What happens when you click on the double arrows in the upper right of the UD page after a reboot?  The "Mount" buttons should go gray for a short period and then turn Orange.

Nothing changes.  They stay grey,  Thanks!

[Sprort]

I'm having an issue with UD not working anymore RE: SMB/NFS. I had an SMB share for years and randomly it disappeared. The share was pointed to a NAS on my local network, which is still working perfectly fine (no issues access it at all from any other device). Now, UD will let me go through the steps of adding an SMB share, but after it says Success, nothing happens. I also seem to have a persistent mount under /mnt/remotes/ but it is inaccessible and shows an incorrect amount of directories. I have uninstalled, rebooted, reinstalled, updated the OS and repeated. I'm at my wits end, any help would be greatly appreciated.

[dlandon]

Post diagnostics.

[Sprort]

6 hours ago, dlandon said:

Post diagnostics.

 

diagnostics.zip

[dlandon]

10 hours ago, Arrior said:

Nothing changes.  They stay grey,  Thanks!

That is a problem.  The ping status should be updating and it doesn't seem to be working.  I would like to get to the bottom of this issue.

 

What browser are you using?

 

Reboot and then run this command in a terminal while the "Mount" buttons are gray and show any output:

/usr/local/emhttp/plugins/unassigned.devices/scripts/get_ud_stats ping

This will tell me if the ping status update is running OK.  This should turn the "Mount" buttons Orange.  Let me know if it doesn't.

 

When you're on the UD page, Hold the crtl and shift keys, and press the 'j' key.  You'll see a dialog pop up.  Let me know if you see any errors.

[dlandon]

3 hours ago, Sprort said:

 

diagnostics.zip 85.33 kB · 0 downloads

Remove the NerdPack plugin and reboot.  NerdPack is not compatible with 6.11.

[insomniak]

Having an issue where the whole server times out and becomes unresponsive. Narrowed it down to Unassigned Devices since I uninstalled all plugins and reinstalled one at a time. When UD is installed and disks are not mounted yet everything is ok. However when I go to mount a drive, the UI is stuck in "mounting" on that one drive i initiated and the server becomes unresponsive. Cannot SSH in either. I had logs opened and was able to capture the last screenshot.

 

I notice it says that "dev8" and "dev3" are being used so theirs duplicates in UD section. However after I rebooted, there were no more duplicate ID's and trying to mount the drive still lead to the same unresponsiveness. 

 

I've attached diagnostics for reference too. This only started happening in the past week so any help is appreciated. 

afterdark-diagnostics-20221130-0734.zip

[Arrior]

3 hours ago, dlandon said:

That is a problem.  The ping status should be updating and it doesn't seem to be working.  I would like to get to the bottom of this issue.

 

What browser are you using?

Firefox

Reboot and then run this command in a terminal while the "Mount" buttons are gray and show any output:

/usr/local/emhttp/plugins/unassigned.devices/scripts/get_ud_stats ping

This will tell me if the ping status update is running OK.  This should turn the "Mount" buttons Orange.  Let me know if it doesn't.

 

When you're on the UD page, Hold the crtl and shift keys, and press the 'j' key.  You'll see a dialog pop up.  Let me know if you see any errors.

Using FireFox.

 

So, I entered it in terminal and nothing happened in the terminal just got a new line. IT did not mount the shares. When I did crtl + Shift J I got:

Key event not available on some keyboard layouts: key=“i” modifiers=“accel,alt,shift” id=“key_browserToolbox”

Edited November 30, 2022 by Arrior

[dlandon]

7 minutes ago, Arrior said:

So, I entered it in terminal and nothing happened in the terminal just got a new line. IT did not mount the shares. When I did crtl + Shift J I got:

Key event not available on some keyboard layouts: key=“i” modifiers=“accel,alt,shift” id=“key_browserToolbox”

What browser are you using?

[JorgeB]

32 minutes ago, insomniak said:

I had logs opened and was able to capture the last screenshot.

A permanent syslog would be better, but if 'start tree-log replay' is the last line displayed it suggests a problem with the tree-log, if that's the only problem this might fix it:

 

btrfs rescue zero-log /dev/nvme1n1p1

 

If you rebooted since check that the device is still nvme1n1, then try mounting again.

 

 

[insomniak]

59 minutes ago, JorgeB said:

A permanent syslog would be better, but if 'start tree-log replay' is the last line displayed it suggests a problem with the tree-log, if that's the only problem this might fix it:

 

btrfs rescue zero-log /dev/nvme1n1p1

 

If you rebooted since check that the device is still nvme1n1, then try mounting again.

 

 

I attached the syslog.

 

After checking, the device is still nvme1n1. I'll try the suggestion above if that's the case but hoping the syslog shows something too. 

syslog-192.168.164.2.log

[JorgeB]

9 minutes ago, insomniak said:

I'll try the suggestion above if that's the case but hoping the syslog shows something too. 

It does, it looks like the problem it the Intel 10GbE NIC, not the btrfs filesystem:

 

Nov 30 06:15:47 afterdark kernel: BTRFS info (device nvme1n1p1): start tree-log replay
Nov 30 06:16:17 afterdark kernel: ------------[ cut here ]------------
Nov 30 06:16:17 afterdark kernel: NETDEV WATCHDOG: eth0 (ixgbe): transmit queue 18 timed out

 

Maybe that NVMe device having some conflict with the NIC, can you try without the NIC or with disabled?

[insomniak]

12 minutes ago, JorgeB said:

It does, it looks like the problem it the Intel 10GbE NIC, not the btrfs filesystem:

 

Nov 30 06:15:47 afterdark kernel: BTRFS info (device nvme1n1p1): start tree-log replay
Nov 30 06:16:17 afterdark kernel: ------------[ cut here ]------------
Nov 30 06:16:17 afterdark kernel: NETDEV WATCHDOG: eth0 (ixgbe): transmit queue 18 timed out

 

Maybe that NVMe device having some conflict with the NIC, can you try without the NIC or with disabled?

The NIC is the onboard motherboard one so I'll try to disable I'll give that a try this evening once I'm home. In the mean time, further down in the logs starting at 7:44:01, I had tried to mount the NVME again after a server reboot and it didn't note the same type of log display about the NIC. Main thing I saw was CPU 10 stalls so not sure if that plays into it at all either. 

 

 

Nov 30 07:44:01 afterdark kernel: BTRFS info (device nvme1n1p1): start tree-log replay
Nov 30 07:44:12 afterdark flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Nov 30 07:44:30 afterdark kernel: rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 10-... } 21156 jiffies s: 3181 root: 0x1/.
Nov 30 07:44:30 afterdark kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x400/.
Nov 30 07:44:30 afterdark kernel: Task dump for CPU 10:
Nov 30 07:44:30 afterdark kernel: task:kworker/u64:4   state:R  running task     stack:    0 pid:12618 ppid:     2 flags:0x00004008
Nov 30 07:44:30 afterdark kernel: Workqueue: btrfs-cache btrfs_work_helper
Nov 30 07:44:30 afterdark kernel: Call Trace:
Nov 30 07:44:30 afterdark kernel: <TASK>
Nov 30 07:44:30 afterdark kernel: ? __schedule+0x59e/0x5f6
Nov 30 07:44:30 afterdark kernel: ? _raw_spin_unlock+0x14/0x29
Nov 30 07:44:30 afterdark kernel: ? chksum_update+0x13/0x1e
Nov 30 07:44:30 afterdark kernel: ? crc32c+0x2f/0x62
Nov 30 07:44:30 afterdark kernel: ? folio_wait_bit_common+0x1ce/0x241
Nov 30 07:44:30 afterdark kernel: ? ___slab_alloc+0x288/0x590
Nov 30 07:44:30 afterdark kernel: ? __load_free_space_cache+0x215/0x452
Nov 30 07:44:30 afterdark kernel: ? folio_mkclean+0x5c/0xa1
Nov 30 07:44:30 afterdark kernel: ? page_vma_mkclean_one.constprop.0+0x138/0x138
Nov 30 07:44:30 afterdark kernel: ? virt_to_slab+0x5/0x19
Nov 30 07:44:30 afterdark kernel: ? memcg_slab_free_hook+0x4b/0xf9
Nov 30 07:44:30 afterdark kernel: ? sysvec_apic_timer_interrupt+0x92/0xa6
Nov 30 07:44:30 afterdark kernel: ? asm_sysvec_apic_timer_interrupt+0x16/0x20
Nov 30 07:44:30 afterdark kernel: ? native_queued_spin_lock_slowpath+0x81/0x1d0
Nov 30 07:44:30 afterdark kernel: ? do_raw_spin_lock+0x14/0x1a
Nov 30 07:44:30 afterdark kernel: ? __btrfs_remove_free_space_cache+0xe/0x2d
Nov 30 07:44:30 afterdark kernel: ? load_free_space_cache+0x230/0x2dd
Nov 30 07:44:30 afterdark kernel: ? caching_thread+0x7e/0x43d
Nov 30 07:44:30 afterdark kernel: ? move_linked_works+0x2f/0x6a
Nov 30 07:44:30 afterdark kernel: ? pwq_adjust_max_active+0x88/0xa7
Nov 30 07:44:30 afterdark kernel: ? btrfs_work_helper+0x114/0x2a5
Nov 30 07:44:30 afterdark kernel: ? process_one_work+0x1ab/0x295
Nov 30 07:44:30 afterdark kernel: ? worker_thread+0x18b/0x244
Nov 30 07:44:30 afterdark kernel: ? rescuer_thread+0x281/0x281
Nov 30 07:44:30 afterdark kernel: ? kthread+0xe7/0xef
Nov 30 07:44:30 afterdark kernel: ? kthread_complete_and_exit+0x1b/0x1b
Nov 30 07:44:30 afterdark kernel: ? ret_from_fork+0x22/0x30
Nov 30 07:44:30 afterdark kernel: </TASK>
Nov 30 07:45:02 afterdark kernel: rcu: INFO: rcu_preempt self-detected stall on CPU
Nov 30 07:45:02 afterdark kernel: rcu: #01110-....: (60000 ticks this GP) idle=b59/1/0x4000000000000000 softirq=52533/52533 fqs=12875 
Nov 30 07:45:02 afterdark kernel: #011(t=60001 jiffies g=89909 q=938305 ncpus=32)
Nov 30 07:45:02 afterdark kernel: NMI backtrace for cpu 10

 

 

[JorgeB]

4 minutes ago, insomniak said:

Main thing I saw was CPU 10 stalls so not sure if that plays into it at all either. 

It's strange, another option to try and troubleshoot would be to swap that NVMe device with a different one and see if the issue follow the device or the slot.

[Arrior]

1 hour ago, dlandon said:

What browser are you using?

Firefox but have used Chrome as well.

[dlandon]

2 minutes ago, Arrior said:

Firefox but have used Chrome as well.

A recent update may have changed things.  Check in another browser and see if the issue persists.

[dlandon]

3 hours ago, Arrior said:

So, I entered it in terminal and nothing happened in the terminal just got a new line. IT did not mount the shares.

That command does not mount the shares.  All it does is ping all remote shares and updates the "online" status.  The grayed "Mount" buttons should then turn Orange on the next update of the UD page.  If it doesn't update on its own, a manual refresh would enable the "Mount" buttons.

 

If it just terminated without any errors, then that seems to be working.

 

I'm suspecting the UD page refresh.  That's why I'm asking about the browser you are using.

[Arrior]

1 hour ago, dlandon said:

A recent update may have changed things.  Check in another browser and see if the issue persists.

Tried Chrome. No change. I'm confused in terms of what recently changed? Unraid, the plug in? Sorry, not sure if there is something else you want me to do. They are still greyed out.

[dlandon]

Just now, Arrior said:

Tried Chrome. No change. I'm confused in terms of what recently changed? Unraid, the plug in? Sorry, not sure if there is something else you want me to do. They are still greyed out.

Nothing that I know of changed to cause this.  I'm don't see the issue on either of my servers.

 

Manually refresh the UD page.  Any change?  Does the "Main" page with the array disks refresh and change the stats?  It should refresh every 3 seconds.

 

At this point, the best thing to do is for me to Team Viewer into your server so I can have a look.  If you're interested, PM me and we can make arrangements.