Unassigned Devices - Managing Disk Drives and Remote Shares Outside of The Unraid Array

[Lev]

33 minutes ago, dlandon said:

@Lev I have an idea.  Can you show me the results of the command 'ls /dev/disk/by-id'?

 

EDIT: With the scsi plugin installed.

 

Here's the output of NOT Installed (left) and Installed (right) and the lines that are different.

 

 

There are two things going on in this, I'll start with the most straight forward.

1. Lines 15-18 on the left are lines 23-26 on the right, but out of order. These are four unassigned Seagate SAS disks using JBOD pass-through on a LSI 3108 RAID controller. Below is a table I created showing just these four drives in aligned order to make it less confusing.
   
   
   
   Best to focus on any of these four drives, but I'll explain what other more complicated thing going on is, because it may give you a clue, or it may just confuse you. In that case just ignore what I'm about to say.
    
2. Lines 19-26 on the left are lines 15-22 on the right, but out of order. These are four logical RAID disks on the same LSI 3108 RAID Controller. All four of these disks are 'Assigned' successfully to UnRAID on the left (plugin Not Installed) but on the right are 'UnAssigned' (plug Installed) because it no longer matches what value UnRAID has stored to flag the disk as 'Assigned'. However as you can see UnRAID is able to detect all the SAS and RAID drives as they are visible in the drop down selection list and makes no difference whether the plug-in is installed or not.
   
   screenshot below with dynamix.scsi.device plugin NOT installed. Notice UnRAID assignments and detection is working. UD detection is working.

 

screenshot below with dynamix.scsi.device plugin INSTALLED... Notice UnRAID no longer sees assigned SAS & RAID disks, however detection of those disks is working. UD detection is NOT working.

 

 

 

[dlandon]

@Lev Are you showing the complete 'ls' output?  It appears something is missing.

[dlandon]

1 hour ago, cluengas said:

Unfortunately the new short path did not work everywhere.

It looks like you are using a default mount point - /mnt/disks/S3ESNX0K197057Y.  Why don't you use a mount point that makes more sense than a cryptic number and set it in UD so it won't change?  Click on the partition mount point to change it.

 

I have no idea about your docker container issues.

[Lev]

1 hour ago, dlandon said:

@Lev Are you showing the complete 'ls' output?  It appears something is missing.

 

yes. I just ran again, same output confirmed.

 

ls /dev/disk/by-id

 

[RealActorRob]

On 1/3/2022 at 5:28 PM, dlandon said:

Those are stored on the flash drive in the 'preclear_reports' folder.

 

If you still have issues, post a screen shot showing all the disk info.

All I had to do was click the red 'X' by preclear on the Main tab and FORMAT is now orange.

 

Not using it just yet, but looks good to go if/when needed.

[Lev]

@dlandon

TL:DR

Two problematic inconsistencies I've found with the naming convention for disk/by-id/scsi-*
 

Either one or a mix of both maybe the root cause, please check. Supporting research is underneath in how I concluded this.
 

1. If the _ underscore is being used as a delimiter in UD functions, it does not have strong consistency with SATA and with SCSI there is no consistency in its use in the naming convention for disk/by-id
    
2. My guess is UD is unaware that the plugin once installed is altering the naming convention of disk/by-id/ specifically for SCSI devices, replacing the expected 'ID_SERIAL' with 'ID_SCSI_SERIAL' as this key is only encountered and present with SCSI disks, not ATA. See UDEVADM output below.
    
3. UnRAID detection in the drop down list is consistent in naming convention 'ID_MODEL'(not Unique)_'ID_SCSI_SERIAL'(Unique!) in both situations, plugin installed or NOT installed, my guess is the UnRAID detection function may have a conditional IF 'ID_BUS' = 'scsi' then use  'ID_SCSI_SERIAL' instead of 'ID_SERIAL' that aligns with how the plugin also uses 'ID_SCSI_SERIAL'for only SCSI disks.

 

 

Supporting Research

It was not easy to map the two different ID's for the disk that were generated with the plugin installed and with it removed to make this simple table.

 

 

Why? lsscsi and all it's various command options will not output it, it's UDEVADM and some logic code.

 

Here's what's interesting, and why I'm sharing this... notice the naming convention naming convention  is not the same between the SATA drives and the SAS/SCSI drives. In multiple ways.

 

Pay close attention to the underscores between the keys, I think they are the delimiter and this maybe where some function is expecting one or two delimiters to be present to return the expected values. This inconsistency in naming convention looks like something that could cause some problems like what we are seeing.

 

SATA disk

disk/by-id/ata-WDC_WD140EDGZ-11B2DA2_3HG65VTN

Naming convention: 'ID_BUS'(not Unique)-'ID_SERIAL'(Unique!)

 

root@tower2:~# udevadm info --query=all --name=/dev/sdg
P: /devices/pci0000:00/0000:00:01.0/0000:01:00.0/host1/target1:0:52/1:0:52:0/block/sdg
N: sdg
S: disk/by-id/ata-WDC_WD140EDGZ-11B2DA2_3HG65VTN
S: disk/by-id/wwn-0x5000cca29dc2d03d
S: disk/by-path/pci-0000:01:00.0-scsi-0:0:52:0
E: DEVLINKS=/dev/disk/by-id/ata-WDC_WD140EDGZ-11B2DA2_3HG65VTN /dev/disk/by-id/wwn-0x5000cca29dc2d03d /dev/disk/by-path/pci-0000:01:00.0-scsi-0:0:52:0
E: DEVNAME=/dev/sdg
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/host1/target1:0:52/1:0:52:0/block/sdg
E: DEVTYPE=disk
E: ID_ATA=1
E: ID_ATA_DOWNLOAD_MICROCODE=1
E: ID_ATA_FEATURE_SET_APM=1
E: ID_ATA_FEATURE_SET_APM_ENABLED=0
E: ID_ATA_FEATURE_SET_HPA=1
E: ID_ATA_FEATURE_SET_HPA_ENABLED=1
E: ID_ATA_FEATURE_SET_PM=1
E: ID_ATA_FEATURE_SET_PM_ENABLED=1
E: ID_ATA_FEATURE_SET_PUIS=1
E: ID_ATA_FEATURE_SET_PUIS_ENABLED=0
E: ID_ATA_FEATURE_SET_SECURITY=1
E: ID_ATA_FEATURE_SET_SECURITY_ENABLED=0
E: ID_ATA_FEATURE_SET_SECURITY_ENHANCED_ERASE_UNIT_MIN=65538
E: ID_ATA_FEATURE_SET_SECURITY_ERASE_UNIT_MIN=66774
E: ID_ATA_FEATURE_SET_SMART=1
E: ID_ATA_FEATURE_SET_SMART_ENABLED=1
E: ID_ATA_ROTATION_RATE_RPM=7200
E: ID_ATA_SATA=1
E: ID_ATA_SATA_SIGNAL_RATE_GEN1=1
E: ID_ATA_SATA_SIGNAL_RATE_GEN2=1
E: ID_ATA_WRITE_CACHE=1
E: ID_ATA_WRITE_CACHE_ENABLED=1
E: ID_BUS=ata
E: ID_MODEL=WDC_WD140EDGZ-11B2DA2
E: ID_MODEL_ENC=WDC\x20WD140EDGZ-11B2DA2\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
E: ID_PART_TABLE_TYPE=gpt
E: ID_PART_TABLE_UUID=b7673fa9-3326-4df7-a432-ef769fc82c7e
E: ID_PATH=pci-0000:01:00.0-scsi-0:0:52:0
E: ID_PATH_TAG=pci-0000_01_00_0-scsi-0_0_52_0
E: ID_REVISION=85.00A85
E: ID_SERIAL=WDC_WD140EDGZ-11B2DA2_3HG65VTN
E: ID_SERIAL_SHORT=3HG65VTN
E: ID_TYPE=disk
E: ID_WWN=0x5000cca29dc2d03d
E: ID_WWN_WITH_EXTENSION=0x5000cca29dc2d03d
E: MAJOR=8
E: MINOR=96
E: SUBSYSTEM=block
E: USEC_INITIALIZED=33560058

 

SAS/SCSI disk (Plugin NOT installed)

Full Identifier: ST12000NM002G_ZLW0QFJV0000C0438211_35000c500ca3d8cbf

Naming convention: 'ID_BUS'(not Unique)-'ID_MODEL'(not Unique)_'ID_SCSI_SERIAL'(Unique!)'_'ID_SERIAL'(Unique!)

 

disk/by-id/scsi-35000c500ca3d8cbf (there's not a single _ delimiter)

Naming convention: 'ID_BUS'(not Unique)-'ID_SERIAL'(Unique!

 

SAS/SCSI disk (Plugin INSTALLED)

Full Identifier: ST12000NM002G_ZLW0QFJV0000C0438211_35000c500ca3d8cbf

Naming convention: 'ID_BUS'(not Unique)-'ID_MODEL'(not Unique)_'ID_SCSI_SERIAL'(Unique!)_'ID_SERIAL'(Unique!)

 

disk/by-id/scsi-ST12000NM002G_ZLW0QFJV0000C0438211 (there's one _ delimiter)

'ID_BUS'(not Unique)-'ID_MODEL'(not Unique)_'ID_SCSI_SERIAL'(Unique!)

 

A few things jump out at me...
 

disk/by-id/ naming convention is changed by the plugin to use 'ID_SCSI_SERIAL' instead of 'ID_SERIAL'
 

On a SCSI disk, 'ID_SCSI_SERIAL' is strictly the SCSI serial number integer data type,  'ID_SERIAL' is a data type which appears to a data conversion of the hexidecmal WWN of the disk. See UDEVADM output below for SCSI SAS and then scroll back up to compare against ATA. It's a mess.

 

Comparing the two UDEVADM outputs you'll notice the ATA device where 'ID_SERIAL'  returns a long string data type, with _ underscores as delimiters for 2 other data keys, 'ID_MODEL'_'ID_SERIAL_SHORT' Also 'ID_MODEL' WDC contains a _ delimiter with value returned to complicate things further. Ugh.
 

root@tower2:~# udevadm info --query=all --name=/dev/sdk
P: /devices/pci0000:00/0000:00:01.0/0000:01:00.0/host1/target1:0:62/1:0:62:0/block/sdk
N: sdk
S: disk/by-id/scsi-35000c500ca3d8cbf
S: disk/by-id/wwn-0x5000c500ca3d8cbf
S: disk/by-path/pci-0000:01:00.0-scsi-0:0:62:0
E: DEVLINKS=/dev/disk/by-id/scsi-35000c500ca3d8cbf /dev/disk/by-id/wwn-0x5000c500ca3d8cbf /dev/disk/by-path/pci-0000:01:00.0-scsi-0:0:62:0
E: DEVNAME=/dev/sdk
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/host1/target1:0:62/1:0:62:0/block/sdk
E: DEVTYPE=disk
E: ID_BUS=scsi
E: ID_MODEL=ST12000NM002G
E: ID_MODEL_ENC=ST12000NM002G\x20\x20\x20
E: ID_PATH=pci-0000:01:00.0-scsi-0:0:62:0
E: ID_PATH_TAG=pci-0000_01_00_0-scsi-0_0_62_0
E: ID_REVISION=E003
E: ID_SCSI=1
E: ID_SCSI_SERIAL=ZLW0QFJV0000C0438211
E: ID_SERIAL=35000c500ca3d8cbf
E: ID_SERIAL_SHORT=5000c500ca3d8cbf
E: ID_TYPE=disk
E: ID_VENDOR=SEAGATE
E: ID_VENDOR_ENC=SEAGATE\x20
E: ID_WWN=0x5000c500ca3d8cbf
E: ID_WWN_WITH_EXTENSION=0x5000c500ca3d8cbf
E: MAJOR=8
E: MINOR=160
E: SUBSYSTEM=block
E: USEC_INITIALIZED=33453336

UnRAID detection in the drop down list is consistent in naming convention 'ID_MODEL'(not Unique)_'ID_SCSI_SERIAL'(Unique!) in both situations, see screenshot below, does not matter if plugin installed or NOT installed, it appears as UnRAID has the condition IF 'ID_BUS' = 'scsi' then use  'ID_SCSI_SERIAL' instead of 'ID_SERIAL' is my guess for its detection function.

 

 

 

Edited January 5, 2022 by Lev
fixed case for data keys to be consistent in using all CAPS as they are defined in UDEVADM

[adamf]

8 hours ago, dlandon said:

Your log is so full of these messages, I can't find anything about UD mounting a disk:

Jan  4 16:35:38 Tower kernel: eth0: renamed from vetheb13b06
Jan  4 16:35:38 Tower kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth33e51aa: link becomes ready
Jan  4 16:35:38 Tower kernel: docker0: port 1(veth33e51aa) entered blocking state
Jan  4 16:35:38 Tower kernel: docker0: port 1(veth33e51aa) entered forwarding state
Jan  4 16:35:38 Tower kernel: docker0: port 1(veth33e51aa) entered disabled state
Jan  4 16:35:38 Tower kernel: vetheb13b06: renamed from eth0
Jan  4 16:35:38 Tower kernel: docker0: port 1(veth33e51aa) entered disabled state
Jan  4 16:35:38 Tower kernel: device veth33e51aa left promiscuous mode
Jan  4 16:35:38 Tower kernel: docker0: port 1(veth33e51aa) entered disabled state
Jan  4 16:36:38 Tower kernel: docker0: port 1(veth447cdf3) entered blocking state
Jan  4 16:36:38 Tower kernel: docker0: port 1(veth447cdf3) entered disabled state
Jan  4 16:36:38 Tower kernel: device veth447cdf3 entered promiscuous mode
Jan  4 16:36:38 Tower kernel: docker0: port 1(veth447cdf3) entered blocking state
Jan  4 16:36:38 Tower kernel: docker0: port 1(veth447cdf3) entered forwarding state
Jan  4 16:36:38 Tower kernel: docker0: port 1(veth447cdf3) entered disabled state
Jan  4 16:36:38 Tower kernel: eth0: renamed from veth430d7c6

These happen about every minute and swamp the log.

 

You'll need to clear this up.  You also need to update to the latest version of UD.

Not sure what those are. The only dockers i have running are cloudflared and jellyfin... Any ideas on what that might be?

I am not using IPv6 (at least to my knowledge), so I wonder what that message is.... 

-Adam

Edited January 5, 2022 by adamf

[austriancoder]

Hi all

 

I am using the UD plugin to share some NFS root filesystems and ran into a little problem. I need to edit /etc/exports by hand to get NFS root working.

 

"/mnt/disks/testing-nfs-root" -async,no_subtree_check,fsid=200 *(sec=sys,rw,insecure,anongid=100,anonuid=99,all_squash)

 

to

 

"/mnt/disks/testing-nfs-root" -async,no_subtree_check,fsid=200 *(sec=sys,rw,insecure,anongid=100,anonuid=99,no_root_squash)

 

Is there a way to do this in an automatic way?

 

 

Thanks

[dlandon]

6 hours ago, adamf said:

Not sure what those are. The only dockers i have running are cloudflared and jellyfin... Any ideas on what that might be?

I am not using IPv6 (at least to my knowledge), so I wonder what that message is.... 

-Adam

One of the networking experts will have to jump in here to help.

[dlandon]

1 hour ago, austriancoder said:

Hi all

 

I am using the UD plugin to share some NFS root filesystems and ran into a little problem. I need to edit /etc/exports by hand to get NFS root working.

 

"/mnt/disks/testing-nfs-root" -async,no_subtree_check,fsid=200 *(sec=sys,rw,insecure,anongid=100,anonuid=99,all_squash)

 

to

 

"/mnt/disks/testing-nfs-root" -async,no_subtree_check,fsid=200 *(sec=sys,rw,insecure,anongid=100,anonuid=99,no_root_squash)

 

Is there a way to do this in an automatic way?

 

 

Thanks

Why do you need to make that change?

 

Post your diagnostics so I can investigate further.  I need to see what version of Unraid and NFS you are using, plus some other details.

[austriancoder]

13 minutes ago, dlandon said:

Why do you need to make that change?

 

Post your diagnostics so I can investigate further.  I need to see what version of Unraid and NFS you are using, plus some other details.

 

This change is needed for disk-less Linux boot. I have a small board farm where I run some GPU driver conformance tests for mesa3d. I spoke at XDC2021 about my CI work - there is even a nice/outdated picture of my setup in the presentation. Here is a nice and easy guide about diskless network booting.

 

I am not sure what the ideal solution could look like. The simplest could a be script that can be called after /etc/exports was modified by UD.

 

With the unmodified /etc/exports line this is how the NFS boot looks like. The problematic lines are these two:

 

22-01-05 13:17:19 R SERIAL> + mount -t tmpfs tmpfs /tmp
22-01-05 13:17:19 R SERIAL> /init-stage1.sh: 17: cannot create /etc/resolv.conf: Permission denied

 

After this the boot hangs (as expected). With the modified line the device boots nicely and runs the unit tests.

 

tower-diagnostics-20220105-1407.zip

[dlandon]

21 minutes ago, austriancoder said:

I am not sure what the ideal solution could look like.

Go to the UD settings and set 'NFS Security' to 'Private'.  Then enter this line:

*(sec=sys,rw,insecure,anongid=100,anonuid=99,no_root_squash)

in the 'Rules'.

[Squid]

7 hours ago, adamf said:

Not sure what those are. The only dockers i have running are cloudflared and jellyfin... Any ideas on what that might be?

I am not using IPv6 (at least to my knowledge), so I wonder what that message is.... 

-Adam

Those messages happen when a container starts (and stops).  Is either of those (or any other) continually restarting due to a problem with them?

[austriancoder]

2 hours ago, dlandon said:

Go to the UD settings and set 'NFS Security' to 'Private'.  Then enter this line:

*(sec=sys,rw,insecure,anongid=100,anonuid=99,no_root_squash)

in the 'Rules'.

 

Works \o/

 

Should I have found this by reading the docs etc.? Btw .. I know dumb question .. where can I find some sort of documentation?

 

[dlandon]

10 minutes ago, austriancoder said:

Should I have found this by reading the docs etc.? Btw .. I know dumb question .. where can I find some sort of documentation?

Not a dumb question.  When on the UD or settings pages, click the 'Help' and you'll see some documentation.  The other place is the first and second posts of this very long thread.

[adamf]

13 hours ago, adamf said:

Not sure what those are. The only dockers i have running are cloudflared and jellyfin... Any ideas on what that might be?

I am not using IPv6 (at least to my knowledge), so I wonder what that message is.... 

-Adam

Ok. removed Cloudflared which seemed to be causing all he log crazy stuff... here are my diagnostics agin.

 

-Adam

tower-diagnostics-20220105-1456.zip

[dlandon]

1 hour ago, adamf said:

Ok. removed Cloudflared which seemed to be causing all he log crazy stuff... here are my diagnostics agin.

Now we can get down to the UD disk issue.  Please unmount and then remount the disk you were having issues with and post the diagnostics again.

[adamf]

10 hours ago, dlandon said:

Now we can get down to the UD disk issue.  Please unmount and then remount the disk you were having issues with and post the diagnostics again.

Done.

 

-Adam

tower-diagnostics-20220106-0212.zip

[dlandon]

4 hours ago, adamf said:

Done.

 

-Adam

tower-diagnostics-20220106-0212.zip 359.47 kB · 0 downloads

Here's your issue:

Jan  6 02:11:48 Tower unassigned.devices: Mount drive command: /sbin/mount -t 'hfsplus' -o force,rw,users,umask=000 '/dev/sdb1' '/mnt/disks/My_Book'
Jan  6 02:11:49 Tower kernel: hfsplus: Filesystem was not cleanly unmounted, running fsck.hfsplus is recommended.  mounting read-only.

 

[adamf]

2 hours ago, dlandon said:

Here's your issue:

Jan  6 02:11:48 Tower unassigned.devices: Mount drive command: /sbin/mount -t 'hfsplus' -o force,rw,users,umask=000 '/dev/sdb1' '/mnt/disks/My_Book'
Jan  6 02:11:49 Tower kernel: hfsplus: Filesystem was not cleanly unmounted, running fsck.hfsplus is recommended.  mounting read-only.

 

Ah. Ok. so how do I fix that? what is fsck.hfsplus?

 

Thanks.

 

-Adam

[wgstarks]

Just now, adamf said:

Ah. Ok. so how do I fix that? what is fsck.hfsplus?

 

Thanks.

 

-Adam

The easiest would probably be to attach the drive to a mac and run First Aid in Disk Utility.

[TheSkaz]

 

can you use UD to share out a folder through NFS? if so, are there instructions somewhere?

[wgstarks]

4 minutes ago, TheSkaz said:

 

can you use UD to share out a folder through NFS? if so, are there instructions somewhere?

Be sure NFS is enabled in Settings>NFS and then set NFS export settings in Settings>Unassigned Devices.

[wgstarks]

You’ll also need to click the settings cog for the device and turn on sharing.

[TheSkaz]

13 minutes ago, wgstarks said:

You’ll also need to click the settings cog for the device and turn on sharing.

 

19 minutes ago, wgstarks said:

Be sure NFS is enabled in Settings>NFS and then set NFS export settings in Settings>Unassigned Devices.

 

I have enabled NFS in both places, I am not trying to share out a "device", but a mountpoint/folder. for example, I want to share out /fast/ and /tank/