February 22, 201115 yr I have a hard drive dock. I'd like to add a SNAP event when I plug in a drive. In order to do that I need to always be able to tell what port (or something) it is. Even if I might add or remove other drives or even sata adapter cards. Is there anything that stays the same for a port whether it's one of the motherboard ports or a port on a adapter card? I don't mean the hard drive id, I'm trying to get the port id.
February 22, 201115 yr The answer to this is "depends" on your hardware, but keep reading... In the early days before sata, it was easy: hda/hdb was the "primary master/slave", hdc/hdd were "secondary master/slave", etc., and this mapping was fixed. Then came sata which was built atop SCSI (using so-called libata subsystem). Now SATA drives had "scsi" style identifiers: sda, sdb, etc. But, like pure scsi devices, the assignment of device identifiers (e.g., "sda") depended on order of discovery by the OS, which depended on disk spin up among other things. So one particular disk might be sda after boot up, but then be sdb on the next boot, maybe back to sda on the third boot etc. - very annoying. Recognizing this, the linux subsystem for assigning device id (called 'udev') provides so-called "/dev/disk/by-path" symbolic links which are supposed to remain constant from boot to boot - indeed, up until unRAID version 5.0, this worked pretty well. However, starting with version 5.0 development, two issues have surfaced. First, the 'udev' developers switched the discovery code from scripts to C code. In the process, they didn't bother to implement SAS "by-path" discovery - so no more "by-path" for SAS controllers (without using an unofficial patch for udev). Second, it seems for many controllers now, even 'by-path' is not remaining constant from boot-to-boot. Rather than wait around for udev developer to acknowledge and fix this problem, starting with 5.0-beta5, unRAID no longer uses the concept of "slots" (that is, fixed port addresses). This has resulted in a fairly significant code and documentation change, however, in the end I think this will work out ok. So.. to answer your question: for version 5.0 and later, answer is "no", for 4.7 and below, it's "yes".
February 23, 201115 yr Author The answer to this is "depends" on your hardware, but keep reading... In the early days before sata, it was easy: hda/hdb was the "primary master/slave", hdc/hdd were "secondary master/slave", etc., and this mapping was fixed. Then came sata which was built atop SCSI (using so-called libata subsystem). Now SATA drives had "scsi" style identifiers: sda, sdb, etc. But, like pure scsi devices, the assignment of device identifiers (e.g., "sda") depended on order of discovery by the OS, which depended on disk spin up among other things. So one particular disk might be sda after boot up, but then be sdb on the next boot, maybe back to sda on the third boot etc. - very annoying. Recognizing this, the linux subsystem for assigning device id (called 'udev') provides so-called "/dev/disk/by-path" symbolic links which are supposed to remain constant from boot to boot - indeed, up until unRAID version 5.0, this worked pretty well. However, starting with version 5.0 development, two issues have surfaced. First, the 'udev' developers switched the discovery code from scripts to C code. In the process, they didn't bother to implement SAS "by-path" discovery - so no more "by-path" for SAS controllers (without using an unofficial patch for udev). Second, it seems for many controllers now, even 'by-path' is not remaining constant from boot-to-boot. Rather than wait around for udev developer to acknowledge and fix this problem, starting with 5.0-beta5, unRAID no longer uses the concept of "slots" (that is, fixed port addresses). This has resulted in a fairly significant code and documentation change, however, in the end I think this will work out ok. So.. to answer your question: for version 5.0 and later, answer is "no", for 4.7 and below, it's "yes". I'm going to have to think about how that affects SNAP in general on unRAID version 5. Are you planning on using the unofficial patch for udev in 5.0? But back to the question at hand... I'm not sure what a slot equates to. I looked at a drive connected to my BR10i device to see if I could tell what port on that device the drive is connected to. The second modalias has id numbers for the manufacturer lsi and the partiicular adapter BR10i (isnt' that equiv to a lun?). But there's no indicator of what port on the card it's using (I changed ports 3 times while checking). Of course I can use the device id from a hard drive itself (on 4.7 at least) but what I want to be able to do is set an action to occur when a drive (any drive) is plugged into that port on that adapter card. Because it can be any drive I can't use the drive id to detect it. Is there really no way to determine port number on an adapter or port number for the integrated sata chipset? udevadm info -a -p $(udevadm info -q path -n /dev/disk/by-id/*SATA*Q58*) ATTRS{modalias}=="scsi:t-0x00" ATTRS{modalias}=="pci:v00001000d00000058sv00001014sd00000394bc01sc00i00" ATTRS{modalias}=="pci:v00001022d00009603sv00000000sd00000000bc06sc04i00"
February 23, 201115 yr I think the unofficial patches were in 5.0 beta 3, and you know how that turned out... As for identifying the physical port on a SAS controller, it seems quite complicated. Through the various mailing lists and posts I came across looking into udev and sas and by-path entries, there doesn't seem to be anything in particular for that. One possible issue is supposedly the slot identifier, if you can call it that, always increments on the addition of a new drive. I think that means if you hotplug a drive in, then remove it, then hotplug it back in, it gets an incremented slot id.
February 23, 201115 yr Author Wow, that is shocking. I guess the most granual it will work is to detect any drive being plugged into a particular adapter. That severely limits use for designated ports. Dang. Ok, how about is there any way to get a unique serial number for the adapter in case there are two of the same adapter installed?
February 23, 201115 yr Here's one of the threads talking about a similar issue -- http://kerneltrap.org/mailarchive/linux-scsi/2010/3/5/6831083/thread#mid-6831083 I can't find the other posting that had some udev rules that were a little more consistent that someone else had used.
February 23, 201115 yr Wow, that is shocking. I guess the most granual it will work is to detect any drive being plugged into a particular adapter. That severely limits use for designated ports. Dang. Ok, how about is there any way to get a unique serial number for the adapter in case there are two of the same adapter installed? Right, well this has been a pain-in-the-neck for a while. For example, here is the documentation I added at the top of rc.udev for unRAID version 4.7 and below (can't remember exactly when I had to make this change): # tmm - invoke udevd with env variable UDEVD_MAX_CHILDS=1. This seems to fix a nasty bug where # a race would occur between various disk drivers in obtaining a scsi host number. This resulted # in disks being "reordered" or "missing" because the "disk=<identifier>" lines in the disk.cfg # file were not really being persisent. This was mainly observed when sata_sil24 would conflict # with ata_piix. The causes udevd to strictly serialize all events & this seems to solve the # problem, but I don't fully understand the mechanism This trick doesn't work anymore with latest udev, and I'm tired of fighting this battle and hence, no more slots. I'm not sure exactly how you implemented SNAP, but the /dev/disk/by-id still is ok for all device types. Once 5.0-beta5 comes out you might also get some ideas about how to manage non-array devices. Really, we should probably take this thread to the 5.0 Development boards since there might be some support I can build into unRAID to make it easier for you - and you can make a 5.0 SNAP plugin
February 23, 201115 yr Now remember, even going through /sys/devices/pci* things can change on reboots. I think this is hits on what the one message in the previously linked thread was talking about, looking at the PHY and PORT items. lrwxrwxrwx 1 root root 0 Feb 22 20:49 sda -> ../devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda/ lrwxrwxrwx 1 root root 0 Feb 22 20:44 sdb -> ../devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0/block/sdb/ lrwxrwxrwx 1 root root 0 Feb 22 20:26 sdc -> ../devices/pci0000:00/0000:00:1f.2/host2/target2:0:0/2:0:0:0/block/sdc/ lrwxrwxrwx 1 root root 0 Feb 22 20:26 sdd -> ../devices/pci0000:00/0000:00:1f.2/host3/target3:0:0/3:0:0:0/block/sdd/ lrwxrwxrwx 1 root root 0 Feb 22 20:26 sde -> ../devices/pci0000:00/0000:00:1f.2/host4/target4:0:0/4:0:0:0/block/sde/ lrwxrwxrwx 1 root root 0 Feb 22 20:40 sdf -> ../devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.3/1-1.3:1.0/host10/target10:0:0/10:0:0:0/block/sdf/ lrwxrwxrwx 1 root root 0 Feb 22 20:32 sdg -> ../devices/pci0000:00/0000:00:01.0/0000:01:00.0/host11/port-11:0/end_device-11:0/target11:0:0/11:0:0:0/block/sdg/ lrwxrwxrwx 1 root root 0 Feb 22 20:32 sdh -> ../devices/pci0000:00/0000:00:01.0/0000:01:00.0/host11/port-11:1/end_device-11:1/target11:0:1/11:0:1:0/block/sdh/ lrwxrwxrwx 1 root root 0 Feb 22 20:49 sr0 -> ../devices/pci0000:00/0000:00:1f.2/host5/target5:0:0/5:0:0:0/block/sr0/ ls -al /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/host11 total 0 drwxr-xr-x 16 root root 0 Feb 22 20:32 ./ drwxr-xr-x 4 root root 0 Feb 22 20:32 ../ drwxr-xr-x 3 root root 0 Feb 22 20:50 bsg/ drwxr-xr-x 4 root root 0 Feb 22 20:50 phy-11:0/ drwxr-xr-x 4 root root 0 Feb 22 20:50 phy-11:1/ drwxr-xr-x 4 root root 0 Feb 22 20:50 phy-11:2/ drwxr-xr-x 4 root root 0 Feb 22 20:50 phy-11:3/ drwxr-xr-x 4 root root 0 Feb 22 20:50 phy-11:4/ drwxr-xr-x 4 root root 0 Feb 22 20:50 phy-11:5/ drwxr-xr-x 4 root root 0 Feb 22 20:50 phy-11:6/ drwxr-xr-x 4 root root 0 Feb 22 20:50 phy-11:7/ drwxr-xr-x 5 root root 0 Feb 22 20:32 port-11:0/ drwxr-xr-x 5 root root 0 Feb 22 20:32 port-11:1/ drwxr-xr-x 2 root root 0 Feb 22 20:50 power/ drwxr-xr-x 3 root root 0 Feb 22 20:50 sas_host/ drwxr-xr-x 3 root root 0 Feb 22 20:50 scsi_host/ lrwxrwxrwx 1 root root 0 Feb 22 20:50 subsystem -> ../../../../../bus/scsi/ -rw-r--r-- 1 root root 4096 Feb 22 20:50 uevent ls -al /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/host11/phy-11:0/ total 0 drwxr-xr-x 4 root root 0 Feb 22 20:50 ./ drwxr-xr-x 16 root root 0 Feb 22 20:32 ../ lrwxrwxrwx 1 root root 0 Feb 22 20:53 port -> ../port-11:0/ drwxr-xr-x 2 root root 0 Feb 22 20:53 power/ drwxr-xr-x 3 root root 0 Feb 22 20:53 sas_phy/ -rw-r--r-- 1 root root 4096 Feb 22 20:53 uevent
February 23, 201115 yr Author Wow, that is shocking. I guess the most granual it will work is to detect any drive being plugged into a particular adapter. That severely limits use for designated ports. Dang. Ok, how about is there any way to get a unique serial number for the adapter in case there are two of the same adapter installed? Right, well this has been a pain-in-the-neck for a while. For example, here is the documentation I added at the top of rc.udev for unRAID version 4.7 and below (can't remember exactly when I had to make this change): # tmm - invoke udevd with env variable UDEVD_MAX_CHILDS=1. This seems to fix a nasty bug where # a race would occur between various disk drivers in obtaining a scsi host number. This resulted # in disks being "reordered" or "missing" because the "disk=<identifier>" lines in the disk.cfg # file were not really being persisent. This was mainly observed when sata_sil24 would conflict # with ata_piix. The causes udevd to strictly serialize all events & this seems to solve the # problem, but I don't fully understand the mechanism This trick doesn't work anymore with latest udev, and I'm tired of fighting this battle and hence, no more slots. I'm not sure exactly how you implemented SNAP, but the /dev/disk/by-id still is ok for all device types. Once 5.0-beta5 comes out you might also get some ideas about how to manage non-array devices. Really, we should probably take this thread to the 5.0 Development boards since there might be some support I can build into unRAID to make it easier for you - and you can make a 5.0 SNAP plugin I don't know exactly what the term *slot* refers to. Is it a single drive port or a whole adapter? Sometimes I don't get all the terminology until I have a need to. I believe you said that anything relying on /dev/disk/by-id is still going to work in version 5.0 as before - correct? That's good news. SNAP currently relies totally on /dev/disk/by-id. At some point I'll join the 5.0 Dev boards for SNAP. However, this particular thread is not really about SNAP as it currently exists but is about a new feature I'd like to add to SNAP if possible.
February 23, 201115 yr Author BRiT..the link you gave me was talking about the port number like below. For the /sys/block/sdd row shown twice below, the port-7 refers to the adapter itself, not a port on the adapter. Two different drives on the integrated sata ports: lrwxrwxrwx 1 root root 0 Feb 23 03:01 /sys/block/sdb -> ../devices/pci0000:00/0000:00:11.0/host3/target3:0:0/3:0:0:0/block/sdb/ lrwxrwxrwx 1 root root 0 Feb 23 03:01 /sys/block/sdc -> ../devices/pci0000:00/0000:00:11.0/host5/target5:0:0/5:0:0:0/block/sdc/ On a single drive of a BR10i sata adapter. I changed the cable and rebooted, but no changes. lrwxrwxrwx 1 root root 0 Feb 23 03:01 /sys/block/sdd -> ../devices/pci0000:00/0000:00:02.0/0000:02:00.0/host7/port-7:0/end_device-7:0/target7:0:0/7:0:0:0/block/sdd/ lrwxrwxrwx 1 root root 0 Feb 23 03:34 /sys/block/sdd -> ../devices/pci0000:00/0000:00:02.0/0000:02:00.0/host7/port-7:0/end_device-7:0/target7:0:0/7:0:0:0/block/sdd/ It sort of looks like on the integrated ports a different host# for each port. But on the sata adapter there wasn't anything unique on two different ports (cable p2 or p3). Is this happening because it's a BR10i and not yet fully supported or would any sata adapter have the same issue?
February 23, 201115 yr The message I linked to earlier mentioned having to look at the "phy-#:#" portion as well. I believe that's the physical device connection on SAS controllers. Whereas with /portN:M/ refers to the logical host adapter and logical device id (N referring to the adapter, and M referring to the logical device id). ls -al /sys/devices/pci0000\:00/0000\:00\:01.0/0000\:01\:00.0/host11/port-11\:0/ total 0 drwxr-xr-x 5 root root 0 Feb 22 21:13 ./ drwxr-xr-x 16 root root 0 Feb 22 21:13 ../ drwxr-xr-x 7 root root 0 Feb 22 21:13 end_device-11:0/ lrwxrwxrwx 1 root root 0 Feb 23 04:21 phy-11:0 -> ../phy-11:0/ drwxr-xr-x 2 root root 0 Feb 23 04:21 power/ drwxr-xr-x 3 root root 0 Feb 23 04:21 sas_port/ -rw-r--r-- 1 root root 4096 Feb 23 04:21 uevent Plugging in a drive to the second mini-sas cable "P4" produced the following. It used port11:2, the next logical id, but was mapped to the PHY I expected (the 8th one counting from 0 to 7). lrwxrwxrwx 1 root root 0 Feb 23 04:27 sdi -> ../devices/pci0000:00/0000:00:01.0/0000:01:00.0/host11/port-11:2/end_device-11:2/target11:0:2/11:0:2:0/block/sdi/ ls -al /sys/devices/pci0000\:00/0000\:00\:01.0/0000\:01\:00.0/host11/port-11\:2/ total 0 drwxr-xr-x 5 root root 0 Feb 23 04:27 ./ drwxr-xr-x 17 root root 0 Feb 22 21:13 ../ drwxr-xr-x 7 root root 0 Feb 23 04:27 end_device-11:2/ lrwxrwxrwx 1 root root 0 Feb 23 04:28 phy-11:7 -> ../phy-11:7/ drwxr-xr-x 2 root root 0 Feb 23 04:28 power/ drwxr-xr-x 3 root root 0 Feb 23 04:27 sas_port/ -rw-r--r-- 1 root root 4096 Feb 23 04:27 uevent Unplugging then replugging in a drive to the second mini-sas cable "P3" produced the following. It used port11:2, the next logical id, but was mapped to the PHY I expected (the 7th counting from 0 to 7). lrwxrwxrwx 1 root root 0 Feb 23 04:33 sdi -> ../devices/pci0000:00/0000:00:01.0/0000:01:00.0/host11/port-11:3/end_device-11:3/target11:0:3/11:0:3:0/block/sdi/ ls -al /sys/devices/pci0000\:00/0000\:00\:01.0/0000\:01\:00.0/host11/port-11\:3/ total 0 drwxr-xr-x 5 root root 0 Feb 23 04:33 ./ drwxr-xr-x 17 root root 0 Feb 22 21:13 ../ drwxr-xr-x 7 root root 0 Feb 23 04:33 end_device-11:3/ lrwxrwxrwx 1 root root 0 Feb 23 04:34 phy-11:6 -> ../phy-11:6/ drwxr-xr-x 2 root root 0 Feb 23 04:34 power/ drwxr-xr-x 3 root root 0 Feb 23 04:33 sas_port/ -rw-r--r-- 1 root root 4096 Feb 23 04:33 uevent Unplugging then replugging in a drive to the first mini-sas cable "P4" produced the following. It used port11:4, the next logical id, but was mapped to the PHY I expected (the 4th counting from 0 to 7). lrwxrwxrwx 1 root root 0 Feb 23 04:37 sdi -> ../devices/pci0000:00/0000:00:01.0/0000:01:00.0/host11/port-11:4/end_device-11:4/target11:0:4/11:0:4:0/block/sdi/ ls -al /sys/devices/pci0000\:00/0000\:00\:01.0/0000\:01\:00.0/host11/port-11\:4/ total 0 drwxr-xr-x 5 root root 0 Feb 23 04:37 ./ drwxr-xr-x 17 root root 0 Feb 22 21:13 ../ drwxr-xr-x 7 root root 0 Feb 23 04:37 end_device-11:4/ lrwxrwxrwx 1 root root 0 Feb 23 04:38 phy-11:3 -> ../phy-11:3/ drwxr-xr-x 2 root root 0 Feb 23 04:38 power/ drwxr-xr-x 3 root root 0 Feb 23 04:37 sas_port/ -rw-r--r-- 1 root root 4096 Feb 23 04:37 uevent Also note that the previous ports are not available, port11:2, port11:3, but all the phy-N-M are! ls -al /sys/devices/pci0000\:00/0000\:00\:01.0/0000\:01\:00.0/host11/ total 0 drwxr-xr-x 17 root root 0 Feb 22 21:13 ./ drwxr-xr-x 4 root root 0 Feb 22 21:13 ../ drwxr-xr-x 3 root root 0 Feb 23 04:20 bsg/ drwxr-xr-x 4 root root 0 Feb 23 04:20 phy-11:0/ drwxr-xr-x 4 root root 0 Feb 23 04:20 phy-11:1/ drwxr-xr-x 4 root root 0 Feb 23 04:20 phy-11:2/ drwxr-xr-x 4 root root 0 Feb 23 04:20 phy-11:3/ drwxr-xr-x 4 root root 0 Feb 23 04:20 phy-11:4/ drwxr-xr-x 4 root root 0 Feb 23 04:20 phy-11:5/ drwxr-xr-x 4 root root 0 Feb 23 04:20 phy-11:6/ drwxr-xr-x 4 root root 0 Feb 23 04:20 phy-11:7/ drwxr-xr-x 5 root root 0 Feb 22 21:13 port-11:0/ drwxr-xr-x 5 root root 0 Feb 22 21:14 port-11:1/ drwxr-xr-x 5 root root 0 Feb 23 04:37 port-11:4/ drwxr-xr-x 2 root root 0 Feb 23 04:20 power/ drwxr-xr-x 3 root root 0 Feb 23 04:20 sas_host/ drwxr-xr-x 3 root root 0 Feb 23 04:20 scsi_host/ lrwxrwxrwx 1 root root 0 Feb 23 04:20 subsystem -> ../../../../../bus/scsi/ -rw-r--r-- 1 root root 4096 Feb 23 04:20 uevent As for support the BR10i/LSI1068E is fully supported by Linux with the mptsas driver which should conform to the same SAS standards as others, so whatever you're seeing in /sys/ should be nearly identical to how it handles any other SAS devices. The only part that isn't fully supported on the controller for unRAID is within emhttp dealing with temps and spindown/up. Under a pure unRAID 5.0 beta 4 version, I had the SAS controller show up as host11 and also as host4 and also as host7, so it did jump around between reboots when there was absolutely no changes to the system.
Archived
This topic is now archived and is closed to further replies.