Fusion IO drivers

[Energen]

So you expect limetech to implement something that isn't even compatible in the first place, is not maintained by the original manufacturers, is not built into the native linux package that unraid runs on, create, develop, and maintain something that would have to be rebuilt every time there was an update?  Because 5 people have some obscure, obsolete hardware?

 

I mean, maybe limetech would be interested in doing this... I can't speak for them.  But they need to maintain stability for that mass market that you mentioned, not so much for some old stuff that very few people seem to have.

[joetlawson]

I expect them to respond to a customer that paid to use there software! Really if you have nothing positive to offer then stay out of it. Are you the Limetech police? Come on man no reason to try and be a jerk because I'm frustrated they did not even take the time to respond to me. That seems to be common with a lot of companies these days unfortunately.

Edited August 18, 2020 by joetlawson

[Energen]

Ah, my mistake, you're one of these cry baby millennials that think everyone owes you something, I apologize for hurting your feelings.  You mention about paying for a product, ok so you paid for your IO devices so go back to that company for driver support.. don't take out your frustration on Unraid.   Oh wait, you can't, because the hardware is obsolete.  Yet you want it to magically be brought back to life just for you.   Typical.

[joetlawson]

I'm far from a millennial . Honestly just go away if I pay for something I at least expect a response. Let me guess that is to much to ask for taking my hard earned money. Come on get real I'm new to Unraid and was not aware it was not naively supported.  You know what maybe Limetech could of took the 30 seconds to respond to my ticket and I may of understood and would not have posted. As far as your assumptions I'm not even going to bother to lower myself to your level. Thank god Limetech has people like you policing the forums.  How could they ever get by without advanced members like you. Oh wait by taking there money and not even bothering to acknowledge them.  There is an idea you should try to do the same thing!

[itimpi]

There is a lot of discussion (including by Limetech) earlier in this thread about the problems of adding FusionIO support.   It sounds as if it is likely to be quite a bit more work than simply adding drivers (assuming they even compile on the latest kernel - many 3rd party drivers do not).

[joetlawson]

8 minutes ago, itimpi said:

There is a lot of discussion (including by Limetech) earlier in this thread about the problems of adding FusionIO support.   It sounds as if it is likely to be quite a bit more work than simply adding drivers (assuming they even compile on the latest kernel - many 3rd party drivers do not).

Thanks for the feedback! I'm not sure who is who in here. 

[itimpi]

4 minutes ago, joetlawson said:

Thanks for the feedback! I'm not sure who is who in here. 

Limetech is the Unraid owner/original developer and as such can be considered an author active source.

 

it is also worth noting that Limetech employees only sporadically visit the forum so you should never assume they will see any particular post.

[joetlawson]

1 hour ago, itimpi said:

Limetech is the Unraid owner/original developer and as such can be considered an author active source.

 

it is also worth noting that Limetech employees only sporadically visit the forum so you should never assume they will see any particular post.

Yeah I was not assuming they would respond to a post. This was a ticket I opened with there tech support. They never responded too it after about 4-5 months.

[itimpi]

27 minutes ago, joetlawson said:

Yeah I was not assuming they would respond to a post. This was a ticket I opened with there tech support. They never responded too it after about 4-5 months.

Limetech are normally good about responding within days so it is always possible the reply got caught in a spam trap or something similar.

[joetlawson]

1 hour ago, itimpi said:

Limetech are normally good about responding within days so it is always possible the reply got caught in a spam trap or something similar.

Yeah that is always possible.

[jonnygrube]

As a technical note, FusionIO/WesternDigital are still supporting their products with drivers current as of 01/30/2020. 

Energen....these are still great cards, and hella not obsolete for this application.

I have a ioDrive2 and a SX350 that can do ~2GB/s+ R/W at 8W idle through a ESXi6.7 VM with super low latency like nvme.

If i were to make a guess, I'd say there are prolly ~50-60 of us in the forums that would integrate FusionIO prducts into our builds and at least that many more that would be inclined to buy these cheap used cards in the future for that purpose. 

No, we aren't a majority... but it's not 5 or 6 or a dozen.  There's dozens of us...

Tom, the CEO and admin, chimed in on this thread and put the ball in our court.
 

On 2/16/2020 at 11:59 AM, limetech said:

You could put the code and Makefile in a github repo and once other issues are sorted (such as udev rules) we can look at cloning repo and adding to Unraid OS.  However, if a newer kernel comes along and now driver won't build, we'll have no choice but to file an Issue in the repo and omit the driver until issue is resolved.

 

If we want to merge this into his repo, somebody will need to work on the 5.X kernel integration.

I would suggest following this proxmox guide as a starting point. -> https://forum.proxmox.com/threads/configuring-fusion-io-sandisk-iodrive-iodrive2-ioscale-and-ioscale2-cards-with-proxmox.54832/

Edited August 18, 2020 by jonnygrube
additional content

[joetlawson]

15 hours ago, jonnygrube said:

As a technical note, FusionIO/WesternDigital are still supporting their products with drivers current as of 01/30/2020. 

Energen....these are still great cards, and hella not obsolete for this application.

I have a ioDrive2 and a SX350 that can do ~2GB/s+ R/W at 8W idle through a ESXi6.7 VM with super low latency like nvme.

If i were to make a guess, I'd say there are prolly ~50-60 of us in the forums that would integrate FusionIO prducts into our builds and at least that many more that would be inclined to buy these cheap used cards in the future for that purpose. 

No, we aren't a majority... but it's not 5 or 6 or a dozen.  There's dozens of us...

Tom, the CEO and admin, chimed in on this thread and put the ball in our court.
 

 

If we want to merge this into his repo, somebody will need to work on the 5.X kernel integration.

I would suggest following this proxmox guide as a starting point. -> https://forum.proxmox.com/threads/configuring-fusion-io-sandisk-iodrive-iodrive2-ioscale-and-ioscale2-cards-with-proxmox.54832/

Like I said I'm still learning UnRaid but if I can be of help in anyway just let me know. This is good info for sure. It's a great OS I just know it would be better if I could use these for cache drives as I have a modular power supply since it is a Dell server and have no other good options for cache drives and that is why I purchased these not knowing it would be such a pain to get them working. A little frustrating having had these for months now and not being able to use them.

[MortalMonkey]

I've got 2 of the HP branded Fusion-IO Drive2 1.2TB that I would love to install as cache disks as well.  Gonna stick around and see if there is anything I can do to help.  

[Sillyvain]

If this option was possible I would switch to unraid with all my 32 x bl460C G8 Servers in a heartbeat.   All My BL460C G8 are attached to a D2200SB Storage blade which means a total of 14 hard drives each servers ( 2x hard drives on the BL460C G8 and 12 hard drives on the D2200SB = 14 hard drives)  I have a 1.2 TB Fusion IO integrated in each of them..

Edited November 29, 2020 by Sillyvain

[juchong]

Hi folks! I've been able to get the driver to compile on 4.x kernels (Unraid 6.8.3) cleanly using the Unraid-Kernel-Helper docker. I'm working out how to get unRAID to recognize the drives now. Stay tuned!

[Cmc0619]

I put in a feature request with the guys maintaining the open source driver to change the dev name as that appears to be the only sticking point.  Their reply was that they're unable to control the block names.  They did give a suggestion about mapping names in udev but I think I read early in this thread that that had already been tried.  It looks, to me anyway, like we're reliant on Lime to change the logic of their device identification logic to include the fio* prefixes.

 

Here's the thread on git if anyone is interested in reading their full responses.

Link

[joetlawson]

@Cmc0619 thanks for the info I have already given up on this one. I do find this comment interesting though.

 

bplein commented 4 hours ago

I'd like to use these drives in an Unraid server but their software only looks for devices named under /dev/sd* or /dev/nvme*. Is there a way to change how these drives present? Maybe something "simple" as a udev rule file that could be created? Or a barring that switch that could be specificied on module load?

 

This is best left to the Unraid folks... they are the ones making assumptions about what a device is named. I'd go as far as saying "lsblk correctly identifies /dev/fio* as block devices, why doesn't UNRAID" and let them sort it out. That's a bug!

[Cmc0619]

On 12/31/2020 at 1:05 PM, Cmc0619 said:

I put in a feature request with the guys maintaining the open source driver to change the dev name as that appears to be the only sticking point.  Their reply was that they're unable to control the block names.  They did give a suggestion about mapping names in udev but I think I read early in this thread that that had already been tried.  It looks, to me anyway, like we're reliant on Lime to change the logic of their device identification logic to include the fio* prefixes.

 

Here's the thread on git if anyone is interested in reading their full responses.

Link

And then a wild coder appears!  There are updates to the case I opened on the github.  After initial statements that it wasn't possible.  An enterprising, sharp coder by the nym of 'snuf' has posted a patch that he believes will get the Fusion IOs visible in Unraid.  See my link above for the comments.  Unfortunately, I'm away from my home server for 3 weeks.  Can anyone try this out and confirm if it works?

 

[joetlawson]

16 minutes ago, Cmc0619 said:

And then a wild coder appears!  There are updates to the case I opened on the github.  After initial statements that it wasn't possible.  An enterprising, sharp coder by the nym of 'snuf' has posted a patch that he believes will get the Fusion IOs visible in Unraid.  See my link above for the comments.  Unfortunately, I'm away from my home server for 3 weeks.  Can anyone try this out and confirm if it works?

 

Awesome I'm fairly new to Unraid and not sure how to do this so I'm not the right guy to test this but this is awesome.

[jc179]

Tried compiling with the latest GIT release (v4.2) as I am kernel 4.19 on unraid 6.8.3 still. Managed to compile it, but needed to modify the .sh scripts - I assume no one else has run into this, so it must be something weird with my unraid build env. Basically some instances of "cat" seems to be broken reporting 'standard output: No such file or directory'.... Very odd, not going to dwell on it for now, managed to work around this, and load the module (left some notes below as to what I did) .

 

I added the script you linked to my /lib/udev/rules.d/95-fio.rules , but don't see any new devices show up. There wasn't any info on how exactly to use this either, so I may need to run something to kick off the scirpt and create the new udev devices. 

 

I am a little tempted to manually populate and run the below and just see, but I will wait :

/usr/bin/fio-status --field iom.format_uuid /dev/%k", SYMLINK+="disk/by-id/fio-%c

/usr/bin/fio-status --field iom.format_uuid /dev/%P", SYMLINK+="disk/by-id/fio-%c-part%n

 

 

If interested, here is the error output from the make 

 

make[1]: Leaving directory '/mnt/user/dev/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16'
./kfio_config.sh -a x86_64 -o include/fio/port/linux/kfio_config.h -k /lib/modules/4.19.107-Unraid/build -p -d /mnt/user/dev/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kfio_config -l 0
cat: standard output: No such file or directory
1609869599.051  Exiting
Preserving configdir due to '-p' option: /mnt/user/dev/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kfio_config
make: *** [Makefile:88: include/fio/port/linux/kfio_config.h] Error 1

kfio_config.sh complains that "cat: standard output: No such file or directory"

which is ridiculous, as cat certainly exists, and is in the path... 

 

[email protected]:/mnt/user/dev/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16# env | grep PATH
MANPATH=/usr/local/man:/usr/man
NODE_PATH=/usr/lib64/node_modules
PATH=.:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin

 

[email protected]:/mnt/user/dev/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16# which cat
/usr/bin/cat
 

Here is the line which it's stopping at, I added an echo to identify:

    : "${TMP_OUTPUTFILE:=${CONFIGDIR}/kfio_config.h}"
    echo "cat instance 5"
    cat <<EOF


if I modify the line with the cat commands in kfio_config.sh: 

from this 
"cat <<EOF"
to
"echo <<EOF"
... to use echo instead of cat, we progress !! 

So the change now looks like:

    : "${TMP_OUTPUTFILE:=${CONFIGDIR}/kfio_config.h}"
    echo "cat instance 5"
    #cat <<EOF
    echo <<EOF

and re-run the line from the make file script, it completes now!

 

./kfio_config.sh -a x86_64 -o include/fio/port/linux/kfio_config.h -k /lib/modules/4.19.107-Unraid/build -p -d /mnt/user/dev/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kfio_config -l 0

 

So, stop and resume the original make command, which finally finishes:

 

Ok: altered version for iomemory-vsl.ko appended 2a18ca0

### 

 

I don't see any new /dev/sdX devices, I assume the linked script goes somewhere like /lib/udev/rules.d/95-fio.rules 

 

###

insmod iomemory-vsl.ko

##

dmesg -T | tail -100

[Tue Jan  5 13:38:33 2021] <6>fioinf VSL configuration hash: 8f82ea05bdf1195cb400fb48e4ef09fc49b3c1aa
[Tue Jan  5 13:38:33 2021] <6>fioinf
[Tue Jan  5 13:38:33 2021] <6>fioinf Copyright (c) 2006-2014 Fusion-io, Inc. (acquired by SanDisk Corp. 2014)
[Tue Jan  5 13:38:33 2021] <6>fioinf Copyright (c) 2014-2016 SanDisk Corp. and/or all its affiliates. (acquired by Western Digital Corp. 2016)
[Tue Jan  5 13:38:33 2021] <6>fioinf Copyright (c) 2016-2018 Western Digital Technologies, Inc. All rights reserved.
[Tue Jan  5 13:38:33 2021] <6>fioinf For Terms and Conditions see the License file included
[Tue Jan  5 13:38:33 2021] <6>fioinf with this driver package.
[Tue Jan  5 13:38:33 2021] <6>fioinf
[Tue Jan  5 13:38:33 2021] <6>fioinf ioDrive driver 3.2.16.1731 4870ad45b7ea 2a18ca0 a loading...
[Tue Jan  5 13:38:33 2021] <6>fioinf ioDrive 0000:04:00.0: mapping controller on BAR 5
[Tue Jan  5 13:38:33 2021] <6>fioinf ioDrive 0000:04:00.0: MSI enabled
[Tue Jan  5 13:38:33 2021] <6>fioinf ioDrive 0000:04:00.0: using MSI interrupts
[Tue Jan  5 13:38:33 2021] <6>fioinf ioDrive 0000:04:00.0.0: Starting master controller
[Tue Jan  5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Adapter serial number is 602555
[Tue Jan  5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Board serial number is 602555
[Tue Jan  5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Adapter serial number is 602555
[Tue Jan  5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Default capacity        640.000 GBytes
[Tue Jan  5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Default sector size     512 bytes
[Tue Jan  5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Rated endurance         10.00 PBytes
[Tue Jan  5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: 85C temp range hardware found
[Tue Jan  5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Firmware version 7.1.17 116786 (0x700411 0x1c832)
[Tue Jan  5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Platform version 10
[Tue Jan  5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Firmware VCS version 116786 [0x1c832]
[Tue Jan  5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Firmware VCS uid 0xaeb15671994a45642f91efbb214fa428e4245f8a
[Tue Jan  5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Powercut flush: Enabled
[Tue Jan  5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: PCIe power monitor enabled (master). Limit set to 24.750 watts.
[Tue Jan  5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Thermal monitoring: Enabled
[Tue Jan  5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0.0: Hardware temperature alarm set for 85C.
[Tue Jan  5 13:38:34 2021] <6>fioinf ioDrive 0000:04:00.0: Found device fct0 (Fusion-io ioDrive 640GB 0000:04:00.0) on pipeline 0
[Tue Jan  5 13:38:36 2021] <6>fioinf Fusion-io ioDrive 640GB 0000:04:00.0: probed fct0
[Tue Jan  5 13:38:36 2021] <6>fioinf Fusion-io ioDrive 640GB 0000:04:00.0: sector_size=512
[Tue Jan  5 13:38:36 2021] <6>fioinf Fusion-io ioDrive 640GB 0000:04:00.0: setting channel range data to [2 .. 4095]
[Tue Jan  5 13:38:36 2021] <6>fioinf Fusion-io ioDrive 640GB 0000:04:00.0: Found metadata in EBs 168-169, loading...
[Tue Jan  5 13:38:36 2021] <6>fioinf Fusion-io ioDrive 640GB 0000:04:00.0: setting recovered append point 169+198180864
[Tue Jan  5 13:38:36 2021] <6>fioinf Fusion-io ioDrive 640GB 0000:04:00.0: Creating device of size 640000000000 bytes with 1250000000 sectors of 512 bytes (213422 mapped).
[Tue Jan  5 13:38:36 2021] fioinf Fusion-io ioDrive 640GB 0000:04:00.0: Creating block device fioa: major: 254 minor: 0 sector size: 512...
[Tue Jan  5 13:38:36 2021]  fioa: fioa1 fioa2
[Tue Jan  5 13:38:36 2021] <6>fioinf Fusion-io ioDrive 640GB 0000:04:00.0: Attach succeeded.





[email protected]:/mnt/user/dev# /usr/bin/fio-status

Found 1 ioMemory device in this system
Driver version: 3.2.16 build 1731

Adapter: Single Adapter
        Fusion-io ioDrive 640GB, Product Number:FS1-004-640-CS, SN:602555, FIO SN:602555
        External Power: NOT connected
        PCIe Power limit threshold: 24.75W
        Connected ioMemory modules:
          fct0: Product Number:FS1-004-640-CS, SN:602555

fct0    Attached
        ioDrive 640GB, Product Number:FS1-004-640-CS, SN:602555
        PCI:04:00.0, Slot Number:5
        Firmware v7.1.17, rev 116786 Public
        640.00 GBytes device size
        Internal temperature: 54.14 degC, max 54.63 degC
        Reserve space status: Healthy; Reserves: 100.00%, warn at 10.00%
        Contained VSUs:
          fioa: ID:0, UUID:0630fa34-0505-1441-9d5b-6d79afe74709

fioa    State: Online, Type: block device
        ID:0, UUID:0630fa34-0505-1441-9d5b-6d79afe74709
        640.00 GBytes device size

 

Edited January 5, 2021 by jc179
fixed spelling

[Cmc0619]

1 hour ago, jc179 said:

 

I don't see any new /dev/sdX devices, I assume the linked script goes somewhere like /lib/udev/rules.d/95-fio.rules 

 

I don't think you'll see them show up as /dev/sdX devices.   snuf's analysis looks like he determined that /dev/disk/by-id is what unraid uses and that's what his udev rules create.  Does anything show up in the GUI?  When I add a new drive it shows up in Unassigned Devices on the main GUI page.

 

[jc179]

You know what, I was so fixated with that I forgot to look at what the script is actually doing, and looked in the wrong place. 

 

I do see the card shows up under /dev/disk/by-id  !!!

 

[email protected]:/dev/disk/by-id# ls -al
total 0
drwxr-xr-x 2 root root 1100 Jan  5 14:59 ./
drwxr-xr-x 7 root root  140 Dec 30 13:17 ../
lrwxrwxrwx 1 root root   10 Jan  5 14:59 fio-0630fa34-0505-1441-9d5b-6d79afe74709 -> ../../fioa
lrwxrwxrwx 1 root root   11 Jan  5 14:59 fio-0630fa34-0505-1441-9d5b-6d79afe74709-part1 -> ../../fioa1
lrwxrwxrwx 1 root root   11 Jan  5 14:59 fio-0630fa34-0505-1441-9d5b-6d79afe74709-part2 -> ../../fioa2
(I remove all other unecessary info here, eg other disks we're not interested in).

 

Checking in the Unraid GUI, there is nothing new. I didn't have any new unassigned disks, I even stopped the array to have a look, no changes... dang. Was really hoping this would work!  Maybe Unraid 6.9 might behave differently? Will see if I can give that a try at some point... 

 

There are indeed two partitions on my fio card, and this has mapped them correctly -  I can  mount and talk to the fio card a-ok  as well. The issue might lie that even using the by-id, when mounted, seems to revert back to a /dev/fioa device....

 

mount /dev/disk/by-id/fio-0630fa34-0505-1441-9d5b-6d79afe74709-part2 /a

 

/dev/fioa2      597G  113M  596G   1% /a
/dev/fioa2 on /a type fuseblk (rw,allow_other,blksize=4096)

 

Wish I had better news...

 

Jonathan

 

 

[jc179]

Messing around further, I tried using a loop device back to the /dev/fioa, but even *that* doesn't show up in the Unraid GUI, lsscsi also doesn't see it, probably because it is major #7, (major #8, hdd block devices, losetup won't do it ).  

 

again, all a guess... 

 

mknod /dev/sdn b 7 200
losetup /dev/sdn /dev/fioa
udevadm control --reload-rules
udevadm trigger
 

[email protected]:/# fdisk -l /dev/sdn
Disk /dev/sdn: 596.5 GiB, 640000000000 bytes, 1250000000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5D116A01-F6AC-45FB-94A9-C3881252FA51

Device     Start        End    Sectors  Size Type
/dev/sdn1     34      32767      32734   16M Microsoft reserved
/dev/sdn2  32768 1249996799 1249964032  596G Microsoft basic data
[email protected]:/#
 

 

Perhaps if we could modify the error checking of losetup (opensource recompile), so that this would work, then this might show up, with the caveat of a hefty performance impact, but I'd still use it honestly...

 

crude quick speed test:

[email protected]:/# dd if=/dev/fioa of=/dev/null bs=1024k count=20000
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 14.4936 s, 1.4 GB/s

 

[email protected]:/# dd if=/dev/sdn of=/dev/null bs=1024k count=20000
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 24.5716 s, 853 MB/s
[email protected]:/#
 

This fails... 

[email protected]:/dev# mknod /dev/sdo b 8 200
[email protected]:/dev# losetup /dev/sdo /dev/fioa

losetup: /dev/sdo: failed to set up loop device: No such device or address
 

Maybe someone else has another idea to try, maybe emhttpd could be patched to recognize the fioa device or something else... I don't know... sucks... Just want to make this work, id even take half speed... lol. 

 

Is like pandora's magic box here...

 

Jonathan

[Cmc0619]

Let's phone a friend.

 

@limetech are we on the right path here with any of this?

[ChatNoir]

5 hours ago, Cmc0619 said:

Let's phone a friend.

 

@limetech are we on the right path here with any of this?

You will not reach a user this way.

You have to type the name (at least the first part), then wait to be prompted by a dropdown menu and select from there. @Cmc0619