dAigo

Ok, I will see what I can do. Out of curiosity, can unraid/btrfs give a higher priority to the "copy" of the data that is on a specific disk in case of a read request? Or is it random/round robin? That could remove the SATA/AHCI penalty in while reading the data. (booting VMs/starting Programs) But it would probably still be a waste I think Definitly correct, however, I would argue that nvme and ahci PCIe are both kind of bottlenecked by the qemu/kvm overhead and btrfs, at least in non-enterprise enviroments like unRAID. According to Intel, NVMe has "only" 3µs lower latency, wich is great (50%!) but in the end only a portion of the io-wait time... If you compare the "avarage service time" under "Heavy Load" of the Samsung SM951 NVMe/AHCI versions, they are at 278µs (NVMe) vs 281µs (AHCI), which is exactly what intel told us 3 Years earlier (and both are faster than the 256GB NVMe 950 Pro) And that were all bare-metal tests, add virtualization latency with consumer hw on top and you problably wont notice any difference between AHCI/NVMe as long as they both are using the same PCIe lanes. The protocolls differ in many ways, and if perfomance is similar, compatibility could make AHCI PCIe SSDs the better choice right now. Which is why those AHCI PCIe SSDs were mostly OEM, high speed, high compatibility but only short term (3-5 years is ja good timeframe for an OEM like HP/Lenovo) Its just, that NVMe is the obvious winner in the long run, with PCIe4.0, 3D-Nand/3D XPoint, better/faster controller -> 320GBit/s ~ DDR3/DDR4 speed for storage... (5 years +) interesting things to know about PCIe/m.2 on "older" chipsets: (x99/z97/...) - all the onboard goodies (storage/audio/LAN) are connected to the chipset - the connection to that chipset varies, but is shown on wikipedia - x99 Chipset uses DMI 2.0 to connect to the CPU, so everything that is NOT in a PCIe Slot will share a max of 20GBit/s - the m.2 slot can be either shared with the SATA Express (or SATA 9&10) or directly connected (through an additional controller), as an example m.2 on Asus X99WSIPMI is shared, but the Asus Z97DELUXE has an additional "ASMedia® SATA Express controller" that is connected directly to the CPU and the m.2 port shares its bandwith. - DMI2.0 uses PCIe2.0 which has a theoretical max. of 380k IOPS, which could also bottleneck a PCIe3.0 (x4) SSD that is rated for 460k IOPS - a SATA SSDs should have 6 GBit/s each (so 24 GBit/s total with 4) - INTEL 750 is listed with 2.4 GByte/s max. but Benchmarks show up to 2.7 GByte/s (1Byte=8Bit -> 19,2 GBit/s up to 21,6Gbit/s) - PCI3.0 is not only faster but also uses a better optimized "encoding" which results in less overhead and higher IOPS What does that mean? 1) A NVMe m.2 SSD like the Samsung 950 or Intel 750 could be bottlenecked to "only" 380k IOPS, when used in PCIe2.0 as in DMI2.0. 2) If you consider other components like LAN (1-2 GBit), SATA (6Gbit), USB3 (5GBit) its probably safe to say, that it would be rare to see a NVMe disk using its full potential while sharing 20Gbit/s with other components. 3) AHCI or NVMe on a m.2 slot that is connected through DMI2.0 would probably make no difference at all. *) basicly Ferrari vs Corvette with a lot of traffic All Skylake Chipsets (Z/H/Q170, Q/B150, B110) use DMI 3.0 (so PCI3.0) and "newer/better" x99/z97 mainboards have the needed additional controller to make sure PCI3.0 and no shared lanes are used. In that case NVMe should be "slightly" faster than AHCI, depending on the workload and the system. But yes, "in general" you are right, I just wanted to share some insight into the NVMe/AHCI/PCIe/Chipset dilemma At least that is what I think is true about nvme today, I could be wrong and there will be changes in the future.

Short answer: Yes you can run a VM on that drive outside of the array. At least I can (and I currently do) and I see no reason why other systems should behave different. You could also use passthrough for better perfomance, but passthrough always depends on more components. In my case it worked fine, same procedure as a GPU. Longer answer: You wont be able to do everything through the Web-Interface, so a basic understanding about ssh/cli commands and/or linux in general will help. You would need to identify the name of your device, be able to format it as xfs (or ext4) and mount it somewhere under /mnt. Once you do that, you can choose the drive like any other disk on your server when creating VMs through the GUI. It would defenitly help to know how you automate the mounting procedure for every reboot of your server. If you are able to do all of that, your are done in 5-20 minutes and can start your VM. But since you are a "first timer", I would asume, that these steps are not exactly familiar to you? I could post the way I did it, that does not mean its safe or the best/easyest way, but it worked for me. (and others I spoke to) I definetly had a very good learning experience, I still dont know everything I should know about what i have done, but I know more than before You would still do it at your own risk, so create backups of whatever is on your ssd or dont put any valuable date on it. I dont think "official" NVMe support is something we will see in the next 4-6 Month, maybe some beta features or plugins, but from what I have seen so far, there are a lot of things that needs to be evaluated before it can be supported. But that is just my personal opinion, i have no knowledge how fast lime-tech can get where ever it is they want to be, before they support new stuff.

I'll try this soon. Any luck with this gfjardim? I'm in the same situation where I have a 256gb nvme drive I want to mount for 2 windows vms but am unable to currently. I am more then happy to help test this and it may be very helpful to unraid itself if this works. Regards, Jamie Edit: as a side note, the nvme drives show up in lsblk as a disk but not in lsscsi. So it may be worth looking through lsblk for devices of type disk, and checking if their mounted or not? I'm unsure how you do this currently It would still use /dev/ but may be a way to look for them quickly? I haven't touched Linux in a good while so I am most likely wrong Unfortunatly, it seems nvme is a little bit "more different" than one would expect. From what I saw, when I looked through the source code, he is using /dev/sdX (and hdX), but not /dev/nvme (as suspected) But he also uses /dev/disk/by-id symlinks, probably because unraid itself uses these symlinks and he needs to see wich disk is in fact "unassigned". The thing is, that id wont change, even if your reorganize how your disks are plugged into your mainboard, because its supposed to be unique (serial number etc.). Which is problably why unraid uses it. /dev/sda, sdb and probably even /dev/nvme can change. I mean, if you have two identical disks how do you know which one is /dev/nvme0 and which is /dev/nvme1? unlike sata ports. pci slots usually dont even have a number on consumer mainboards... but I digress. I dont know if its a bug or if there are good reasons for it, but right now, the kernel does not create a "by-id" symlink. (at least on my system) It seems, that this need fixing in the kernel. (Source) So I guess, its either a lot of new code needed, wich may end up useless/harmfull if the kernel/unraid changes the nvme behavior or unraid needs to change things, so that a nvme disk behaves like any normal disk, which would need proper testing and therfore some time. It seems, that unRaid 6.2 may add some new stuff regarding sdds, so I guess we wait and see how much love nvme will get.

That seems correct to me, which is why I not only mentioned cache settings, but also the "dataplane" setting. Look at the document behind the [ftp=ftp://www.linux-kvm.org/images/a/a7/02x04-MultithreadedDevices.pdf]"multithread"[/ftp] link in my post. It seems to me, that data-plane uses 1 IOThread per device as a default. (Page 9) I was trying to point out, that you may add threads that way until your bug is fixed. So.. in theory... if you have two 8-core cpus (2x 4+4) best performance would be to create a vm with 16 seperate disks (raw images on your ssd), and create a raid0 in the vm. That way, qemu could use 16 threads, which should be (theoreticaly) distributed across all 16 "cores" and therefore run at the same time... Well, until you run from the i/o bottleneck of the disk to the i/o bottleneck of your cache/host/physikal hardware. (a 16-disk software raid in a vm does sound rather io/interrupt heavy, maybe tone it down^^) Something like that is what i think happened with my tests. (just a gut feeling according to my current knowledge) I get almost twice as many 4k random IOs when I add data-plane. But because that only helps with the IO of the virtual disk, as soon as I hit the single cache/memory "thread", there is a bottleneck. Therefore, if I remove/ignore as much memory/cache IOs (by "ignoring" the "flush" requests/IOs of the guest), the disk performance goes up even higher. The Windows driver forbids me to deactive write cache, otherwise i would test that. In passthrough I had "near bare-metal" perfomance, wich would almost double the sequential performance. I am with you on "why passthrough in a virtual system" but depending on the hardware/costs passthrough is the better Option (see GPU-passthrough -> spend 10k+ on a GRID GPUs + VMware to get a shared vGPU seems unreasonable for gaming ) That was obviosly not the case with mechanical drives, but SSDs exposed those bottlenecks. High IO requirements and affordable high IO hardware (pci/nvme flash storage) are pushing the IO development in qemu. From what I understand, they are/were trying the "data-plane" approach with diffrent devices, such as network/memory because it seems everything is limited by I/Os -> Threads. Oh btw. there are version .110 VirtIO drivers available, I did not notice any change in perf., but who knows... So ... before I am getting off topic , you should try "data-plane" to see if it changes anything. Rest is offtopic, just my thoughts in generall If its not a generall unRAID/qemu issue (most people seem ok), maybe its your hw config? (yeah, that rabbid hole ) While XEONS are great at what they do, I don't know how well unraid/qemu works with dual socket cpus. That may introduce additional overhead/cache issues. But that NUMA stuff gets really complicated. you pinned your vcpu to your second cpu, but what about the memory? Can qemu make sure, that the memory of your second cpu is used/prioritzed? what about the host cache? on wich cores are the qemu threads running? Xeons are great for multithreaded applications, but even if you could seperate the IO into ONE additional thread, that thread would run on a 2.2Ghz core... an i7 with 3.8-4.5Ghz may still be faster. Depending on your workload, it may be of no diffrence to create one or two additional threads, you would need a larger scale. Thats why I love the new skylake... more/enough pci-lanes for storage, more/enough memory for vms, but still a very high ghz per core ratio. Only thing missing is ecc memory and dual cpu, but 8 "cores" are usualy "enough". Maybe your hardware has badly "optimized" linux drivers and there is an issues with the host<->device communication? (Could also be the reason for the passthrough issues...) Maybe because its a Dell controller with a LSI firmware, on a HP mainboard? the LSI 2008 is a good controller, but rather old. The LSISAS2008 seems to have some issues with TRIM/SSDs. Do you use IT or IR firmware? Did you try another controller? (onboard maybe) Maybe the write-cache is slow/damaged on the controller? HP is great for workstation/server hardware, but they are a pita when it comes to campatibility. At work, we had a case where two external Tape drives were damaged (fw update/reset didn't help) and replaced until hp pointed to the "IBM" sas controller (LSI2008...) and said we should change that to one that is listed in the compatibility list of the tape-drive (HP with LSI2008...). We did that and had now issues ever again. We usually try to avoid mixing hp with other vendors, if the customer does not have the money for "all HP", we use "no HP". Thats why supermicro gets so pupular, they are mixing very well with other vendors.

Since you are waiting anyway... i am testing different settings right now, so I thought I share them, maybe it helps to find the bottleneck/bug. Be aware, I am no veteran on that topic, just posting my observations and personal conclusions. (which may be wrong) Depending on how brave you are with your "guinea pig'ing" and how often you are willing to do backups, you could go ahead and try different "cache" options in qemu and windows. IBM: Best practice: KVM guest caching modes In my case "unsafe" is the fastest, according to AS SSD... Funny thing is, I have better results using "none" cache, than the default "writeback", as if there is some kind of overhead/bottleneck that slows things down when qemu caches and waits for the ACK (O_SYNC issues?)... Most SSDs only have normal power loss protection. some kind of ssd-internal journal that ensures data-integrity, but does not prevent loosing data in your cache in case of a sudden powerloss. Some Intel SSDs (320, 750) have additional capacitators that should work as a little battery in case of a sudden powerloss. A battery-backed or flash-backed sata controller may also add some "safety". When or if at all the cache is actually written depends on many factors, qemu alone has four and the windows guest has additional three... so 12 possible combinations. I disabled any cache flushing, so neither windows ("Turn of write-buffer flushing") nor qemu ("unsafe") ever tells the ssd to write the cached data to its flash an waits for the ok. I asume/hope, that the SSD will do that, when IO is low, which should be fairly often in a io-light desktop/gaming usage... The Problem is, caching itself makes benchmarking very questioanable, since you never know if you are actually testing the perfomance of your hardware or just its ability to use its given cache... Maybe your host has not enough resources to properly cache? You should try and use at least 3GB files in AS SSD, at least I get results that are more usable, especially in a vm. In addition to that, and since single/multithreading was mentioned, you may want to try out [ftp=ftp://public.dhe.ibm.com/linux/pdfs/KVM_Virtualized_IO_Performance_Paper_v2.pdf]virtio-blk-data-plane[/ftp]. According to that document (from 2013...), "normal" VirtIO had a IOPS Cap of 140k iops. while that seems "enough" for a sata ssd, it shows, that VirtIO was not designed vor high i/o. I asume driven by enterprise demands (which are io heavy) that option gets more love than the "legacy" virtio, because it seems, that it is, in one way another..., capable of multithreading However, there are things I do not understand myself right now, so i may have come to false conclusions, wich could explain why the following does not make sense: - data-plane is intended für virtio-blk/virtio-scsi but in my case, it worked best with the normal virtio. - caching: - disabling host-cache ("none") speeds things up, but I would guess host-cache should be the hosts memory and therefore be much faster? (probably depends on disk/ssd speed) - most of all: the 4k random i/o are highest (with my settings), when the guest only has very limited memory (512MB). - 1GB Memory halfes the 4k reads and 2GB+ then halfes the 4k writes as well. (I can reproduce that with win8, 10, Server2012 and Server 2016 prev) - so if windows runs out of memory to cache and qemu ignores/has no caching, the system gets faster, sounds strange. The cache settings increased my sequential perfomance (1,5x - 2x), while data-plane helped with 4k Random i/o (1,5x-3x -> depending on guest memory), your results may vary. my xml: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> ... ... <devices> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='unsafe' io='threads'/> <source file='/mnt/nvme/Rechner/Rechner_vDisk1.img'/> <backingStore/> <target dev='hda' bus='virtio'/> <boot order='1'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> ... ... </devices> <qemu:commandline> <qemu:arg value='-set'/> <qemu:arg value='device.virtio-disk0.scsi=off'/> <qemu:arg value='-set'/> <qemu:arg value='device.virtio-disk0.config-wce=off'/> <qemu:arg value='-set'/> <qemu:arg value='device.virtio-disk0.x-data-plane=on'/> </qemu:commandline> </domain>

Hi, is there an easy way to add nvme disks (pcie ssds) to the plugins list? Since they are not yet supported as cache disks, this awesome plugin would be ne next best thing Right now they dont show up, you are probably not looking into /dev/nvme* for disks? Not nesseseraly all features, but at least auto-mount (for VMs) would be nice. If you dont have access to a nvme drive, I could test it for you or provide console output to your device search pattern. Dont mind if it's too much work, it'll probably work without the plugin. Anyway, great plugins so far, keep it up

Thanks, I'll do that. I know, but even if my questions won't get answered, others's may be. I kept the title very broad for that reason. Would it be usefull to open a "Feature request" so that we can keep track of any progress or test things out if it seems usefull? I think that depends on your Hardware. I choose the Intel SSD over the faster and cheaper Samsung 950 Pro for compatibility, safety reasons. I mean, Samsung doesn't even have a datasheet for the 950... Who knows what features might be missing to support passthrough, raid and what ever a "normal" consumer wont need. And the ASUS-Mainboard, because it has support for the Intel SSD. (with the "Hyper-Kit") I have an option in the bios to specify if i use the hyper-kit in the M.2 slot (and therefore an nvme SSD) or not. It may otherwise run in some M.2/AHCI legacy mode, who knows. As far as i know, the Mainboard/PCIe/BIOS can have impact on PCI-Passthrough, so your results may vary. In my case, the answer is yes, I can. Just for you, installed Win10 x64 on the mentioned Intel 750 NVMe SSD (M.2 -> Hyper-Kit -> U.2). (in under 5 Minutes I might add ) I used uefi/ovmf, rest is default. And of course edited the .xml to add the SSD passthrough, since only GPU/USB passthrough has support through the web interface. No additional drivers were needed (but Intel recommends to install them at some point) I had some trouble with the bootmanager and uefi, I need to select the windows-efi-bootfile manually. But I have these on other VMs as well, problably related and not nvme specific. Win7 has no native nvme support (afaik), so a hotfix/driver would probably be needed to install I asume. Linux should work if the kernel has nvme support. Mac I don't know. I also can't say how safe it is to passthrough the ssd, if there are issues with the passthrough, the filesystem on you systemdrive could be damaged. Its your own risk I would say. It may or may not be safer to format/mount the drive in unRAID and place your virtual disks there. At least you could place (and boot) any OS on there, even if there is no native nvme support (Win7) If I find the time, I may run some benchmarks to compare passthrough vs. unRAID mounted. So until the disk leaves the testbench and goes live, I could try things out if there are questions.

EDIT: As of March 11, 2016 unRAID Server Release 6.2.0-beta18 NVMe is *supported* for cache and array usage. THIS IS BETA SOFWARE -------------------- While every effort has been made to ensure no data loss, **use at your own risk!** Original Post: In the last 5 years, i usualy found a "good enough" answer on the forums to any question i could not answer for myself, but not this time. And so I hope this thread, if it goes anywhere, may help other people in the future. Distant future probably, as the tpoic is not that hot even now with Intel/Samsung pushing NVMe to consumers/enthusiasts. I found varius posts regarding NVMe disks, but they seem to get nowhere: 1) Probaly off Topic therefore no Answer 2) Same Guy, new Thread -> still No Answer 3) PCIe: AHCI vs. NVMe From the 6.1.2 Notes i took: My guess so far is, that PCIe SSDs with AHCI should work, because in the end they are recognized as SATA Disks. (/dev/sdX) But since unRAID was always developed with the SATA Protocoll in mind, if we are lucky NVMe functionality is "unknown" and if we are not its "not working"? Enough theorie, here is my current situation. I installed an Intel 750 NVMe PCIe SSD and was hoping that it could be selectable in the GUI as a cache/data disk, but its not. No big deal, hoping doesn't mean expecting. It is however correctly recognized as a PCI device and even as a "disk". (probably thanks to the update in 6.1.2?) And as it is correctly identified as a "disk" i would asume, that its possible to add it as an additional drive besides the array/cache until further support is implemented. (which I am trying to avoid) Normaly I use the Plugin "unassigned devices" from gfjardim to manage devices outside of the array, but it seems he had no reason so far to add support for NVMe. I may ask him if he thinks he could add that. (maybe just add /dev/nvme* as a search pattern?) And if I would run the disk besides the Array, are there recommendations for the filesystem? In a cache pool, it would obviosly be btrfs, but it seems xfs would be the more common way and is at least mentioned in some spec sheets: Intel - NVMe Filesystem Recommendations (last paragraph - "Be sure to turn off the discard option") So, what is the "officall" state of NVMe in unRAID? (as cache/data/parity) Does it work? (as in "Report a Bug if it does not work.") Could it work? (as in "It's new tech, send us additional logs/debugs and we might get it to work.") Should it work? (as in "unRAID makes no difference between SATA/AHCI and NVMe.") If it works, but not as a part of the array, the next questions would be: Would you think using a NVMe disk is "safe" in unRAID or should I wait for newer kernel/modules/filesystem versions? And if it doesn't work, is there any planned roadmap for further support or should I just create a "Feature request"? I would understand if that topic is not very high on the list, especially if you would need to recode the whole storage system. And I know that "PCIe AHCI SSDs" and even "normal SATA SSDs" are usualy more than enough, my two old Intel X25-M SSDs are still good enough for normal "desktop use", but as a tech nerd, I want to see what the new stuff could do. Its unbelievable what you guys packed into v6 of unRAID, so many shiny toys (docker, kvm, cache pools). And to top that, Intel (skylake) and NVidia (maxwell2) are getting so much more efficient that its hard, to push it to the limits. (But I can't keep myself from pushing ) I may need a "little" Microsoft (Exchange) and Cisco (UCM) Lab in the near future (for work), would be nice to keep that running while playing Fallout4 at 1440p/144Mhz, while listening to music from the local media server, while monitoring everything via snmp, while ... oh, its getting off topic ... Well its the first post, some things needed to be said That beeing said, any and all feedback would be appreciated