Should my NVME be this slow?

[craigr]

So I followed Spaceinvader's "How to passthrough and boot from an NVMe controller for bare metal performance in KVM/unRAID" for my WD SN750 and I'm not getting anywhere near the speed that he is getting from his Samsung drive.  There is no special WD NVMe driver that I can find as with the Samsung, but even before Spaceinvader installed the driver for Samsung his speeds are much better than mine.  I know the SN750 is a bit slower, but only by 10% or less.

 

My full system specs are in my signature, but this is on a Supermicro X11SSH-LN4F with E3-1275 v5 @ 3.60GHz.

 

The only thing I did differently is to use the "VFIO-PCI Config" app to stub my NVMe drive because I do not want it to be seen by unRAID at all or in unassigned devices.  Are these speeds acceptable for my motherboard and NVMe drive?

------------------------------------------------------------------------------
CrystalDiskMark 7.0.0 x64 (C) 2007-2019 hiyohiyo
                                  Crystal Dew World: https://crystalmark.info/
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
Sequential 1MiB (Q=  8, T= 1):  1729.866 MB/s [   1649.7 IOPS] <  4843.69 us>
Sequential 1MiB (Q=  1, T= 1):  1276.371 MB/s [   1217.2 IOPS] <   820.22 us>
    Random 4KiB (Q= 32, T=16):   921.287 MB/s [ 224923.6 IOPS] <  2190.89 us>
    Random 4KiB (Q=  1, T= 1):    35.604 MB/s [   8692.4 IOPS] <   114.63 us>

[Write]
Sequential 1MiB (Q=  8, T= 1):  1692.193 MB/s [   1613.8 IOPS] <  4943.04 us>
Sequential 1MiB (Q=  1, T= 1):  1328.853 MB/s [   1267.3 IOPS] <   787.54 us>
    Random 4KiB (Q= 32, T=16):   455.348 MB/s [ 111168.9 IOPS] <  4428.32 us>
    Random 4KiB (Q=  1, T= 1):    66.335 MB/s [  16195.1 IOPS] <    61.32 us>

Profile: Default
   Test: 1 GiB (x5) [Interval: 5 sec] <DefaultAffinity=DISABLED>
   Date: 2020/07/30 21:45:42
     OS: Windows 10 Professional [10.0 Build 19041] (x64)

Thanks again!

craigr

[craigr]

Ah, I think that it must be because my MB only supports PCI-e 2x, not PCI-e x4.  Does this sound correct to you?

 

Quote

Question

Can you please confirm if X11SSH-F supports M.2 NVMe drives?

 

Answer

It supports NVMe M.2 devices, but be aware it only has a x2 link instead of a x4, so the maximum throughput is reduced from 4 GBps to 2GBps. Note that SATA M.2 drives on this board are not supported.

 

https://www.supermicro.com/support/faqs/faq.cfm?faq=22446

 

So this makes me wonder if I should try and save some money by returning the SN750 and getting a drive that is PCI-e 2x instead.  Any recommendations for a PCI-e NVMe that is 2x and not also SATA?

 

Thanks again,

craigr

[testdasi]

Some pointers

• First and foremost and most importantly: Don't expect benchmark results in real life load!
   
• Your CDM and SIO's CDM aren't line-for-line comparable.
  • Your 1st line - not comparable to SIO (you use Q8T1 sequential, SIO uses Q32T1 sequential)
  • Your 2nd line = SIO's 3rd line (Q1T1 sequential)
  • Your 3rd line - not comparable to SIO (you use Q32T16 4k, SIO uses Q32T1 4k)
  • Your 4th line = SIO's 4th line (Q1T1 4k)
• That is before we even consider you run CDM 7.0.0 and SIO runs CDM 5.2.1
  • CDM website explicitly said "Benchmark result is NOT compatible between different major version."
     
• Running an x4 in x2 PCIe slot (same PCIe generation e.g. 3.0 vs 3.0) will only be a limitation if you can put a load that is higher than the x2 max speed.
  • That is everything will be identical (within margin of error) until you can reach about 1.8 - 1.9 GB/s (theoretically 2GB/s but deducting overhead).
  • In real life load, you will rarely ever get to 1.8GB/s.
    • Most real life consumer load rarely has queue depth above 4 so Q8 is pretty unrealistic and Q32 has zero value outside of bragging rights (or in enterprise settings).
      • There's a reason CDM 7.0 has "Real World Performance" test that only runs Q1T1.
    • Most real life consumer load is random (4k) and outside of copying large media files / iso images rarely ever sequential.
    • Most real life consumer load is usually mixed (simultaneous read + write) instead of purely read or purely write
       
• There is no widely-available PCIe x2 NVMe that is worth buying. x4 is so ubiquitous that economies of scale dictates that it costs manufacturers more to manufacture x2 devices so they have to cut costs elsewhere.

 

So TL;DR: benchmark looks fine. No need to make any changes.

[craigr]

Thanks!  I will sleep easy on this one tonight.

 

Kind regards,

craigr