I have an HPE DL380 Gen9 server with 2 PCIe adapters for my NVMe drives

There are 2x WD Blue SN570 1TB drives setup in a cache pool, using 1 for redundancy

 

I was noticing that when transferring to my shares from Windows I was getting only around ~500 MB/s when it was NVMe -> NVMe transfer.

Running iperf I saw that I was getting ~900MB/s connection so I knew that I should be getting more than 500 MB (not sure why it's not the full 10gig here but not my priority atm)

 

I was unable to get the diskspeed docker container to work with my NVMe drive since it's active in a cache pool so I decided to run a dd speed test on one of my VMs and got the following result:

 

When running dd from Unraid I get more than double the VM speed:

 

Shouldn't my NVMe drives be working at full speed on the VMs and through my SMB shares?

Here are the SMB share speeds I'm getting:

Which seem to align more with the VM speeds for some reason

 

Any insight as to why I'm not getting the full speeds via SMB share and VMs would be great, attached are diagnostics, thank you!

diagnostics-20240227-2317.zip

Try testing with a disk share (or exclusive share).

20 hours ago, JorgeB said:

Try testing with a disk share (or exclusive share).

With a disk share I get pretty much expected speeds via SMB transfer to the drive:

 

Edit: I let it run a little longer and the speeds did not maintain:

 

On my VM I am getting slow speeds still but I didn't change anything on where the VM disk is stored (should I have? I already have it stored under a cache only share so it should be on this disk share if I'm understanding correctly?):

 

Also, is a disk share required to get the full speeds for some reason or was this just a test to make sure network/hardware is capable? I want to make sure I'm able to use my cache -> array share still

Edited February 29, 20242 yr by alexhoopes

3 hours ago, alexhoopes said:

Edit: I let it run a little longer and the speeds did not maintain:

That suggests the device is not able to keep up, initial writes are cached to RAM, but then it can only write as fast as they are flushed to the storage.

17 hours ago, JorgeB said:

That suggests the device is not able to keep up, initial writes are cached to RAM, but then it can only write as fast as they are flushed to the storage.

Do you know what part of my device wouldn't be able to keep up?

 

The drive on the server is rated for 1200MB/s write and it's connected via an x8 PCIe Gen3 lane, I see that ~1200MB/s write when doing the initial dd write test on Unraid (see original post) but not on the VMs or shares itself

6 hours ago, alexhoopes said:

Do you know what part of my device wouldn't be able to keep up?

The NVMe devices.

 

6 hours ago, alexhoopes said:

The drive on the server is rated for 1200MB/s write

I think you'll find they are rated up to 1200MB/s, and once the small pseudo SLC cache is full they will be much slower than that.

 

Jorge is on target. Working as designed. The 1TB Blue 570 has a pretty small static cache, then performance drops off a cliff.  https://www.tomshardware.com/reviews/wd-blue-sn570-review/2.  Your testing seems to concur with the testing from Tom's.

On 3/1/2024 at 1:17 AM, JorgeB said:

The NVMe devices.

 

I think you'll find they are rated up to 1200MB/s, and once the small pseudo SLC cache is full they will be much slower than that.

 

Did not know about SLC prior to this, thank you! I will be shopping for different drives

 

Is it safe to assume this is why the VMs were gettign poor speeds too?

10 hours ago, alexhoopes said:

Is it safe to assume this is why the VMs were gettign poor speeds too?

Possibly.