Slow SSD cache speeds

[Typrix]

I've recently set up my first Unraid build and I am experiencing slow read speeds when using the Cache, even within the same machine.

 

Here's the setup (Unraid 6.8.3):

 

Unraid Cache Pool (Raid 0 encrypted btrfs)

• Samsung 860 EVO
• Samsung 860 EVO

VM — Windows 10 (same machine)

• Samsung 970 EVO Plus (NVMe controller passthrough)

Here are the speeds I observed:

 

(Write) VM > Cache: 600-700 MB/s

(Read) Cache > VM: 150-200 MB/s (with dips to 25-50 MB/s)

(Control) VM > VM: 2+ GB/s

 

Transfers within the VM seem fine so it's probably something about the Unraid setup itself. Also I find it interesting that the read speeds are much slower than the write speeds. What gives?

 

Let me know if there is any other information I can provide that could help.

 

Edited August 12, 2020 by Typrix

[testdasi]

It's generally a good idea to attach diagnostics (Tools -> Diagnostics -> attach full zip file in your next post).

 

Do you really need encryption? It's hard to gauge if performance is expected or not with encryption variable thrown in.

 

Also I'm not sure about what performance to expect from your 960 Evo Plus. I remember the 960 Evo was a very long time ago. Not sure what's the diff with the seemingly newer "Plus" version. Cache -> VM would be subjected to any deficiency in write speed of the NVMe. VM->VM would be cached in RAM so it may be less of an impact.

 

Lastly, Samsung SSD seems to benefit from 1MiB alignment, which is only implemented in 6.9.0 (currently beta25 but I am using it in my production server as it's quite stable).

The 1MiB alignment is NOT backwards compatible with pre-6.9.0 Unraid + you will need to wipe the SSD's to change partition alignment so keep those things in mind if you wanna try.

[testdasi]

At around what time did you do your test? You syslog has several instances of balance and mover happening so could just be things happening at the same time?

 

With regards to 1MiB alignment, it is applicable particularly to Unraid because Unraid prior to 6.9.0 uses the 4k alignment for both HDD and SSD. While I don't totally understand the exact mechanism, that 4k alignment doesn't go well with SSD's, and particularly with Samsung ones. So moving to 1MiB alignment seems to help with performance, CPU load during IO and general wear.

 

Windows defaults to 1MiB alignment so if you formatted your NVMe under Windows then it would be correctly aligned.

[Cabbit]

I'm having very similar issues - was going to make a new thread but I think we have the same problem.

Using a 1TB NVME drive as cache and have another as a part of the array. Testing transfers on my gaming PC that also has an NVME drive.

Writing to either the cache or directly to the array drive results in reads of 4Gbps and Writes of 7Gbps.

I was on 6.8.3 and just upgraded to 6.9.0-beta25 - however this only made things worse. Dropping reads from 5 to 4gbps.

 

A single stream iPerf yields around 8-9Gbps, two streams maxes it out fully (9-9.5gbps)

 

My NVME drives are MBR 4K - I'm not really sure how to format them to 1MiB alignment. Any input on that would be awesome!

[testdasi]

 

2 hours ago, Cabbit said:

I'm having very similar issues - was going to make a new thread but I think we have the same problem.

Using a 1TB NVME drive as cache and have another as a part of the array. Testing transfers on my gaming PC that also has an NVME drive.

Writing to either the cache or directly to the array drive results in reads of 4Gbps and Writes of 7Gbps.

I was on 6.8.3 and just upgraded to 6.9.0-beta25 - however this only made things worse. Dropping reads from 5 to 4gbps.

 

A single stream iPerf yields around 8-9Gbps, two streams maxes it out fully (9-9.5gbps)

 

My NVME drives are MBR 4K - I'm not really sure how to format them to 1MiB alignment. Any input on that would be awesome!

You should read the 6.9.0-beta25 release note. It has information regarding how to quickly wipe your SSD to reformat with 1MiB alignment.

Repeat: WIPE your SSD i.e. all data is gone so make sure you have things backed up.

 

With regards to slower network speed, check your xml to see if you are using virtio-net (virtual) network adapter. Change it to virtio + change your machine type to v5.0 (e.g. i440fx-5.0 or Q35-5.0).

 

[Cabbit]

1 hour ago, testdasi said:

 

You should read the 6.9.0-beta25 release note. It has information regarding how to quickly wipe your SSD to reformat with 1MiB alignment.

Repeat: WIPE your SSD i.e. all data is gone so make sure you have things backed up.

 

With regards to slower network speed, check your xml to see if you are using virtio-net (virtual) network adapter. Change it to virtio + change your machine type to v5.0 (e.g. i440fx-5.0 or Q35-5.0).

 

 

Awesome thanks, was able to reformat with 1MiB.

However it didn't seem to improve the transfer speeds.

 

What I meant was my computer transferring files over SMB to a share, not VM.

My computer can iPerf to the server at a full 10G, but files are only transferring at 5-7Gbps.

Might open a new thread then.

[trurl]

2 hours ago, Cabbit said:

transferring files over SMB to a share

Are you sure the share you are transferring to is cached? The default is NO.

[Cabbit]

54 minutes ago, trurl said:

Are you sure the share you are transferring to is cached? The default is NO.

Yep, it's cached