Slow NvME Cache Pool

[Dr. Ew]

I'm trying to figure out what the issue is with my Cache Pool. I have both 40GbE and 10GbE NIC's, but I mainly utilize 10GbE, so I'll refer to that, for now. 

 

My NvME Cache Pool isn't saturating the 10GbE line. My unassigned array of Spinners does saturate the line though. So, I am deducing it isn't a network issue. 

 

I have 6 2TB NvME's in  a Supermicro server. Client machine is an i9 extreme and/or xeon-w. I have an array of 24 spinners, LSI MegaRAID, attached as an unassigned device. This array hits 1.03Gb/s, no problem. I haven't put more than 2 of the 6 NvME's in the cache pool yet, so i am testing with a RAID 1 config with Intel 760P. It is maxing out aroud 750Mb/s write. 

 

What could be the cause for this slow NvME transfer rate?

[jonp]

Try this: 

 

1. Stop the array.

2. Go to Settings > Global Share Settings. 

3. Turn Enable disk shares to On.

4. Start the array

5. Navigate to your server from the client machine using file explorer.

6. Double click the "cache" share.

7.  Double click the share name you wish to write to.

8.  Write a file to the share and note the performance.

 

Let me know if you saturate the link doing this.

 

All the best,

 

Jon

[Dr. Ew]

On 3/15/2019 at 2:35 AM, jonp said:

Try this: 

 

1. Stop the array.

2. Go to Settings > Global Share Settings. 

3. Turn Enable disk shares to On.

4. Start the array

5. Navigate to your server from the client machine using file explorer.

6. Double click the "cache" share.

7.  Double click the share name you wish to write to.

8.  Write a file to the share and note the performance.

 

Let me know if you saturate the link doing this.

 

All the best,

 

Jon

I’m finally getting around to trying this, this AM. I’ll let you know how it goes.

 

i have a suspicion the raid5 BTRFS isn’t configured properly. Transfer speed is one thing, but when I run a benchmark in a Mc that’s on the Cache pool, It’s only benchmarking at the speed of one drive.

[jonp]

I’m finally getting around to trying this, this AM. I’ll let you know how it goes.
 
i have a suspicion the raid5 BTRFS isn’t configured properly. Transfer speed is one thing, but when I run a benchmark in a Mc that’s on the Cache pool, It’s only benchmarking at the speed of one drive.

Did you figure this out?

Sent from my Pixel 3 XL using Tapatalk

[Dr. Ew]

On 4/2/2019 at 11:58 AM, jonp said:

Did you figure this out?

Sent from my Pixel 3 XL using Tapatalk
 

I finally got to thoroughly test yesterday and today. My trials are up on 3 of 5 servers and I’m not sure what to do. As you can see I’ve got quite a bit in these machines, and while $650 for the licenses won’t kill me, I’d rather not spend on a system I may have to scratch in a few days.

 

 

Testing Methodology

-Windows 10 latest release & MacOS Mojave latest release

-Windows Tested SMB utilizing crystal diskmark on mounted shares, atto benchmrk on mounted shares, and LakeHorn Network Speed Tester.

-MacOS tested SMB and AFP with DiskMark, Black Magic Speed Test, Lakehorn Network Speed Tester.

 

-Conducted 20 tests for each protocol, for each software. Total of 60 tests per protocol. I then removed outliers, and took average.

 

-Test utilizes 1gb file for testing

 

-I also conducted the same tests with 100mb and 5gb file. Initial results were +-10% of the 1gb test, so I decided to not further test with different file sizes, until I get initial input. 

 

The results are very strange.

 

 

 

nAR 1

24 core Xeon, 256gb RAM

Chelsio T580 Nic -> 10/40GbE

Mellanox Connect-X 4 -> 100GbE

Data Array -> 8 12tb Seagate IronWolf Pro

Cache Array -> 6 2tb Intel 660p in RAID 5

Unassigned NvME -> 2tb Intel 660p

LSI HDD Array RAID 6 -> Unassigned -> 24 2tb Seagate Barracuda

LDI SSD Array RAID 6 -> 8 4tb Samsung EVO SSD

 

nAR2

16 core Xeon, 128gb RAM

Chelsio T580 Dual Port 40/10GBe

Mellanox Connectx-3

Data Array -> 8 12tb Seagate IronWolf Pro

Cache Array -> 11 1tb SSD’s Samsung, WD mixed in RAID 5 on LSI HBA

Cache Array 2 -> 2 1tb WD Black RAID 0

 

nAR3

Mellanox ConnectX-3 10gbe Only

16 core Xeon, 128gb RAM

Data Array -> 8 Hitachi 4tb SAS Drives

Cache Array -> 16 480gb SSD’s Samsung, Crucial, WD, PNY mixed 

Unassigned -> 2 Intel 1tb 770p

 

Currently using the i9 client to test Windows and Hackintosh MacOS (t580 NIC), and Testing MacOS Proper with iMacPro. 

 

Last night testing with MacOS on both client machines was abysmal. 

 

On all tests, I was getting max 190Mb/s write and 500Mb/s read. I tested this morning and again just now, to verify results are much better, although nothing changed. 

 

Next post will contain results.

 

[Dr. Ew]

Today’s results:

 

Results listed in Megabytes Per Second Read/Write. Tested sequential read/write.

 

MacOS

nAR1 

Cache Only, AFP -> 430/1020

Share, Cache, AFP -> 229/1004

Share, No Cache, AFP-> 330/990

 

Cache Only SMB -> 550/1050

Share, Cache, SMB -> 530/1012

Share, No Cache, SMB-> 516/1036

 

Unassigned NvME -> 990/1008

 

Nar 1 HDD Array LSI->  1007/1033

nAR 1 SSD Array LSI ->  1050/1050

 

nAR2 

Cache Only ->  540/209

Traditional Share/Cache Yes -> 545/289

Share, No Cache -> 650/225

NvME Unassigned -> 920/980

 

 

nAR 3

Traditional Share/Cache Yes -> 47/275

Share, No Cache ->35/189

NvME Unassigned -> 873/350

 

 

Windows

nAR1 -> 

Share, Cache Yes -> 895/1195

Cache Only -> 779/803

Share, No Cache -> 800/950

nAR 1 HDD Array -> 850/900

nAR1 SSD Array -> 786/1170

Unassigned NvME -> 1000/1010

 

nAR2

Cache Only -> 1030/253

Traditional Share/Cache Yes -> 1030/288

Share, No Cache -> 1020/280

NvME Unassigned -> 1034/1116

 

nAR3

Cache Only -> 34/275

Traditional Share/Cache Yes -> 34/284

Share, No Cache -> 50/304

NvME Unassigned -> 1006/1023

 

 

The most peculiar:

 

nAR1 ->

Slow read speeds across the board.

 

Read speed significantly worse in MacOS.

 

*****unusually high read/write performance out of Data Array. I can’t explain this. 

 

nAR 2 -> 

the opposite is true here, read outperforming write considerably

on both Mac and Windows.

 

very low write speed across the board, except for the unassigned NvME.

 

nAR3 - incredibly low read speed across the board, except for on the unassigned NvME. -> unusually high write speed on data array with no cache

 

not sure what to make of it

Edited April 4, 2019 by Dr. Ew

[Dr. Ew]

Testing again to verify the peculiarities.

[Dr. Ew]

I made some headway. I reworked a few things on nAR1. After re-testing, it looks like I am getting synthetic benchmarks in the 800-1000+ R/W range on both platforms. 

 

Still struggling to get the 40GbE network set, so I can test further. 

 

I re-ran disk benchmark in a windows VM, which is on the cache pool. The benchmark is clocking 11,850 megabytes per second read, spot on to where I was looking to be. However, write speed is clocking in right around 3,000 megabytes per second. Random r/w for the pool is decent, @ 750/600.

Edited April 9, 2019 by Dr. Ew

[Dr. Ew]

And, I restarted the server this morning, and I am back to the same slow transfer speeds on the cache pool. 

 

I've tried both the 6x NvME Cache Pool, and the 16 1tb SSD Cache Pool. Both are realizing roughly the same read/write speed.

 

NvME Cache -> R/W -> 550/285MB/s

2.5" SSD Cache -> R/W -> 525/265MB/S

 

Is anyone else having similar trouble?

[jonp]

That's a heck of a lot of testing you've done, but I don't suspect you're going to get a large amount of feedback from other users in here.  The vast majority of Unraid users run on a 1gbps network with traditional HDDs in the array and SSDs in a software-managed btrfs cache pool.  No RAID controllers; no 10gbe, no 40gbe; no 100gbe; mostly no NVMe (though more users are starting to leverage those).  All of that said, here's what I can tell you about what you're seeing in testing:

 

1)  User shares add overhead to performance.

When you are writing to a share, you're going through the linux path of /mnt/user/ShareX.  This means that all disks that participate in a share will have their relevant data for that share displayed when navigating to that path.  If you navigate to /mnt/diskX/ShareX, you are circumventing the user share file system in favor of direct disk access (same happens if you go to /mnt/cache/ShareX). This will alleviate a lot of overhead and improve data speeds.  You can access disks in the array or the cache without going through /mnt/user by turning on Disk Shares, though you lose the flexibility of writing through user shares as a result.

 

2)  We've tuned the OS for 10gbe performance, but you should still test it.

Using iperf is the best way to verify that you're getting full 10gbe access speeds through the network.  There are many guides online for how to do this, and its a vital step in ensuring maximum performance.  We also haven't tested or tuned anything for more than 10gbe, so not sure what else you or we may need to do to improve things there.

 

3)  You're definitely on the bleeding edge here.

As mentioned at the beginning of the post, a hardware configuration such as this is not typical for 99.999% of the Unraid user community.  We don't market or sell Unraid for use with your type of hardware, though we don't do anything to prevent you from using it either.  Definitely open to feedback on how we can improve though.

 

[JorgeB]

12 hours ago, Dr. Ew said:

2.5" SSD Cache -> R/W -> 525/265MB/S

I get around 700/800MB/s R/W with my SSD cache pool.

 

 

Edited April 11, 2019 by johnnie.black
typo

[jonp]

Something else worth noting:  any tests involving the use of a hardware-assisted RAID aren't really something we can help with.  We can only help with what we code / manage which is the software-defined array and cache pool.

 

Did you do the tests where you were writing directly to the cache pool and not through a user share?

[srfsean]

Have there been any developments in this use case? I have 4 970 Pro NVMe drives in a Raid 6 (I believe) cache pool on 10Gbe network and am seeing zero difference in read write speeds compared to the array. No matter the cache use setting. Occasionally the array has better transfer speeds than the NVMe cache(only a few MB/s more). I’m assuming this is the smb user share overhead. Everything seems to be right around 500Mb/s no matter the settings used. Iperf confirms 10Gbe connection speed is capable of being fully saturated. I’ve tried transferring from RAM disk, SSD and NVMe to array (cache set to Yes, No and Prefer). No matter the setting it’s like the cache does nothing for speed. I understand I could enable disk shares and probably see much better speeds but is there any intention or ability to have unraid function as intended through user shares? Faster transfer to cache and then move to array later. 

[DarkMan83]

Currently i've searched the Forums for exactly this problem i think.

I'm on unraid v6.9-beta25.

For Serverspecs see my sig.

 

I have an NVME cachepool and that one is set to the share i'm transferring to.

I've also tuned the...

vm.dirty_background_ratio: 9%

vm.dirty_ratio: 10%

 

As i have 128 GB of ECC RAM, this should use ~12,8 GB RAM as cache.

The file i'm transferring is ~11 GB.

 

Using: /mnt/user/ShareName

The maximum speed i get is ~500 MB/s and the cache has no IO going on, as the RAM is utilized.

After the file is completly transferred, it gets flushed to the cache at ~500 MB/s.

 

Using: /mnt/cache

If i transfer the same file again i get ~1 GB/s and it is again written to the RAM.

After the file is completly transferred, it gets flushed to the cache at ~2 GB/s.

 

I wonder why the 500 MB/s manifesting as a magic barrier, regardless of what anyone tried.

 

Greetings

Dark

[Andyman]

I was having similar experiance

 

I am on 6.9.0.  I have tested a variety of cachepools and I could never get over 500MB on sequential reads on FIO

 

-1x 500GB Samsung Evo btrfs/xfs

-8x 250GB ssd in RAID 10 btrfs

-8x 120GB ssd in RAID 0 btrfs

-1x 250GB WD black nvme btrfs/xfs

 

I am on 1G lan so I use fio to test on the pools directly.  I eventually figured out I needed to increase iodepth to properly saturate the pools and the speeds are now what I expected them to be

 

 

 

[mata7]

andy can you please tell me how do you to increase iodepth on unraid, im having the same problem, i cant pass 400mb/s when transfer files to the cache, Crucial P5 2TB and intel X520-DA2/X520-SR2, thanks