Dr. Ew Posted March 15, 2019 Share Posted March 15, 2019 I'm trying to figure out what the issue is with my Cache Pool. I have both 40GbE and 10GbE NIC's, but I mainly utilize 10GbE, so I'll refer to that, for now. My NvME Cache Pool isn't saturating the 10GbE line. My unassigned array of Spinners does saturate the line though. So, I am deducing it isn't a network issue. I have 6 2TB NvME's in a Supermicro server. Client machine is an i9 extreme and/or xeon-w. I have an array of 24 spinners, LSI MegaRAID, attached as an unassigned device. This array hits 1.03Gb/s, no problem. I haven't put more than 2 of the 6 NvME's in the cache pool yet, so i am testing with a RAID 1 config with Intel 760P. It is maxing out aroud 750Mb/s write. What could be the cause for this slow NvME transfer rate? Quote Link to comment
jonp Posted March 15, 2019 Share Posted March 15, 2019 Try this: 1. Stop the array. 2. Go to Settings > Global Share Settings. 3. Turn Enable disk shares to On. 4. Start the array 5. Navigate to your server from the client machine using file explorer. 6. Double click the "cache" share. 7. Double click the share name you wish to write to. 8. Write a file to the share and note the performance. Let me know if you saturate the link doing this. All the best, Jon Quote Link to comment
Dr. Ew Posted March 30, 2019 Author Share Posted March 30, 2019 On 3/15/2019 at 2:35 AM, jonp said: Try this: 1. Stop the array. 2. Go to Settings > Global Share Settings. 3. Turn Enable disk shares to On. 4. Start the array 5. Navigate to your server from the client machine using file explorer. 6. Double click the "cache" share. 7. Double click the share name you wish to write to. 8. Write a file to the share and note the performance. Let me know if you saturate the link doing this. All the best, Jon I’m finally getting around to trying this, this AM. I’ll let you know how it goes. i have a suspicion the raid5 BTRFS isn’t configured properly. Transfer speed is one thing, but when I run a benchmark in a Mc that’s on the Cache pool, It’s only benchmarking at the speed of one drive. Quote Link to comment
jonp Posted April 2, 2019 Share Posted April 2, 2019 I’m finally getting around to trying this, this AM. I’ll let you know how it goes. i have a suspicion the raid5 BTRFS isn’t configured properly. Transfer speed is one thing, but when I run a benchmark in a Mc that’s on the Cache pool, It’s only benchmarking at the speed of one drive.Did you figure this out?Sent from my Pixel 3 XL using Tapatalk Quote Link to comment
Dr. Ew Posted April 4, 2019 Author Share Posted April 4, 2019 On 4/2/2019 at 11:58 AM, jonp said: Did you figure this out? Sent from my Pixel 3 XL using Tapatalk I finally got to thoroughly test yesterday and today. My trials are up on 3 of 5 servers and I’m not sure what to do. As you can see I’ve got quite a bit in these machines, and while $650 for the licenses won’t kill me, I’d rather not spend on a system I may have to scratch in a few days. Testing Methodology -Windows 10 latest release & MacOS Mojave latest release -Windows Tested SMB utilizing crystal diskmark on mounted shares, atto benchmrk on mounted shares, and LakeHorn Network Speed Tester. -MacOS tested SMB and AFP with DiskMark, Black Magic Speed Test, Lakehorn Network Speed Tester. -Conducted 20 tests for each protocol, for each software. Total of 60 tests per protocol. I then removed outliers, and took average. -Test utilizes 1gb file for testing -I also conducted the same tests with 100mb and 5gb file. Initial results were +-10% of the 1gb test, so I decided to not further test with different file sizes, until I get initial input. The results are very strange. nAR 1 24 core Xeon, 256gb RAM Chelsio T580 Nic -> 10/40GbE Mellanox Connect-X 4 -> 100GbE Data Array -> 8 12tb Seagate IronWolf Pro Cache Array -> 6 2tb Intel 660p in RAID 5 Unassigned NvME -> 2tb Intel 660p LSI HDD Array RAID 6 -> Unassigned -> 24 2tb Seagate Barracuda LDI SSD Array RAID 6 -> 8 4tb Samsung EVO SSD nAR2 16 core Xeon, 128gb RAM Chelsio T580 Dual Port 40/10GBe Mellanox Connectx-3 Data Array -> 8 12tb Seagate IronWolf Pro Cache Array -> 11 1tb SSD’s Samsung, WD mixed in RAID 5 on LSI HBA Cache Array 2 -> 2 1tb WD Black RAID 0 nAR3 Mellanox ConnectX-3 10gbe Only 16 core Xeon, 128gb RAM Data Array -> 8 Hitachi 4tb SAS Drives Cache Array -> 16 480gb SSD’s Samsung, Crucial, WD, PNY mixed Unassigned -> 2 Intel 1tb 770p Currently using the i9 client to test Windows and Hackintosh MacOS (t580 NIC), and Testing MacOS Proper with iMacPro. Last night testing with MacOS on both client machines was abysmal. On all tests, I was getting max 190Mb/s write and 500Mb/s read. I tested this morning and again just now, to verify results are much better, although nothing changed. Next post will contain results. Quote Link to comment
Dr. Ew Posted April 4, 2019 Author Share Posted April 4, 2019 (edited) Today’s results: Results listed in Megabytes Per Second Read/Write. Tested sequential read/write. MacOS nAR1 Cache Only, AFP -> 430/1020 Share, Cache, AFP -> 229/1004 Share, No Cache, AFP-> 330/990 Cache Only SMB -> 550/1050 Share, Cache, SMB -> 530/1012 Share, No Cache, SMB-> 516/1036 Unassigned NvME -> 990/1008 Nar 1 HDD Array LSI-> 1007/1033 nAR 1 SSD Array LSI -> 1050/1050 nAR2 Cache Only -> 540/209 Traditional Share/Cache Yes -> 545/289 Share, No Cache -> 650/225 NvME Unassigned -> 920/980 nAR 3 Traditional Share/Cache Yes -> 47/275 Share, No Cache ->35/189 NvME Unassigned -> 873/350 Windows nAR1 -> Share, Cache Yes -> 895/1195 Cache Only -> 779/803 Share, No Cache -> 800/950 nAR 1 HDD Array -> 850/900 nAR1 SSD Array -> 786/1170 Unassigned NvME -> 1000/1010 nAR2 Cache Only -> 1030/253 Traditional Share/Cache Yes -> 1030/288 Share, No Cache -> 1020/280 NvME Unassigned -> 1034/1116 nAR3 Cache Only -> 34/275 Traditional Share/Cache Yes -> 34/284 Share, No Cache -> 50/304 NvME Unassigned -> 1006/1023 The most peculiar: nAR1 -> Slow read speeds across the board. Read speed significantly worse in MacOS. *****unusually high read/write performance out of Data Array. I can’t explain this. nAR 2 -> the opposite is true here, read outperforming write considerably on both Mac and Windows. very low write speed across the board, except for the unassigned NvME. nAR3 - incredibly low read speed across the board, except for on the unassigned NvME. -> unusually high write speed on data array with no cache not sure what to make of it Edited April 4, 2019 by Dr. Ew Quote Link to comment
Dr. Ew Posted April 4, 2019 Author Share Posted April 4, 2019 Testing again to verify the peculiarities. Quote Link to comment
Dr. Ew Posted April 8, 2019 Author Share Posted April 8, 2019 (edited) I made some headway. I reworked a few things on nAR1. After re-testing, it looks like I am getting synthetic benchmarks in the 800-1000+ R/W range on both platforms. Still struggling to get the 40GbE network set, so I can test further. I re-ran disk benchmark in a windows VM, which is on the cache pool. The benchmark is clocking 11,850 megabytes per second read, spot on to where I was looking to be. However, write speed is clocking in right around 3,000 megabytes per second. Random r/w for the pool is decent, @ 750/600. Edited April 9, 2019 by Dr. Ew Quote Link to comment
Dr. Ew Posted April 10, 2019 Author Share Posted April 10, 2019 And, I restarted the server this morning, and I am back to the same slow transfer speeds on the cache pool. I've tried both the 6x NvME Cache Pool, and the 16 1tb SSD Cache Pool. Both are realizing roughly the same read/write speed. NvME Cache -> R/W -> 550/285MB/s 2.5" SSD Cache -> R/W -> 525/265MB/S Is anyone else having similar trouble? Quote Link to comment
jonp Posted April 10, 2019 Share Posted April 10, 2019 That's a heck of a lot of testing you've done, but I don't suspect you're going to get a large amount of feedback from other users in here. The vast majority of Unraid users run on a 1gbps network with traditional HDDs in the array and SSDs in a software-managed btrfs cache pool. No RAID controllers; no 10gbe, no 40gbe; no 100gbe; mostly no NVMe (though more users are starting to leverage those). All of that said, here's what I can tell you about what you're seeing in testing: 1) User shares add overhead to performance. When you are writing to a share, you're going through the linux path of /mnt/user/ShareX. This means that all disks that participate in a share will have their relevant data for that share displayed when navigating to that path. If you navigate to /mnt/diskX/ShareX, you are circumventing the user share file system in favor of direct disk access (same happens if you go to /mnt/cache/ShareX). This will alleviate a lot of overhead and improve data speeds. You can access disks in the array or the cache without going through /mnt/user by turning on Disk Shares, though you lose the flexibility of writing through user shares as a result. 2) We've tuned the OS for 10gbe performance, but you should still test it. Using iperf is the best way to verify that you're getting full 10gbe access speeds through the network. There are many guides online for how to do this, and its a vital step in ensuring maximum performance. We also haven't tested or tuned anything for more than 10gbe, so not sure what else you or we may need to do to improve things there. 3) You're definitely on the bleeding edge here. As mentioned at the beginning of the post, a hardware configuration such as this is not typical for 99.999% of the Unraid user community. We don't market or sell Unraid for use with your type of hardware, though we don't do anything to prevent you from using it either. Definitely open to feedback on how we can improve though. Quote Link to comment
JorgeB Posted April 11, 2019 Share Posted April 11, 2019 (edited) 12 hours ago, Dr. Ew said: 2.5" SSD Cache -> R/W -> 525/265MB/S I get around 700/800MB/s R/W with my SSD cache pool. Edited April 11, 2019 by johnnie.black typo Quote Link to comment
jonp Posted April 11, 2019 Share Posted April 11, 2019 Something else worth noting: any tests involving the use of a hardware-assisted RAID aren't really something we can help with. We can only help with what we code / manage which is the software-defined array and cache pool. Did you do the tests where you were writing directly to the cache pool and not through a user share? Quote Link to comment
srfsean Posted June 18, 2020 Share Posted June 18, 2020 Have there been any developments in this use case? I have 4 970 Pro NVMe drives in a Raid 6 (I believe) cache pool on 10Gbe network and am seeing zero difference in read write speeds compared to the array. No matter the cache use setting. Occasionally the array has better transfer speeds than the NVMe cache(only a few MB/s more). I’m assuming this is the smb user share overhead. Everything seems to be right around 500Mb/s no matter the settings used. Iperf confirms 10Gbe connection speed is capable of being fully saturated. I’ve tried transferring from RAM disk, SSD and NVMe to array (cache set to Yes, No and Prefer). No matter the setting it’s like the cache does nothing for speed. I understand I could enable disk shares and probably see much better speeds but is there any intention or ability to have unraid function as intended through user shares? Faster transfer to cache and then move to array later. Quote Link to comment
DarkMan83 Posted July 24, 2020 Share Posted July 24, 2020 Currently i've searched the Forums for exactly this problem i think. I'm on unraid v6.9-beta25. For Serverspecs see my sig. I have an NVME cachepool and that one is set to the share i'm transferring to. I've also tuned the... vm.dirty_background_ratio: 9% vm.dirty_ratio: 10% As i have 128 GB of ECC RAM, this should use ~12,8 GB RAM as cache. The file i'm transferring is ~11 GB. Using: /mnt/user/ShareName The maximum speed i get is ~500 MB/s and the cache has no IO going on, as the RAM is utilized. After the file is completly transferred, it gets flushed to the cache at ~500 MB/s. Using: /mnt/cache If i transfer the same file again i get ~1 GB/s and it is again written to the RAM. After the file is completly transferred, it gets flushed to the cache at ~2 GB/s. I wonder why the 500 MB/s manifesting as a magic barrier, regardless of what anyone tried. Greetings Dark Quote Link to comment
Andyman Posted October 28, 2020 Share Posted October 28, 2020 I was having similar experiance I am on 6.9.0. I have tested a variety of cachepools and I could never get over 500MB on sequential reads on FIO -1x 500GB Samsung Evo btrfs/xfs -8x 250GB ssd in RAID 10 btrfs -8x 120GB ssd in RAID 0 btrfs -1x 250GB WD black nvme btrfs/xfs I am on 1G lan so I use fio to test on the pools directly. I eventually figured out I needed to increase iodepth to properly saturate the pools and the speeds are now what I expected them to be Quote Link to comment
mata7 Posted March 16, 2021 Share Posted March 16, 2021 andy can you please tell me how do you to increase iodepth on unraid, im having the same problem, i cant pass 400mb/s when transfer files to the cache, Crucial P5 2TB and intel X520-DA2/X520-SR2, thanks Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.