10Gbe unRAID to unRAID (Peer-to-Peer)

[TechMed]

Hello Everyone,

 

I am having speed issues with my repurposed 10Gbe Mellanox cards setup in/on an unRAID peer-to-peer network; server to server, not server to client. One card is a dual port and one is a single port, and I am transferring files from my one unRAID server to the another unRaid server. I am using Krusader to affect the transfers as I like the easy interface between the two unRAID servers. Using Krusader's indicator in the 'file copy' window, the transfers start out at ~600 MiBs 👏, then in 1 to 2 SECONDS drop to ~125 MiBs 😭. I am transferring between two SSD’s, each rated to sustain ~500 Gbps read/write speeds.

 

While these 10Gbe cards are the third NIC in the rig(s), they are not part of the bd0 which are the two on-board NIC's. They are on an entirely different subnet and are connected via copper SFP+.

 

In reading through other posts, I saw where adjusting the “Desired MTU” to 9000 might help, nope. I also made sure I am in the correct PCIe slot(s). I am (really) guessing here, but it seems like this might have something to do with “caching” since the transfers speeds are fantastic for about 2 seconds, then take a nose dive. This is where my research and skills stop. I would really like to get some better use (speeds) out of these cards. ANY troubleshooting help would be appreciated!

 

HW:

SM – X10DAC

SM – X9SCM-F

Mellanox MNPH29C-XTR (Dual)

Mellanox MNPA19-XTR (Single)

 

unRAID 6.6.6 on each server

 

Additional info: I forgot to mention that when I copy from the spinning drive(s), the speed drops even lower... ~55 MiBs!

Edited December 20, 2018 by TechMed
Additional Info

[JorgeB]

1 hour ago, TechMed said:

I am transferring between two SSD’s, each rated to sustain ~500 Gbps read/write speeds.

What model SSDs are you using? Sometimes people assume up to 500MB/s means 500MB/s sustained writes, very few SSDs are capable of that.

[TechMed]

Crucial MX500's...

 

From Crucial site:

Form Factor: 2.5-inch internal SSD

Total Capacity: 500GB

Warranty: Limited 5-year

Specs: 500GB 2.5-inch internal SSD • SATA 6.0Gb/s • 560 MB/s Read, 510 MB/s Write

Series: MX500

Product Line: Client SSD

Interface: SATA 6.0Gb/s

Device Type: Internal Solid State Drive

[Alphahelix]

The amount of RAM of the receiving unRAID can play a role here. I see in my system that unRAID blast away full speed until my RAM is filled. As Johnni.Black mention the speed of your SSD is also a possibility. Yet with 10Gbe you should see arround 1Gb/sec (theoretical max is 1,25Gb/sec) when filling up the RAM. But only if the sending unRAID can read the data fast enough. So as I see it it can be both the read speed and the write speed that is the bottleneck.

[TechMed]

The Sender has 32Gb and the receiver has 64Gb.

 

I just ran iperf3 as well and the speed is there...

 

 

 

 

 

[JorgeB]

10 minutes ago, TechMed said:

Crucial MX500's...

That's one of the better ones, still it can't sustain 500MB/s, it can do around 400MB/s, are the SSDs being regularly trimmed? If yes test the actual write speeds with the script below, before testing lower the RAM cache to minimum by typing the below on the console to not interfere with the results:

 

sysctl vm.dirty_ratio=2
sysctl vm.dirty_background_ratio=1

 

The copy the script to your flash drive and run the test with:

 

/boot/write_speed_test.sh /mnt/cache/test.dat

 

 

 

 

 

 

 

write_speed_test.sh

Edited December 20, 2018 by johnnie.black

[TechMed]

@johnnie.black I think something is missing here...

 

Or am I just misreading what you wrote (that is a possibility, it is almost 'midnight' for me

[Alphahelix]

Looks to me like the cards are working 9.89 Gbits/sec is in my book close enough.

I think you should look at your storage in both ends. Maybe you could try and create a 11Gb RAM drive on both of the and and send a 10Gb file to test. That way you can test:

RAM->RAM

SSD->RAM

HDD->RAM

RAM->SSD

RAM->HDD

 

With that you can see where to put your shares. Just remember to run a few test for each and average the values.  Just remember that one big file is most of the time faster than many smaller ones.

[Alphahelix]

I have tried myself, but no luck. BUT I have heard that the temp folder in unRAID can use 50% of the free RAM. I do not know if that is something you can use. I see it as the best way to test in you scenario, especially because after a reboot all data in RAM is cleared.

[TechMed]

18 minutes ago, johnnie.black said:

That's one of the better ones, still it can't sustain 500MB/s, it can do around 400MB/s, are the SSDs being regularly trimmed? If yes test the actual write speeds with the script below, before testing lower the RAM cache to minimum by typing the below on the console to not interfere with the results:

 

sysctl vm.dirty_ratio=2
sysctl vm.dirty_background_ratio=1

 

The copy the script to your flash drive and run the test with:

 

/boot/write_speed_test.sh /mnt/cache/test.dat

 

 

 

 

 

 

 

write_speed_test.sh

 

Yes, both drives are being trimmed by Dynamix SSD TRIM plugin.

 

Okay, so let me see if I have this correct:

Copy the script to the 'boot' of the flash drive

go to out to terminal and enter the two 'sysctl...' lines from above

then execute the write_speed_test from terminal

 

I am assuming the script will generate "test.dat' ?

Then post test.dat here?

 

So, other than me fat fingering something, is there any way that these commands or script can fubar either server?

I'm just a bit nervous about the one is all. It has everything on it.

 

Do I need to run anything to reset the RAM cache when done?

[JorgeB]

3 minutes ago, TechMed said:

I am assuming the script will generate "test.dat' ?

Then post test.dat here?

Yes, not need to post, it will display the speeds in the end.

 

4 minutes ago, TechMed said:

Do I need to run anything to reset the RAM cache when done?

It will reset at reboot, or type:

 

sysctl vm.dirty_ratio=20
sysctl vm.dirty_background_ratio=10

 

[TechMed]

Thank you sir!

 

Putty-ing in now...

 

****

Here is the result:

10240000000 bytes (10 GB, 9.5 GiB) copied, 96.7382 s, 106 MB/s
write complete, syncing
removed '/mnt/cache/test.dat'

****

 

Soooo... guess the drive(s) is not as fast as I thought!? How can they be so slow?

And I goofed, the SSD's are the MX300's, but still supposed to be as fast...

 

[JorgeB]

the SSD's are the MX300's

That explains it.

 

 

[TechMed]

okay. Just to be clear, the MX300's are 'spec'd' to be able to do ~500 Mbs, but are only doing ~100 Mbs?

 

 

If so, anyone want to by a few gently used MX300's? 😉

[TechMed]

Any thoughts on truly fast SSD's?

I thought I could trust Crucial...

[JorgeB]

They can write up to 500MB/s, only when the SLC cache is empty, once that is exhausted speed are much slower, though I would still expect better then 100MB/s, but clearly they are what's limiting you're speed.

 

IMO the MX500 and 860EVO are currently the best, though only the 1TB capacities can sustain 500MB/s writes.

[TechMed]

*SIGH*

 

Of course it means new drives now...

 

I am not doubting you at all on your info, just curious how only the 1Tb+ models are the only ones that can sustain the higher speeds, when the spec's read the same for the 250Mb - 2Tb models?

Or does it come down to that old adage of 'truth in advertising'?

[JorgeB]

The larger models have more dies and can spread the writes in parallel, hence why they are faster, though I meant the 500GB models are the smallest capacity that can sustain 500GB/s, this for the Samsung 850 or 860EVO, the Crucial MX500 should be similar, around 400/450MB for the 500GB model, e.g. this is for the 850EVO, 860EVO is similar:

 

 

https://www.anandtech.com/show/8747/samsung-ssd-850-evo-review/2

 

Turbowrite is the small SLC cache that lasts for a few seconds only, after that speed will decrease a lot for the smallest capacity models.

 

 

Edited December 20, 2018 by johnnie.black

[TechMed]

@johnnie.black

(just wanted to say thank you for all the assistance before I forget!

Also, I am writing this during a very boring conference call so, I apologize if it is a bit disconnected)

 

So, if I am reading all this correctly (been to a few sites in between our posts and your clip), what we all need to be paying attention to as we move towards these higher networking speeds, is the Sequential Write speeds, and the Read to a lesser degree, so that we can saturate our networks.

 

Looking at your clip, the 500+Gb Samsung's have the 500+ MB/s Write speeds.

Looking at the one I found (below), all the 2Tb are basically the same.

https://www.tomshardware.com/reviews/crucial-mx500-ssd-review-nand,5390-2.html

 

If that is the case, then this excerpt from the Crucial spec sheet indicates the Sequential Write, across the models, is the same. Although, is that because of the SLC cache and the fact that it is bigger on the larger drives?

But, if the following quote is correct, this "dynamic SLC" could be a nice boost to consistent performance, correct?

"Recent advances in Micron NAND technology enable the SSD firmware to achieve acceleration through on-the-fly switching between SLC and TLC modes to create a high-speed SLC pool that changes in size and location with usage conditions."

 

 

So, if the above is to be believed, then we (unRAID users) should be able to utilize any of these SSD's and still get the 500+ Mbs Sequential Writes.

I for one only need the 500Gb model. BUT, I want the high write speeds. Do you feel the specs from Crucial (above) indicate that it has the ability to consistently and constantly write at ~510 MB/s? I mean there is no point in upgrading if the end result is going to be the same as the MX300's since they have specs (real world) indicating they should be able to do 350-375MB/s Sequential Write speeds.

 

UGH!

 

Liking this resource: https://www.userbenchmark.com/Faq/What-is-sequential-write-speed/45

 

[JorgeB]

38 minutes ago, TechMed said:

"Recent advances in Micron NAND technology enable the SSD firmware to achieve acceleration through on-the-fly switching between SLC and TLC modes to create a high-speed SLC pool that changes in size and location with usage conditions."

From what I understand this works best when the SSD is mostly empty, still it should be able to sustain around 400/450MB/s even when the SLC cache is exhausted.

[1activegeek]

I'm chasing down similar issues with my disks, so thank you for this thread! 

 

I'm running the WD Blue 500GB (as seen in this comparison to the EVO850 500GB: https://ssd.userbenchmark.com/Compare/WD-Blue-3D-500GB-vs-Samsung-850-Evo-500GB/m337874vs3477). By all accounts that I can see, I should be able to be hitting these higher speeds as well, yet my output from that script just showed diminishing returns as the file got larger (see below).

 

Any overall advice on where I could be seeing a different bottleneck? 

 

writing 10240000000 bytes to: /mnt/cache/test.dat
603738+0 records in
603738+0 records out
618227712 bytes (618 MB, 590 MiB) copied, 5.00111 s, 124 MB/s
1136801+0 records in
1136801+0 records out
1164084224 bytes (1.2 GB, 1.1 GiB) copied, 10.0158 s, 116 MB/s
1604982+0 records in
1604982+0 records out
1643501568 bytes (1.6 GB, 1.5 GiB) copied, 15.1463 s, 109 MB/s
2052681+0 records in
2052681+0 records out
2101945344 bytes (2.1 GB, 2.0 GiB) copied, 20.0228 s, 105 MB/s
2518845+0 records in
2518845+0 records out
2579297280 bytes (2.6 GB, 2.4 GiB) copied, 25.0144 s, 103 MB/s
2864068+0 records in
2864068+0 records out
2932805632 bytes (2.9 GB, 2.7 GiB) copied, 30.0163 s, 97.7 MB/s
3243564+0 records in
3243564+0 records out
3321409536 bytes (3.3 GB, 3.1 GiB) copied, 35.0189 s, 94.8 MB/s

[JorgeB]

5 hours ago, 1activegeek said:

I'm running the WD Blue 500GB

Is that the original WD Blue or the newer WD Blue 3D? Those look like pre-3D speeds, but either way the SSD is the limit, get a faster one.

[1activegeek]

Yes they are the 3D models (WDS500G2B0A-00SM50 - from the drive itself). Is it possible there is something else physical on the board or something perhaps that can be holding it back? I'm just confused what is going to be faster, since according to benchmark comparisons between these drives and something like the EVO850's that everyone boasts. Granted I'm using the PEAK benchmark here, but mine was showing close to that 122MB/s mark in the 4k write. 

 

Sorry to be trying to pin suggestions on you, I'm just lost (I think similar to the posts above) about what is being advertised vs delivered and how I can understand what should be performing at the higher levels expected? 

 

[mrbilky]

Samsung has a 5 year warranty and the iops are a bit higher the WD has a 3 year warranty but what I found more revealing was when I looked the two up and compared them the WD drive weighs less than half of what the Samsung 860 evo does not sure if thats a typo but it came in at 1.31 ounces and the Samsung comes in at 3.04 would be interested to know what thats all about

[JorgeB]

51 minutes ago, 1activegeek said:

Is it possible there is something else physical on the board or something perhaps that can be holding it back?

Are you regularly trimming the SSD? Except for that can't think of anything else, except it's not that fast with sustained writes, benchmarks mostly test the small SLC cache, so sometimes can be misleading.