November 25, 20223 yr Hi everybody, I got 2x 10 GbE ethernet cards + a DAC cable to connect my computer to Unraid server (2x Mellanox Technologies MT26448). I tried to run iperf3 and performances are good enough for me, but when I try to copy any file from my PC to my Unraid server, it goes really slow... I'm trying to copy some .vdi drive images from VirtualBox folder (~10-30 gb each one) and they are saved into a SATA3 SSD drive that performs pretty well... The destination drive in Unraid is the RAID-1 SSD cache pool, both SSDs are connected to the SATA 3 port on the motherboard (SuperMicro X10SLL-F) and both of them are almost brand new (Bought both of them 3 weeks ago)... I think the problem can't be in the Windows PC as it has a Ryzen 3950X, 64 gb of ram and all drives are NVME SSD + 1x SATA3 SSD. As you can see here, it seems speed is fine from iperf3 tests (with antivirus -Kaspersky- enabled)... Already tried disabling antivirus, but I cannot understand why it goes at around 50 mb/s... It seems initially speed is qiet good (~900 mb/s) but immediately it drops down to 35-50 mb/s)... I attach also diagnostic zip file, maybe it could help! endys-diagnostics-20221125-0149.zip Thanks everybody! Edited November 25, 20223 yr by endystrike
November 25, 20223 yr Author already tried also, but results didn't improve significantly (~60 mb/s): - enable direct IO - changing MTU/Jumbo size to 9000 both in Windows and Unraid, then rebooted both - checked SATA speed in the Unraid supermicro server Edited November 25, 20223 yr by endystrike
November 25, 20223 yr Author Also turned on disk cache write as it was disabled on all drives (even if there was lo power loss).
November 25, 20223 yr Author I also rerun iperf3 after applying the changes and from both sides it seems 10 gbit works, but copying files the problem still persists and still get ~60 mb/s of avg speed... endys-diagnostics-20221125-0334.zip
November 25, 20223 yr make sure that "flow control" is enabled on all cards (and switches if there are any). Windows has a bad habit to do an initial burst, and if not paced out properly it assumes that the line is crippled on the way and slows down. Also watch the error counters for the cards, Even "direct connection" cables are not safe from faults and maybe you have got one that is not really properly supported by your cards? This slow speed is usually a result of lost packets and therefor needed resends. Iperf is a good tool, but you should retry it with much larger packet sizes to see if a lot of data can be transmitted too.
November 25, 20223 yr Author Hi @MAM59, thank you so much for helping me! I checked and on Windows "Flow control" was already enabled. I honestly don't know where is the problem now, I really don't have any other idea about how to fix this issue...
November 25, 20223 yr That might just be a device limitation, enable turbo write and transfer directly to the array, you should get >100MB/s
November 25, 20223 yr Author 1 hour ago, JorgeB said: That might just be a device limitation, enable turbo write and transfer directly to the array, you should get >100MB/s Thank you for your help @JorgeB Unfortunately, after enabling turbo write, the transfer speed is still the same... Btw, I'm writing to the SSD cache pool, so why turbo write should improve speed in this case? Thank you!
November 25, 20223 yr 1 minute ago, endystrike said: Btw, I'm writing to the SSD cache pool, so why turbo write should improve speed in this case? Because you are are using Kingston SA400, which are some of the slowest SSDs available, not all SSDs can sustain 500MB/s writes, if fact most can't, unfortunately I missed that your array also won't be very fast since there's a 1TB disk, but pretty sure you are seeing low speeds due to device limits, that's why it starts reasonably fast, while it's being cached to RAM then it slows down when it's limited by the device write speed.
November 25, 20223 yr Author 15 minutes ago, JorgeB said: Because you are are using Kingston SA400, which are some of the slowest SSDs available, not all SSDs can sustain 500MB/s writes, if fact most can't, unfortunately I missed that your array also won't be very fast since there's a 1TB disk, but pretty sure you are seeing low speeds due to device limits, that's why it starts reasonably fast, while it's being cached to RAM then it slows down when it's limited by the device write speed. Well, it seems you're right: SSD speed is trashy... I did a test and I tried to copy and paste in the same cache drive an img of a VM... After the first 2-3 minutes, it goes from 300 mb/s to 50-60 mb/s... It seems I've to ask refund to Amazon for the SSDs and get a different model...
November 25, 20223 yr Author @JorgeB any advice for a better SSD? I'm looking for Crucial MX 500 or BX 500 or Samsung 870 Evo, but at this point I wanna be sure I'll get a decent drive before getting something not good again... Thank you! Edited November 25, 20223 yr by endystrike
November 25, 20223 yr Author @JorgeB does the fact SSDs are in a pool of 2 identical drives affect the speed? If there would be only 1 of them would it go faster? Thank you
November 25, 20223 yr And don't expect 500MB/s, 300MB/s would be a good result, possibly more depending on the rest of the hardware.
November 25, 20223 yr Author 42 minutes ago, JorgeB said: And don't expect 500MB/s, 300MB/s would be a good result, possibly more depending on the rest of the hardware. Ok thank you so much, I'll probably try also configuring the cache pool as RAID-0 instead of RAID-1 then, so that the backup time of my PC will decrease a lot...
November 26, 20223 yr Author @JorgeB it seems the problem was due to the Kingston SA400 SSD, with Samsung SSDs speed is fine... Now I've just another problem: SSD temperature... It seems they get hot while transferring files... What's a good threshold value? Thank you!
November 27, 20223 yr I usually set most of my SSDs to 60/65 warning/critical, though some models get other than that, critical temp is usually 70C for most SSDs, so just adjust as needed.
November 27, 20223 yr I know it looks a bit gimmicky, but I put a Icy Box IB-M2HSF-703 on my NVMe. It is a heatsink with a 20mm fan. Idle temps now are 38C. Before installing, it was low 50C at idle, and would get hot enough to throttle down transfer speeds. Edited November 27, 20223 yr by ConnerVT
November 27, 20223 yr Author @ConnerVT thank you for that! Unfortunately I don't have any M2 slot on the mobo, and my pci-e slots are full so I can't put any M2 drive in my build... I had 3 pci-e slot on my Supermicro X10SLL-F: 1x pci 8x in a 16x slot where I put my Nvidia Quadro P620 for Plex 1x pci slot 8x in 8x slot where I put LSI SAS 9211-8i 1x pcie 4x in a 8x slot where I put my Mellanox 10 gbe eth card... So the only solution for me was to use sata ssd and now I set the cache pool as raid-0 to enhance the performances...
November 27, 20223 yr Doh! I assumed the cache was NVMe. Must remember not to post until after I drink my morning coffee. 😆
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.