endystrike Posted November 25, 2022 Share Posted November 25, 2022 (edited) Hi everybody, I got 2x 10 GbE ethernet cards + a DAC cable to connect my computer to Unraid server (2x Mellanox Technologies MT26448). I tried to run iperf3 and performances are good enough for me, but when I try to copy any file from my PC to my Unraid server, it goes really slow... I'm trying to copy some .vdi drive images from VirtualBox folder (~10-30 gb each one) and they are saved into a SATA3 SSD drive that performs pretty well... The destination drive in Unraid is the RAID-1 SSD cache pool, both SSDs are connected to the SATA 3 port on the motherboard (SuperMicro X10SLL-F) and both of them are almost brand new (Bought both of them 3 weeks ago)... I think the problem can't be in the Windows PC as it has a Ryzen 3950X, 64 gb of ram and all drives are NVME SSD + 1x SATA3 SSD. As you can see here, it seems speed is fine from iperf3 tests (with antivirus -Kaspersky- enabled)... Already tried disabling antivirus, but I cannot understand why it goes at around 50 mb/s... It seems initially speed is qiet good (~900 mb/s) but immediately it drops down to 35-50 mb/s)... I attach also diagnostic zip file, maybe it could help! endys-diagnostics-20221125-0149.zip Thanks everybody! Edited November 25, 2022 by endystrike Quote Link to comment
endystrike Posted November 25, 2022 Author Share Posted November 25, 2022 (edited) already tried also, but results didn't improve significantly (~60 mb/s): - enable direct IO - changing MTU/Jumbo size to 9000 both in Windows and Unraid, then rebooted both - checked SATA speed in the Unraid supermicro server Edited November 25, 2022 by endystrike Quote Link to comment
endystrike Posted November 25, 2022 Author Share Posted November 25, 2022 Also turned on disk cache write as it was disabled on all drives (even if there was lo power loss). Quote Link to comment
endystrike Posted November 25, 2022 Author Share Posted November 25, 2022 I also rerun iperf3 after applying the changes and from both sides it seems 10 gbit works, but copying files the problem still persists and still get ~60 mb/s of avg speed... endys-diagnostics-20221125-0334.zip Quote Link to comment
MAM59 Posted November 25, 2022 Share Posted November 25, 2022 make sure that "flow control" is enabled on all cards (and switches if there are any). Windows has a bad habit to do an initial burst, and if not paced out properly it assumes that the line is crippled on the way and slows down. Also watch the error counters for the cards, Even "direct connection" cables are not safe from faults and maybe you have got one that is not really properly supported by your cards? This slow speed is usually a result of lost packets and therefor needed resends. Iperf is a good tool, but you should retry it with much larger packet sizes to see if a lot of data can be transmitted too. 1 Quote Link to comment
endystrike Posted November 25, 2022 Author Share Posted November 25, 2022 Hi @MAM59, thank you so much for helping me! I checked and on Windows "Flow control" was already enabled. I honestly don't know where is the problem now, I really don't have any other idea about how to fix this issue... Quote Link to comment
JorgeB Posted November 25, 2022 Share Posted November 25, 2022 That might just be a device limitation, enable turbo write and transfer directly to the array, you should get >100MB/s 1 Quote Link to comment
endystrike Posted November 25, 2022 Author Share Posted November 25, 2022 1 hour ago, JorgeB said: That might just be a device limitation, enable turbo write and transfer directly to the array, you should get >100MB/s Thank you for your help @JorgeB Unfortunately, after enabling turbo write, the transfer speed is still the same... Btw, I'm writing to the SSD cache pool, so why turbo write should improve speed in this case? Thank you! Quote Link to comment
JorgeB Posted November 25, 2022 Share Posted November 25, 2022 1 minute ago, endystrike said: Btw, I'm writing to the SSD cache pool, so why turbo write should improve speed in this case? Because you are are using Kingston SA400, which are some of the slowest SSDs available, not all SSDs can sustain 500MB/s writes, if fact most can't, unfortunately I missed that your array also won't be very fast since there's a 1TB disk, but pretty sure you are seeing low speeds due to device limits, that's why it starts reasonably fast, while it's being cached to RAM then it slows down when it's limited by the device write speed. Quote Link to comment
endystrike Posted November 25, 2022 Author Share Posted November 25, 2022 15 minutes ago, JorgeB said: Because you are are using Kingston SA400, which are some of the slowest SSDs available, not all SSDs can sustain 500MB/s writes, if fact most can't, unfortunately I missed that your array also won't be very fast since there's a 1TB disk, but pretty sure you are seeing low speeds due to device limits, that's why it starts reasonably fast, while it's being cached to RAM then it slows down when it's limited by the device write speed. Well, it seems you're right: SSD speed is trashy... I did a test and I tried to copy and paste in the same cache drive an img of a VM... After the first 2-3 minutes, it goes from 300 mb/s to 50-60 mb/s... It seems I've to ask refund to Amazon for the SSDs and get a different model... Quote Link to comment
endystrike Posted November 25, 2022 Author Share Posted November 25, 2022 (edited) @JorgeB any advice for a better SSD? I'm looking for Crucial MX 500 or BX 500 or Samsung 870 Evo, but at this point I wanna be sure I'll get a decent drive before getting something not good again... Thank you! Edited November 25, 2022 by endystrike Quote Link to comment
Solution JorgeB Posted November 25, 2022 Solution Share Posted November 25, 2022 MX500 and 870EVO are good options, Samsung should be slightly faster. 1 Quote Link to comment
endystrike Posted November 25, 2022 Author Share Posted November 25, 2022 @JorgeB does the fact SSDs are in a pool of 2 identical drives affect the speed? If there would be only 1 of them would it go faster? Thank you Quote Link to comment
JorgeB Posted November 25, 2022 Share Posted November 25, 2022 Should be about the same, if using identical devices. 1 Quote Link to comment
JorgeB Posted November 25, 2022 Share Posted November 25, 2022 And don't expect 500MB/s, 300MB/s would be a good result, possibly more depending on the rest of the hardware. 1 Quote Link to comment
endystrike Posted November 25, 2022 Author Share Posted November 25, 2022 42 minutes ago, JorgeB said: And don't expect 500MB/s, 300MB/s would be a good result, possibly more depending on the rest of the hardware. Ok thank you so much, I'll probably try also configuring the cache pool as RAID-0 instead of RAID-1 then, so that the backup time of my PC will decrease a lot... Quote Link to comment
endystrike Posted November 26, 2022 Author Share Posted November 26, 2022 @JorgeB it seems the problem was due to the Kingston SA400 SSD, with Samsung SSDs speed is fine... Now I've just another problem: SSD temperature... It seems they get hot while transferring files... What's a good threshold value? Thank you! Quote Link to comment
JorgeB Posted November 27, 2022 Share Posted November 27, 2022 I usually set most of my SSDs to 60/65 warning/critical, though some models get other than that, critical temp is usually 70C for most SSDs, so just adjust as needed. Quote Link to comment
ConnerVT Posted November 27, 2022 Share Posted November 27, 2022 (edited) I know it looks a bit gimmicky, but I put a Icy Box IB-M2HSF-703 on my NVMe. It is a heatsink with a 20mm fan. Idle temps now are 38C. Before installing, it was low 50C at idle, and would get hot enough to throttle down transfer speeds. Edited November 27, 2022 by ConnerVT 1 Quote Link to comment
endystrike Posted November 27, 2022 Author Share Posted November 27, 2022 @ConnerVT thank you for that! Unfortunately I don't have any M2 slot on the mobo, and my pci-e slots are full so I can't put any M2 drive in my build... I had 3 pci-e slot on my Supermicro X10SLL-F: 1x pci 8x in a 16x slot where I put my Nvidia Quadro P620 for Plex 1x pci slot 8x in 8x slot where I put LSI SAS 9211-8i 1x pcie 4x in a 8x slot where I put my Mellanox 10 gbe eth card... So the only solution for me was to use sata ssd and now I set the cache pool as raid-0 to enhance the performances... Quote Link to comment
ConnerVT Posted November 27, 2022 Share Posted November 27, 2022 Doh! I assumed the cache was NVMe. Must remember not to post until after I drink my morning coffee. 😆 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.