As title suggest, I'm having a really annoying issue where I cannot get my Windows to Unraid transfers via SMB above 300MB/s using an NVME cache and NVME drive on Windows. This issue has been going on for quite awhile now and across multiple hardware changes (incl. full CPU/Mobo/RAM change). But I finally have some time to dig in and fix it. I've spent the better part of a day troubleshooting. 

 

Log/Diagnostic file is attached. 

 

I've tried all the usual tweaking on Windows and Unraid with no difference.  See more details on that below.  The most odd thing here is I have a spare (well lots of spare) server that I created a brand new Unraid USB on and booted it up with a single 256GB nvme drive and standard Mellanox ConnectX3 card.  I did absolutely no tweaks on the Unraid system. Just simply created an array using the single NVME drive, created and export a share using that array, and did the same 18GB MKV file transfer from Windows.  I immediately hit ~700MB/s on the transfer and it stayed consistent.  I moved that same NVME drive (which is in a PCIE to NVME adapter) to my main server, created a new pool with it, and created a new share that only uses that new pool. Same transfer capped out at 300MB/s---the same speed as the other transfers. 

 

Another oddity here is transferring to a share that uses a SATA SSD as cache has the same exact transfer limit.   

 

Oh and its not the network cards because the card that I used in the new server setup came from my main server. I figured the NIC could be bad so I replaced with a dual SFP+ ConnectX3 card. Also used different SFP+ direc cables with no luck.  Tried swapping to different ports on my Unifi switch, same results.  All three devices (windows PC, main Unraid server, and test server) are connected to the same Unifi pro switch. 

 

Based on the above and below testing, I've concluded it is some sort of configuration error in my main Unraid server. I've look at all settings in the main Unraid server and made sure they matched up to the new test server that had the normal speed.  The server's config has been the same since like 2013/14 when I first started using Unraid. It's transferred from/to multiple USB drives and at least three prior system overhauls. 

 

At this point I'm hoping someone has a saving grace idea. Otherwise the only other option I see is to reset Unraid to factory (while hopefully preserving my Docker and VM items). 

 

Main server details:

1. Tyan S8030GM2NE Board
2. AMD EPYC 7302P 
3. 64GB DDR4 ECC RAM
4. Mellanox dual ConnectX3 10G SFP+ NIC
5. Main array with 6 data and 2 parity drives 
6. WD Red 500GB SSD -- Cache -- Used for docker, appdata, VMs, etc. 
7. Two 1TB nvme drives in RAID 0 (I've tried each on their own)--Samsung 980 (non-pro) and Corsair P3 (non-plus)
8. Silicon Valley Gen3 256GB NVME (installed for testing purposes)

 

Other troubleshooting I've tried:

1. iPerf3 -- I've tested this six ways from sunday and all my network tests look normal
   1. Windows to main Server -- Normal test shows ~9.45 gig transfers
   2. Windows to main Server -- 4 parallel connections total about the same 9.45 gig
   3. Windows to main server -- reverse test has same results
   4. Windows to main server -- Bdir has same results
   5. Windows to test server -- Running all the same tests as above I get practically the same results (test server is fractionally faster)
2. I've done all the normal network tweaking
   1. MTU all on devices and Unifi network set to 9014. Interestingly I did not change the MTU on the test server when it hit ~700MB/s. 
   2. Changed all the settings on Windows NIC (flow control disabled, RSS enabled, enabled all the offload options, etc.). These settings worked for the 700MB/s test transfer. 
3. I've ran the transfers on the main server with all Docker contains stopped, VMs stopped, and then disabling both Docker and VM services. Made absolutely no difference. 
4. Transfers to the SSD Cache, NVME Cache, and test NVME drive all have the same transfer speeds (270-300MB/s) for the same file coming from the same location on Windows nvme drive. 
5. Transferring between the NVME cache and the test NVME drive using Krusader (using shares that use the two caches as primary) I get 500-600 MiB/s which seems plenty normal. 

 

camaro-diagnostics-20240823-2308.zip

Edited August 24, 20241 yr by bryce2113

I have same problem in longtime ago, unable to solve. Since then I haven't use NVMe as cache and use RAM cache and directly write to stripe spinner disk pool.

 

Just make test again after update 6.12.13, it got full speed on SMB writing to NVMe ( already minimize cache level and confirm it flush to NVMe in same time )  ....... Anyway, just FYR

 

Edited August 24, 20241 yr by Vr2Io

What people tend to forget is that TCP and SMB both reserve Bandwith for running connections. "Reserve" means, they half the speed for two clients and so on.

The limits may rise again during longer transfers, but every new client kicking in will instantly lower the throughput of all others.

 

Some say "but hey, I'm alone on this server now!", but usually they are wrong. The reduction also happens for "internal" connections because they also use the LAN layer to get to the data.

I for instance am running a vm to record sat and cable shows from tv. The VM store the files onto a share of the main array where others (cutting programs, encoders and so on) can pick them up an process later on.

Of course this counts in as a user therefore I hardly get over 500MB/s transfers as long as the VM is up and running.

 

So if you wanna benchmark, make sure the box is free of everybody and everything that could affect it.

 

Did you try transferring to a disk share? (or exclusive share)

On 8/24/2024 at 4:20 AM, JorgeB said:

Did you try transferring to a disk share? (or exclusive share)

 

I just turned on Disk shares and copied same file to the main NVME cache and achieved ~700MB/s.  See below image.  What does this point to as the issue? I've created user shares on the main server that only have the same NVME drive for the share and get get the same slow speeds. 

 

On 8/24/2024 at 1:42 AM, MAM59 said:

What people tend to forget is that TCP and SMB both reserve Bandwith for running connections. "Reserve" means, they half the speed for two clients and so on.

The limits may rise again during longer transfers, but every new client kicking in will instantly lower the throughput of all others.

 

Some say "but hey, I'm alone on this server now!", but usually they are wrong. The reduction also happens for "internal" connections because they also use the LAN layer to get to the data.

I for instance am running a vm to record sat and cable shows from tv. The VM store the files onto a share of the main array where others (cutting programs, encoders and so on) can pick them up an process later on.

Of course this counts in as a user therefore I hardly get over 500MB/s transfers as long as the VM is up and running.

 

So if you wanna benchmark, make sure the box is free of everybody and everything that could affect it.

 

As noted, I've run same transfer with VM and Docker services off with the the same exact results.   And I'm the only user and no other services run on the server when VM/Dockers are off. 

1 hour ago, bryce2113 said:

What does this point to as the issue?

Going through FUSE (user shares) is always slower, whenever possible use disk/exclusive shares for better performance.

58 minutes ago, JorgeB said:

Going through FUSE (user shares) is always slower, whenever possible use disk/exclusive shares for better performance.

Why does the test transfer on the other server using user shares achieve the expected ~700MB/s but my main server does not?

Have you checked the CPU usage while transferring? Specifically single core and not just CPU %. Fuse + SMB uses a b*tt load of resources and as single threaded.

 

Not saying this is the problem but might be a part of it.

11 hours ago, bryce2113 said:

Why does the test transfer on the other server using user shares achieve the expected ~700MB/s but my main server does not?

It can vary with the hardware used, but disk shares should still be faster.

15 hours ago, Yock said:

Have you checked the CPU usage while transferring? Specifically single core and not just CPU %. Fuse + SMB uses a b*tt load of resources and as single threaded.

 

Not saying this is the problem but might be a part of it.

 

Yes and the CPU looks to be barely doing anything. Its AMD EPYC 7302P. The test server is running dual Xenon 2697A v4.  Much higher core count in the test server but the IPC on the EPYC trounces it. 

For those transfer speeds what matters most is the CPU single threaded performance, it won't use more than one core.

6 hours ago, JorgeB said:

It can vary with the hardware used, but disk shares should still be faster.

 

It looks like I have a hardware issue of some kind. On the main server I unplugged the HDD array and used the Unraid USB from the test server.  Essentially re-created the test I did with my other server but on my main server.  I had the same ~300MB/s transfer speed whereas on the test server I got ~700MB/s.  

 

I can't explain it either. The BIOS on the main server is up to date and I've even reset all BIOS settings to Optimized Defaults through this process.  The issue has also persisted when I moved Unraid from a dual Xenon 2997 V2 setup to the current AMD EPYC 7302P.  I'm wondering if there is something within Unraid that's not utilizing the EPYC CPU properly? 

3 minutes ago, JorgeB said:

For those transfer speeds what matters most is the CPU single threaded performance, it won't use more than one core.

It's not even pinning one CPU core though during the transfer.  That and the single threaded performance of the EPYC 7302P is only slightly lower than the single threaded performance of the Xenon 2697A v4.  The performance difference in single thread IPC should not equate to a more than 50% decrease in transfer speed? 

 

 

4 minutes ago, bryce2113 said:

The performance difference in single thread IPC should not equate to a more than 50% decrease in transfer speed? 

No, but it's expected to be slower.

On 8/27/2024 at 3:55 PM, bryce2113 said:

 

Yes and the CPU looks to be barely doing anything. Its AMD EPYC 7302P. The test server is running dual Xenon 2697A v4.  Much higher core count in the test server but the IPC on the EPYC trounces it. 

Maybe the CPU is stuck in low power state or something, a simple thing i would try is install the "tips and tweaks" plugin and put the CPU in performance mode to see if it makes any difference, at least then you will know if CPU is the problem, even if the transfers only go up by a few MB/s.

 

While i hope it doesn't come to this, it's pretty simple to backup and restore docker containers with with the "Appdata Backup" plugin

On 8/28/2024 at 10:16 AM, Yock said:

Maybe the CPU is stuck in low power state or something, a simple thing i would try is install the "tips and tweaks" plugin and put the CPU in performance mode to see if it makes any difference, at least then you will know if CPU is the problem, even if the transfers only go up by a few MB/s.

 

While i hope it doesn't come to this, it's pretty simple to backup and restore docker containers with with the "Appdata Backup" plugin

 

Thanks.  This sent me down a whole rabbit whole of tweaking & testing different settings, both in Unraid and in the Bios.  TLDR--I'm not getting almost 400MB/s transfer speed. I think most/all of this was from Bios tweaks to Cstate, ACPI, HSMP support, etc. settings in my board's BIOS. I can confirm with grep that the CPU's cores are not boosting up and down as expected. I did not test with grep before but have a suspicion that the BIOS/board was controlling the the Cstates and boosting.  

 

Interestingly disabling NIC offload in the TipsAndTweaks settings halved my transfer limit to about 200MB/s. 

 

At this point I'm going to leave it alone until I get the itch to dig deeper. Something is still up since I should be able to match the 700MB/s transfer I get on the test server which has slower and much older hardware overall. 

Just out of curiosity i've been doing some testing my self. Running 3 NVME's in a raidz1 configuration.
My server seems to be unable to get any higher than 1.3GB/s in combined transfer speeds, what i mean with that is if i move a file read and write speed combined will never exceed 1.3GB/s, looking at CPU usage no thread is even close to maxed out.

 

Ran a "dd if=/dev/zero of=/mnt/cache/temp/test.img bs=16G count=20 oflag=dsync" a couple of times and got results ranging from 1.2 - 1.8GB/s. Looking at the speed of the disks while test ran they individually hit 1.2GB/s until settling at the before mentioned speeds (probably because of the SSDs internal cache getting full)

 

Copying a file from my PC to the server to a fuse layer folder 600MB/s, moving to an exclusive share i max out my 10Gpbs nic.
Copying a file internally on the server from fuse layer to fuse layer i get 300MB/s, from exclusive to exclusive i get 600MB/s.

Rounded the numbers a little.

My server is on a Ryzen 3900x. Before it became my server i used it as my work/gaming computer and the speeds were way higher than that so should not be a saturated PCIe problem.

 

Makes me think that Unraid has some serious overhead somewhere even on exclusive shares.

 

@bryce2113 Run the "dd if=/dev/zero of=/mnt/cache/temp/test.img bs=16G count=20 oflag=dsync status=progress" (change the destination in the command to fit your server)  and see what the internal results are. Might point to if it's internal or network problem.
Btw. as you can see i'm far from an expert but the result of this should at least be interesting.

 

Edit: Changed some numbers i f.ed up.

Edited September 8, 20241 yr by Yock

Did you ever figure this out? I seem to be running into the same problem. SMB writes to nvme cache seem to cap out at about 350MB/s, however in the reverse direction I can pull my full speed. Iperf3 is showing 9Gbits to Unraid. At some point I had it writing at 700MB/s but I don't know what happened.

Hi,

 

I also had this problem (main problem was slow SMB transfer to my Unraid server (stuck at around 25-30 MB/sec)) and it seems to be finally fixed.

 

My Hardware is not the problem at all (Ryzen 9 / actual Ryzen 7, NVMEs, etc).

 

I have 2.5GBit NICs and also some 2.5GBit Switches (Aliexpress).

 

Tried a lot and it's fixed now. I think changing MTU to 9000 on the Unraid Server + Clients fixed it finally, but here a list of what I did:

 

--------------------------------------------------------------------------------
 

https://www.youtube.com/watch?v=t6MZGboDMZs&t=9s

--> all of suggested tweaks
--------------------------------------------------------------------------------
SMB Performance Tuning

--------------------------------------------------------------------------------
Permit exclusive Shares --> Yes
MTU 9000 (Unraid) + Windows clients

--------------------------------------------------------------------------------

On 9/8/2024 at 12:49 AM, Yock said:

@bryce2113 Run the "dd if=/dev/zero of=/mnt/cache/temp/test.img bs=16G count=20 oflag=dsync status=progress" (change the destination in the command to fit your server)  and see what the internal results are. Might point to if it's internal or network problem.
Btw. as you can see i'm far from an expert but the result of this should at least be interesting.

 

Edit: Changed some numbers i f.ed up.

Picking this back up again. I ran this command, but changed to use a 16GB .mkv file. Max transfer was right around 400MB/s, which is coincidentally the max speed I'm seeing over SMB.  No clue what's going on other than this likely points to some internal issue within Unraid.  Which I pretty much already knew since iPerf results are right around ~9gbs. 

 

After 10+ years on Unraid I'm seriously consider switching to TrueNAS to get speed back. 

On 10/17/2024 at 6:06 PM, bpham said:

Did you ever figure this out? I seem to be running into the same problem. SMB writes to nvme cache seem to cap out at about 350MB/s, however in the reverse direction I can pull my full speed. Iperf3 is showing 9Gbits to Unraid. At some point I had it writing at 700MB/s but I don't know what happened.

Nope, still having the same issue. 

I played around some more and am certain this is likely a FUSE issue. I enabled Exclusive Shares in Global Share Settings. I then created a share using only my NVME Pool has the Primary Storage, Secondary Storage set to None.  SMB transfer was ~700MB/s.  On the same share I selected the Array as the Secondary Storage, which skips Exclusive Shares option (i.e. uses FUSE again). Transfer speeds ~400MB/s. 

 

Not sure there is a great permanent way to avoid FUSE with a share that uses an NVME as primary but relies on Mover and the HDD array for storage. 

7 hours ago, bryce2113 said:

Not sure there is a great permanent way to avoid FUSE with a share that uses an NVME as primary but relies on Mover and the HDD array for storage. 

You can use the disk share.

On 12/9/2024 at 5:32 AM, bryce2113 said:

I played around some more and am certain this is likely a FUSE issue. I enabled Exclusive Shares in Global Share Settings. I then created a share using only my NVME Pool has the Primary Storage, Secondary Storage set to None.  SMB transfer was ~700MB/s.  On the same share I selected the Array as the Secondary Storage, which skips Exclusive Shares option (i.e. uses FUSE again). Transfer speeds ~400MB/s. 

 

Not sure there is a great permanent way to avoid FUSE with a share that uses an NVME as primary but relies on Mover and the HDD array for storage. 

In Unraid Beta 7 you can set the CPU governor to performance, that gains me about 200MB/s.

You can also do it with the "tips and tweaks" plugin.