10Gbe Slower than expected

[Robot]

Hi! I work with 4K video (extensive editing and a lot of exports), and I've always wanted to switch to 10Gbe.
I finally did it! Yay!! I purchased:

- ASUS XG-C100C (For the unRAID server)
- Netgear XS508M-100EUS (as a Switch)
- My working machine is a Mac Mini M1 (which has 10Gbe built in). (Planning on getting a Mac Studio at some point).

Now, my speeds aren't BAD, but I was expecting more. Any ideas what might be going on here?

 

While copying a single 50GB video export, I'm getting speeds around 900MB/s. See image here.
It is saying 8,1Gbps, which should be 1012MB/s, but Mac OS reports between 880 and 920MB/s. Also checks out while timing the transfer.

When running a Blackmagic Speedtest, I'm getting numbers around 800MB/s Write and 1000MB/s read. See here.

Considering 1Gbe is pretty efficient and I was getting regularly ~115MB/s (920Mbps), which is just a 8% loss, I was expecting a maximum of 10% less speed, that is 9Gbps, just a little over 1.1GB/s.

IMPORTANT: This is NOT a disk limitation. My cache disk is a Corsair MP400 4TB drive, which gives speeds over 3GB/s both write and read.

 

Thank you!!

Edited April 19, 2022 by Robot

[1812]

take a look at this: 

 

[Robot]

Hi! Thanks for the response!

 

I was getting excited reading the post, but then I realized that it doesn't really apply to my case, sadly.

I'm already getting the speeds the OP of that threads finally manages to get (a little better, actually).

I ran an iperf3 yesterday, and I got 8.31Gbps, which honestly seemed a little off compared to what you see on the forums.

All my shares are set to public, since only I have access to my network.
BUT, I decided to try out a private share with a user, and running an iperf3 today got me 9.41Gbps...

 

I restarted my Mac and connected again just as a guest, and I'm still getting 9.41Gbps, not sure what happened yesterday.

 

That being said, tough, transfer speeds are still the same. Around 900+ MB/s.

Again, it is not bad, but I expect at least 1.1GB/s from a 9.41Gbps connection.

If anything has any idea what is going on, I would appreciate!
(Can't really try out from a Windows machine as of now)

[Robot]

Hello again,

After looking into this a bit more, I found out several things:

 

SHFS Overhead

When working with shares, UNRAID seems to add to SMB transfers an SHFS Overhead (or something like that, didn't quite understand what it is).

 

I'm not sure what this overhead is used for, but makes a performance hit. Apparently, how bad this hit is, depends on the system itself.

In my case, iperf3 shows a connection of 9.41Gbps, which should translate to over 1.1GB/s, but instead I get around 890 - 920MB/s on average.

 

It's not super bad, but could be better. I guess I can thank my performance impact is around just 15-20%.

 

IMPORTANT NOTE: This only affects writing to UNRAID shares. Reading from them does max out the connection. (As long as the disks allow for it).

 

Writing directly to Cache Disks

Following up with the SHFS Overhead, it turns out that you can indeed get 100% network performance, as long as you accept a lot of compromises.

This is achieved by enabling showing all disks shares, and then mounting your Cache Drives.

 

When transferring files to my cache files, I do indeed get over 1.1GB/s transfer speeds, both writing and reading.

Just to clarify, I did NOT accept those compromises. I just enabled the cache share to test out performance.

 

Weird halt times at the end of copying files

With all these tests, I realized that when copying large files (several GB video files) to my UNRAID Server, once the copy gets to the end, it halts for around 10-20s. Then finishes.

I was copying a 65GB file to my server, and it took around 1min 12sec to get to the end of the progress bar, than it halted there for 20s, then finished.

 

Not sure why this happens, but it's consistent. Haven't tried through Windows, I don't have a Windows PC with 10Gbe connection, not yet at least.

 

Terribly slow transfers of certain files/folders

When copying certain files over the 10Gbe connection, transfer speeds are terribly slow.

For instance, transferring an appdata backup in a .tar file, goes at around 5 - 10MB/s.

I guess I'll just have to live with all these, for now at least.

I wonder if all these issues are caused by the SHFS Overhead.

Edited April 25, 2022 by Robot

[SpencerJ]

Hey @Robot,

 

I'm going to tag @jonp and @bonienl in here to see if they have any recommendations for you.

 

 

[Robot]

Thank you, Spencer!

[jonp]

 

Hi @Robot!

 

Certainly seems like you've been doing your research when it comes to optimizing Unraid for 10gbps performance!!  You've gone down all the paths that I did so many years ago (short of modifying sysctl.conf and adjusting all the values there which we do by default in Unraid OS).  That being said, I think you're actually getting pretty good performance out of the current configuration, all things considered.  Let me expand.

 

SHFS Overhead

You are correct that SHFS does add a bit of overhead in order to operate.  SHFS is what allows us to create "user shares" that can span across multiple disks in the array and multiple cache pools.  Every time you go to write data to a share managed by SHFS (read that as anything that goes through /mnt/user or /mnt/user0), SHFS is involved.  User shares also dictate policies like minimum free space (which determines what disk will receive the next file written to that share), allocation methods, and cache participation.  It is the very thing that makes managing data on Unraid so easy!!  The downside is that there is a bit of overhead, but I'll be honest, you're getting a fraction of the overhead I did years ago when I experimented with SMB and 10gbps connections for a LinusTechTips video.  Over the years, SHFS has made a lot of progress in terms of optimizations and I'm glad to see that the hit isn't as severe as it used to be, though I can understand why you want more.  Your use-case is very demanding.

 

Writing directly to Cache Disks

And once again you are correct in that writing directly to the cache (bypassing SHFS entirely) will deliver full performance as expected.  This is because SHFS doesn't have to step into every transaction and manage it.

 

Weird halt times at the end of copying files

This is news to me, but I have asked the team to look into it and attempt to reproduce it.  If there is a bug, we'll have to reproduce it to figure out how to solve it.

 

Terribly slow transfers of certain files/folders

Please expand on test examples here.  Size of the file, length of the transfer in time, etc.  This isn't something I've witnessed first hand.

 

Longer Term Solution

At some point, maybe we allow you to create a primary pool of storage that isn't dependent on SHFS?  That'd be neat, eh? 

[Robot]

Hi @jonp! Thanks for replying!

 

9 hours ago, jonp said:

SHFS Overhead

[...]
The downside is that there is a bit of overhead, but I'll be honest, you're getting a fraction of the overhead I did years ago when I experimented with SMB and 10gbps connections for a LinusTechTips video. 
[...]

 

I see, and I understand. But yes, as you say, in the end I'm getting pretty good performance, at least on Mac. I hope to also equip my Windows machine with 10Gbe at some point, I hope the overhead loss is as low as with the Mac!

But as you said too, I was expecting to push it even harder

 

9 hours ago, jonp said:

Writing directly to Cache Disks

And once again you are correct in that writing directly to the cache (bypassing SHFS entirely) will deliver full performance as expected.  This is because SHFS doesn't have to step into every transaction and manage it.

 

Yeah... although it's not worth the compromises to be honest

 

9 hours ago, jonp said:

Weird halt times at the end of copying files

This is news to me, but I have asked the team to look into it and attempt to reproduce it.  If there is a bug, we'll have to reproduce it to figure out how to solve it.

This has happened when copying over laaaarge files, I'm speaking >20GB at least. And I'd swear it happened after having copied several files.

I noticed this for the first time when I was doing my first "speed tests" in the 10Gbe connection. I selected a random 50GB video export and copied it over. I deleted it using Krusader and copied again. And again. Some times to different folders and/or shares. Just to check speed. In between copies I might  do something else.

 

I don't know how to exactly reproduce it. I'm copying a 51GB file right now just to test, and speeds are all over the place. They jump between 300 and 900MB/s constantly.

I did the copy again, although through terminal so I could precisely time it:

File is 51GB according to <du -h>, but macos reports 54,88GB.

--> It took 1 minute 58 seconds, which is around 450MB/s depending on what size you choose as correct.

 

9 hours ago, jonp said:

Terribly slow transfers of certain files/folders

Please expand on test examples here.  Size of the file, length of the transfer in time, etc.  This isn't something I've witnessed first hand.

 

I've encountered this issue some times, can't really tell what scenarios exactly, except for one:

If I run a backup in Carbon Copy Cloner from my main disc to a specific folder in my UNRAID server, it takes several hours. Instead, if doing it to an external NVME plugged in via Tunderbolt, it takes around 2 minutes.

I know 10Gbe maxes out at like 1/3 the speed of the external NVME, but even 2 minutes *10 would be just 20 minutes, not 3-5 hours
I abandoned the idea of backup up to my UNRAID and I'm doing everything to this external NVME.

There are more examples... but I can't be precise. The other day I was copying over some files from my server, such as backups made by CA Restore/Backup (usb, vms, appdata), and it was doing the transfer at 5-10MB/s for some reason. I tried copying some of those files now manually, but it hits 150MB/s and then just finishes.

 

9 hours ago, jonp said:

Longer Term Solution

At some point, maybe we allow you to create a primary pool of storage that isn't dependent on SHFS?  That'd be neat, eh? 

 

Well that would be awesome!! Is it a new feature coming in next release?? 😮
 

Thank you!

[JorgeB]

On 4/25/2022 at 1:58 PM, Robot said:

Weird halt times at the end of copying files

Not a Mac user but there was an issue with OSX forcing sync writes with Samba, not sure if current releases are still affected but worth trying:

 

 

[Robot]

On 4/27/2022 at 10:26 AM, JorgeB said:

Not a Mac user but there was an issue with OSX forcing sync writes with Samba, not sure if current releases are still affected but worth trying:

 

 

 

Mmm... interesting indeed!! I do have an M1 Mac Mini running Big Sur.

I'll have to try it out when the issue happens again regularly. It's so random that sometimes it'll happen just half the times.

 

As soon as the issue seems persistent copying some files over, I'll try this.

[PGC77]

I know this thread is over a year old but because I have two same issues like @Robot:

• Weird halt times at the end of copying files
• Terribly slow transfers of certain files/folders

My setup:

Unraid 6.11.5 (no Unraid Cache drive)

macOS 12.6.6 Monterey (Hackintosh) with 1 TB NVME 

 

Everything is connected via 10Gbe over a Netgear XS708E Switch.

I also have a QNAP 872XT 80TB NAS (more to that later in this post)

 

Black Magic Speed Test gives me about 450-500 MB write and 980-1000 MB read speeds.

 

Weird halt times at the end of copying files

When I copy a large video file (45 GB) from my Mac to the Unraid server, the speed is ok but it halts for about 20 seconds at 43 GB, then it continues.

When measuring with BM Speed Test (4 GB file), there is the same phenomena. After clicking start, the write speed goes up to 450-500 MB. Before switching to the read speed test, it seems like the app is frozen for about 20 seconds, then it continuous with the read test.

 

When coping back from Unraid to the Mac, everything is normal.

 

Terribly slow transfers of certain files/folders

I tried to copy a disk image file (317 GB, created with Carbon Copy Cloner) from my Mac to the Unraid server. The write speed is constantyl between 20-25 MB/s). When I copy the same file to my QNAP NAS, I get normal write speeds.

 

If anyone can help, it would be greatly appreciated.