Uncast Episode 5: All about Unraid 6.10 and 10gbps

[SpencerJ]

 

On episode 5 of the Uncast pod, join @jonp for a sneak peek at some of the security changes being made in Unraid 6.10.

 

Also, there is a deep-dive discussion about 10gbps performance over SMB on Unraid.

 

Need a refresher on Unraid security best practices? Be sure to check out the security best practices blog.

Curious about SMB Multichannel Support? Check this out!

[JorgeB]

On 7/2/2021 at 9:18 PM, SpencerJ said:

there is a deep-dive discussion about 10gbps performance over SMB on Unraid

Enjoyed the podcast, but IMHO more important than using other things to get around the performance penalty introduced by user shares would be to try and improve that, for example I still need to use disk shares for doing internal transfers if I want good performance, and any SMB improvements won't help with that, e.g., transfer of the same folder contents (16 large files) from one pool to another done with pv, first using disk shares, then using user shares:

 

46.4GiB 0:00:42 [1.09GiB/s] [==============================================>] 100%

 

46.4GiB 0:02:45 [ 286MiB/s] [==============================================>] 100%

 

If the base user shares performance could be improved it would also benefit SMB and any other transfers.

 

 

 

 

 

 

[bonienl]

5 hours ago, JorgeB said:

it would also benefit SMB

Yet, in my testing I get a transfer rate of 800 MB/s when copying a 80 GB file from Windows (nvme disk) to a user share on Unraid (nvme RAID0 pool)

 

[JorgeB]

Just now, bonienl said:

Yet, in my testing I get a transfer rate of 800 MB/s when copying a 80 GB file from Windows (nvme disk) to a user share on Unraid (nvme RAID0 pool)

Yeah, user shares performance can vary a lot, I also had some servers/configs in the past were I could get around 600-800MB/s, but lately this is what I get in one of my main servers, there have been a lot o posts from other users with the same issue, between 300-500MB/s when using user shares, 1GB/s+ with disk shares.

[bonienl]

5 minutes ago, JorgeB said:

... between 300-500MB/s when using user shares,

Yup, but this is without the SMB multi channel feature that Jon is talking about

 

Spoiler....

[JorgeB]

14 hours ago, bonienl said:

but this is without the SMB multi channel feature

I don't dispute that, in fact like mentioned I've gotten similar results in the past, but it's not always possible, curiously user shares are currently still faster for me when using SMB than when doing an internal server transfer:

 

 

This is my other main server, better but still not great:

 

 

 

My point was that this has been a known issue for many years now, that affects some users particularly bad, here's just one example, and if this could be improved it would be much better than a setting that will only help SMB, also last time I checked Samba SMB multichannel was still considered experimental, though it should be mostly OK by now, but of course if it's currently not possible to fix the base user share performance than any other improvements are welcome, even if they don't help every type of transfer.

[Dazog]

Would be nice to have this enabled even as a beta to try out (io_uring):

 

 

 

Edited July 8, 2021 by Dazog

[falconexe]

On 7/6/2021 at 3:21 AM, JorgeB said:

 

My point was that this has been a known issue for many years now, that affects some users particularly bad, here's just one example

 

Thanks for pointing out my thread. This is THE ONLY thing that I am disappointed in with UNRAID since 2014. This still plagues me today, unless I use the crazy work around that I discussed in that thread. Without it, I never break 400MB/s, which is particularly sad when you have all the necessary hardware in place to hit 10Gb/s.

[glennv]

Waiting eagerly for the announced blog post with instructions for testers to play with smb multi chan to finaly get all the juice from my 10G . Hoping its not a windows only thing ...... ( osx, linux only user)

[jonp]

On 8/5/2021 at 5:03 PM, glennv said:

Waiting eagerly for the announced blog post with instructions for testers to play with smb multi chan to finaly get all the juice from my 10G . Hoping its not a windows only thing ...... ( osx, linux only user)

 

I'll work on that this week   It shouldn't be a Windows-only thing.

[SpencerJ]

On 8/5/2021 at 3:03 PM, glennv said:

Waiting eagerly for the announced blog post with instructions for testers to play with smb multi chan to finaly get all the juice from my 10G . Hoping its not a windows only thing ...... ( osx, linux only user)

 

Here you are @glennv: https://unraid.net/blog/how-to-beta-test-smb-multi-channel-support

[Lilarcor]

Will I get benefits if my server has 2.5Gb NIC, or it’s for 10Gb only?

[mgutt]

On 7/5/2021 at 2:43 PM, JorgeB said:

If the base user shares performance could be improved it would also benefit SMB and any other transfers.

 

We have a similar discussion in the German forums and we have multiple users with full speed while using user shares:

https://forums.unraid.net/topic/113345-smb-leistung-mangelhaft/?tab=comments#comment-1031606

 

At the moment we are not sure if this is related to specific NICs or the new Unraid version.

 

aio read size = 1
aio write size = 1

 

Why is this disabled in smb.conf of Unraid?

 

 

Regarding my tests this option reduces performance if disabled. So I would say it should be enabled by default and not only for SMB Multichannel scenarios.

 

PS You did not mention in the blog article that SMB Multichannel alone does not work. The adapters on both sides need RSS support. Else you don't benefit from Multichannel from a single 10G connection (but if you use multiple 10G connections).

 

RSS is auto-detected since Samba 4.13, so Unraid 6.10 is a requirement. More details in my Guide:

 

On 8/17/2021 at 12:37 AM, Lilarcor said:

Will I get benefits if my server has 2.5Gb NIC, or it’s for 10Gb only?

 

There are two different things to know:

 

1.) SMB Multichannel enables transfers across multiple channels. This means if you have multiple ethernet ports in the same ip range, SMB automatically splits a file transfer and use all ports at the same time. By that its possible to connect a server with two 1G ports and a client with one 2.5G and transfer with ~200MB/s.

 

2.) The next point is RSS (receive side scaling). RSS allows to split transfers across multiple cpu cores. By that the a 10G transfer does not hit the cpu core limit which bottlenecks the transfer speed. As an SSD is faster in writing multiple threads, even 1G connections will benefit a little bit. Note: RSS works only if the network adapter supports it on both sides and multichannel is enabled. RSS will slow down transfer speeds if the target is a HDD. So consider installing more RAM and raising "vm.dirty_ratio" if you like to transfer directly to HDDs.

 

So what does it mean for your scenario:

It depends. Lets say your server and client both use 2.5G. Then enabling SMB Multichannel alone wouldn't enhance the transfer speeds. But if both support RSS, the transfers will be split across multiple cores and so you benefit a little bit from parallel writes to your SSD, which can be more useful if you transfer very much tiny files.

 

Or let's say our server has two 1G connections and your client 2.5G. Then enabling SMB Multichannel will boost the transfer to ~200MB/s. In this scenario RSS is not needed. PS I tested this scenario with 2x 1G (server) and 1x 10G (client) and it worked only after I disabled RSS on the client side as my servers ethernet adapters did not support RSS (mixed scenarios are not allowed!).

[JorgeB]

17 minutes ago, mgutt said:

Why is this disabled in smb.conf of Unraid?

Because with some disks performance is much worse if it's enable:

https://forums.unraid.net/bug-reports/prereleases/unraid-os-version-690-beta30-available-r1076/?do=findComment&comment=11034

 

[mgutt]

2 minutes ago, JorgeB said:

Because with some disks performance is much worse if it's enable:

Ok, yes. The reason could be the aio max threads setting. Its default is 100. So using an HDD as target, which has a bad I/O performance, could slow down the transfer.

 

I think a better solution would be to test a smaller aio max threads value and if it solves the issue, I would suggest to use the small value by default and the higher value is set, after the user added a cache pool. It still means a reduced performance for HDD transfers, but I think those users won't do this often.

 

Or it will be added as an option to the SMB settings (as SMB Multichannel should be as well).

[JorgeB]

1 hour ago, mgutt said:

Ok, yes. The reason could be the aio max threads setting. Its default is 100. So using an HDD as target, which has a bad I/O performance, could slow down the transfer.

The problem are the reads, not the writes, and just setting it to 1 would cause that.

[mgutt]

46 minutes ago, JorgeB said:

just setting it to 1 would cause that.

1 means something like "enabled" ("Samba will read from files asynchronously when the request size is bigger than this value") and 0 disabled. That's why I said aio max threads. This setting seems to have a different influence:

https://www.samba.org/samba/docs/current/man-html/smb.conf.5.html#AIOMAXTHREADS

[dopeytree]

I don't understand how you can be achieving near 10Gbps transfers with no raid setup as single spinning disk would achieve around 110MB/s possible up to 220MB/s for a fast drive so top out around the 1gig ethernet connection which is 125MB/s...

 

Testing throughput to SSD cache seems silly.

 

Is unraid going to go down the path of RAID5 for sustained 10Gbps transfers?

 

This is where TrueNAS Scale comes into play...

 

Edited August 21, 2022 by dopeytree

[ChatNoir]

1 hour ago, dopeytree said:

Testing throughput to SSD cache seems silly.

Why ?

 

Pools can be used for more than simply caching data that will end up on an Array of HDDs.

Many people use multiple pools with diverse configuration and use case. They can be Single or Multiple drives, using Sata SSDs, NVMEs or HDDs.

 

And if someone has a relevant usecase, why not use one of the appropriated BRTFS RAID profile to focus on speed, capacity, redundancy (or a mix of both).

I can easily imagine a RAID1 or RAID10 of NVME SSDs use to host work files accessed over a 10G LAN.

[dopeytree]

I'm just down the rabbit hole comparing options.. I have gone for a 10GbE setup around the home and want to make sure I don't regret going with unraid in a year or 2...

 

It's going to be a video media file server (backing up drone video work might be 100Gb x4 a week) with built in backup, windows gaming VM & occasionally I'll edit 4k prores video from it on my m1 MacBook pro over the network.

Probably 2.5GbE £20 as I'm not paying £300 for a 10GbE thunderbolt adapter for the mac.

 

I just picked up 4x 12tb normal drives for £550

vs

1x 4tb SSD drive is at least £350 and that's for a sata ssd. An m.2 would be a lot more.

 

Realistically most folks have 2x m.2 slots so they need the biggest they can afford kinda like the parity drive.

I've got a 2tb m.2 laying around and a spare 256gb m.2

 

I suppose realistically having 1tb cache for video editing will work as I usually edit on usb3 1tb ssd's so 2tb should do for now.

I then want the video cache share to be backed up / mirrored to a usb ssd that I can take with me when I'm away.

 

Edited August 21, 2022 by dopeytree

[BVD]

On 7/6/2021 at 4:21 AM, JorgeB said:

I don't dispute that, in fact like mentioned I've gotten similar results in the past, but it's not always possible, curiously user shares are currently still faster for me when using SMB than when doing an internal server transfer:

 

<...>

 

My point was that this has been a known issue for many years now, that affects some users particularly bad, here's just one example, and if this could be improved it would be much better than a setting that will only help SMB, also last time I checked Samba SMB multichannel was still considered experimental, though it should be mostly OK by now, but of course if it's currently not possible to fix the base user share performance than any other improvements are welcome, even if they don't help every type of transfer.

 

Did you ever track this down to determine the cause? I don't have any systems which exhibit this behavior, but I'd be curious to dive into it if you haven't already and are interested...

 

Fuse is a HUGE pain in the nuts to troubleshoot from a performance perspective, but maybe if we could at least track it down to a specific set of variables from the HW/Network side where this ends up being seen (e.g. typically occurs on systems with low max freq., has turbo boost disabled or unavailable, and is dual CPU, just for instance) and maybe even what fuse is doing during that time that's taking so long (enable debug tracing, check out what the fuse, kernel, and filesystem API's are doing, which calls those APIs are making and which are taking the longest, perhaps get a flame graph to make visualizing that part a bit easier).

 

I'd guess the reason behind a simple `mv filename /location` being slower than copying over smb has to do with the fuse interrupt queue getting thrashed - with mv being a single-threaded-with-no-optimizations type of tool whose thread is on the same server doing the fuse ops, you're almost certainly going to have more interrupts than with SMB given it's massive history of perf optimizations and the fact that those optimizations were designed against our use case (network file transfers). I could see how with * either * a limited number of cores (causing high IRQ from the kernel side), or lower frequency (fuse's interrupt queue getting stacked up waiting for cpu to finish it's actions with the filesystem/etc), SMB could be more performant.

 

... Im rambling. It's late. Anyway, hoping to hear your thoughts!

[JorgeB]

2 hours ago, BVD said:

Did you ever track this down to determine the cause?

I identify FUSE as the cause, any deeper dive within FUSE is above my knowledge, I'm more of a hardware guy.

 

2 hours ago, BVD said:

I don't have any systems which exhibit this behavior,

This should be easy to reproduce in any server if you do a transfer with small files, it's where the issue is more noticeable, e.g., map another Unraid server (or any other SMB client) to a user share and do a timed cp, I use a folder with several subfolders with about 10.000 files totaling about 10GB, I always do 3 runs and calculate the average speed, with Unraid v6.10.3 on my test server I'm getting 52.98MiB/s for this transfer, now map to the same server/device but using a disk share instead and repeat the same transfer, now I get 255.43MiB/s.