Hmm, maybe that's it? I'm using an intel 82599 network card, not Mellanox, but you'd think that would be supported pretty well too, I know for TrueNas intel stuff is preferred for some reason though but Mellanox is more common for hobbyist stuff due to used parts being way more available and cheaper.

But it can saturate it on iperf... idk. Guess this might be beyond forum support

1 hour ago, JunctionRunner said:

But it can saturate it on iperf

Not with a single stream, and that's what matters for a single file transfer.

5 minutes ago, JorgeB said:

Not with a single stream, and that's what matters for a single file transfer.

But windows on this machine shows similar single stream iperf results in the same direction as unraid, yet it transfers fine...

And would SMB multichannel not use multiple streams? Also this is not one giant file, it's multiple.

Edited December 6, 20232 yr by JunctionRunner

7 minutes ago, JunctionRunner said:

And would SMB multichannel not use multiple streams?

And is SMB multichannel enabled and working with Unraid? I never managed to get it working.

 

8 minutes ago, JunctionRunner said:

Also this is not one giant file, it's multiple.

Yes, but they are transferred one at a time.

 

Yes, I did enable it, what would be the way to test that it's actually working and enabled in unraid?

If enabling it doesn't actually turn it on, then I guess that would explain why the performance is so much worse.

There is that "SMB Extras" section, maybe that needs some stuff added too, but you'd think the yes/no toggle would yknow, work. And I have done multiple reboots since enabling it.

Found this article, not sure when it was posted or if it's outdated but those smb extras didn't help.
https://unraid.net/blog/how-to-beta-test-smb-multi-channel-support

Edited December 6, 20232 yr by JunctionRunner

51 minutes ago, JunctionRunner said:

Yes, I did enable it, what would be the way to test that it's actually working and enabled in unraid?

There was a Samba status command that would show it was enable or not, same for Windows with a different command, but it's been a while since I tested that, at the time it was with multiple NICs, should not be difficult to find with Google.

 

 

 

 

Well, support pointed me towards the sbmstatus command but I can't seem to find a multichannel option like this example here.

 

I enabled jumbo frames on my router, switch and workstation, set them all to maximum, and I saw single stream performance jump up quite a lot, but it's inconsistent.

These two tests were done, then a few minutes later, slow again in one direction.
 

Still better, but slow.

With a direct disk share I've now seem it peak to near 800MB/s but then it drops quite a lot, often ending back down around 500MB/s, and if I use the unrad array share with the cache enabled, I get basically 500 max reading and writing...

So, I guess I'll see if there's anything else I can do tuning wise, but still, the fact that windows just magically works perfectly in comparison... I thought the windows network stack was supposed to be worse than linux for performance. Maybe SMB multichannel isn't actually working.

 

Edited December 7, 20232 yr by JunctionRunner

If I'm reading the Samba status correctly SMB multichannel is not working, that could be the difference you see compared with Windows, where probably multichannel is working, Unraid should get you similar speeds if a single stream iperf test consistently got you close to line speed.

 

 

Can you post new diagnostics with Multichannel enabled?

Guess I did post my prior reply but it does seem like multichannel isn't enabled.

Here are the diagnostics. I just ran another copy test to the SSD disk share and I actually hit 1GB/s briefly but then it dropped off very fast. For the standard share with cache enabled, same cap as before.

And copying from, just so there's a current reference of each.

 

crap, I did just notice I just the four SSD's in a 2 group 2 device zfs mirror, so I switched to back raid 0, saw worse peak on the ssd disk share, but higher overall. I have switched back to two groups of two devices though because that is the end use, unless I need to buy another four SSDs and have two groups of four devices to be able to maximize speed? I definitely don't want to be running four in raid 0 but buying another four is at least two, likely four weeks away.

It still should be able to get at leas the above speed with only two though so I think the mutichannel issue is where to look.

If I have to wait longer and buy another four SSDs, I guess at least I get more high performance storage capacity, I was thinking of maybe doing it eventually, but I'd like to not _have_ to. I did at least get more than five hours of sleep today, so I shoudn't miss a stupid obvious change like that I made again.

Edited December 8, 20232 yr by JunctionRunner

I'm looking for the Unraid disgnostics zip file created when you run diagnostics on Unraid.

Just now, dlandon said:

I'm looking for the Unraid disgnostics zip file created when you run diagnostics on Unraid.

Sorry, brain farted and forgot to attach.

quantaserver-diagnostics-20231208-0408.zip

Give me the output of 'testmrm -v'.  Post it as a zip because it can be quite long.  It will not be anonymized.  If that's an issue, PM it to me.

6 minutes ago, dlandon said:

Give me the output of 'testmrm -v'.  Post it as a zip because it can be quite long.  It will not be anonymized.  If that's an issue, PM it to me.

Where would I go about running that? Command isn't found in the terminal. Also tried testmem in case it was a typo. Can't find reference to that command searching google or the unraid manual.

Crap, mis-spelled it - 'testparm -v'.

Here you go, ended up being fairly short so I just copied it from the terminal, and I don't see anything to be concerned about not being anonymized.

testparmesean output.txt

Edited December 8, 20232 yr by JunctionRunner

Testparm shows that Multichannel is enabled in samba:

server multi channel support = Yes

 

The issue is Multichannel doesn't work with only one NIC.  You have Unraid setup to only see eth0 according to your network.cfg, and no Bonding:

# Generated settings:
IFNAME[0]="br0"
BRNAME[0]="br0"
BRSTP[0]="no"
BRFD[0]="0"
BRNICS[0]="eth0"
PROTOCOL[0]="ipv4"
USE_DHCP[0]="no"
IPADDR[0]="172.16.1.225"
NETMASK[0]="255.XXX.XXX.0"
GATEWAY[0]="172.16.1.254"
DNS_SERVER1="172.16.1.254"
USE_DHCP6[0]="yes"
DHCP6_KEEPRESOLV="no"
MTU[0]="9198"
SYSNICS="1"

 

NICS have to be either Bonded or Teamed for Multichannel to work.  You would Bond the interfaces in Unraid.

 

Here are the steps I recommend:

• Configure at least two NICs in Unraid and Bond them.  I'm not sure which Bonding mode would be best, but try the 'active backup (1)'.
• Install a plugin called 'Tips and Tweaks' and review the DIsk Cache options.  The default in Linux is to use about 50% of memory for disk caching if I remember correctly.  This will cause file transfers to go very fast at times, then drop down as the buffered data is committed to disk.  Use the default set in Tips and Tweaks to start, and adjust as you feel appropriate.
• Also disable the NIC Flow Control and NIC Offload.  This can sometimes help throughput, especially with Intel NICs.
• Set the NIC RX and TX buffers.  I usually set those to 1024 bytes with 1GB NICs.  More isn't always better.  With high speed NICs like you have, I'm not sure what would work best.

Ah, I see, so windows smb multichannel works with only a single interface, but linux, or at least unraid doesnt for some reason then.

That seems, odd, but it does explain why this has been a complete no go from the very start and why it works immediately well in windows. 

Unfortunately that flakey transceiver I have was my last one (I'll dig around and quadruple check later today), so I'll have to wait for some others I ordered on ebay to arrive. I was going to do two bonded anyhow, but didn't know it was a requirement for multichannel. 

Not sure how many sessions windows opens by default, but hopefully I don't need more than two to achieve the same speeds since I can't add any more 10 gig into this server. Unfortunately it will take a while to arrive.

Edit: I will give it a shot with a 1 gig port bonded for the hell of it when I'm done work today, but I have a feeling that won't be very good.

Edited December 8, 20232 yr by JunctionRunner

My understanding is Multichannel does require Bonded or Teamed NICs unless certain NIC parameters are set.  Unriad does not build the kernel with those parameters set.

 

Don't you have 4 NICs in that machine?

Edited my post, I have two single gig ports so I'll try that when I get home and report back.

Edited December 8, 20232 yr by JunctionRunner

This thread should help to verify if RSS is enabled for your NIC with Unraid, that (or RDMA) is needed for multichannel support with a single NIC:

 

 

Well, I've followed the steps in that guide, and unfortunately it seems to be a no-go.

On the unraid box it only shows these two for my machine

On windows, for some reason the unraid box doesn't show up at all in my list...

My card does support RSS though.

I've rebooted multiple times, both machines with no luck.

My egrep output looks messier than the example but probably because I have more cores/threads.

I am wondering why the unraid server doesn't show up in there... I have drives mapped and I have checked again while running a file transfer.

Oh right, so I tried those tweaks, and that dropped performance by about 50%, so I reverted it all back for now. I figure I adjust and play with those once multichannel is working.

Edited December 9, 20232 yr by JunctionRunner

Well, transceivers arrived so that's good, though this weekend I have to leave to visit family to make them happy instead of working on projects lol, uhg. Testing with the direct share still.

Enabled bonding, active backup, ~700 with dips to mid 400s with the smb extras thing cleared out.

Added back in the smb extras I had before, and rebooted my client and I still see no multichannel connection and speeds varied wildly, getting up to 900 briefly.


Read speeds look fantastic though



Unfortunately after poking at this for a little bit, no luck getting smb multichannel to enable and show in powershell, getting similar results with it turned off completely in unraid both copying to and from the server so once it actually turns on it should work great.


The other thing I'm thinking of, and maybe it's best suited as another thread, is using a virtual machine with stablebit drivepool on it, and sharing out my storage from there.

I need to test with mover tuning more, but if I was able to just map my cache, and my array as separate physical disks, that means that I could do tiered storage with drivepool sill, which means I would be able to pin folders to the faster tier, and unlike mover tuning, it seems more sophisticated in terms of how it handles thresholds, and properly moves oldest files first, etc.

One thing at a time though, first I want to get multichannel actually working, then I'll see if there's a way to get these drives in to a vm at full speed, so far my testing got 200MB/s at best.

Updated iperf as well.

still slow in that one direction with single stream which annoys me

but good with multiple.