I have a Cat6A connection from my Windows 11 desktop (running only NVMEs) to a 10GB switch, to Unraid. Should be 10GB all the way, and by all indications it is - because I can copy a file off the cache to Windows at a full 10G speeds.

Writing a file to Unraid (well, an SMB share that uses the cache as primary) is slow. The fastest I've seen is roughly 240MB/s, but it's mostly below 200.

I have only 1 cache drive on Unraid - a brand-new Samsung 9100 PRO 8TB NVME that has less than 200GB used.

I've read through and made changes to my Windows NIC per this thread but I saw no change in behavior.

I've made some SMB settings changes as well, but again, no change. (Current settings are in a screenshot attached.)

Attached are my diagnostics and a few screenshots from Unraid.

Some quick notes:

• Unraid is 7.2.3 - I've encountering no other issues and this is not a new behavior with this version. It's been like this for a while, I've just been too lazy to try and figure it out.

• My Unraid box is very capable. AMD EPYC 7763 on a Supermicro H12SSL-NT with 512GB of ram and 120TB of space.

• It's not the cabling - I tried a direct connection from my desktop to the switch, no change in behavior.

• I can write from my desktop to my M1 Mac Mini at a solid 10GB - no problems whatsoever.

• I can write to my desktop from both my Mac Mini and Unraid at a solid 10GB.

• My M1 Mac Mini is experiencing the same behavior. Reading from cache - fast. Writing to it? Slow.

Any and all suggestions and guidance would be appreciated!

biggs-diagnostics-20260121-2024.zip

There are two things I would take a look at:

• enable flow control (although it should not be necessary if all members run at 10G speed, but it may help to recover if a packet gets lost/errors out on CAT cable)

• Writing to UNRAID does not always go to the cache. The cache is only used for NEW files, not if old files get overwritten (CAN work, usually does NOT WORK because of race conditions).

The 200-240 Mb/s are likely the speed to directly write onto a normal magnetic disk.

You can try these things to check it out:

• delete the file on the server BEFORE you try to write it (and wait 1-4s to give the filesystem time to update). If speed goes up to almost 10G speed, you know that the 200 are "normal behaviour".

• create a share that ONLY resides on an SSD drive (primary = SSD, secondary = NONE) and check the write speed. If it is as high as expected (~500 MB/s for SATA SSDs, ~1 Gb/s for NVMe SSDs) all is fine too.

Thinking about your enourmous amount of (unused) RAM you might consider to switch some Filesystems to ZFS and work with a very large ARC (RAM) cache someday.

• Writing to UNRAID does not always go to the cache. The cache is only used for NEW files, not if old files get overwritten (CAN work, usually does NOT WORK because of race conditions).

I forgot to mention, I was diligent in renaming files before each test so each system always saw that file as 'new'.

I like the cache-only test, and I did that, sadly to no change in write performance at all.

Attached are two screenshots. One is of the cache-only setting configuration, the second is of a brand-new file being copied to that share.

Any other ideas to try?

Hmm... for further tests pls stick to the cache-only test. This is much more reliable about speed.

And pls, check the flow control settings on both server, switch and client. Could be somebody is stepping on the brake pedal throughout transfer.

In Unraid use the Tips & Tweaks plugin (and DO NOT disable Flow Control), in your switch check the ports menues and in windows you find the settings under the details of the NIC.

afterwards check with "ethtool eth0" on UNRAID, it should look like this:

root@F:~# ethtool eth0

Settings for eth0:

Supported ports: [ FIBRE ]

Supported link modes: 1000baseX/Full

10000baseCR/Full

10000baseSR/Full

Supported pause frame use: Symmetric Receive-only

Supports auto-negotiation: No

Supported FEC modes: Not reported

Advertised link modes: 1000baseX/Full

10000baseCR/Full

10000baseSR/Full

Advertised pause frame use: Symmetric

Advertised auto-negotiation: No

Advertised FEC modes: Not reported

Speed: 10000Mb/s

Duplex: Full

Auto-negotiation: off

Port: FIBRE

PHYAD: 0

Transceiver: internal

Supports Wake-on: d

Wake-on: d

Current message level: 0x00000014 (20)

link ifdown

Link detected: yes

Watch out the field Supported pause frame use: and Advertised pause fram use:

They should not show "none"

(note: some adapters in windows have no setting for this, the feature is always turned on there)

According to Tips n Tweaks, 'Disable NIC Flow Control?' was set to 'Yes' already.

The results of 'ethtool eth0' is:

Settings for eth0:

Supported ports: [ FIBRE ]

Supported link modes: 10000baseT/Full

Supported pause frame use: Symmetric

Supports auto-negotiation: No

Supported FEC modes: Not reported

Advertised link modes: 10000baseT/Full

Advertised pause frame use: No

Advertised auto-negotiation: No

Advertised FEC modes: Not reported

Speed: 10000Mb/s

Duplex: Full

Auto-negotiation: off

Port: Direct Attach Copper

PHYAD: 0

Transceiver: internal

Supports Wake-on: d

Wake-on: d

Current message level: 0x00000007 (7)

drv probe link

Link detected: yes

Attached is a full screenshot of current Tips and Tweaks as well.

16 minutes ago, Magua said:

Advertised pause frame use: No

This makes me worry... it is forbidden somewhere

17 minutes ago, Magua said:

According to Tips n Tweaks, 'Disable NIC Flow Control?' was set to 'Yes' already.

You have got me wrong, it must be set to NO!

(you want to enable the feature, so you have to turn off the disable flag. Its puzzling... double negative)

Ah, understood. Thanks for clarifying.

I've changed that setting to 'No', rebooted and tried a few more tests. It helped, but only about a 25% improvement or so.

Copying from Unraid's NVME to Windows or Mac is still 3X faster than writing.

Current screenshots attached - with the changes highlighted in Tips n Tweaks.

And here's the current 'ethtool eth0' output:

Settings for eth0:

Supported ports: [ FIBRE ]

Supported link modes: 10000baseT/Full

Supported pause frame use: Symmetric

Supports auto-negotiation: No

Supported FEC modes: Not reported

Advertised link modes: 10000baseT/Full

Advertised pause frame use: Symmetric

Advertised auto-negotiation: No

Advertised FEC modes: Not reported

Speed: 10000Mb/s

Duplex: Full

Auto-negotiation: off

Port: Direct Attach Copper

PHYAD: 0

Transceiver: internal

Supports Wake-on: d

Wake-on: d

Current message level: 0x00000007 (7)

drv probe link

Link detected: yes

hmm, next you should turn off "disable NIC offload" in Tips&nTweaks... (again a double negative!)

Why do you make your smart NIC dumb as hell and burden all the work onto the main CPU?

Edited January 22Jan 22 by MAM59

Well - that change made a huge improvement:

I clearly didn't understand that 'No' for disable offload was the preferred setting.

Thanks for the guidance. Still not 10GB, but much better. Any other settings I should look out for?

nothing I can see here anymore.

But 600MB/s is already "good". You cannot expect to get full 10G all the time.

It depends also on other clients, background tasks and so on.

Do a few more transfers and see if the value is constant or if it is fluctuating now and then.

As I said: its normal :-)

(I get 10G here only at certain times of the day too)

As I have already said, you may get more improvements if you reformat the drive an use ZFS as the file system.

(its fairly still empty, so this can be done easily)

Edited January 22Jan 22 by MAM59

Oh I get it - no system is perfect. I do get 50% faster speeds between my Windows desktop and my M1 Mac Mini though.

(Example attached - same file, same network, same switch involved, same Windows source machine.)

I've noticed that, for some reason, Intel 10GbE NICs with Linux don't typically perform as great as for example, Mellanox NICs; with those, I can get line speed in both directions, even with 25GbE and all default settings.

I think I'm finally in good shape (see attached).

The last change I made: Convert the filesystem on the cache from btrfs to xfs.

For my use-cases, I see no reason to have this remain as btrfs and that change, coupled with the Tips N Tweaks changes mentioned above have me properly maxed.

Thanks!

There shouldn't be such a large difference by just changing the fs, but I'm glad it's better now.

xfs? that gives no speed improvements or more security at all. xfs is good "inside the UNRAID array with parity", but not for standalone pool drives.

try zfs

xfs? that gives no speed improvements or more security at all. xfs is good "inside the UNRAID array with parity", but not for standalone pool drives.

try zfs

Clearly that's wrong. It gives a 50% speed improvement!

I realize there is zero security or added functionality, but that's a different concern.

Bottomline ==> changing from btrfs to xfs netted me my full 10 GB performance.

I'll try changing from btrfs to zfs on my secondary NVME and see how that performs.

14 minutes ago, Magua said:

Clearly that's wrong. It gives a 50% speed improvement!

Naah, this is just accidentallly (time of day? remember?)

btrfs is not lamer than xfs, at least not that much that you can notice it.

zfs is also not much faster, it just contains an automatical kicking in RAM cache for reading and writing. This may help network transfers by buffering a lot of data before the underlying disk system blocks the incoming data for a while.

Even NVMes have limited write speeds, they usually contain a RAM cache too. But considering your 512Gb RAM in the computer, this may catch up even big files before they need to slow down for beeing written to the drive.

So my advice is not really about the zfs filesystem directly, it is just a mule carrying the possible speed up RAM cache...

Here's another scenario.

I just formatted my secondary NVME, its an older drive, 2TB - to ZFS. From my same source (machine, drive, everything) I copied the same 5GB file over.

It averaged a hair under 500MB/s

Then I reformatted that same NVME drive to XFS and repeated the exact same test - averaging around 675 MB/s. (Obviously slower than my primary NVME, but this is an older and slower drive.)

Still - all other scenarios the same, XFS is 35% faster than ZFS. 🤷‍♂️

I fully realize this isn't a bullet-proof, scientific test - it's very anecdotal to my situation. I'm not trying to argue - just sharing my findings so that maybe if this comes up with another user they have another option to consider.

Edited January 22Jan 22 by Magua

if this was your first zfs disk, you would have needed to reboot after formatting I think.

The cache is only allocated at boot time, you can see it in the dashboard then.

If there is no entry "ZFS Cache" you need to reboot before you test

(Note: after reboot it will not occupy any RAM, but the dashboard will add this new (orange) topic and you will see it grow (and shrink) when you start your tests)

Btw: I just had also only ~600Mb/s here, although this morning i had stable 1Gb/s. This is because "others" do something on the server too currently, its perfectly normal

Edited January 22Jan 22 by MAM59

if this was your first zfs disk, you would have needed to reboot after formatting I think.

The cache is only allocated at boot time, you can see it in the dashboard then.

If there is no entry "ZFS Cache" you need to reboot before you test

Ah, good to know. I didn't check that.

What the heck - I'll give this another shot and report back...

What the heck - I'll give this another shot and report back...

After another reformat to ZFS, then a full system reboot, I did see an improvement. Initially it was roughly a hair under 500MB/s (without a reboot).

With the reboot I'm seeing around 600MB/s flat.

you should get higher rates in the opposite direction. try it out.