Unraid 6.11.5 - Poor/Capped Samba 4KB performance/IO?

[tyranuus]

Hi Guys, 
I have just setup a 'new' NAS using the following specs running an Unraid trial to get the feel for the platform, and in most ways I really like it. Overall functionality seems good whilst not being overwhelming, the array functionality seems to work well, as does the modularity of it compared to ZFS via TrueNAS for example, and the parity on the main array is really re-assuring that if more than 2 discs don't go at once, there is still parity/redundancy to keep moving and rebuild without losing data.

 

This all being said whilst general read/write on larger files I've noticed 4Kb/small file performance seems distinctly lacking or capped compared to the prior OMV server, and I'm hoping someone could help me figure out why?

 

NAS Specs:

3570K

24GB DDR3 1600

Array:

7x WD Red 3TB via IBM M1015 HBA SAS card (1 being used as parity)

1x Toshiba 7200 RPM 4TB drive via IBM M1015 HBA SAS (also being used as parity)

Unassigned drive:

1x Samsung 860 Evo 1TB connected directly to Intel 6GB/s SATA

1GBps LAN

 

Now, if I run a 1GB crystaldiskmark test on the array network share, I see around 9.5Mbps 4KB 1T1QD reads, which seems incredibly high for HDDs, so some caching is going on, however if I run it on the SSD in the system which is not part of the array and just mounted seperately, I see more or less exactly the same read speed, which should definately not be the case unless there is a bottleneck, the SSD should have a substantial lead over HDDs, 5400RPM ones at that. The SSD is mildly higher, but talking a few hundred KB higher, closer to 10MBps. If the drive was local, I know it'd be hitting around 40MBps read.

Now, initially I'd just put that down to overhead of Samba, frankly the read and write speeds for anything except 4KB 1Q1T are basically maxing gigabit, the bit that concerned me however, is when I run the same test on a HDD in the old server which was running OMV 4, its seeing around 20MBps on 4KB reads, in the same 1T1QD setting.

Now again, there's no way a HDD could produce those sorts of 4KB results, even a 7200RPM drive (usually they'd be less than 1MBps, so there is definately some cache being used or similar, but the old server is running an i3-2110, 8GB RAM, notably inferior hardware, so it seems off that it is returning higher 4KB single thread read results full stop, even if they're not reflective of true drive performance.

 

Any thoughts?

 

 

 

Edited April 19, 2023 by tyranuus
Clarification

[JorgeB]

Try testing on a disk share, FUSE has always some overhead, mostly noticeable with small files or I/Os.

[tyranuus]

Would FUSE still be an issue, as I am testing using an SSD which is not part of the array, and is just an unassigned drive being mounted/shared seperately?

[JorgeB]

If you're not using /mnt/user, i.e., if it's really unassigned, then it already not using FUSE.

[JorgeB]

Samba performance with Unraid (outside FUSE) should be similar to any other OS using Samba, though it will depend of the Samba version, older releases were faster due to less security mitigations, disabling kernel mitigations can sometimes also have a big impact, depending on the CPU.

[tyranuus]

Apologies, I will have to dig later, it is mounted by the unassigned devices plugin, I am not sure if that is using /mnt/user or not  

[JorgeB]

Just now, tyranuus said:

it is mounted by the unassigned devices plugin, I am not sure if that is using /mnt/user or not

That will be outside FUSE.

[tyranuus]

OK thanks

I will try the kernel mitigations disabling and see if that impacts performance. 

 

It just caught me out that OMV4 vs Unraid (whether its cached data or not), was reporting roughly double 4KB single thread performance compared to Unraid via SAMBA, using considerably slower/less capable hardware, on the same network.

[tyranuus]

Unfortunately disabling the CPU Kernel mitigations did nothing for the 4KB performance, if anything speed seemed to decrease very slightly, so it does not seem to be CPU overhead causing this issue.

 

I mean, I'm not devastated by close to 10MBps 4KB single thread reads and writes, its still WAY higher than any single HDD even locally can achieve so the SSD drive should still achieve its goals, but knowing how much performance is being left on the table, not just from having the drive installed physically, but vs OMV4 being able to return double those figures even if its some sort of cache read seems a bit wrong, especially knowing the older server has a weaker CPU, both in terms of cores and clock speeds, and also considerably less (and I believe slower) RAM. 

 

As it is, Unraid is returning very similar figures from the HDD array and the SSD standalone, at least for 4K 1T1QD reads/writes, which suggests a bottleneck elsewhere in Unraid, and as the old server is returning higher figures, with weaker hardware, something seems to be going on.

 

I also checked the dashboard whilst the 4K test was running, and none of the individual CPU cores are getting anywhere near 100% either, so its not a CPU bottleneck either, with one of the cores being pegged around 100% either.

 

I just took a look at the SAMBA configuration on the old server, and it looks like its configured:

Use Sendfile (ON)
Asynchronous IO Enabled

socket options = TCP_NODELAY SO_RCVBUF=65536 SO_SNDBUF=65536
read raw = yes
write raw = yes
max xmit = 65535
getwd cache = yes

 

Doing a quick read on Unraids default SAMBA config, it sounds like its not far off these already, so I'm not sure its even anything in the configuration, but something else going on, almost like something somewhere has a hard limit written in, which is only allowing 9.5-10MBps?

Edited April 19, 2023 by tyranuus

[JorgeB]

What Samba version is the other server using?

[tyranuus]

I am trying to find out.

The server was running OMV 4.1.36, which apparently was based on Debian 9 and released in 2018 if that helps.

 

Edit, if I'm reading the logs correctly, it's using Samba 4.5.1, so a decent chunk older than the Samba 4.17 that Unraid is currently using, but still within the Samba 4.x family.

 

Edit 2: Just realised the other server only has 4GB of RAM, so it's even more relatively anaemic than I had thought, which, even with caching involved, seems strange that i3-2110 and 4GB of RAM is producing a higher 4KB 1T1QD result than this Unraid server running an i5-3570K and 24GB RAM!

Edited April 19, 2023 by tyranuus

[JorgeB]

11 hours ago, tyranuus said:

but still within the Samba 4.x family.

Yeah, but before Samba 4.13 that slowed down everything a lot with all the security issues, Samba 4.17 brought some performance back but Samba 4.18 is expected to bring some more big gains, so retest once Unraid includes that, likely v6.13.

 

https://www.samba.org/samba/history/samba-4.18.0.html

 

Quote

While 4.17 already improved the situation quite a lot, with 4.18 the locking overhead for contended path based operations is reduced by an additional factor of ~ 3 compared to 4.17. It means the throughput of open/close operations reached the level of 4.12 again.

 

[tyranuus]

Thank you, that is good to know.
Is there any rough roadmap to how long it is likely to be before 6.13 hits?

I suspect it won't be inside the trial period

[JorgeB]

First 6.13 beta should be available soon after 6.12 stable is released, but for now cannot give you an ETA nor confirm if Samba 4.18 will be included.

[tyranuus]

Thank you, that is great to know.

I assume realistically once 4.18 is incorporated into a beta/RC, it should just require an unraid OS update to unlock the benefits, and shouldn't require any additional configuration changes? 

[JorgeB]

Correct, just need to update.