Slow SMB reads from user shares fixed by increasing read-ahead on /dev/md (SOLVED)

For posterity and because it did take hours to find the problem... For some reason unraid per default sets low read-ahead values (512) for md devices, at least for my setup (External 3.1 USB drive enclosure, 3 disks).

This affects systems that are used to read large files over SMB shares with some overhead in the setup (External USB drives, iSCSI drives, weird setups like that).

Main Issue:

Copying files from shared folder directly from disk (not cached!) resulted in read speeds of 5-15 mb/s on a 1GB network. Drove me crazy.

Symptoms:
- SMB copy from disk share: ~100+ MB/s (great! So no network issue)
- SMB copy from user share: ~5–20 MB/s wildly fluctuating(huh!?)
- local dd from /mnt/user: normal speed (250 MB/s per disk -> no USB or disk issue)
- hdparm -t /dev/sdX: normal for all drives
- SMART: clean, disks are fine
- filesystem check: clean. xfs is fine on all drives
- Direct I/O setting: no impact
- SMB "sendfile" off: no improvement
- /mnt/user0 shared as a test: only slightly faster

- no CPU bottleneck, only short flickers of 100% (waiting for I/O?)

Key diagnostic:
Slow user-share SMB transfer showed iostat like:
- high disk util ~95–100%
- low throughput ~15 MB/s
- small average read request size ~50 KB

Fast disk-share SMB transfer showed:
- high disk util
- throughput ~100 MB/s
- average read request size ~220 KB

Check what blockdev --getra /dev/md<device> says first. If it´s already high (higher than 512...) then this is not your issue.

Fix:

blockdev --setra 8192 /dev/md<first device>
blockdev --setra 8192 /dev/md<second device>

(or more if you have more disks)

This set the read-ahead size to 4 MB and instantly fixed all performance woes after issuing those commands.

If you want to persist this easiest way is the use the plugin user scripts and set it as "execute on array startup".

Pre-made script:

#!/bin/bash

# 8192 sectors = 4 MiB read-ahead
RA=8192

logger "Setting Unraid md partition read-ahead to ${RA} sectors"

for dev in /dev/md[0-9]*p[0-9]*; do
  if [ -b "$dev" ]; then
    /sbin/blockdev --setra "$RA" "$dev"
    logger "Set read-ahead on $dev to $RA"
  fi
done

If you have a 10GB network and only large files, your might even increase this to a higher value. Test first. This also potentially tanks small file and concurrent performance. If you have 20 users hammering on small files, this may not be a great idea.

Hope someone some day finds this.