Slow write performance, but only to array - 2MB/sec

[voxter]

Hi everyone,

 

I'm seeing slow write performance when writing to the array but not directly to disks.  I've tried a number of suggestions I've found in other posts about similar topics, although I think mine is slightly different.

 

Originally I wondered if it was something related to my parity disk, but I've just removed my parity disk from the array entirely and write performance is still bad even without it.

 

Directly to my disks, I get 170-180MB/sec writes, but writing to the array is between 1.8-2.4MB/sec:

 

Quote

root@UNRAID:/mnt/disk2# dd if=/dev/zero of=test1.img bs=300M count=1 oflag=dsync
1+0 records in
1+0 records out
314572800 bytes (315 MB, 300 MiB) copied, 1.7865 s, 176 MB/s

Quote

root@UNRAID:/mnt/user/Backups# dd if=/dev/zero of=test1.img bs=300M count=1 oflag=dsync
1+0 records in
1+0 records out
314572800 bytes (315 MB, 300 MiB) copied, 129.506 s, 2.4 MB/s

 

All my disks are identical (Seagate Ironwolf 6TB SATA), all connected to the same onboard SATA controller on my Asrock x470 Taichi Motherboard, with an AMD Ryzen 9 3900x with 32GB ram.

 

I've stopped all VM's and docker containers to no avail. I've tried direct IO, turbo write, nothing seems to help.

 

I've also attached a diagnostics zip, if that helps.

 

Any suggestions are very welcome!

 

Thanks :) 

unraid-diagnostics-20200725-1949.zip

[voxter]

Should also mention I'm on latest stable (6.8.3) and my array has 3 disks in it, no cache disk, and have tried both with and without one parity disk - all disks in the array and parity are identical disks.

[JorgeB]

4 hours ago, voxter said:

Directly to my disks, I get 170-180MB/sec writes, but writing to the array is between 1.8-2.4MB/sec:

dd is known to give very bad results with users shares, did you try an actual transfer?

[voxter]

2 hours ago, johnnie.black said:

dd is known to give very bad results with users shares, did you try an actual transfer?

Thanks for the dd tip, was not aware of that! I wonder why raw disk access would be so slow like that?

 

I tested again by creating a 5GB file on my ethernet connected mac, using /dev/zero as the source again, and copying that 5GB file filled with zeroes over SMB ... copying from the mac with rsync to an SMB mounted destination (so i could see transfer speed) reported an average of 22MB/sec which is arguably faster than I reported originally but still nowhere near the 185MB/sec the drives achieve on their own.

 

Copying the file to the same location in /mnt/user/sharename via scp managed to hit about 55MB/sec, almost double what SMB is doing but still almost 3.5x slower than peak disk speed on its own. 

 

Copying the same file using FTP managed to hit 102MB/sec - again nowhere near 185MB either.

 

There must be some strategies here for improving SMB performance, or potentially being able to explain why despite getting faster performance from SCP, then FTP, all three options are still significantly lower than the expected top end.

 

Ideas?

 

 

 

 

 

 

[JorgeB]

18 minutes ago, voxter said:

with rsync to an SMB mounted destination (so i could see transfer speed) reported an average of 22MB/sec

Rsync is not built for speed, still that's slow, make sure you're not using compression (-z), also OSX has some known issues with SMB.

 

19 minutes ago, voxter said:

There must be some strategies here for improving SMB performance

Like mentioned part of the problem might be OSX, while some performance loss is normal using user shares vs disk shares, I can easily get around 800/900MB/s from Windows to Unraid (or vice versa) to user shares.