Slow consistent write speeds

[Gio]

Hello,

 

I've been running unraid for a couple of weeks now and I've had performance issues all the time.
I even reinstalled Unraid, got new harddrives, tried all the performance optimizations and I still have issues.

 

The problem

Performance drops every single transfer. The speed goes from 160 MB/s to 2 MB/s and then goes back again.
Every single transfer has this issue. It happens in docker and it happens when copying on the server using rsync.

However, it does NOT happen when copying from my own drive over the network with SMB.

My first installation also had the same problem but even with SMB, this was fixed by reinstalling.

 

What I did

I tried to copy from cache to array -  this works without any problems. I disabled network offload (this fixed SMB and VM issues).
I increased md_num_stripes to 8192 and I enabled reconstruct write / turbo mode.

 

The share

The share data is located on 2 disks, currently a 10 TB Seagate Ironwolf (ST10000VN0008) and a 4 TB Seagate desktop (ST4000DM000).

The parity is also a 10 TB Ironwolf.

 

Extra info

There are no running docker containers or VM, also the tests were done after a fresh reboot.

 

root@Kratos:/mnt/user/plex/TV Shows/Marvel's Agents of S.H.I.E.L.D/Season 7# rsync -av --progress * /mnt/user/plex
sending incremental file list
Marvel's.Agents.of.S.H.I.E.L.D.S07E03.Alien.Commies.from.the.Future!.WEBDL-1080p.mkv
  1,766,883,328  63%    2.32MB/s    0:07:10  
 

 

Does anyone have any idea how to fix it?
To be honest I am getting really sick of it and I am reconsidering going to FreeNAS with ZFS.

kratos-diagnostics-20200710-1605.zip

Edited July 10, 2020 by Giovanni Kosman

[Squid]

You're going to have to post the diagnostics while the slow transfers are in progress

[Gio]

1 minute ago, Squid said:

You're going to have to post the diagnostics while the slow transfers are in progress

I'm sorry, somehow I managed to post an incomplete post. 
I finished my post with diagnostics.

[Gio]

I did some more testing and made a share that only used 1 disk at a time.
Performance seems fine on every disk except for disk1. I think it's a faulty disk.
The disks came from my old NAS so was to be expected I think.

 

Will test some more with the disk out of the array and keep this post updated.

[itimpi]

1 hour ago, Giovanni Kosman said:

I did some more testing and made a share that only used 1 disk at a time.
Performance seems fine on every disk except for disk1. I think it's a faulty disk.
The disks came from my old NAS so was to be expected I think.

 

Will test some more with the disk out of the array and keep this post updated.

You might want to try installing the DiskSpeed docker to test the disk.    you should be able to do that with the disk still in the array.

[Gio]

32 minutes ago, itimpi said:

You might want to try installing the DiskSpeed docker to test the disk.    you should be able to do that with the disk still in the array.

I'll do that after I moved the files. Currently running the unbalancer plugin so I can test some more later.

[Gio]

So now I actually do not think the disk is the problem.

I emptied the disk and ran a couple of copy tests again, this time no speed decrease at all.

 

@itimpi
I also ran the benchmark as asked, I don't see any issue here either (although it looks like it does only read tests?)

 

 

root@Kratos:/mnt/user/testabc# rsync -av --progress /mnt/user/plex/TV\ Shows/House/Season\ 8/* ./
sending incremental file list
S08E01 - Twenty Vicodin Bluray-1080p.mkv
  3,519,696,181 100%  143.10MB/s    0:00:23 (xfr#1, to-chk=21/22)
S08E02 - Transplant Bluray-1080p.mkv
  3,519,986,886 100%  124.77MB/s    0:00:26 (xfr#2, to-chk=20/22)
S08E03 - Charity Case Bluray-1080p.mkv
  3,519,238,379 100%   76.13MB/s    0:00:44 (xfr#3, to-chk=19/22)
S08E04 - Risky Business Bluray-1080p.mkv
  2,617,081,856  74%   91.85MB/s    0:00:09  

kratos-diagnostics-20200711-0831.zip

Edited July 11, 2020 by Gio

[Gio]

Anyone got an idea? It's still happening.

It's so amazingly annoying, it slows down all my docker instances. 

 

rsync -avh --progress * /mnt/user/plex/Anime/Sword\ Art\ Online/Season\ 3/
sending incremental file list
[MTBB] Sword Art Onlineː Alicization - 01v2 [418A1A46].mkv
          1.83G 100%  216.13MB/s    0:00:08 (xfr#1, to-chk=29/30)
[MTBB] Sword Art Onlineː Alicization - 02v2 [C837BD21].mkv
        847.89M 100%   54.12MB/s    0:00:14 (xfr#2, to-chk=28/30)
[MTBB] Sword Art Onlineː Alicization - 03v2 [05C9C930].mkv
        858.80M 100%  161.06MB/s    0:00:05 (xfr#3, to-chk=27/30)
[MTBB] Sword Art Onlineː Alicization - 04v2 [773FE70D].mkv
        948.98M 100%   48.38MB/s    0:00:18 (xfr#4, to-chk=26/30)
[MTBB] Sword Art Onlineː Alicization - 05v2 [36956C97].mkv
        763.21M 100%  124.02MB/s    0:00:05 (xfr#5, to-chk=25/30)
[MTBB] Sword Art Onlineː Alicization - 06v2 [09F2008F].mkv
        977.36M 100%  126.92MB/s    0:00:07 (xfr#6, to-chk=24/30)
[MTBB] Sword Art Onlineː Alicization - 07v2 [2322F202].mkv
        105.94M  12%    1.44MB/s    0:08:26

 

This is just utter crap. 
 

kratos-diagnostics-20200716-1953.zip