Slow docker download performance? - 6.10.#

[miccos]

Hi
I am kind of at a loss and don't know where else to look.
I am experiencing extremely slow downlaods of docker updates and nzb downloads. I am only getting between 1 and 5 MB/s, my ISP plan is 1000Mbps. Prevoisly I was getting 80-90MB/s.
I was on 6.10.3 as I only seem to have started experiencing the issue early this year, since March maybe. I have down graded to 6.10.2 but still the same speeds.
I use SABnzbd and have tried NZBGet with no difference. I also use two different news groups. I have tested SAB on my pc and the SAB test downloads run at full speed.
I don't use a VPN, other then unraid's implementation of wiregaurd which is to my phone, but i was troubleshooting with my ISP, with traceroutes and they repeatadle said I was using a VPN. Does wireguard route ALL traffic? I have tested with wiregaurd off and my download has still been the same.

Hopefully that makes sense and someone is able to assist.

Attached is the diagnostics from 6.10.3

 

tower-diagnostics-20220628-1425.zip

Edited July 9, 2022 by miccos

[BRiT]

Are you located in Canada or trying to pull anything from there? If so, it's likely because Rogers has royally fraked their network. It's been a major outage across the country since 5am Friday.

 

https://www.cnbc.com/2022/07/08/rogers-network-outage-across-canada-hit-banks-businesses-and-consumers.html

 

[miccos]

I am in Australia, so not impacted by that outage.

[miccos]

I have upgraded to 6.11.0-rc2 to see if this makes any difference by some chance.
Unfortunately it seems to have made no difference.

anyone have any suggestions?

[miccos]

would a HBA card potential cause a slowdown?
It might just be a coincidence but looking back, my slowness seems to have started the day I installed my HBA card (LSI 9207-8i)

My cache is on an nvme drive. 

[miccos]

I have reconnected all my HDDs back to the onboard sata ports and so far my SAB downloads are back to normal, so utilizing the full bandwidth of my internet connection.

Looking at the syslogs the HBA card shows up correctly as far as I understand.

 

mpt2sas_cm0: LSISAS2308: FWVersion(20.00.07.00), ChipRevision(0x05), BiosVersion(07.39.02.00) Aug 17 11:42:23 Tower kernel: mpt2sas_cm0: Protocol=(Initiator,Target), mpt2sas_cm0: Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)

attached is the latest diagnostics zip with setup no using HBA.

Anyone able to provided some advice? Was i meant to change a setting before installing?

tower-diagnostics-20220817-1146.zip

[JorgeB]

That's a little strange, were you downloading directly to the array or was cache also using the HBA? Were normal array writes still OK, just the apps downloads were slow?

 

P.S.

 

Aug 17 11:42:50 Tower kernel: REISERFS warning:  read_super_block: reiserfs filesystem is deprecated and scheduled to be removed from the kernel in 2025

 

You need to convert all reiser disks to xfs, also note that reiser is known to under perform with full disks, but it should be the same with either controller.

[miccos]

Yeah downloading direct to cache, ran common problems to make sure I hadn’t missed anything.

Array writes seemed to be normal. Only really noticed the slower docker update downloads and Sab was clearly slower.

 

Is there any tests I can run on the disk layer that could build more of a picture?
 

How do I convert to xfs and do I loose data if/when I convert?

 

thanks

[JorgeB]

9 minutes ago, miccos said:

Yeah downloading direct to cache

Since cache is NVMe don't see how having the array on the HBA would interfere with downloads to the cache device, are you sure this is reproducible, i.e., if you change the disks to the HBA again you start having the issue again.

 

11 minutes ago, miccos said:

How do I convert to xfs and do I loose data if/when I convert?

 

https://wiki.unraid.net/index.php/File_System_Conversion

[miccos]

This is me making the change back to the on board sata ports. I'll lets some more downloads occur naturally and will convert back to the HBA card and report back.

 

[miccos]

Sab history before switching back to HBA (originally on HBA 3 days and beyond ago).

 

and straight back to throttled downloads  this makes no sense haha.
There is no other activey on the server.


 

[JorgeB]

Very strange, so docker is only using the NVMe device and just by having the array disks on the LSI without being used by anything you have the low download speed correct?

[miccos]

Correct.

Sometimes there will be a spike in download speed but these are random and also still only around 6MB/s.

[miccos]

update:
Still no fix, I have reformatted all drives to XFS and have moved my cache drive over to a SATA SSD.
My downloads still cap at 1-1.5 MB/s when all drives are connected through the HBA card.

I still get normal performace when all array drives are through the on board SATA ports and the SATA SSD (cache) is via the HBA card.

Any suggestions or ideas?

Edited January 8, 2023 by miccos

[JorgeB]

Sorry, never seen a similar issue, I guess you could try with a different controller.

[miccos]

Update, for anyone who is curious.
I swapped from the nvme cache to a sata ssd and still no change. 
I have found tough, that when I have 6/8 SAS to sata connectors plugged into my drives I get the poor sabnzdb (docker) download performance (1.5MB/s). But when I have 4/8 sas to sata connectors (I think that is what these are called, I am not quite sure) my sabnzdb download performance is back where I would kind of expect it (50-60MB/s, I have a 1Gb connection so perfect world and originally this was around 90MB/s).

I will get around to moving back to the nvme drive, hopefully keeping the status quo and after that I will try test to see if it is a falt with one of my SAS to sata connectors or it could be a fault with one of the two SAS plugs on the HBA board I guess too?

[miccos]

I updated my system and swapped out for a new motherboard, cpu and case.
Now I am kind of back to my oringal problem but I am getting 20-25MB/s downlaods

Could this issue be caused by a bad SAS to Sata cable? or a failing drive?

 

tower-diagnostics-20230504-1403.zip

[JorgeB]

I guess it could, but there's nothing relevant logged, at least for now, though a slow disk would likely not leave a trace, you can run the diskspeed docker to at least see if read performance is normal.

[miccos]

Thanks Jorge,
Not sure what I am looking at to be honest.

[JorgeB]

That looks normal, though it can only test read speeds, but write speed should be similar.

[miccos]

Also found this so seems I haven’t been the only one experiencing this issue.