Hey everyone,

I'm having alot of trouble trying to narrow down the issues I'm having with consistant network access speeds via Windows.

 

I have my Unraid connected via fibre 10GBps link to a network switch with 1x other computer with 10GBps and the others computers having 1GBps connections.

The 10GBps computer seems to work well MOST of the time. I can stream 4k video, pretty quick loading RAW image files but can sometimes spend 20seconds just opening a JPEG. Very intermittant.

 

The other 1GBps computer is running Adobe Lightroom and can take up to 1 minute to change between photos (the photos are 30MB RAW files from Canon cameras). It never used to be like this - everything was very fast and speeds were consistant.

 

iPerf3 backwards and fowards shows 6GBps and 4GBps for the 10GB computer and 950MBps and 940MBps for the 1GBps computer to Unraid so no issues there.

 

I did have ALOT of errors being reported by my SSDs which were between 1 and 3 years old. I have now swapped them all for new Samsung 870's. 2x 2TB and 2x 500GB. These are setup as 2x cache pools. 'General Cache' and 'Editing Cache'. I have the editing cache specified for the storage area that Lightroom uses, with the Mover setup to move them to the array when they get old. I understand pretty well how the cache and the array work and I don't think I'm missing anything.

What I have tried:

Replaced all 4x SSDs which made up 2x cache pools. (2 drives and 2 drives).

Reinstalled fresh copy of Unraid on a new USB and setup the basics to keep testing.

Reinstalled older version 6.9.2 and tested with that which doesn't seem to help much. Things are a bit quicker with the old version but I still get slow-downs.

 

I cannot see any bottlenecks on my network or equipment. CPU usages on all computers and network traffic are all very normal. I've even tried using IP address instead of DNS name when accessing via SMB.

 

I'm leaning towards hardware failure of the motherboard or card to do with the drives. I cannot see any errors and I cannot figure out exactly where the problem is.

EDIT: I also ran the DiskSpeed docker to test the drives. Performance across all drives is very consistant and as fast as expected. The new SSDs are working perfectly and it seems so are the drive controllers.

 

Would someone mind taking a look at my diagnostics and see if there's anything obvious that's causing these stalls in access data across the network please?

 

In my mind:

The network seems to have no bottlenecks or latency issues.

The drive controllers seems to be performing well and not reporting errors.

The drives are brand new and no errors.

Is it SMB?

Are there random stalls happening on the hardware somewhere that I don't know about?

 

Thanks everyone,

Tom.

 

 

monas-diagnostics-20221106-1154.zip

Edited November 6, 20223 yr by 1131546

More fuel for the fire:

 

The 1GBps computer can access the photos quite quickly, like it used to be, when I share the files on the 10GBps computer and access them via SMB windows to windows.

So I don't think the problem is isolated to the 1GBps computer - I'm pretty sure its something to do with inconsistent performance of my Unraid server.

 

Is this normal Unraid behaviour?:

I'm copying files from Unraid to the 1GBps computer and its copying at 100MB per sec. Everyone is happy.

I start browsing the same SMB Share using the 10GBps computer and it starts to slow down the copy that is in progress. Fair enough.

I copy a small file from the same SMB share to the 10GBps computer and the 1GBps computer copy just comes to a grinding halt and just sits there for a minute or two. The 10GBps copy finishes copying the small file quite slowly, then the 1GBps copy starts to ramp up speed again.

 

I also get random pauses in the file copy from Unraid to either computer when copying 100GB of images. The pauses can be anywhere from 30secs to 2minutes.

 

This is doing my head in!

Thanks!

Tom

Edited November 6, 20223 yr by 1131546
More info

You shouldn't bonding 10G & 1G NIC.

Edited November 6, 20223 yr by Vr2Io

2 minutes ago, Vr2Io said:

You shouldn't bonding 10G & 1G NIC.

No bonding. Only 1 10GBps NIC.

The Unraid server is connected to Switch A via a 10GBps fibre.

There is another computer connected to switch A via a 10GBps fibre.

Switch A is connected to Switch B via 1GBps link and the other computers are connected to Switch B.

 

Hope that makes sense.

 

Actually they are bonding in MAC address 18:c0:4d:b9:df:b1, pls try toggle the bonding setting or delete network-rules.cfg file, then setting the 10G NIC be eth0, and try again.

 

bond0: flags=5443<UP,BROADCAST,RUNNING,PROMISC,MASTER,MULTICAST>  mtu 1500
        ether 18:c0:4d:b9:df:b1  txqueuelen 1000  (Ethernet)
        RX packets 329170  bytes 45232652 (43.1 MiB)
        RX errors 0  dropped 85  overruns 0  frame 0
        TX packets 1895322  bytes 2715835227 (2.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

 

eth0: flags=6147<UP,BROADCAST,SLAVE,MULTICAST>  mtu 1500
        ether 18:c0:4d:b9:df:b1  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

 

eth1: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
        ether 18:c0:4d:b9:df:b1  txqueuelen 1000  (Ethernet)
        RX packets 329170  bytes 45232652 (43.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1895322  bytes 2715835227 (2.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

 

# PCI device 0x10ec:0x8168 (r8169)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="18:c0:4d:b9:df:b1", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x15b3:0x1003 (mlx4_core)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="f4:52:14:87:bd:80", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

 

 

Edited November 6, 20223 yr by Vr2Io

2 hours ago, Vr2Io said:

Actually they are bonding in MAC address 18:c0:4d:b9:df:b1, pls try toggle the bonding setting or delete network-rules.cfg file, then setting the 10G NIC be eth0, and try again.

 

 

That's interesting. I didn't even think to check that because this was a new install. Must be a default setting?

 

Thanks I'm trying it now and will report back.

 

Tom

 

 

2 hours ago, Vr2Io said:

Actually they are bonding in MAC address 18:c0:4d:b9:df:b1, pls try toggle the bonding setting or delete network-rules.cfg file, then setting the 10G NIC be eth0, and try again.

 

 

I've disabled bonding and bridging. Made the 10GBps Mellanox card eth0. Trying to keep it simple as possible. I've disabled VMs and Dockers too (not that any were installed).

 

That seems to have improved the 'recovery time' for when a transfer is stalled, however I still feel Unraid can only do 1 thing at a time.

 

If I'm copying files, then I browse for another file, it just reduces the copy speed to 0MBps for about 30 seconds, then recovers. My windows computer just freezes while waiting for it.

 

I've attached a new diagnostic.

 

 

Thanks again,

Tom.

 

 

monas-diagnostics-20221106-1600.zip

Edited November 6, 20223 yr by 1131546

6 hours ago, 1131546 said:

I've disabled bonding and bridging. Made the 10GBps Mellanox card eth0. Trying to keep it simple as possible. I've disabled VMs and Dockers too (not that any were installed).

Network fine now.

 

6 hours ago, 1131546 said:

If I'm copying files, then I browse for another file, it just reduces the copy speed to 0MBps for about 30 seconds, then recovers. My windows computer just freezes while waiting for it.

1. Does both end also 10G NIC ?  otherwise bottleneck there. Highest transfer speed ?

2. Does actual writing happen in SSD instead array disk ? For array (spinner disk) storage performance increase, pls also turn on array reconstruct write mode. 

 

Edited November 6, 20223 yr by Vr2Io

6 hours ago, Vr2Io said:

1. Does both end also 10G NIC ?  otherwise bottleneck there. Highest transfer speed ?

Unraid server is 10G Mellanox card to Ubiquity 10G switch with 10G SFPs and fibre about 2meters long. No bottlenecks there from what I've tested with iperf and it used to work just great.

  

6 hours ago, Vr2Io said:

2. Does actual writing happen in SSD instead array disk ? For array (spinner disk) storage performance increase, pls also turn on array reconstruct write mode. 

When I copy off the array (spinners), i get the expected 130MBps or so, but still get the stalls when asking unraid to do something else.

When I copy of the SSD cache, things are faster as expected with SSD, but the behaviour is the same with the stalls.

Speeds can also vary between 0MBps and the top speeds of the drives for a minute or two with no pattern that I've been able to find.

 

This is interesting: I started a parity check last night - and it seems the server rebooted 10mins after starting. I woke up this morning to find it saying 'Aborted'.

Maybe it's a power supply issue...??

Argh!

 

Thank you again, Vr2lo

Tom.

 

Your real problem was stalls, it is quite common that small stalls (drop out) happen, but if too long period, this usually mean storage issue rather then network issue and hard to advice how to shoot.

1 hour ago, Vr2Io said:

Your real problem was stalls, it is quite common that small stalls (drop out) happen, but if too long period, this usually mean storage issue rather then network issue and hard to advice how to shoot.

Thanks for your help - I'll keep on trying to isolate the issue.

 

Otherwise, I'm just going to replace all hardware and fingers crossed that fixes it!

 

Thanks again Vr2lo.

Tom.