NFS slowing down over time

[Grobalt]

I have 2 unraid systems which i currently "clone". Copying 40TB of large files from one to the other. Default is v3, but i changed to NFS v4 and restartet the nfs server as well -> behaves same.

 

When i start the nfs service it starts copying with around 180-200 MB/s over 2.5gbit ethernet. After like 12 hours it is around 80-100 MB/s left. Stopping and Starting nfs service again revices original speed.

 

Any idea ? I can find some google results with same problem but no solution.

- NFS simply gradually slows down over time

- NFSv4.1 mount is extremly slow until remount

- NFS low performance after some activity

- Severe NFS performance degradation after LP #2003053

- https://access.redhat.com/solutions/6271621 ( i cannot read the solution)

 

Both running 6.12.3

Edited August 2, 2023 by Grobalt

[dlandon]

31 minutes ago, Grobalt said:

I have 2 unraid systems which i currently "clone". Copying 40TB of large files from one to the other. Default is v3, but i changed to NFS v4 and restartet the nfs server as well -> behaves same.

How did you change from NFSv3 to NFSv4 in Unraid?  Post diagnostics.

[Grobalt]

/etc/nfsmount.conf

 

# Protocol Version [3,4]
# This defines the default protocol version which will
# be used to start the negotiation with the server.
Defaultvers=4

 

then restartet nfs service

unraid-diagnostics-20230802-1455.zip

[dlandon]

That controls what version is used as the starting point for auto negotiation of a client.

 

I need some information on your setup.  I assume you are mounting NFS shares remotely and transferring files over the LAN.  How are you mounting the NFS shares?  Fstab or UD?

[Grobalt]

adding NFS share on main page and click on mount :)

[ChatNoir]

8 hours ago, Grobalt said:

adding NFS share on main page and click on mount

Then using the Unassigned Devices plugin.

[dlandon]

So you are using UD to mount a remote NFS share, then you use a file manager on the local machine to copy from /mnt/remotes/... to desired destination on the local machine?  What is the remote server?  Another Unraid server?  What file manager are you using?

[Grobalt]

as shown in post 1 - both are unraid, both are running same version and i copy via MC in a putty instance

[dlandon]

1 hour ago, Grobalt said:

as shown in post 1 - both are unraid, both are running same version and i copy via MC in a putty instance

I asked because your log does not show any UD mounts to either NFS or CIFS remote shares.

 

Several observations.

 

You have some parity errors:

Aug  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=288
Aug  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=296
Aug  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=304
Aug  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=312
Aug  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=320
Aug  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=328
Aug  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=336
Aug  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=344
AuAug  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=280
g  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=352
Aug  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=360
Aug  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=368
Aug  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=376
Aug  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=384
Aug  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=392
Aug  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=400
Aug  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=408
Aug  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=416
Aug  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=424
Aug  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=432
Aug  1 03:00:15 unraid kernel: md: recovery thread: P incorrect, sector=440

 

You have some Fix Common Problems issues:

Jul 29 20:44:06 unraid root: Fix Common Problems: Warning: Write Cache is disabled on disk2
Jul 29 20:44:06 unraid root: Fix Common Problems: Warning: Write Cache is disabled on disk3
Jul 29 20:44:06 unraid root: Fix Common Problems: Warning: Write Cache is disabled on disk5

 

You are using a realtek NIC.  The drivers for Realtek NICs ar notorious for having issues with Linux because they are not well maintained.

 

My suggestions:

• Set the NFS version in UD Settings to NFSv4 so you don't have to edit the '/etc/nfsount.conf' file after each reboot.
• Fix the FCP issues.
• Fix the parity errors.
• Change NICs.  Intel NICs work very well in Unraid.
• You said you were using NFS mounts.  In the next release of UD, I've made some adjustments in NFS mounts that may help.  Most notable the 'noatime', and 'nodiratime' that won't do disk writes on every file access.

[Grobalt]

Yes, parity will be done after all files are written, I upgraded all disks and have another copy of the files. But an additional nic will be more power draw, not the best option. And no write cache is default at enterprise data disks, therefore I enable that in the go.cfg

[dlandon]

Update UD, unmount and remount your remote shares and see if it makes any difference.

 

1 hour ago, Grobalt said:

But an additional nic will be more power draw, not the best option. And no write cache is default at enterprise data disks

Don't enterprise disks draw more power?  And you're concerned over power draw of a NIC?

[Grobalt]

no they dont draw more power and most of the time they are spin down. Spin up and spin down causing issues on drives is a myth  I have customers with more then 30k disks with a "zero watt storage" design where disks are used like a tape without any problem.