network copy to UnRAID faster than storage


Recommended Posts

My setup:

Netgear GSM7328Sv2 Layer-3 Switch with two SFP+ 10Gbit modules installed that convert to RJ45 with Cat6 (no a or e)

On each of the Cat6 ends there is a computer:

One is the UnRAID server (with a 10Gbit ASUS XG-C100C PCI card, MTU of 9000, a M2 cache drive and several seagate HDDs, the main HDD samba share does not use the cache and the mover in order to directly write to the array).

The other is a Windows 10 computer (one HDD and one M2 ssd inside, with a 10Gbit ASUS XG-C100C PCI card).

 

The old problem:

Copying from the Windows Computer to the UnRAID Server failes after a few seconds and the network "freezes". (every time, no copying possible at all.)

Let me explain: I sit on the Windows Computer and start a ~60GB folder copy (there are files in it from 1MB to 8GB). After a few seconds the speed drops down to 0KB/s and the UnRAID server does not respond anymore. That means I still get a pong on a ping, but I can not call the webinterface of the unraid, not open any samba share and also no SSH into the server. All from that one Windows computer.

But in exact this time I can do everything from a second, other windows computer (SSH works, Samba Share read and write works, the webinterface can be called and used, ...).

So I figured out that this must be a Windows problem.

I read many discussions on `Super User` and `Unix & Linux` from stackexchange and tried my best ...

 

The new/current problem:

... After all that reading I tried configuring a few windows settings:

  • under the `Network Settings` I changed from the `128-Bit encryption` to the `40- or 56-Bit encryption`
  • I checked the `Receive Window Auto-Tuning Level` with `netsh interface tcp show global` and set it to experimental with `netsh interface tcp set global autotuninglevel=experimental`
  • under the network interface settings I disabled `Computer can turn off the device to save energy`
  • MTU is set to 1500

And also configured the network interface card on windows:

  • Energy-Efficient Ethernet -> Disabled (default was `Disabled`)
  • Interrupt Moderation -> Disabled (default was `Enabled`)
  • Interrupt Moderation Rate -> Off (default was `Adaptive`)
  • Flow Control -> Disabled (default was `Rx & Tx Enabled`)
  • all `Receive Buffers` -> 4096 (default was `512`)
  • all `Transmit Buffers` -> 8184 (default was `2048`)
  • all `TCP/UDP Checksum Offload` -> Disabled (default was `Rx & Tx Enabled`)
  • Receive Side Scaling -> Enabled (default was `Enabled`)
  • Priority & VLAN -> VLAN Enabled (default was `Priority & VLAN Enabled`)
  • Jumbo Packet -> Disabled (default was `Disabled`)
  • all `Recv Segment Coalescing` -> Disabled (default was `Enabled`)
  • Downshift retries -> Disabled (default was `4`)
  • IPv4 Checksum Offload -> Disabled (default was `Rx & Tx Enabled`)

 

Now I can copy files from Windows to UnRAID's M2 cache with a 100/0 change for success. Copying to M2 will always work. (I created a second samba share that only uses the M2 cache drive in order to verify my thoughts.)

But I can copy files from Windows to UnRAID's HDD (no mover, no cache) with only a 60/40 change for success. Copying to the HDD will fail in around 40% of the tries.

 

So I figured that Windows tries to copy files to unraid faster than the HDD is able to write and that if the HDDs will not keep up fast enough, the copy failes.

That also would explain why copying directly to the M2 cache will always success because ot the higher write speeds.

 

I installed the UnRAID stats plugin and captured this:

1210572860_networkcopy.thumb.PNG.b56faa44c858f642af3e877021f80a3d.PNG

The Array writes without a drop. But the network drops, so I guess I am write: Windows tries to send data faster to UnRAID than the array can write.

(This screenshot carries on like so: The storage continues around 150MB/s without a drop and the Network will peak up to around 200MB/s from time to time and then drop back to 0. This copying in particular succeeded.)

 

Additional:

All copying from windows were copying from a HDD inside Windows.

Copying from the M2 ssd inside Windows to UnRAID makes the network peak even worth: starting at 750MB/s and then dropping down like cracy. Here a failed copy is more likly then from a HDD. I guess only around 20% of the time this will work, so really bad. (A faster timeout because of a more cracy amount for caching somethere? don' know...)

 

My questions:

  1. Where is the data, that is sent too fast, cached? (The RAM does not goes significant higher)
  2. Is it a bad idea to go with Cat6 Rj45 cables to a SFP+ port?
  3. How can I achieve that the overall network speed for writing is lower so that it will not drop? (From Windows viewpoint: better lower but continually write speeds)
  4. Are jumbo packets needed? Turing them on crashes any copying attempt. If yes:
    1. There is a jumbo packet setting inside the network interface on Windows, but also I can set the MTU via cmd/bash. Should I set both ot only one?
    2. The interface setting can be `Disabled`, `9014 Bytes`, `4088 Bytes`, `2040 Bytes`, `16348 Bytes`. The MTU via cmd/bash can be set to anything. Should I set both to `9014` or the MTU to `9000` and the interface to `9014`?
      1. Setting the MTU to 9000 and the Interface to 9014 and I can no longer access the Webinterface and the samba shares
      2. Setting the MTU to 1500 and the Interface to 9014 and I can access the Webinterface and the samba shares
    3. The netgear switch can set a port to a maximum MTU of `9216`, not any higher.
    4. My general Problem in this quesiton 3 is: switch, interface and mtu. Three places to set a size and I have no idea where the headers are part ot the size and where not. So setting what of these three places to what setting.
  5. Setting the `autotuninglevel` to `normal` goes back to the old problem: no copying possible.
  6. Did I changed too many settings in the network card? What should I revert to default?
  7. Can I change network interface settings in UnRAID? e.g. can I change `Flow Control` in UnRAID or other settings? And if so, should I?
Edited by christopher2007
Link to comment

Unraid does use a RAM cache to buffer writes directly to the Array.  Once this RAM cache is full, the speed will drop to the array write speed.   Writes directly to the array seems to peak at about 50MB/s for very large files.  If you are copying small files, the speed will drop dramatically.  (File overhead on both ends are responsible for this...)

 

As I remember, the RAM cache does not show up on the RAM usage graphs since the RAM used is any RAM that is free.  The more RAM you have the larger the RAM cache can be as a general rule.  

 

You also have to remember that your switches have storage designed into them because they have to be able to buffer potential differences in Network speed.  i.e., 10Gb/s to 1Gb/s.   Make sure that the network switch firmware is up to date.

 

Many Unraid users are using SSD cache disk/arrays to speed transfers to the server.  If you decide to try this, you need to look at the write speed of the SSD's since you are using 10Gb/s networking.  There are significant differences between drives that you will see when using 10Gb/s networking.

 

 

 

 

Edited by Frank1940
  • Thanks 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.