I'm getting low write speed (11-12MB/s)

[squarefrog]

I am running Unraid on a HP N40L and connecting from a Mac over SMB, through a Linksys E4200 router. I have 3 drives, and 1 parity drive, running Unraid 5.0.4

 

Copying files with rsync is gaining me pretty awful write speeds - between 11 and 12 MB/s. Unraid reports that it's using Gigabit:

 

root@Tower:~# dmesg | grep -i duplex
tg3 0000:02:00.0 eth0: Link is up at 1000 Mbps, full duplex

root@Tower:~# ethtool eth0
Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: on
        Supports Wake-on: g
        Wake-on: g
        Current message level: 0x000000ff (255)
        Link detected: yes

 

As is the Mac:

 

 

This is much lower than the expected 40MB/s I see other people getting. What can I do to see if one of my drives is failing, or if something else is blocking speed?

 

If the cables were not rated for gigabit, would both machines auto link at 100Mb or would they still report 1Gb?

[teamhood]

Check out this thread that I started a few years ago. There is some great information on rsync configuration and I even go step by step how to add.

 

http://lime-technology.com/forum/index.php?topic=13432.30

[garycase]

This is almost always due to a bad cable pair in one of your Ethernet cables, which results 100Mb transfers.    When this happens your data transfer speed is limited to 11-12 MB/s ... which is exactly what you're seeing.

 

The ports themselves still show Gb, as they're "handshaking" with each other and recognize that they're Gb capable; but the cabling simply doesn't support that.

 

It's also possible you have a bad switch port.  If there's no separate switch other than the 4-port one built into your router, try using a different router port.    I'd then use a different set of Ethernet cables.  [You can replace one at-a-time if you want to try and isolate the specific cable that's causing the problem]    Be sure you try ALL of the cables involved -- if you've got the system wired as it should be, that will include a short cable between the UPS and the server.

 

[squarefrog]

Thanks both for the replies.

 

@teamhood: why would this be an rsync issue?

 

@garycase: so if one of the cables is bad, both systems look like they are connected at Gb speeds?

[garycase]

The cables aren't "bad" ... only one of the pairs is.    The NIC's "handshake" to determine the capabilities of the other device ... and Yes, they can negotiate at Gb speeds but only actually be able to transfer at 100Mb if there's any discontinuity in the cable.  ANY cable in the path can cause this, so you need to either replace them all; or try one-at-a-time to isolate the issue unless you happen to have a good cable tester.

 

Note that this can also be caused by a defective switch port (including the 4-port switch built-in to the router).

 

VERY simple to determine if this is the issue => simply replace the cables.

 

 

[Frank1940]

You can also reboot the router and switches.  They are all really small computers and sometimes a reboot will 'fix' some otherwise weird issues with networking speeds.  (It only takes a couple of minutes to do so it is quick and simple and you can that out as be a issue quickly.)

[squarefrog]

@garycase great to know I'll try that. Unfortunately all the rest of my cables are in storage, I'll see if there are any at work I can borrow.

 

@Frank1940 thanks - I did try rebooting them all first

[Frank1940]

Was this a sudden reduction in speed or is this your first experience with using rsync?  (I have never used rsync so I am not that familiar with it.)

 

But I have backup files from my windows machines using both the Win copy command and the Teracopy plugin for Win which adds data verification.  I have  also uploaded BlluRay ISO files (~25GB)using ImgBurn.  These are my observations:

 

1--  Copying a single very large file (>15GB) results in very high speed (~100MB/s) initially until the RAM cache buffer on my unRAID servers become full then the speed drops back and the final average will be in the neighborhood of 40MB/s.

 

2-- Copying large numbers of files, the dynamics change considerably depending on what the mix is.  If the files are fairly large in size (say, at least 10-25MB), the speed is not too slower than a single large files. 

 

3-- Copying very large numbers (>10,000) of small files (say <5MB), then the overhead for the file space allocation and name generation start to become a real issue and the speed drops back to about what you are seeing. 

 

Another factor totally unrelated to the actual copy process, is that the reiserfs file system is reputed to really slow down figuring out where to write files when the drive is close to be full.  I seem to recall a number something 95%...

[squarefrog]

Was this a sudden reduction in speed or is this your first experience with using rsync?  (I have never used rsync so I am not that familiar with it.)

That is a good question. I can't say for certain. I usually start an rsync task then go and do household chores so I rarely actively look at it. Also this time I added the `--progress --human-readable` flags, so it's certainly more noticeable.

 

1--  Copying a single very large file (>15GB) results in very high speed (~100MB/s) initially until the RAM cache buffer on my unRAID servers become full then the speed drops back and the final average will be in the neighborhood of 40MB/s.

This would make sense, and mirror what other users are getting without a cache drive.

 

2-- Copying large numbers of files, the dynamics change considerably depending on what the mix is.  If the files are fairly large in size (say, at least 10-25MB), the speed is not too slower than a single large files. 

It was a large number of files, but they were all 5GB or more so they should have copied quickly.

 

3-- Copying very large numbers (>10,000) of small files (say <5MB), then the overhead for the file space allocation and name generation start to become a real issue and the speed drops back to about what you are seeing. 

I'd definitely expect overhead issues, but not for large files. Incidentally, I connected over AFP to copy some files to a share that resides on a single drive. This is even slower: 6-7MB/s but that drive is slow, and almost full so I forget about it.

 

Another factor totally unrelated to the actual copy process, is that the reiserfs file system is reputed to really slow down figuring out where to write files when the drive is close to be full.  I seem to recall a number something 95%...

 

I don't think the drives are 95% full yet, but again this could make sense. After I move house I'll be adding another drive and at the same time converting my array to one of the new Unraid 6 formats.

 

I think I should mark this as solved as there is little I can do before I move next month. Thanks all for the support.