Just built an unRAID box, and moving my files over from two separate Windows boxes. I'm seeing two very different patterns from the boxes (see attached screenshots). unRAID box is set up with cache drive.

 

The "A" box copies at ~35MB/S

 

The "B" box copies at ~5-8MB/S.

 

 

 

I'm guessing it's not an unRAID issue, since nothing changed between the two writes. I've looked at all my Win7 settings, and they appear identical. Is this normal? Any ideas on how to fix?

Check for NIC driver updates from the manufacturer, restart the Windows machine that is not working correctly, what speed is the NIC in the bad windows machine running at, check the cable (for gigabit speed it should be CAT5e or CAT6).

Check for NIC driver updates from the manufacturer, restart the Windows machine that is not working correctly, what speed is the NIC in the bad windows machine running at, check the cable (for gigabit speed it should be CAT5e or CAT6).

 

Thanks prostuff. I've done all the above, no improvement. Copies between the two Windows machines saturate the GBE link and run @ ~80-90MB/s.

I don't think it's the hardware... here's a screenshot of something interesting:

 

 

I booted up a VM instance of Windows Home Server INSIDE the "B" box. Let's call that B.1. I then tried to copy from B.1 to unRAID, and the network utilization smoothes out. I then tried to copy from the host B box to unRAID, and the spikiness reduced a little bit, but still all over the place. See above - in the foreground window, the left side of the trace is from B, the right side of the trace is from B.1. In the background you see the B.1 VM with it's internal trace. (The "dip" in the B.1 transfer rate was when I tried to do a screen capture...)

 

What's causing this? Are there differences between the way WinServer 2003 and Win7 is set up for network transfers?

 

Also - I can't seem to get a transfer to complete from B to unRAID. It starts to copy, and then after a while it throws a network connection lost error, with 3 options - Retry, Skip, or Cancel. Hitting retry repeated doesn't solve the issue, but hitting cancel will cancel immediately. This doesn't happen from A or from B.1.  ???

 

I'm also seeing a lot of dropped packets on the unRAID side... what could be causing this?

 

NIC info (from ethtool)

Settings for eth0:

Supported ports: [ TP MII ]

Supported link modes:  10baseT/Half 10baseT/Full

                        100baseT/Half 100baseT/Full

                        1000baseT/Half 1000baseT/Full

Supports auto-negotiation: Yes

Advertised link modes:  10baseT/Half 10baseT/Full

                        100baseT/Half 100baseT/Full

                        1000baseT/Half 1000baseT/Full

Advertised auto-negotiation: Yes

Speed: 1000Mb/s

Duplex: Full

Port: MII

PHYAD: 0

Transceiver: internal

Auto-negotiation: on

Supports Wake-on: pumbg

Wake-on: g

Current message level: 0x00000033 (51)

Link detected: yes

 

NIC driver info (from ethtool -i)

driver: r8169

version: 2.3LK-NAPI

firmware-version:

bus-info: 0000:00:0a.0

 

Ethernet config info (from ifconfig)

eth0      Link encap:Ethernet  HWaddr 00:14:6c:c1:af:17 

          inet addr:192.168.10.200  Bcast:192.168.10.255  Mask:255.255.255.0

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:437288120 errors:0 dropped:536687 overruns:0 frame:0

          TX packets:160651054 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:596402093 (568.7 MiB)  TX bytes:53383818 (50.9 MiB)

          Interrupt:17 Base address:0x6000

 

Jumbo frames enabled on some hardware and not on others could cause problems.  Check your NIC properties on the Windows boxes and see if they are enabled on one and not the other.  Try disabling them if they are and see if that makes a difference.  High numbers of dropped packets usually means bad drivers, faulty hardware, or faulty cabling.  Try moving the offending machine to a different port on the switch and/or new cables.

Thanks, Spectrum. I've checked the jumbo-frames setting previously on the Windows boxes and they are not enabled on either computer. I'll double-check all other hardware and settings (on the router, etc.) when I get home. Will also try to troubleshoot to see if one particular device is causing more RX dropes. I have a couple more Qs in the meantime:

 

1. Can unRAID run jumbo frames? How do I check what it's set at?

2. Does the fact that I'm only getting RX drops indicate anything?

3. I'm doing some reading on dropped packets - usually indicating network congestion. I'm running all my drives and the GBE NIC on the PCI bus. Is it possible that PCI bus saturation could lead to dropped RX packets (and therefore I don't see TX dropped packets)??

 

Thanks

From http://en.wikipedia.org/wiki/TCP_global_synchronization:

 

TCP has automatic recovery from dropped packets, which it interprets as congestion on the network (which is usually correct). The sender reduces its sending rate for a certain amount of time, and then tries to find out if the network is no longer congested by increasing the rate again subject to a ramp-up. This is known as the slow-start algorithm.

 

Almost all the senders will use the same time delay before increasing their rates. When these delays expire, at the same time, all the senders will send additional packets, the router queue will again overflow, more packets will be dropped, the senders will all back off for a fixed delay... ad infinitum.

 

This pattern of each sender decreasing and increasing transmission rates at the same time as other senders is referred to as "global synchronization" and leads to inefficient use of bandwidth, due to the large numbers of dropped packets, which must be retransmitted.

 

Is this what I'm seeing with the spikiness?

If you were seeing the TCP algorithm backing off due to congestion up you should see more of a ramp in network traffic than a spike.  It would be like a ramp going up slow then a fast dropoff.  Also it would eventually level out and find the Goldie Lox point (just right) and things would be ok.  Actually, when I first saw your post this was my 1st thought, but the pic you posted of the traffic pattern doesn't support it.

 

I'm not sure about unRAID supporting Jumbo Frames; haven't seen anything that says yea or nea on that.

 

It's theoretically possible to saturate the PCI bus with a GBE NIC, but I wouldn't expect it to get saturated from just one other system connecting.  Possible, just not probable.  Also if you were saturating the PCI bus you would see it from any system, not just one.

 

My bet is that you'll see that dropped packet counter increasing when your transferring from the suspect system and not the other.

It's theoretically possible to saturate the PCI bus with a GBE NIC, but I wouldn't expect it to get saturated from just one other system connecting.

 

I was thinking that since all my drives are attached to SATA cards on on the PCI bus, the combination of network activity and disk activity would saturate the bus.

 

Also if you were saturating the PCI bus you would see it from any system, not just one.

 

Yes, of course. I clearly wasn't thinking straight.    Further validated since transfers from the other system has faster overall throughput, which means the bus isn't saturated yet.

 

Also - since I'm getting steady network utilization from the VMware box (B.1) running inside the offending box (B), it seems likely not a hardware issue. Both Windows boxes A and B are connected to the same switch, which is attached to the rest of my house wiring, so that would potentially eliminate downstream physical layer items as sources.

 

I will try the following tonight and see if it makes a difference:

 

1. Completely uninstall and re-install the NIC drivers on B box.

2. Switch to a different NIC on B box.

 

Anything else I should try? The only other difference I can think of is that box B was originally a Vista installation that was upgraded in place to Win7. There are some utilities that I've been running on the network that still reports Windows Vista as the install. I've read that Vista was much worse when it came to network performance and also had problems copying large files to SMB shares. Short of complete reinstall, though, not sure what I can do to change that.

 

ooh FACEPALM

Did you have any music or videos playing on the offending box when you were making the transfer?  Player doesn't matter....

ooh FACEPALM

Did you have any music or videos playing on the offending box when you were making the transfer?  Player doesn't matter....

 

No - no music or videos *playing* but iTunes was launched and running in the background (as it always is). No streaming occurring, though, during the file copies. I can try copying with iTunes off as well. I've also tried both with VM running and not, and that makes no difference.From the original trace, you can see that CPU was at 0% and RAM was at 56%, both within tolerance. The second trace comparing B to B.1 also shows no network activity between copies.

 

I've also disabled the virtual NICs that were installed with VM ware. Interesting, I don't see any network activity when I copy files from B to B.1. Does it only show packets that move down the wire?

iTunes in the background may very well be it.  Win7 will throttle your network bandwidth if you have audio or video playing because "you don't want your media interrupted do you?"  The defaults are waay to conservative and tank network transfer speeds.  Here is the kb article and the registry key to change to make it all better.  This applies to Vista and Win7.  Yet one more thing to try lol.

iTunes in the background may very well be it.  Win7 will throttle your network bandwidth if you have audio or video playing because "you don't want your media interrupted do you?"  The defaults are waay to conservative and tank network transfer speeds.  Here is the kb article and the registry key to change to make it all better.  This applies to Vista and Win7.  Yet one more thing to try lol.

 

Oh my... if I'm reading that KB article correctly, it throttles to about 1/8 maximum speed (10 packets/millisecond)? That could explain a lot. Can't wait to try it when I get home.

Yeah it really won't have much effect on a 100Mb network but it absolutely guts a Gb network.  Took me forever to find out why files were transferring so smurfing slow when I first installed Vista way back when.  Ended up it was because the first thing I do when I sit down at my computer is start some sort of music playing  I have mine set to 50 and I have never had a hiccup in multimedia that I could attribute to networking.  Another interesting tidbit I found today while looking for that KB article is Compound TCP.  I don't have any more info other than what is there and at Wikipedia, but I have enabled it on my Win7 dev box and will run like that for awhile and see what happens.

I wonder whether I'm experiencing something similar.

 

When copying large files (say, 8GB) to unRAID (always from ubuntu), the file transfer rate will, initially, hit 50+MB/s.  However, every few seconds, the transfer appears to pause for a couple of seconds (the disk activity light on the unRAID box becomes more persistent), then the transfer will continue, showing a rate of around 40MB/s.

 

Is it likely that these pauses are caused by buffering activity on the unRAID server?  I am running a 7200rpm cache drive which has provided a significant boost to write speeds.

 

I'm also seeing an unexpectedly high number of dropped packets (considering that there is little other traffic on the network, and most is throttled by 100Mb bottlenecks elsewhere on the network).  The ubuntu box and unRAID are both configured to Gb network, with two Gb switches in the path.

 

root@Tower:~# ifconfig
eth0      Link encap:Ethernet  HWaddr 70:71:bc:28:f5:6d  
          inet addr:10.2.0.100  Bcast:10.2.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:79653250 errors:0 dropped:3460 overruns:0 frame:0
          TX packets:93642468 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1075533232 (1.0 GiB)  TX bytes:2646633071 (2.4 GiB)
          Memory:fe400000-fe420000 

Yeah it really won't have much effect on a 100Mb network but it absolutely guts a Gb network.  Took me forever to find out why files were transferring so smurfing slow when I first installed Vista way back when.  Ended up it was because the first thing I do when I sit down at my computer is start some sort of music playing   I have mine set to 50 and I have never had a hiccup in multimedia that I could attribute to networking.  Another interesting tidbit I found today while looking for that KB article is Compound TCP.  I don't have any more info other than what is there and at Wikipedia, but I have enabled it on my Win7 dev box and will run like that for awhile and see what happens.

 

So as a follow-up:

 

I tried turning off iTunes and everything else that was running on that box, and it didn't help the situation at all. I completely uninstalled and re-installed the NIC driver and it helped a little, but still seeing dropped packets. The box was connected using a CAT5 cable, which I switched to CAT5e, but I was still seeing errors and couldn't determine whether the rate of errors was slowing down.

 

I finally decided it was one of those weird software settings issues among computers. Here's my logic:

 

The players:

Box "A" - fresh install windows 7 on new hardware.

Box "B" - Windows 7 upgraded in place over Vista back in the day.

Box "B.1" - Windows 2003 virtual machine running on "B"

"unRAID" - 5 year old hardware with unRAID recently installed

 

Copy from A to B - fast, sometimes approaches 80+MB/s. [eliminates hardware failure on A and B]

Copy from A to unRAID - as fast as you can expect (up to 50MB/s to cache and ~30MB/s to array). Some dropped packets during copy seen on unRAID. [this rules out hardware failure on unRAID since I can successfully copy at decent speeds]

Copy from B to unRAID - slow and spiky as seen in prior posts. Lots of dropped packets. Various troubleshooting methods improved situation a little, but didn't solve the problem.

Copy from B.1 to unRAID - improved performance, no spikiness. [this rules out hardware failure on B and any hardware or setting failures along the copying path (switches, router, cabling, etc.), since B.1 is virtual and uses the same hardware as B.

 

My conclusion was the the install of Win7 on top of old Vista left some nasty network protocol configurations, and that I would probably need to re-install Win7 fresh in order to further troubleshoot.

 

I ended up retiring the old unRAID hardware, then rebuilt B hardward as the new unRAID box. This upgraded me from all PCI bus to all SATA/PCIe bus.

 

Result: stable copies/read from all over the network. Copying to array disk shares now in the 40MB/s range. Copying to array user shares in the 38MB/s range. Copying to cache in the 50MB/s (this is limited by the cache disk being old IDE drive -- copying directly to SATA disk without parity shows 60MB/s+). Absolutely no more network errors.

 

Settings for eth0:
Supported ports: [ TP MII ]
Supported link modes:   10baseT/Half 10baseT/Full 
                        100baseT/Half 100baseT/Full 
                        1000baseT/Half 1000baseT/Full 
Supports auto-negotiation: Yes
Advertised link modes:  10baseT/Half 10baseT/Full 
                        100baseT/Half 100baseT/Full 
                        1000baseT/Half 1000baseT/Full 
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000033 (51)
Link detected: yes

NIC driver info (from ethtool -i)
driver: r8169
version: 2.3LK-NAPI
firmware-version: 
bus-info: 0000:04:00.0

Ethernet config info (from ifconfig)
eth0      Link encap:Ethernet  HWaddr 00:16:17:d3:9e:0f  
          inet addr:192.168.10.200  Bcast:192.168.10.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:59170551 errors:0 dropped:0 overruns:0 frame:0
          TX packets:34068840 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1558987284 (1.4 GiB)  TX bytes:1407306613 (1.3 GiB)
          Interrupt:29 Base address:0xa000