X9SCM-F slow write speed, good read speed

[moose]

I'm running 5.0-rc8a and having very slow write speeds with my X9SCM-F motherboard.  I'm experiencing ~ 1.2 MB write speed and ~ 110 MB read speed.  Parity check is ~ 140 MB/s.

 

I have two drives: (1) 3TB parity and (1) 2TB data connected directly to the (2) 6Gb ports on the motherboard.  Both disks (parity & data) are empty.

 

Complete rig specs are in my signature (with this problem I am running bare metal, no ESXi), but note that I removed the 2x SAS2LP-MV8, 1x SASLP-MV8 and EXPI9301CT (add-on SATA controller/Ethernet cards) from the motherboard and still experience the above write slow speed.  (Note:  I also tried with the parity and data drive using a SAS2LP-MV8 controller in a PCI-E x8 slot with the same write speed results.)

 

I have a 1Gb network and the X9SCM-F is connected via LAN1 port only (Intel 82579LM).

 

Edit: This issue is solved with 5.0-rc15. 

syslog-2012-09-21.txt

Smart_Status_Report_parity.txt

Smart_Status_Report_data.txt

[dgaschk]

Collect a syslog after making a transfer. Do a test without parity.

[moose]

I created a 1GB "dummy.txt" file and transferred it to the unRAID server from a Win7 PC via the 1Gb LAN connection with no parity.  Attached is the syslog (without parity) after the file transfer was complete.  The observed write speed was ~ 1.4 MB/s.

 

I hope you didn't want me to attach a syslog after transferring with parity since now I think I would have to rebuild parity and that would take a bit of time.  I can do that if needed.

syslog-2012-09-22_no_parity.txt

[dgaschk]

What does "ifconfig" and "ethtool eth0" show?

[moose]

results below.

 

root@Moose:~# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:25:90:74:76:d0  
          inet addr:192.168.0.31  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:791743 errors:0 dropped:152 overruns:0 frame:0
          TX packets:149471 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1122162714 (1.0 GiB)  TX bytes:16276648 (15.5 MiB)
          Interrupt:16 Memory:f7900000-f7920000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:209 errors:0 dropped:0 overruns:0 frame:0
          TX packets:209 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:28690 (28.0 KiB)  TX bytes:28690 (28.0 KiB)

root@Moose:~# ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes:   10baseT/Half 10baseT/Full 
                        100baseT/Half 100baseT/Full 
                        1000baseT/Full 
Supports auto-negotiation: Yes
Advertised link modes:  10baseT/Half 10baseT/Full 
                        100baseT/Half 100baseT/Full 
                        1000baseT/Full 
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
Link detected: yes

[dgaschk]

The receive path is dropping packets. Try a new Ethernet cable, switch port, switch...

[moose]

Thanks dgaschk, will do.  The wire closet (and switches) are upstairs and the unRAID server is in the basement.  I retrofitted my house with CAT6.  The CAT6 between the wire closet and basement is bundled in conduit, however I don't have all the CAT6 ends terminated in the basement.

 

What I'll do is move the unRAID server upstairs and plug directly into the 1Gb switch with a new Monoprice CAT6 jumper and do the same with the WIN7 machine (directly into the switch with a new CAT6 jumper) and retest/troubleshoot to see if I can eliminate the dropped packet problem with the RX path.

 

I appreciate the assistance.  Thank you!

[moose]

update:  with new CAT6 jumpers - Win7 client <-> GS748Tv4 (1 Gb switch) <-> unRAID server, I am getting the same results.  Cabling does not appear to be the problem.  I'll continue to troubleshoot.

[marcusone]

I get tons of dropped packets and it doesn't seem to be cable or switch related nor does it affect my speed at all.

 

Sent from my SGH-I727R using Tapatalk 2

[dgaschk]

The loss is only about 0.2%. As long as its below 1% it should be ok. Dropped packets are not unusual but most systems have zero. The dropped packets are not causing the dramatic slow down but since they are on the same path as the slow transfers they may be a result of the real issue.

[switchman]

Try a different switch if you have one.

[BetaQuasi]

I'd try a different switch, and also try the other NIC on the X9.  Seen various issues with the 82579LM, so could be another problem with it here.

[moose]

I was having trouble getting the LAN2 NIC on the X9SCM-F to work and found a thread on the hardforum website (http://hardforum.com/showthread.php?p=1038927544) that indicated some people had the EPROM of the NIC get corrupted when the x9scm-f bios was upgraded from v1.x to v2.x, so I contacted Supermicro and they emailed me an EPROM flash utility to fix the corrupted NIC EPROM.

 

I followed the instructions/re-flashed and on-board LAN2 worked, however I still got the slow write speed (~1.2 MB).  I tried on-board LAN1, however now it doesn't work.  I re-flashed the motherboard bios to 2.0a, and cleared CMOS by unplugging AC, shorting JBT1, plugging AC back in.  Same results, LAN1 doesn't work, LAN2 works with ~1.2 MB write speed.

 

I then used a fixed IP for the WIN7 client and connected the WIN7 client directly to LAN2 of the x9scm-f with a new CAT6 jumper (no switch) and got the same ~1.2 MB write speed.

 

I then attempted to try a different LAN adapter (Intel expi9301ctblk), however unRAID only detects the on-board LAN2 (verified via MAC address output of ifconfig command).

 

It seems the X9SCM-F is corrupted.  I'm not sure if there is something that I can do to reset the bios, or if I should downgrade the bios to version 1.1a (I did see that some bricked their boards by attempting to downgrade from v2.x to v1.x bios), or should I RMA the board to Supermicro?  Any advice?

 

I see that dikkiedirk is also asking about the procedure downgrading x9scm-f bios from 2.0a to 1.1a here: http://lime-technology.com/forum/index.php?topic=22810.0

 

update:  I successfully downgraded the bios from v2.0a to v1.1a, however still have the slow write speed and LAN1 still doesn't work in unRAID. No dropped RX packets.

update #2:  I disabled the two on-board LAN ports by moving JPL1/JPL2 to pins 2-3 and got the Intel expi9301ctblk LAN adapter to work in unRAID, however still having the slow write speed problem.  No dropped RX packets.

[dgaschk]

I was having trouble getting the LAN2 NIC on the x9scm-f to work and found a thread on the hardforum website (http://hardforum.com/showthread.php?p=1038927544) that indicated some people had the EPROM of the NIC get corrupted when the x9scm-f bios was upgraded from v1.x to v2.x, so I contacted Supermicro and they emailed me an EPROM flash utility to fix the corrupted NIC EPROM.

 

I followed the instructions/re-flashed and on-board LAN2 worked, however I still got the slow write speed (~1.2 MB).  I tried on-board LAN1, however now it doesn't work.  I re-flashed the motherboard bios to 2.0a, and cleared CMOS by unplugging AC, shorting JBT1, plugging AC back in.  Same results, LAN1 doesn't work, LAN2 works with ~1.2 MB write speed.

 

I then used a fixed IP for the WIN7 client and connected the WIN7 client directly to LAN2 of the x9scm-f with a new CAT6 jumper (no switch) and got the same ~1.2 MB write speed.

 

I then attempted to try a different LAN adapter (Intel expi9301ctblk), however unRAID only detects the on-board LAN2 (verified via MAC address output of ifconfig command).

 

It seems the x9scm-f is corrupted.  I'm not sure if there is something that I can do to reset the bios, or if I should downgrade the bios to version 1.1a (I did see that some bricked their boards by attempting to downgrade from v2.x to v1.x bios), or should I RMA the board to Supermicro?  Any advice?

 

I see that dikkiedirk is also asking about the procedure downgrading x9scm-f bios from 2.0a to 1.1a here: http://lime-technology.com/forum/index.php?topic=22810.0

 

update:  I successfully downgraded the bios from v2.0a to v1.1a, however still have the slow write speed and LAN1 still doesn't work in unRAID. No dropped RX packets.

update #2:  I disabled the two on-board LAN ports by moving JPL1/JPL2 to pins 2-3 and got the Intel expi9301ctblk LAN adapter to work in unRAID, however still having the slow write speed problem.  No dropped RX packets.

 

Where are you copying from? Is there another client you can test?

[moose]

I only have one WIN7 client with a 1Gb connection and that is where I am copying from.  I do have another XPSP3 client (but with a 100Mb NIC) and it's also getting the slow write speed ~ 1.2 MB/s.

 

From the WIN7 client, I can write the same 1GB test file to my other unRAID server (v4.7 in my sig) at ~ 19 MB/s (w/parity), so the client and LAN don't seem to be the problem.

 

I'm considering replacing v5.0-rc8a with v4.7 on the flash drive in the x9scm-f box and seeing if that has any difference.

[moose]

I installed v4.7 on the x9scm-f and got a 20 MB/s write speed from the WIN7 client.  This is similar setup as the previous few tests with v5.0-rc8a, no parity, v1.1a bios on x9, EXPI9301CT 1GB adapter, x9 on-board SATA controller for (1) data drive.  I wonder why the disparity in write speed between v4.7 and v5.0-rc8a.  If I use the v5.0-rc8a AIO file and add my existing pro key file, that is the correct way to start a new v5.0-rc8a server, right?

[SidebandSamurai]

I had slow write speeds also to my unraid when I was on 4.7.  What I did to get better performance is I set the flow control to RX & TX enabled.  Usually by default these settings in Windows 7 are Disabled.  I improved my performacne by 10MB/s  It went from 9-10MB to 19-21MB by just changing that one setting.

 

According to what you have posted, It appears that unraid has already enabled flow control on the unraid side, so it would not hurt to enable it on the Windows7 Side also.

 

--Sideband Samurai

[moose]

I checked the Ethernet adapter in the WIN7 client and flow control was already enabled.

[SidebandSamurai]

Lets eliminate the network card for a second. 

 

Take your GB dummy file (I would use a media file like a movie instead, 1GB or greater).  using the console copy the file from one drive to the next and monitor the speed.  DO NOT USE windows, just the console.  I am checking the performance of your drives to make sure you don't have something gummed up on your array card or motherboard.  I believe you could use Midnight Commander, and Midnight Commander I believe will tell you the throughput.

 

I was not sure, if you tried a new network card or not.  but if you have not, then install a new network card.  I would recommend an Intel card because those seem to be most reliable.  Then run the test again, this time copying using your windows client.  Also try using WINSCP, to see if there is any difference in speed.  Only have one NIC connected to your switch.  Having both connected without bonding them together can cause network communication problems.

 

--Sideband Samurai

[moose]

Unfortunately I only have 1 data disk on this server.  I've tried multiple Intel network cards (on-board and add-on) and SATA controllers (on-board and add-on).  Based on chronological testing in this thread, I've ruled out the network, adapters, controllers and WIN7 client.  The problem is with v5.0-rc8a and the X9SCM-F, however numerous others with this setup don't have the same problem.

 

Would preclearing drives in a v4.7 box create any problems?  I don't believe so.  I did experience slow zeroing (step #2 - writing to disk) when attempting to preclear in the v5.0-rc8a box, so I precleared the parity and data drives in a v4.7 box, so maybe this was the first clue of slow write speeds, however when that same data drive is used in the x9SCM-F with v4.7 I get a 20MB/s write speed, which is ok.

 

I tried WinSCP to transfer a 2GB iso file from the WIN7 client to the X9SCM-F/v5.0-rc8a and had the same result ~ 1.2MB/s write speed.

 

testing components include

network cards:  Intel 82574L (on-board), Intel EXPI9301CT (add-on)

SATA controllers:  Intel Cougar Point 6 port SATA AHCI Controller (on-board) tried multiple 6Gb and 3Gb ports, SAS2LP-MV8 (add-on)

clients:  WIN7, XPSP3

 

I'm out of ideas at the moment.

[dgaschk]

Un-assign parity and test.

[moose]

Parity is already unassigned.  Please see attached screenshot.

 

I haven't reassigned the 3TB parity disk since your post on 9/22/12 asking me to try without parity because reassigning parity would require a parity rebuild (multiple hour delay).

 

[dgaschk]

What does "hdparm -tT /dev/sd?" show?

[moose]

Tower login: root
Linux 3.4.11-unRAID.
root@Tower:~# hdparm -tT /dev/sda

/dev/sda:
Timing cached reads:   25090 MB in  1.99 seconds = 12582.67 MB/sec
Timing buffered disk reads: 434 MB in  3.01 seconds = 144.35 MB/sec
root@Tower:~# exit

[moose]

I have performed the following test scenarios:

 

1.  Formatted a Lexar 8Gb flash drive, then installed 5.0-rc8a AIO (no add-ons, except un-menu).  Preclearing a 2TB drive yields a ~ 1MB/s write in step #2 (zeroing the disk)

2.  Stopped the preclearing process (CTRL-C) and downgraded 5.0-rc8a to 4.7 by overwriting only the "bzimage" and "bzroot" files on the flash drive.

3.  Preclearing the same 2TB drive yields a ~ 90MB/s write in step #2 (zeroing the disk).

 

The 2TB drive is connected directly to one of the X9SCM-F on-board SATA ports.  Could there be a kernel incompatibility with the X9SCM-F motherboard and E3-1270 processor?  Why would there be such a disparity between the preclear write speeds?