X9SCM-F slow write speed, good read speed

[moose]

Further testing has produced the following results:

 

• Typical preclear write speeds (~70 MB/s to 90MB/s) in step #2 (zeroing the disk), with 5.0-rc5 or earlier RCs or Betas
  
• Upgrading to 5.0-rc6 or 5.0-rc8a produces a preclear write speed (~1MB/s) in step#2 (zeroing the disk).
  

 

With each beta or RC change, all I am doing is replacing the bzimage and bzroot files on the flash drive (nothing else) and the only add-on is unmenu.

 

I'm going to take a wild guess and question whether there is some incompatibility with my processor (E3-1270).  Is anyone running a X9SCM-F motherboard and E3-1270 processor and successfully able to achieve normal disk write speeds?

[limetech]

That ST2000DL003-9VT166 is an "Advanced Format" drive, which means it needs to have sectors aligned on 4K boundaries.

 

Is this a "test server" without any user data yet?  If so, I would do this:

 

First, Stop array and verify the linux device identifier assigned to each disk which is the string inside () on the Main page, for your server it appear one disk is (sda) the other is (sdb).

 

Next, from command line, type these commands:

dd if=/dev/zero bs=1M count=1 of=/dev/sda

dd if=/dev/zero bs=1M count=1 of=/dev/sdb

(Notice difference between these is the "sda" or "sdb" at the end of the line, use whatever identifiers have been assigned for your two drives.)

This will clear the first megabyte of each disk, effectively wiping out the existing MBR and file system (ie, make it appear 'unformatted').

 

Now power down server.  Remove flash and re-install unRaid OS 5.0-rc8a from scratch.

 

Plug flash back into server and reboot.  Server should come up with no drives assigned.

Navigate to Settings/Disk Settings and verify "Default partition format" is set to "MBR:4K-aligned" (it should already be set like that).

 

Next, assign your two drives to disk1 and disk2, do not assign parity just yet.  Start server and both drives should appear 'unformatted'.  Go ahead and click 'Format' button and let it finish.

 

Now do some speed tests to the disk1 and disk2 shares.

 

If this solved the problem, then you can Stop array, go to Utils/New Config and clear out the configuration.  You should now be able to assign your 3TB drive to parity and your 2TB drive to disk1.  Start array and let it finish the parity sync while you are preparing your Thanksgiving turkey.

[moose]

Thank you limetech for the detail to troubleshoot/resolve.

 

I'm currently out of town until Friday, however before I left town, I started 4k boundary preclearing the 3TB and 2TB drives in the "test server" with 5.0-rc5 since 5.0-rc5 had no write speed issues based on my earlier testing.

 

So when I get back on Friday, I will continue with the steps you've outlined above to see if this resolves the write speed problem.

[Clink]

My system is showing similar symptoms as moose's, with even lower write speeds of around 500kBps; the hardware is different (SAS card is LSI 1068e based, motherboard is an Intel dq57tm). It makes no difference whether the data drive is connected to the card or to the motherboard; the parity drive was always connected to the latter.

• copy via console shows the same speed, so it isn't network related
  
• hdparm shows all disks at 100+ MBps (except flash of course)
  
• rebuild parity speed is good (80 to 100 MBps)
  
• read speed is good (60+ MBps via network)
  
• 5.0-beta12a has good write speeds (~30MBps)
  
• extras loaded were unmenu and samba 3.6.8 on the rc8a setup
  
• the syslog shows nothing worth of note (and nothing at all during the cp /dev/zero speed tests)
  

Currently I am back to 5.0-beta12a and everything is fine. In all other ways the upgrade to 5.0-rc8a was uneventful except for the more strict samba user rights checking.

 

[moose]

I followed the steps (verbatim) outlined by limetech...no difference in results.  I still get the ~ 1MB/s write speed when writing directly to either disk1 or disk2.

 

1.  Issue the "dd if=/dev/zero bs=1M count=1 of=/dev/sdX" commands from console, power down the unRAID server.

2.  Quick formatting the flash drive.

3.  Unzipping 5.0-rc8a AiO to the flash.

4.  Running the "make_bootable.bat" command on the flash.

5.  Installing the flash, booting the unRAID server, navigating to the main page.

6.  Assigning disk1 and disk2, no parity disk is assigned.

7.  Formatting disk1 and disk2.

8.  Testing read/write speeds from a Win7 client directly to disk1 and disk2.  Same slow write speed results for both disks.

 

When writing to each disk (one at a time), I got the 1 MB/s write speed, however when writing to one disk and then attempting to write to the second disk (both disks at the same time), the second disk write was initially unresponsive, then started writing at ~ 20 KB/s until I cancelled the operation to write to the first disk, then the write to the second disk started to speed up to the ~ 1 MB/s threshold.  It seems like something is saturated at the 1 MB/s write threshold.

[limetech]

I followed the steps (verbatim) outlined by limetech...no difference in results.  I still get the ~ 1MB/s write speed when writing directly to either disk1 or disk2.

...

In your sig it shows you have three types of SATA ports:

a) those on your motherboard

b) those on the AOC-SAS2LP-MV8

c) those on the AOC-SASLP-MV8

 

Let's reduce this further.  Please physically disconnect all your drives (remove the data cable).  Then connect just one drive to say motherboard port and test speed.  Then move that same drive to the SAS2LP and test, then move to the SASLP and test, all with only that one drive connected.  Let me know if you see the same speed issue on all three port types.

[moose]

a) 3TB drive to 6 Gb motherboard SATA port

b) 3TB drive to SAS2LP SATA port

c) 3TB drive to SASLP SATA port

d) 3TB drive to 3 Gb motherboard SATA port

 

I followed the suggested test sequence...only the 3TB drive installed, one cable connected to the 3 TB drive.  Cases (a), (b), © and (d) produced the same slow write speed.  The motherboard has 6 Gb and 3 Gb SATA ports so I tested each of these ports independently, added a 4th test sequence (d).

 

It did appear that with some of the test cases that the write speed started at ~ 10 MB/s for a split second (initial burst) and quickly dropped/settled to the ~ 1 MB/s write speed within 5 seconds or so and remained at ~ 1MB/s until the 1 GB test file completed writing to the unRAID server.

 

Edit:  FYI - reverting 5.0-rc8a back to 5.0-rc5 (by replacing the bzimage and bzroot files) solves the slow write problem with all 4 SATA port scenarios.

[moose]

I'm not sure if any 5.0-rc8a users have a E3-1270 processor (in a X9SCM-F motherboard), however attached are screenshots of the X9SCM-F bios config options for my processor, E3-1270, is there anything here that might be causing the 5.0-rc8a slow write problem?

 

I changed the "power technology" option from "energy efficient" to "custom" and a few extra menu options were presented (so I took a screenshot with "custom").  The second screenshot is the processor "turbo boost technology" options.  If anyone sees a parameter that might be set incorrectly which might explain the slow write speed with 5.0-rc8a , please let me know so I can modify and retest to see if it resolves the problem.

 

syslog is also attached.

 

Just to reiterate, with 5.0-rc5 and earlier releases, write speed is acceptable/normal.

syslog.txt

[limetech]

Please try one more experiment: Assign one of your hard drives to the 'cache' disk and then see if writes are slow or normal to this drive.

 

The big change from -rc5 to -rc6 was an upgrade of the linux kernel from 3.0.x to 3.4.x (in order to fix some other issues).  In looking through the kernel configuration nothing jumps out that should account for reduced disk writes however.  This problem could be an unRaid driver issue and above test would provide a big clue.

[moose]

Following steps were completed:

[*]Issue the "dd if=/dev/zero bs=1M count=1 of=/dev/sdX" commands from console, power down the unRAID server.

[*]Quick format the flash drive and unzip "5.0-rc8a AiO.zip".

[*]Copy "pro2.key" file to /config (add cache drive functionality).

[*]Run "make_bootable.bat" command on the flash drive.

[*]Install flash, boot unRAID server, navigating to the main page.

[*]Assign disk1 (2TB) as data and disk2 (3TB) as cache, no parity disk assigned.

[*]Format disk2 (cache), was no option to format disk1 (data).

[*]Test read/write speeds from a Win7 client directly to disk2 (cache).  Same slow write speed result (~ 1 MB/s) and normal (fast) read speed result.

I would have thought I would have been able to format disk1 (data); I don't believe this would make any difference with this test, but I'll start over from step #1 and see if the data disk is able to be formatted in step #7 on another full test sequence.

 

Edit:  Same result (slow write) on second full test sequence.

[limetech]

In the m/b bios, the setting for "Active Processor Cores" is set to [All].  What are your other choices here?  Can you try changing this to say 2 or even 1?  The linux kernel is configured to support up to 8, but worth a shot.

 

Also somewhere I think you mentioned at one time that parity sync did take place at expected rate?

[moose]

The E3-1270 CPU has (4) cores, complete specs here: http://ark.intel.com/products/52276/Intel-Xeon-Processor-E3-1270-8M-Cache-3_40-GHz.  I set the BIOS to enable only (1) CPU core and still the ~ 1 MB/s write speed to the cache and data disks.  Yes, the parity check speed was ~ 140 MB/s in earlier testing.  I'll re-test the parity check speed.

Edit:  Parity sync (w/ 3TB parity drive and 2TB data drive) is running at ~ 90 MB/s.  Parity check (non-correcting) is running at ~ 110 MB/s.

[dgaschk]

Try a data disk rebuild.

[moose]

The data disk rebuild is rebuilding the data disk at ~ 40 to 50 MB/s.

 

The parity and data disks are both plugged into the 6 Gbps (on-board) SATA ports of the X9SCM-F (Intel Cougar Point 6-port SATA AHCI Controller, (2) 6 Gbps & (4) 3 Gbps SATA ports).  The parity drive is rated at 6 Gbps and data drive is rated at 3 Gbps, so it seems logical that the data disk rebuild would have approximately 1/2 the "parity sync" and "parity check" speeds, correct?

[dgaschk]

It should be about the same. Compare the completion times.

[Helmonder]

Moose: How did you end up with this ?  I have the same motherboard and am also experiencing slow write speeds... When copying within the array (through console MC) I get like 5 ot 6 MB-s.

[moose]

Helmonder, I noticed this when I first loaded 5.0-rc8a and started to run the preclear script on the two drives I have in my test unRAID server.  Step #2 (write zeros to the disk) was painfully slow, so I precleared the drives in the 4.7 server.  Then once 5.0-rc8a was setup with the two precleared drives, I noticed that I when I would write files to the array it was again very slow but the read speed was good.

 

Through various testing with multiple 5.0 Betas and RCs, it appears that with 5.0-rc6 (and onward) that the slow write speed is present and with 5.0-rc5 and below write speed is good.

 

Looking at the unRAID release log (http://lime-technology.com/wiki/index.php/Release_Notes), one change (as Tom noted above) was a change to kernel 3.4.11, which fixed various other issues.  Tom also indicated that he reviewed the kernel configuration and did not see anything obvious.  Tom is looking into the problem.

 

I did see you have a similar thread but a slightly higher write speed (5 - 6 MB/s).  I'm sure this will get resolved.

 

(Also FWIW - In one of the earlier 5.0-rc8a tests I upgraded Samba from 3.6.7 to 3.6.8 using the process outlined in the main 5.0-rc8a announcement thread - no change in write speed performance.)

[Helmonder]

Helmonder, I noticed this when I first loaded 5.0-rc8a and started to run the preclear script on the two drives I have in my test unRAID server.  Step #2 (write zeros to the disk) was painfully slow, so I precleared the drives in the 4.7 server.  Then once 5.0-rc8a was setup with the two precleared drives, I noticed that I when I would write files to the array it was again very slow but the read speed was good.

 

Through various testing with multiple 5.0 Betas and RCs, it appears that with 5.0-rc6 (and onward) that the slow write speed is present and with 5.0-rc5 and below write speed is good.

 

Looking at the unRAID release log (http://lime-technology.com/wiki/index.php/Release_Notes), one change (as Tom noted above) was a change to kernel 3.4.11, which fixed various other issues.  Tom also indicated that he reviewed the kernel configuration and did not see anything obvious.  Tom is looking into the problem.

 

I did see you have a similar thread but a slightly higher write speed (5 - 6 MB/s).  I'm sure this will get resolved.

 

(Also FWIW - In one of the earlier 5.0-rc8a tests I upgraded Samba from 3.6.7 to 3.6.8 using the process outlined in the main 5.0-rc8a announcement thread - no change in write speed performance.)

 

Speed has now been consistent for a few hours at 2,3 MB/s ..

[ptmuldoon]

I seem to be having the same problem as Moose with similar hardware.  Long story short is that I am new to unraid and had thought I had an issue with transfer speeds (See this post Here if interested)

 

After PM’s Moose if he had solved his problem, he recommend commenting here as he still have the same issue.

 

I’ve ‘stripped my system down to the point where I boot from a USB, and have 3 drives connected straight to the MB, and have cut out my M1015 card for now.

 

When running the latest 5.0-rc8a, I get good read speeds.  But the write speeds are terrible, and less than 1MB.

 

Now, when I downgrade to either 5.0-rcr or to the 4.7 stable.  The write speeds improve dramtically back to 100MB/sec.  This is both when attempting to preclear an WD 640GB drive that is seen as “New” by Unmenu

 

My PC specs are as following

Norco 4020 Case

SuperMicro MBD-X9SCM-F-O

Bios Version 2.14.1219

Intel E3-1230 Processor

32GB ECC DDR3 UDIMM Kingston Memory

 

Removed from the System at this point

IMB ServeRaid M1015 Controller

Intel RES2SV240 24-Port 6Gb/s RAID SAS Expander

 

Only Drives in System currently

2 x 640GB WD Caviar

1 x 2TB WD (WD20EADS)

 

 

I also need to double check.  The SuperMicro MBD-X9SCM-F-O has 2 onboard LANs.  When using the Unraid 5 versions (both rc5 and rc8a), I need to connect to Lan 2. But Unraid 4.7 works on Lan1. 

 

I took a bunch of screen shots of the bios, as well screen shots of the preclear write speeds on both 5.0-rc5 and 5.0-rc8a and uploaded them all to Megashare here as well

 

Screenshots

 

And if interested, the actual manual to the MB can be found here

 

MBD-X9SCM-F-O Manual

 

[Helmonder]

How are your speeds when are you copying thru console, from disk to disk using MC ?

 

That cuts out the network completely, for me that is really slow...

 

We are all using the same motherboard.. What bios version are you running ?  I am running 1.1a myself and there is a 2.0 available.. I always try to avoid upgrading biosses unless I have an issue.

 

Bios version is visible in the unraid system info.

[moose]

We are all using the same motherboard.. What bios version are you running ?

 

With earlier testing I've had the same slow write results with BIOS v1.1a, v2.0a and v2.0b (current) on the X9SCM-F.

[ptmuldoon]

I can't find the bios version in the Unraid System Info.  Which 'button' would you that be under? 

 

But I do know that I am on bios version 2.0a.  And I believe Moose is using the latest 2.0b.  Thus, it could be something in the bios settings, which was why I uploaded a bunch of screen shots of the bios settings.

 

But here's a pic of my Bios Version.

 

I can look into changing it to 1.1a as well.  I don't remember why I upgraded it. 

 

Edit:  There website only lists the latest 2.0b bios.  I emailed them to try and get the older 1.1a version, but I a quick google search found them here: http://hardforum.com/showthread.php?t=1666761

 

Think I should downgrade the bios?

[moose]

I can't find the bios version in the Unraid System Info.  Which 'button' would you that be under? 

 

But I do know that I am on bios version 2.0a.  And I believe Moose is using the latest 2.0b.  Thus, it could be something in the bios settings, which was I uploaded a bunch of screen shots.

 

But here's a pic of my Bios Version.

 

I can look into changing it to 1.1a as well.  I don't remember why I upgraded it. 

 

Edit:  There website only lists the latest 2.0b bios.  I emailed them to try and get the older 1.1a version, but I a quick google search found them here: http://hardforum.com/showthread.php?t=1666761

 

Think I should downgrade the bios?

 

When I initially had the slow write problem, I downgraded from v2.0a to v1.1a => problem still existed.  I then went back to v2.0a and then to v2.0b (current) => problem still exists, so I don't think switching BIOS makes a difference.

 

If you do downgrade BIOS (at least to v1.1a), you need to follow a specific sequence to avoid other issues.  Here is where I discuss going from v2.0a to v1.1a BIOS:  http://lime-technology.com/forum/index.php?topic=22675.msg202154#msg202154

 

You can find your BIOS version in unRAID in the "system log", "Utils" -> "System Log", then scroll through the log sequence to locate.

[ptmuldoon]

I confirmed in the system bios that I am at bios Version 2.0a.  But now you scared me off of downgrading the Bios to 1.1a in fear's of bricking the MB, and that it didn't help you either.   

 

I'm going to reassembly the box and pass things back through the M1015 controller as well to ensure I still have good speed with Unraid 5.0-rc5 as well.

 

EDIT.  I reassembled/added my M1015 card and sas expander back in, and with 5.0-rc5, the writes are just fine.  But with 5.0-rc8a, there back under 1/MBS

 

[dgaschk]

Check BIOS settings. See my sig.