X9SCM-F slow write speed, good read speed

[Glimmerman911]

Using a Supermicro MBD-X9SCL+-F, Core i3 2120T 2.6G, 16GB RAM.  2x Seagate ST3000DM001, 2x WD WDBAAY0030HNC-NRSN (but only writing the Seagates).

 

I see no slowdown.

 

Writes to disk share: around 40MB/sec

Writes to user share: around 39MB/sec

 

Writes to cache disk or non-parity protected array peg the network at around 98MB/sec

 

I did have to "fix" win7 network throttling per this:

http://www.daveherbert.info/network-bandwidth-throttling-in-windows-7

 

This is actually the best performance I've ever measured.  Hmm...

 

I upgrade to rc11, and my write performance is back down to 12MB/s.

 

Did you implement all three fixes from this website?  ie the 2 registry settings plus the MMCSS?

How do I find those registry items to modify them?

When I open regedit I see:

 

HKey_Classes_Root

HKey_Current_User

HKey_ Local_Machine

HKey_Users

HKey_Current_Config

[RobJ]

Did you implement all three fixes from this website?  ie the 2 registry settings plus the MMCSS?

How do I find those registry items to modify them?

When I open regedit I see:

 

HKey_Classes_Root

HKey_Current_User

HKey_ Local_Machine

HKey_Users

HKey_Current_Config

 

If you aren't comfortable editing the registry, then you really should not edit the registry, it's too dangerous.  If you have some experience doing it, then just remember that HKLM is HKey_ Local_Machine, and that should be all you need.  Preferably, you should back up your registry first, and/or create a Restore point.

[Glimmerman911]

Local_machine, thank you.  Registry backed up then all three suggestions from the website implemented.

 

Copy of a 4.3 gig file copied at 27 MB/s with upper and lower bounds of 24-28 MB/s. 

 

 

[dgaschk]

Using a Supermicro MBD-X9SCL+-F, Core i3 2120T 2.6G, 16GB RAM.  2x Seagate ST3000DM001, 2x WD WDBAAY0030HNC-NRSN (but only writing the Seagates).

 

I see no slowdown.

 

Writes to disk share: around 40MB/sec

Writes to user share: around 39MB/sec

 

Writes to cache disk or non-parity protected array peg the network at around 98MB/sec

 

I did have to "fix" win7 network throttling per this:

http://www.daveherbert.info/network-bandwidth-throttling-in-windows-7

 

This is actually the best performance I've ever measured.  Hmm...

 

Does anyone with the X9SCL board have slowdown issues? Are all of the problem boards X9SCM?

[PeterB]

Does anyone with the X9SCL board have slowdown issues? Are all of the problem boards X9SCM?

 

As far as I know, the only difference between the SCM and SCL is the C202/C204 chipset.  I would be very surprised if there is any difference in performance!

 

I have just installed an SCM-iiF, but not had time to check the performance.

 

Edit to add:

Okay, I've now done a quick test ...

Writing a 4.7GB file to a cached user share, from my Ubuntu desktop, I get a steady 80MB/sec transfer speed reported.

Writing the same file to an uncached share, it starts off at a healthy 50MB/sec but, by the time the transfer completes, it is down to 12.6MB/sec.

 

During the uncached copy, I did notice that the drive activity light on the parity drive was on solid, whereas the data drive light was flickering.  The other point was that the the drives were still busy for about a minute after the desktop was reporting 100% completion.

 

Hardware spec in my .sig.  The parity and cache drives are on SATA2 motherboard ports, the data drive is on an AOC-USAS2 port.

[mrow]

Im running an Ivy Bridge Xeon (E3-1230v2).  Im wondering if this affects only Ivy Bridge Xeons.  Ive seen folks with Sandy Bridge procs and i3 procs seem to not have this issue.

 

 

Just installed this same CPU last night with two new unbuffered 8GB sticks of RAM. I'm now having the slow write issue. Previously had a Sandy Bridge i3-2120 with two registered 4GB sticks of RAM with no issues at all.

 

I first noticed the issue when sabnzbd downloads and extractions were super slow. Ran "sysctl vm.highmem_is_dirtyable=1" and it sped them up. Will have to try transferring a large file over the network later tonight to see if it's sustained.

[moose]

Im running an Ivy Bridge Xeon (E3-1230v2).  Im wondering if this affects only Ivy Bridge Xeons.  Ive seen folks with Sandy Bridge procs and i3 procs seem to not have this issue.

 

 

Just installed this same CPU last night with two new unbuffered 8GB sticks of RAM. I'm now having the slow write issue. Previously had a Sandy Bridge i3-2120 with two registered 4GB sticks of RAM with no issues at all.

 

I first noticed the issue when sabnzbd downloads and extractions were super slow. Ran "sysctl vm.highmem_is_dirtyable=1" and it sped them up. Will have to try transferring a large file over the network later tonight to see if it's sustained.

 

Interesting observation.  I have a Sandy Bridge Xeon CPU.  I'm running un-buffered (ECC) memory too (and as mentioned previously have the slow write issue, minus the workaround).  The X9SCM-F MB specs "technically" call for un-buffered ECC RAM per SuperMicro website.  X9SCM-F specs

[StevenD]

Im running an Ivy Bridge Xeon (E3-1230v2).  Im wondering if this affects only Ivy Bridge Xeons.  Ive seen folks with Sandy Bridge procs and i3 procs seem to not have this issue.

 

 

Just installed this same CPU last night with two new unbuffered 8GB sticks of RAM. I'm now having the slow write issue. Previously had a Sandy Bridge i3-2120 with two registered 4GB sticks of RAM with no issues at all.

 

I first noticed the issue when sabnzbd downloads and extractions were super slow. Ran "sysctl vm.highmem_is_dirtyable=1" and it sped them up. Will have to try transferring a large file over the network later tonight to see if it's sustained.

 

Interesting observation.  I have a Sandy Bridge Xeon CPU.  I'm running un-buffered memory too (and as mentioned previously have the slow write issue, minus the workaround).  The X9SCM-F MB specs "technically" call for un-buffered ECC RAM per SuperMicro website.  http://www.supermicro.com/products/motherboard/xeon/c202_c204/x9scm-f.cfm

im running ECC ram.

 

 

[Helmonder]

I am using memory advised by supermicro on their website and have the slow write issue when I do not use the dirtyable parameter, even with only 4 gig on board.

[mrow]

I am using memory advised by supermicro on their website and have the slow write issue when I do not use the dirtyable parameter, even with only 4 gig on board.

 

 

Is it unbuffered or registered?

[mrow]

Im running an Ivy Bridge Xeon (E3-1230v2).  Im wondering if this affects only Ivy Bridge Xeons.  Ive seen folks with Sandy Bridge procs and i3 procs seem to not have this issue.

 

 

Just installed this same CPU last night with two new unbuffered 8GB sticks of RAM. I'm now having the slow write issue. Previously had a Sandy Bridge i3-2120 with two registered 4GB sticks of RAM with no issues at all.

 

I first noticed the issue when sabnzbd downloads and extractions were super slow. Ran "sysctl vm.highmem_is_dirtyable=1" and it sped them up. Will have to try transferring a large file over the network later tonight to see if it's sustained.

 

Interesting observation.  I have a Sandy Bridge Xeon CPU.  I'm running un-buffered memory too (and as mentioned previously have the slow write issue, minus the workaround).  The X9SCM-F MB specs "technically" call for un-buffered ECC RAM per SuperMicro website.  http://www.supermicro.com/products/motherboard/xeon/c202_c204/x9scm-f.cfm

im running ECC ram.

 

If you're still using the RAM in your signature that RAM is unbuffered as well.

[Helmonder]

I am using memory advised by supermicro on their website and have the slow write issue when I do not use the dirtyable parameter, even with only 4 gig on board.

 

 

Is it unbuffered or registered?

 

unbuffered

[StevenD]

Im running an Ivy Bridge Xeon (E3-1230v2).  Im wondering if this affects only Ivy Bridge Xeons.  Ive seen folks with Sandy Bridge procs and i3 procs seem to not have this issue.

 

 

Just installed this same CPU last night with two new unbuffered 8GB sticks of RAM. I'm now having the slow write issue. Previously had a Sandy Bridge i3-2120 with two registered 4GB sticks of RAM with no issues at all.

 

I first noticed the issue when sabnzbd downloads and extractions were super slow. Ran "sysctl vm.highmem_is_dirtyable=1" and it sped them up. Will have to try transferring a large file over the network later tonight to see if it's sustained.

 

Interesting observation.  I have a Sandy Bridge Xeon CPU.  I'm running un-buffered memory too (and as mentioned previously have the slow write issue, minus the workaround).  The X9SCM-F MB specs "technically" call for un-buffered ECC RAM per SuperMicro website.  http://www.supermicro.com/products/motherboard/xeon/c202_c204/x9scm-f.cfm

im running ECC ram.

 

If you're still using the RAM in your signature that RAM is unbuffered as well.

 

Yes, that's what I'm using as that's what SuperMicro recommended.

 

[PeterB]

System spec as in my .sig (with Sandy Bridge Xeon).

 

I achieve a sustained write speed of around 26MB/s, writing to a disk share, from Ubuntu desktop, over nfs.  I believe that this is a 'normal' figure, not subject to the slowdown which is being reported.

 

Now, I made an error in placing my order - I meant to get the Ivy Bridge processor (I deliberately purchased 1600 memory), and my Ivy Bridge replacement should arrive in about a week.  I will then be in a position to confirm whether the simple swap to IB introduces the slowdown.

 

However, I suspect that the conditions which give rise to the problem are more complex than just SB/IB swap.

[StevenD]

 

However, I suspect that the conditions which give rise to the problem are more complex than just SB/IB swap.

 

We shall see. Im looking forward to that test.  I dont have a Sandy Bridge laying around or I would try it.

[tcharron]

Just installed this same CPU last night with two new unbuffered 8GB sticks of RAM. I'm now having the slow write issue. Previously had a Sandy Bridge i3-2120 with two registered 4GB sticks of RAM with no issues at all.

 

+1

I upgraded from an i3-2120 (Sandy bridge I think) to an i7-3770 (Ivy Bridge).  My slow write problems only occurred with the i7.  I didn't notice the slowdown until after I had sold the i3, but I know that I had no write performance issues with the i3.

 

[Helmonder]

I have an i3 2120t and I have the slow write issue

[PeterB]

Yes, as I already suspected, it's not just an obvious single component which is causing the slow writes to occur.

 

As I understand it, the problem only occurs when writing to a non-cached share (eg a disk share), but writing to a single disk (eg cache disk) goes at 'normal' speed.  This could mean that the problem relates to the md driver in some way, or it's some contention issue when multiple drives are being written.  So far, the problem has only been discussed in relation to writes (because a non-cached write is the only time multiple drives are accessed?).  What happens if we force multiple, overlapping, reads?  Also, if I understand correctly, the problem only afflicts 'over the network' writes - perhaps the network interface is implicated in some way?

 

Tom seems to have been quiet on this issue recently - iirc, he has set up an X9SCM system and cannot provoke the issue.

 

I think that the only chance we have of understanding the cause is to build a table of system configurations and record the transfer speeds.  Can someone devise a set of standardised tests that anyone can configure, so that we can measure and compare performance?

 

I think we need to obtain figures for single drive and multiple drive, reads and writes, local and over the network.  These results need to be correlated against hardware configuration including: Mobo and bios version, processor, size and type of ram, type of network interface (and bios/firmware version), type and number of disk controllers and which interface the drives under test are connected to.  Also, we need to know whether the 'mem=' parameter and the 'himem_is_dirtyable' configuration have been set.  Do we need to also collect information regarding network topology (switch type) and client machine? ... yes, perhaps the client m/c and network protocol is relevant.

 

My concern is that we may get a number of reports from people who experience the problem, but that the difficulty will be to persuade those who aren't affected to also contribute - I think that 'good' reports may well be as important as 'bad' reports in order for us to track this down.

 

Reading back, I see that one of the earliest posts reports that the write phase of a preclear was also running slowly - if this is the case, it probably exonerates client m/c, network connection and multiple drive activity (but we should still capture that information, if possible).

 

Any comments or further suggestions?

[Helmonder]

Not true. Issue occurs in-array, without network, when copying from disk to disk (or with user shares)

[PeterB]

Not true. Issue occurs in-array, without network, when copying from disk to disk (or with user shares)

 

Hence my penultimate paragraph!

[downloadski]

I did setpup zfs guru in my X9SCM-F, and sustained write rates of 108 Mbyte/sec via the lan to the server.

This makes me think it is not an issue with the mainbord/chipset, but with some piece of software.

Zfsguru is based on freebsd

[dheg]

I don't have the issue and im willing to collaborate on testing

 

Sent from my GT-I9100 using Tapatalk 2

 

 

[RobJ]

From what I've seen, I would not be surprised to find out we have more than one slow write issue, or perhaps there is a primary slow write issue, and a few users have system problems or a drive having problems, slowing down writes.  What is bad about that is that we could postulate a particular cause with specific symptoms, and a user would indicate that "no, can't be that, because it does not behave that way on my machine", only to find out later they have a different issue and we have been sidetracked by their input.  It seems vital to gather good profile info as has been suggested, both the user equipment involved, and the symptoms seen, so that we can define the problem set or sets more exactly.

 

I'll propose a tentative primary problem set to be defined as:

* Supermicro X9SCM motherboard (but may be other similar boards)

* certain writes are slowed to near 1MB/s

* RAM size of at least 4GB (may also be 8GB or 16GB)

* the "mem=4095M" parameter restores full write speed

 

That rather narrowly defines a specific issue, and hopefully with more input, you may be able to both narrow it further (exactly which types of writes are slow, etc), and/or widen it further (other specific motherboards, other speed deficits, etc).  But what is important is being able to find a limited set of characteristics that indicate which slow write issue is involved on a particular users equipment (if indeed there is more than one issue).  For now, the effectiveness of the "mem=4095M" parameter seems to be the one defining characteristic of the primary slow write problem.

 

So I would not eliminate any types of writes or other hardware involvement (such as network involvement) yet, just because they don't seem to apply to a particular user.

 

For equipment, you want to know:

* Motherboard model and BIOS version

* RAM - sizes and types

* Drive controllers

* Network NIC model

* UnRAID version(s)

* Installed addons and plugins

For behavior, you want to know both the write speed now and what you believe it should be, based on actual previous experience (not on what you've heard it should be):

* Local writes to Cache disk

* Local writes to cached User Share

* Local writes to non-cached User Share

* Local writes to onboard Disks

* Local writes to disks on various different controllers

* Network writes to Cache disk

* Network writes to cached User Share

* Network writes to non-cached User Share

* Network writes to onboard Disks

* Network writes to disks on various different controllers

* How the write speeds change as a result of the mem parameter and perhaps the dirty switch

 

And I'm sure there are other things to profile, that's just a starting list.

[PeterB]

Indeed RobJ - wise words.

 

As an example, I have one particular drive which slows almost to a standstill between 75% and 85% of its capacity (removed from my server for exactly this reason).  If I was to test with this drive, it would completely mess up our results!

 

Should we be confining our initial reports to the X9SCM(-F), or should we also be interested in other boards in the X9 series - for instance, my X9SCM-iiF?  It does seem that there are some reports which would seem to match our symptoms, where other mobos are in use.

 

The other thing which concerns me is that we must define how we are to measure the speeds.  If we simply report what is displayed on the client machine, it could be very confusing.  For example, my file writes (as reported on Ubuntu desktop) start off at 50+MB/s, but slow down after 1GB, to around 26MB/s.  The actual write to the disk continues for several seconds after Ubuntu reports 100% completion.  I presume that this is due to some inherent caching somewhere - but I could report a speed of 53MB/s, or 26MB/s, or anything inbetween.  Is there some way that we can measure the actual 'write to disk' speed, or is that not important?  Perhaps what we are really interested in is the 'perceived' (sustained) speed, as seen from the client.

[Frank1940]

PeterB

 

I see the same thing on my main server.  I understand that unRAID (and perhaps, the underlying Linux OS) will use all unused memory to buffer disk writes.  If this is the case, the behavior that both of us have observed is exactly what one would expect.  i.e., a burst of fast write speed at the beginning as the cache is being filled, which then slows down to a more moderate pace after the cache fills up, and finally the originating machine 'indicates' that the write has finished (as it has no  more data to transfer) but the data in the cache still has to be written to the disk array.

 

This behavior is very prominent when using ImgBurn to write Blu-ray iso's. 

 

Frank