X9SCM-F slow write speed, good read speed

[Helmonder]

free -l shows:

 

             total       used       free     shared    buffers     cached
Mem:      16609360    5696284   10913076          0      10976    4807496
Low:        767852     665480     102372
High:     15841508    5030804   10810704
-/+ buffers/cache:     877812   15731548
Swap:            0          0          0

 

My system has 32gig onboard.

 

As far as I can see with all my plugins running and a big file transfer going I have enough memory to go around, including low mem...

[moose]

I can also confirm that with 16GB RAM (2x 8GB sticks in slots 2A and 2B) results in a 35-40 MB/s write speed (with Parity disk) for a 1 GB test file.  When I re-tested with the 3rd 8GB stick in slot 1A (24GB RAM total), the slow write speed returned.  I'm running 5.0-rc8a, BIOS 2.0b and no ESXi/pass through.

 

Edit:  just saw that Tom asked me to re-test with 16MB RAM on 12/3/12 and I didn't follow through...sorry Tom.  I still think there might be a kernel/memory conflict that might be able to be improved.

[Helmonder]

I can also confirm that with 16GB RAM (2x 8GB sticks in slots 2A and 2B) results in a 35-40 MB/s write speed (with Parity disk) for a 1 GB test file.  When I re-tested with the 3rd 8GB stick in slot 1A (24GB RAM total), the slow write speed returned.  I'm running 5.0-rc8a, BIOS 2.0b and no EXSi/pass through.

 

Edit:  just saw that Tom asked me to re-test with 16MB RAM on 12/3/12 and I didn't follow through...sorry Tom.  I still think there might be a kernel/memory conflict that might be able to be improved.

 

Wauw... sounds like were zooming into something...

[ptmuldoon]

Can someone confirm I did upgrade/switch from rc5 to rc8a correctly.  I believe all that is necessary is changing/overwriting the boot/bzimage and boot/bzroot files?  The web GUI say rc8a, but just want to confirm I did really upgrade correctly, and am getting good write speeds. 

 

I just put my last stick of memory back in, and am running 32GB of memory again, and in preclearing my smaller 640GB WD, I'm still getting good write speeds.

[moose]

Can someone confirm I did upgrade/switch from rc5 to rc8a correctly.  I believe all that is necessary is changing/overwriting the boot/bzimage and boot/bzroot files?  The web GUI say rc8a, but just want to confirm I did really upgrade correctly, and am getting good write speeds. 

 

I just put my last stick of memory back in, and am running 32GB of memory again, and in preclearing my smaller 640GB WD, I'm still getting good write speeds.

 

Yes this is my understanding of how to correctly upgrade/downgrade between releases (just replace bzimage and bzroot files).

[dgaschk]

Can someone confirm I did upgrade/switch from rc5 to rc8a correctly.  I believe all that is necessary is changing/overwriting the boot/bzimage and boot/bzroot files?  The web GUI say rc8a, but just want to confirm I did really upgrade correctly, and am getting good write speeds. 

 

I just put my last stick of memory back in, and am running 32GB of memory again, and in preclearing my smaller 640GB WD, I'm still getting good write speeds.

 

Yes this is my understanding of how to correctly upgrade/downgrade between releases (just replace bzimage and bzroot files).

 

unRAID main indicates the running version.

[ptmuldoon]

So that means I did upgrade to rc-8a correctly, and I am getting good write speeds with preclear using 32GB of ram.

 

All I did what shutdown and test with 1 stick of Ram at a time.

 

Should I cancel the preclear, and try copying a large file across disks with MC?    Everything I'm doing for testing as well, is strictly internally on the machine/unraid with no lan issues to conflict with.

 

EDIT.  I'm copying a 15GB file across disks with MC and getting speeds of about 14 MB/s.  And this is with 32GB of RAM in the system.

[moose]

So that means I did upgrade to rc-8a correctly, and I am getting good write speeds with preclear using 32GB of ram.

 

All I did what shutdown and test with 1 stick of Ram at a time.

 

Should I cancel the preclear, and try copying a large file across disks with MC?    Everything I'm doing for testing as well, is strictly internally on the machine/unraid with no lan issues to conflict with.

 

EDIT.  I'm copying a 15GB file across disks with MC and getting speeds of about 14 MB/s.  And this is with 32GB of RAM in the system.

 

I'm only getting good array (data+parity) write speeds with 16GB RAM.  24GB or 32GB RAM results in slow writes.  Also 32GB RAM with preclear resulted in slow write speed with my setup.  Maybe this is the the kernel/memory issue that Tom referenced in reply #70 and maybe the difference in our processors causes different limits of RAM to behave differently...just a guess.

[ptmuldoon]

I'm only getting good array (data+parity) write speeds with 16GB RAM.  24GB or 32GB RAM results in slow writes.  Also 32GB RAM with preclear resulted in slow write speed with my setup.  Maybe this is the the kernel/memory issue that Tom referenced in reply #70 and maybe the difference in our processors causes different limits of RAM to behave differently...just a guess.

 

Except I was previously getting very poor write speeds as well with rc-8a and 32GB of ram.  Its only today that I'm getting good preclear and write speeds.  And all I did was overwrite the bzimage and bzroot file to upgrade and than test with 1 stick of memory at time.  All I can think is maybe my resetting the RAM did something? 

[dgaschk]

The workaround for this issue is to use 16G RAM maximum.

[moose]

The workaround for this issue is to use 16G RAM maximum.

 

I'm planning to use ESXi on this computer (X9SCM-F & E3-1270), just wanted to get unRAID successfully working first.  If I use the full 32GB RAM in the motherboard and ESXi but limit unRAID to less RAM (4-8 GB total), theoretically should this work?  I know...the answer is probably "test it and find out". 

[Helmonder]

I will test that out tomorrow morning.

[ptmuldoon]

The workaround for this issue is to use 16G RAM maximum.

 

Until today, I would have agreed with you.  But for reasons I can't explain I am now getting preclear speeds of 100+MB/s and when I copied a 15GB file from disk 1 to disk 3, it moved at ~ 14MB/s.  This is with 32GB in the system, and virtually the same setup as moose.    This is also all 'internally' on the system, with no LAN in the way.  Just disk to disk within unraid via SSH.

 

I'm planning to use ESXi on this computer (X9SCM-F & E3-1270), just wanted to get unRAID successfully working first.  If I use the full 32GB RAM in the motherboard and ESXi but limit unRAID to less RAM (4-8 GB total), theoretically should this work?  I know...the answer is probably "test it and find out". 

 

As far as I know the amount of RAM you set up your VM with is not an impact on the write speeds.  My ESXi UnRaid VM is set up with 4GB of Ram.  And again, I can't explain it.  Last week, this setup of 4GB for the VM with 32GB in the system gave me less than 1MB write speeds with Unraid-Rc8a.    However today, after testing 1 Stick of Memory at a time, I am now getting good write speeds.

 

I'll be happy to post screen shots of anything if it helps.    I am still using Bios Version 2.0a.

[dgaschk]

Shouldn't the 14 be closer to 40?

[Helmonder]

Just removed two sticks of 8igs from my MB, am now booting with 16gig (in the blue slots). Will keep you posted.

[Helmonder]

Well... I can confirm there is a definite difference... Am now running with 16gig and copying speeds in the array are around 9 MB/s. That is still low however, it is somewhat of a workaround that makes it less needed to revert to RC5 for large transfers. I used to get 20 up to 30 MB/s when copying in-array.

 

This is however a big help sofar so thanks to all.. Hopefully it also is a pointer to finding out what is the actual root cause.

[Helmonder]

SOB... Getting back to my system a few minutes later I see transfer has now been dropped back to 2 MB/s ..

 

EDIT: And confirmed several hours later: still at 2 to 3 MB/S. So I do not know why removing memory worked with one of you, but not with me.. Could there be something else you changed when you removed memory ?

[ptmuldoon]

Its get's a little stranger.

 

Yesterday with Rc8, I swear I had internal array transfer speeds of 9 - 10 MB/s.    Now this morning, I've transferred two movies, both around 15GB each.  And there transferring at 30 - 40 MB/s.  I find it starts about 40MB/s and than slows down to about 33MB/s a minute later.

 

All I did yesterday was turn off the machine, and add 1 stick of memory at a time.  I know I did a 'full install' of RC5 originally on my usb drive, and than later overwrote the bzimage and bzroot field.

 

Helmonder.  What king of Memory are using?  I know the board says to use Unbuffered, and I remember the NewEgg reviews said you need Unbuffered if you wanted to use 32GB of Ram.

[dgaschk]

Have y'all checked SMART reports and logs?

[Helmonder]

Have y'all checked SMART reports and logs?

 

Yes, they are all fine.

[Helmonder]

Its get's a little stranger.

 

Yesterday with Rc8, I swear I had internal array transfer speeds of 9 - 10 MB/s.    Now this morning, I've transferred two movies, both around 15GB each.  And there transferring at 30 - 40 MB/s.  I find it starts about 40MB/s and than slows down to about 33MB/s a minute later.

 

All I did yesterday was turn off the machine, and add 1 stick of memory at a time.  I know I did a 'full install' of RC5 originally on my usb drive, and than later overwrote the bzimage and bzroot field.

 

Helmonder.  What king of Memory are using?  I know the board says to use Unbuffered, and I remember the NewEgg reviews said you need Unbuffered if you wanted to use 32GB of Ram.

 

I am using: kvr1333d3e9sk2/16g (thats a kit of two 8gb ones). At the moment I have two strips in, yesterday I had four.

 

That memory came recommended by supermicro so that should not be it..

 

Now you placing the memory one by one and restarting should not make a difference, the whole unraid setup is  created at bootup so there should be nothing that benefits from the onebyone approach..

 

Now you say you also started out with a completely clean usb stick, maybe there is a clue there, some artefact of an old plugin or addon that plays havoc.. I'll see if I can find something..

[Helmonder]

Right.... Struck something.. If it holds that is..

 

I completely removed all the contents out of my "packages" directory and rebooted. I am now getting copy speeds of 15 to 20 MB/s in-array.

 

For some reason I now have no plugins anymore, they should have reinstalled at boot I would have thought..

 

EDIT: All my "unplugged" plugins by influencer have refused to download (bad download), I redownloaded the plugins and put them in the plugins dir, now they install. They probably have changed since the first time I downloaded them.

 

I have everything installed and enabled again. Speed -has- dropped to 9.26 MB/s. I actually saw it going down slowly every time I added a plugin.

 

I installed a newer version of the JRE plugin needed for airvideo.

 

Theorizing: Every plugin costs (low) memory, so loading plugins puts a strain on memory and makes stuff slower. I have cleared out my packages directory and that might mean that less stuff is installed now making copying a bit faster.

 

This even makes the "remove memory" thing fit, adding more memory puts a strain on low memory (it needs to be paged).

 

This basically would point to RC8 (with its new kernel) using more low memory then the previous kernel..

 

On the other hand... I would not be amazed if transfers drop again, they did last time... And 9.6 MB/s is still to slow..

 

EDIT 2: Speeds seem to be holding, I left the system on its own for a while and am now copying the files back that I copied before, speeds are between 13 and 20 MB/s. What really catches is that the transfers start out at 20 MB/s and then get slower (within a minute), while typing the transfer speed had gone down to 9 MB/s

 

EDIT3: And speeds (now 5 minutes later) are back to 5 MB/s. Whatever is happening gets worse by time..

[Helmonder]

Found something that is now causing my transfers to go quicker and quicker:

 

1) Disabled all my plugins

2) run the following command as root: sync; echo 3 > /proc/sys/vm/drop_caches

 

My transfers are now up to 25 MB/s...

 

Please note I am googling my way thru this, I basically do not know what I am doing..

 

This would aid in the conclusion that it is a memory related issue..

 

Time to go to bed now.. Maybe someone with real knowledge can piece this stuff together :-)

[ptmuldoon]

Helmonder.  Since you mentioned plugins you made me remember, and who knows......

 

Yesterday, my transfer rates on the array were about 9-10MB/s.  But I installed SimpleFeatures last night.  Today, those transfer rates are at 35-40MB/S

 

i agree, adding 1 module of memory at time should not impact Unraid.  I only mentioned it thinking perhaps the BIOS was somehow seeing them differently as they got added.

 

The only plugins I have installed at this point are OpenSSH, SimpleFeatures and Screen.

[Helmonder]

Helmonder.  Since you mentioned plugins you made me remember, and who knows......

 

Yesterday, my transfer rates on the array were about 9-10MB/s.  But I installed SimpleFeatures last night.  Today, those transfer rates are at 35-40MB/S

 

i agree, adding 1 module of memory at time should not impact Unraid.  I only mentioned it thinking perhaps the BIOS was somehow seeing them differently as they got added.

 

I only plugins I have installed at this point are OpenSSH, SimpleFeatures and Screen.

 

I love simplefeatures but I do not susepect it beiing able to increase transfer speed :-)  (just kidding).  have it installed to so that makes our setups the same.

 

In the meantime my transfers are at 15 MB/s again..