A super quick question.  If I transfer a folder/files about 3GB of less I get full gigabit speeds (100-130MB/s).  But as soon as it eclipses that amount it drops, sometimes to 20-30MB/S or even below 10.  I am using 3x250GB HDD's, 1 in parity and the other 2 as disk 1 and 2.  16GB of DDR2 ECC memory and dual Xeon X5460's.  Why does it drop, my switch, PC and server are all gigabit capable.

Because the memory Linux is allocating to use as a writecache is full, and subsequent writes are going directly to your parity parity protected array.  If the 250 gig drives are hdd and not ssd and are probably slow you're actually getting fairly good speed

So 16GB isn't enough for HDD's?  If I had SSD's what kind of performance could I expect?

Your single best performance for writes will be to always get a cache drive.  Either ssd or hdd

Which I have but not the software.  Would 2 Cache drives help?

Here is what actually happens when data is written to a disk in the parity array:

 

unRAID reads the data to be written to, reads the corresponding parity, calculates the change the new data will make to the parity, then writes the data and writes parity. This is why writing is slow and one of the reasons for the cache disk feature. Caching user share writes will first write the data to cache, not involving parity at all (and also not protected by parity at all) and then move it to the parity array at a later scheduled time.

 

And as already mentioned, an initial burst of speed when writing is due to data being cached in memory. That can only last so long before further writing has to wait on the memory-cached data to be written to disk, either the slower parity array or to a cache disk.

 

Hope that helps clear things up for you somewhat. If you have more questions, please keep this explanation in mind when thinking about what to ask.

 

Personally, I don't really care about write speeds so I never cache any of my user shares. Typically, data is written to my array totally unattended anyway, either from downloads my dockers are doing, or from scheduled backups from other computers on the network. YMMV of course.

I'm not sure whether this is your main server or a test setup - 3 250GB drives is a small array.  If this is a test setup it's worth noting that newer/bigger/faster hard drives will be somewhat faster than what you are seeing with those old 250GB drives.  I regularly see 35MB/s - 55MB/s writing to my parity protected array but I'm using much newer 6TB and 3TB drives.  The old 250GB spinner I use as a cache drive is actually so slow that writes to the parity protected array are almost as fast as writes to the cache drive.  Writes to an SSD cache drive will obviously be the fastest option.

Which I have but not the software.  Would 2 Cache drives help?

What do you mean by not the software?

.... And worth noting that the CPU limits write speed too.... Despite a 850 evo cache, the fastest my D525 atom sees is 40 mb/s