XFS write performance

[nightanole]

One of the things my freenas buddies make fun of me for is the zfs and the 25-35meg write performance of unraid.  What performance is everyone getting out of a xfs array?  I have a 7200rpm 3tb parity drive and on its best day it can do 45 meg a sec over the network, but normally its 30-35meg.

[HellDiverUK]

Uh...112MB/s read and write.  What are you running unRAID on?  A Pentium 3?

[BobPhoenix]

Uh...112MB/s read and write.  What are you running unRAID on?  A Pentium 3?

Your getting that with a parity drive and writes to the array?

[itimpi]

One of the things my freenas buddies make fun of me for is the zfs and the 25-35meg write performance of unraid.  What performance is everyone getting out of a xfs array?  I have a 7200rpm 3tb parity drive and on its best day it can do 45 meg a sec over the network, but normally its 30-35meg.

Note that the way unRAID works means you will never get great performance writing to a parity protected array as every write involves both a read and a write to both the parity drive and the disk holding the file to be written.  Improving performance for writes is one of the justifications for adding a cache drive.  With a cache drive installed performance seems to start getting more like 80-100MB/sec which is where network speed starts becoming the limiting factor.  A cache drive need not be very large to give this advantage unless you copy a LOT of files to the array every day.

 

BTW:  If I remember correctly, FreeNAS is one of the NAS options that requires a LOT of RAM to perform properly - I have seen figures of 1GB per TB of storage mentioned.

[gundamguy]

 

BTW:  If I remember correctly, FreeNAS is one of the NAS options that requires a LOT of RAM to perform properly - I have seen figures of 1GB per TB of storage mentioned.

 

When I was doing research on the various options for my NAS system, I also saw that 1GB per TB of RAM requirement. In fact it's worse than that as it has to be ECC RAM as ZFS was intentionally designed for use with ECC RAM.

 

FreeNAS doesn't make you use ECC RAM, but if your running ZFS without ECC RAM you are at high risk of bitrot.

 

[HellDiverUK]

Uh...112MB/s read and write.  What are you running unRAID on?  A Pentium 3?

Your getting that with a parity drive and writes to the array?

 

Yep.  FX8320E with 16GB RAM, Toshiba MC04 5TB data and parity drives.  The 8 core 4GHz CPU is probably helping things along somewhat...

[nightanole]

Uh...112MB/s read and write.  What are you running unRAID on?  A Pentium 3?

Your getting that with a parity drive and writes to the array?

 

Yep.  FX8320E with 16GB RAM, Toshiba MC04 5TB data and parity drives.  The 8 core 4GHz CPU is probably helping things along somewhat...

 

Yea id say you earned it with a 8 core 16 gig of ram and $500 a piece enterprise drives

 

 

But back to the question no one has shed light on, does switching to XFS have a measurable increase in write speeds?

[itimpi]

But back to the question no one has shed light on, does switching to XFS have a measurable increase in write speeds?

What feedback I have seen suggests that there is no significant speed change in most cases.  The one place that XFS does seem to improve things is creating new files as the disk gets very near full (where reiserfs is known to noticeably slow down).

[garycase]

Uh...112MB/s read and write.  What are you running unRAID on?  A Pentium 3?

Your getting that with a parity drive and writes to the array?

 

Yep.  FX8320E with 16GB RAM, Toshiba MC04 5TB data and parity drives.  The 8 core 4GHz CPU is probably helping things along somewhat...

 

There's simply no way you're writing to the protected array at 112MB/s -- the drives aren't physically capable of doing this (SSDs could, but that's not what you have in the array).    Remember that writing a single sector involves 4 disk I/O's, with a full platter rotation between the 1st and 2nd I/O for each of the involved disks (the data disk you're writing to and the parity disk).

 

I assume you're using a cache drive ... this would make these speeds easy to achieve, but this also means your data isn't fault tolerant until it is later moved to the protected array (at much lower speeds).

 

[Squid]

Unless unRaid is caching all the writes in RAM before sending it out to the drives.  I easily hit 80 Meg/s for the first gig or 2 on my P.O.S. 2nd server (4gig) before it drops down to 20 Meg/s (no cache drive - straight to user shares)

[garycase]

That's true -- if the amount of data written is always less than UnRAID can cache, then the perceived write speed could be at those speeds.    With 16GB of RAM, that's a possibility.  But the actual writes are NOT happening at those speeds.

 

As I'm sure you know, the 80Mb/s you see "for the first gig or 2" isn't actually being written to the disk at that rate ... it's just being cached in the server's memory.

 

 

 

 

[Squid]

That's true -- if the amount of data written is always less than UnRAID can cache, then the perceived write speed could be at those speeds.    With 16GB of RAM, that's a possibility.  But the actual writes are NOT happening at those speeds.

 

As I'm sure you know, the 80Mb/s you see "for the first gig or 2" isn't actually being written to the disk at that rate ... it's just being cached in the server's memory.

lol Yeah thats a no brainer

 

[HellDiverUK]

 

There's simply no way you're writing to the protected array at 112MB/s -- the drives aren't physically capable of doing this (SSDs could, but that's not what you have in the array).   

 

The MC04ACA500E is capable of a constant 170MB/s tranfer rate.  These are not WD Reds...

 

https://storage.toshiba.eu/cms/en/hdd/specialty_drives/product_detail.jsp?productid=650

 

 

I do have a cache drive, but the folders wit the big files (Movies in particular) are not cached, mostly because my cache drive is too small (its a 120GB SSD that also has Docker and all my download/torrent folders and appdata on it, including ~15GB of Plex library data).

 

I was able to populate the Movies folder across gigabit at pretty much full gigabit speeds from my workstation.  Just under 800GB of movies read from a 1TB WD Black in the workstation to the unRAID array.  Windows 8.1 averaged the transfer at 112MB/s.

[garycase]

... The MC04ACA500E is capable of a constant 170MB/s tranfer rate.  These are not WD Reds...

 

Yes, I'm well aware of the drives' capabilities.  Nevertheless, you can NOT write to the protected array at over 100Mb/s.    The sustained rate of one drive isn't the issue -- there are FOUR disk I/O's involved with each write to the array ... with a full platter rotation between each pair of them.  Seek times and rotational latency are the issue -- not how fast the data can transition to/from the platters (even WD Reds can sustain well over 150MB/s -- the areal density is the same for both the Reds and your MC04's ... the only difference is the rotational rate).

 

If you're seeing those speeds, it's either because you're not writing enough data to get past UnRAID's initial disk caching in RAM  [With 16GB of RAM, you could, for example, easily write an entire 4.7GB DVD and it would APPEAR to write at very high speed, limited essentially only by your network speed.]  OR because the cache drive is in fact being used.

 

Assuming it's not the latter, then a very large write (e.g. 40GB) would show the actual speed you can sustain to the array.    With 1TB/platter 7200 rpm drives I suspect you'll get in the range of 50Mb/s, which is a VERY good write speed to the protected array.

 

 

 

[garycase]

... I was able to populate the Movies folder across gigabit at pretty much full gigabit speeds from my workstation.  Just under 800GB of movies read from a 1TB WD Black in the workstation to the unRAID array.  Windows 8.1 averaged the transfer at 112MB/s.

 

You must have done this transfer before you assigned a parity drive.    There's simply no way you can copy 800GB to the protected array in 2 hours !!

 

[hackztor]

Figure most people get 30MB-40MB/s without a cache drive. The parity is being built and that is why it is slow. With a cache drive you can see max about 120MB/s saturating the gigabit link. It is done because of the way unraid is designed with parity and not stripping the data. ZFS has huge ram requirements and I would rather put a cache drive in anyday then try to get 16-32GB of ram. I use a 500gb samsung 850 evo as my cache  (vday special 170 dollars) and it works amazing. I previously used a hard drive and it was fine until I started using plex and the drive would not spin down to idle. I went through 3 hard drives due to this and thought it was time to just bite the bullet and enjoy ssd as cache. It sped up plex artwork loading as well.

[nightanole]

Figure most people get 30MB-40MB/s without a cache drive. The parity is being built and that is why it is slow. With a cache drive you can see max about 120MB/s saturating the gigabit link. It is done because of the way unraid is designed with parity and not stripping the data. ZFS has huge ram requirements and I would rather put a cache drive in anyday then try to get 16-32GB of ram. I use a 500gb samsung 850 evo as my cache  (vday special 170 dollars) and it works amazing. I previously used a hard drive and it was fine until I started using plex and the drive would not spin down to idle. I went through 3 hard drives due to this and thought it was time to just bite the bullet and enjoy ssd as cache. It sped up plex artwork loading as well.

 

Thats rather funny considering unraid is the only flavor i know of that does spin down the drives, zfs etc have them always turning.

[garycase]

A 500GB SSD is indeed a very nice cache 

... it should easily support writes that saturate the Gb link -- and it's large enough that it can probably handle just about everything you throw at it.

 

[hackztor]

Its a side affect of plex. If you actually logged in and clicked the spin down it spins down but then will spin up again randomly due to plex. All the rest of the drives spin down correctly just not cache since plex lived on it.

 

Figure most people get 30MB-40MB/s without a cache drive. The parity is being built and that is why it is slow. With a cache drive you can see max about 120MB/s saturating the gigabit link. It is done because of the way unraid is designed with parity and not stripping the data. ZFS has huge ram requirements and I would rather put a cache drive in anyday then try to get 16-32GB of ram. I use a 500gb samsung 850 evo as my cache  (vday special 170 dollars) and it works amazing. I previously used a hard drive and it was fine until I started using plex and the drive would not spin down to idle. I went through 3 hard drives due to this and thought it was time to just bite the bullet and enjoy ssd as cache. It sped up plex artwork loading as well.

 

Thats rather funny considering unraid is the only flavor i know of that does spin down the drives, zfs etc have them always turning.

[lionelhutz]

To answer your first question, I don't believe XFS will give any significant write speed improvement.

 

You could use reconstruct-write mode. You should be able to double your write speeds. Use this command to turn it on,

 

mdcmd set md_write_method 1

 

The downside is that it'll spin all your disks for any write.

 

Without using this setting I can easily write at speeds around 60MBps so I'd expect to be able to saturate my gigabit connection with it on.

 

Use this command to set it back to the default write mode,

 

mdcmd set md_write_method 0

[HellDiverUK]

... I was able to populate the Movies folder across gigabit at pretty much full gigabit speeds from my workstation.  Just under 800GB of movies read from a 1TB WD Black in the workstation to the unRAID array.  Windows 8.1 averaged the transfer at 112MB/s.

 

You must have done this transfer before you assigned a parity drive.    There's simply no way you can copy 800GB to the protected array in 2 hours !!

 

When beta14 comes along I'll retest.  I'm pretty sure I was getting the rates I remember, but I was dosed with the flu, so maybe my memory is fuzzy.  I will report back.

[gubbgnutten]

That's true -- if the amount of data written is always less than UnRAID can cache, then the perceived write speed could be at those speeds.    With 16GB of RAM, that's a possibility.  But the actual writes are NOT happening at those speeds.

 

As I'm sure you know, the 80Mb/s you see "for the first gig or 2" isn't actually being written to the disk at that rate ... it's just being cached in the server's memory.

 

So true. I have 16GiB of RAM in my server, and most transfers clock in at well above 100MB/s (112MB/s typically) but if I issue the "sync" command directly afterwards it takes some time for it to finish...

 

The RAM buffer generally keeps the perceived array writes quick enough for me not to bother with a cache drive and Mover, or setting reconstruct-write manually in everyday use.

[HellDiverUK]

My main reason for using cache is so the main drives can spin down for noise and power saving.  That's the cool feature of unRAID for me.

[Brucey7]

 

 

Thats rather funny considering unraid is the only flavor i know of that does spin down the drives, zfs etc have them always turning.

 

I do have some Thecus NAS 7700's with ZFS file systems and the drives do spin down when not being accessed

[jphipps]

I had done some testing between Reiserfs and XFS, and didn't really see much difference.  This is the output of a DD test from OSX to unraid over NFS:

 

Jeffs-Mac-mini-2:~ jphipps$ dd if=/dev/zero of=/Volumes/Movies/test.out bs=4096 count=6000000

6000000+0 records in

6000000+0 records out

24576000000 bytes transferred in 247.419756 secs (99329174 bytes/sec)

 

Getting about 99MB/s writing about 24GB which should exceed the cache of my 8GB machine, and I don't use a cache drive.