RAIDed unRAID

[bubbaQ]

I have not been very concerned with disk speeds in my unRAID boxes since even with RAMdisks, samba/LAN was the bottleneck.

 

With SMB2 in the new betas, the game has changed.  I'm no longer satisfied with 40 MB/s writes to unRAID.  So I have an idea.

 

Some others have already used Areca controllers to RAID-0 the parity drive.  I'm considering getting a 16-port Areca RAID controller, and doing a series of 2-drive RAID-0 sets on the Areca (with 2TB drives), and then letting unRAID see those 4TB RAIDs as its data and parity disks.

 

Comments?

[WeeboTech]

Without write caching and some kernel tuning I wonder if you will see much improvement.

 

What I did was put 8GB of ram in my machine.

Tune the raid md driver to 2 or 4 times the default value.

 

I altered sysctl vm.dirty_ratio=80

 

This gives me bursts up to 60MB/s on writes. it slows down over time as it settles, but still, I get high bursts which is good for a bunch of small files.

 

Now if you use 7200 RPM 2TB drives, I bet you'll get an improvement, I just wonder if the cost for a 16 port areca is worth it.

 

A 2 port areca for $99 was worth it for me.

 

I guess you could put a few SSD's on the areca for a test to see what the best case scenario could possibly be.

 

I'de love to see the results of your test. If it proves worth while, I have an 8 port areca I'll put into play.

 

[bubbaQ]

My problem is that I regularly am writing/moving large (200GB to 800GB) drive images....so bursting higher doesn't help me.

 

2-drive striped RAID-0 should have more effect than spindle speed increase to 7200 rpm.  I did some testing with a couple of cheap "slow" SSDs and their performance was below that of hard drives with faster datarates.

 

 

[WeeboTech]

The burst works up until about 1GB or so, then it slowly slows down.

So you may find the burst and use of system cache helpful.

[bubbaQ]

The burst works up until about 1GB or so, then it slowly slows down.

So you may find the burst and use of system cache helpful.

 

Bursting 1GB of a 200GB file copy doesn't help much   I've already loaded the server up  with 16GB of RAM for buffers.

 

Another issue with large buffers/cache, is doing a move, and it getting lost in a crash because it is "complete" and the original file removed before the cache is flushed.

 

Another benefit is that I have to verify hashes on these 500GB files each time, and the increased read speeds from a RAID-0 2-disk stripe will definitely help.

 

ps.  Sorry about the edit to your message... fat fingers.  I fixed it.

[bcbgboy13]

With 2TB HDs limit one can use the older generation RAID controllers (with onboard memory and BBUs) - probably way way cheaper than one Areca 16 ports. 

[WeeboTech]

The burst works up until about 1GB or so, then it slowly slows down.

So you may find the burst and use of system cache helpful.

 

Bursting 1GB of a 200GB file copy doesn't help much   I've already loaded the server up  with 16GB of RAM for buffers.

 

Another issue with large buffers/cache, is doing a move, and it getting lost in a crash because it is "complete" and the original file removed before the cache is flushed.

 

Another benefit is that I have to verify hashes on these 500GB files each time, and the increased read speeds from a RAID-0 2-disk stripe will definitely help.

 

ps.  Sorry about the edit to your message... fat fingers.  I fixed it.

 

ya know.. I must be tired, It says 200GB but I mistakenly thought 200MB. My bad, yeah, the burst won't help. But I can say I get a steady 50MB/s when writing on the protected array.

 

And yes the RAID-0 on data should improve the read speed a great deal.

 

As far as it getting lost because the data is in cache. I noticed when I use rsync, It flushes pretty fast.

 

In any case, I have not see tremendous improvements in speed with the RAID-0 parity. They are measurable and worthwhile, but not tremendous as I had expected on sequential writes.

 

Since you're going for the 16 port version, I'm sure there will be a larger cache and a BBU, so I bet you would see big improvements.

 

MY tests showed 10GB writes to a fresh new filesystem start at 92MB/s then slow down to a steady 50MB/s at the end of 10GB.

 

 

 

 

[bubbaQ]

In any case, I have not see tremendous improvements in speed with the RAID-0 parity.

 

I remember.  But did you try RAID-0 parity with a RAID-0 data volume?

[bubbaQ]

With 2TB HDs limit one can use the older generation RAID controllers (with onboard memory and BBUs) - probably way way cheaper than one Areca 16 ports. 

 

Do you have a suggestion?  I just automatically chose Areca since 1) I'm familiar with them 2) reputation, and 3) known unRAID support.

[WeeboTech]

In any case, I have not see tremendous improvements in speed with the RAID-0 parity.

 

I remember.  But did you try RAID-0 parity with a RAID-0 data volume?

 

Not with a raid-0 data volume, I could, but didn't think it was worth the extra cost for my situation.

Don't get me wrong, The change in performance was measurable and worth while for my budget.

 

I hope this works for you, it's just every raid-0 parity test I've done was measurable, but not as high a throughput as I expected.

 

If you have access to a 16 port unit, then test it and post the results.

I would suggest the BBU and the highest cache you can fit in the controller.

I'm really interested to see the results of this. It could change the playing field in high performance areas.

 

 

[WeeboTech]

With 2TB HDs limit one can use the older generation RAID controllers (with onboard memory and BBUs) - probably way way cheaper than one Areca 16 ports.  

 

Do you have a suggestion?  I just automatically chose Areca since 1) I'm familiar with them 2) reputation, and 3) known unRAID support.

 

I've tried the 3ware/AMC controller and the Areca. I've had a allot of experience with the 3ware over the years.

This was my first exposure to Areca and I ended up with two controllers. 8 port and a 2 port.

 

For $99 on a used two port model, it was really worth while.

 

 

I would recommend the Areca.

[bubbaQ]

Just thinking out loud about this idea ..... Some drawbacks of doing this:

 

- volumes will be 4TB, so parity checks will be longer.  The first 2TB of parity check (which involve old spindles) should proceed at near the existing rate,  but should run much faster for the 2TB to 4TB range.

 

- I have no idea how 4K alignment will work for RAID volumes.

 

- have to dedicate 2 spindles to parity, rather than 1, so total data capacity is slightly decreased

 

- RAIDed data disks will be a RAID0 set.  This negates one of the advantages of unRAID that the drives can be removed and read in another system.  This is OK with me since I have software for imaging individual RAIDed drives, and reconstructing the RAID set for forensic cases.  I will maintain most of the spindles as-is (as independent disks) where nightly backups and longer-term storage goes.

 

- it will REALLY suck to have the controller die.

 

As an alternative, I could create a 6GB RAID5 cache drive with the 4 drives.  That would give me fast writes that could fully utilize the LAN now that the unRAID betas have SMB2 (and start considering link aggregation), but that doesn’t help READS after the data has been moved by the mover.  What would be nice, is if the cache disk could retain the copy of the data rather than moving it so it was both a write-cache and a read-cache, and somehow keep track of LRU data and delete a file (after confirming it had been successfully moved to a single spindle) to make room when needed on-the-fly.  So on this 6TB cache disk, I’d want files copied to protected storage on unRAID, and left on the cache till my free space threshold (say 1TB) was reached, at which time the LRU files on the cache drive would be checked (i.e. MD5) to make sure they were successfully copied to a data drive, and then deleted from cache.  Or I could just disable the mover, and move things manually.

 

[WeeboTech]

Or I could just disable the mover, and move things manually.

 

I like the idea of a raid5 cache and a custom mover.

 

You might need to re-enable atime updates, but then you can use that as a basis for determining how old something is and if it should be moved.

 

I think samba has a verbose logging mode that tells you when a file is accessed. 

[Johnm]

Just thinking out loud about this idea ..... Some drawbacks of doing this:

 

- volumes will be 4TB, so parity checks will be longer.  The first 2TB of parity check (which involve old spindles) should proceed at near the existing rate,  but should run much faster for the 2TB to 4TB range.

 

- I have no idea how 4K alignment will work for RAID volumes.

 

- have to dedicate 2 spindles to parity, rather than 1, so total data capacity is slightly decreased

 

- RAIDed data disks will be a RAID0 set.  This negates one of the advantages of unRAID that the drives can be removed and read in another system.  This is OK with me since I have software for imaging individual RAIDed drives, and reconstructing the RAID set for forensic cases.  I will maintain most of the spindles as-is (as independent disks) where nightly backups and longer-term storage goes.

 

- it will REALLY suck to have the controller die.

 

As an alternative, I could create a 6GB RAID5 cache drive with the 4 drives.  That would give me fast writes that could fully utilize the LAN now that the unRAID betas have SMB2 (and start considering link aggregation), but that doesn’t help READS after the data has been moved by the mover.  What would be nice, is if the cache disk could retain the copy of the data rather than moving it so it was both a write-cache and a read-cache, and somehow keep track of LRU data and delete a file (after confirming it had been successfully moved to a single spindle) to make room when needed on-the-fly.  So on this 6TB cache disk, I’d want files copied to protected storage on unRAID, and left on the cache till my free space threshold (say 1TB) was reached, at which time the LRU files on the cache drive would be checked (i.e. MD5) to make sure they were successfully copied to a data drive, and then deleted from cache.  Or I could just disable the mover, and move things manually.

 

 

This is what i was thinking last night and to tired to type...

I am thinking out loud here so see if this helps...

 

few things..

1 if you went the  raid0 multi-unraid..  in theory your 4TB drives would parity check should take about the time of a 3TB array.

 

2 newer arecas handle 4k drives just fine. I am using F4's in mine. It saw them as 4k drives.

 

3 you loose no more space then a set of native 4TB drives.

 

personally, i think this sounds like a logistical nightmare...

In a way, you take take a simple system (unRAID) and toss a level into the mix that could be a pita to manage and will be more likely to fail as you will be doubling the drive fail rate. not to mention many green drives do not like raid (WD's and Seagates come to mind). so depending on what drives you have... this might not even be an option (unless you buy new drives).

 

I hate to say it. but if you are looking for such massive performance... maybe you need to tier your storage. place a second server next to your unRAID with a true raid for priority files.

 

The next option is.. a Multi-tier a single unRAID server. I already know you can use an areca 1222 in unraid. you could build a raid5 or raid6 cash drive and put up to 8 Drives into it.

In theory you could make a 14TB (raw RAID5) (tier1) cache drive. Use the "dont move this folder" option on it. This way you leave priority files on the raid5 and then unraid will "mover" all normal files into the (tier2) unRAID file system at night. this way you get the best of both unraid and hardware raid on box in "a safe and not so crazy way".

 

 

depending how important your data is would probably decide your raid level. you could do 5, 50, 6, 60 or even 10 with or without hot spares with the areca..

 

you would loose your read speed on the "unRAID" portion.. but a single spindle read in unraid is not so bad. if you use the tier method, you should not need those files daily?

 

If your file sizes were not so big, a set of of SSDs would be a way to go.

 

I have had 2 areca raids at home, an 8 drive raid5  and 16 drive raid6 (with hot spare) running for almost 2 years now. both are using samsung F4's.

only one drive has died in that time. The hot spare was already rebuilding itself as i even got the email.

we also have countless areca raid at work with both SAS drives and Hitachi 2tb and 3tb drives. I have never seen a controller die (I am sure it has happened but it is rare is my guess).

 

I would assume you could also do it with an LSI controller. I just like that with the areca, there is no driver or OS client. it has is own OS and web GUI in the card. just plug an RJ45 into it and configure/monitor your raid with any web browser. many LSI's are still hardware assisted

 

[bubbaQ]

If your file sizes were not so big ...

 

... then my problem issues (or at least most of them) would go away!

 

I usually have 2 to 3 TB of "live" active data I am working on at any one time.  After that, the bulk (working images) is moved offline, while support files and analysis are kept on-line, but at that point there is no need for speed.

 

Another issue is making vdi disk for VBox from the original drive images.  That doubles the size of things while the project is active, and is slow when working with parity-protected drives in unRAID.

 

Another reason to do RAIDed unRAID, is to see what unRAID does with high datarate rotational media versus lower datarate SSD.... particularly with the aggressive algorithms and onboard cache of the Areca.  Rotational latency should be a lesser factor in that situation so actual throughput should be able to approach the theoretical maximum. 

 

Consider a drive that reads at 100MB/sec and writes at 75MB/sec.  So theoretically it takes 10 seconds of reading and 13.3 seconds of writing for a total of 23.3 seconds to write 1GB to unRAID.  That means 43MB/sec of maximum theoretical throughput (i.e. ignoring latency).  Using this formula, and I/O specs for your drives, you can easily calculate the maximum theoretical throughput for your system.

 

Since it is possible to get over 90% of that theoretical max with existing drives in unRAID, the rotational latency is not costing much w/r/t drives in that datarate range.  With RAID0 data and parity, throughput should nearly double, so the theoretical throughput should do the same.  Will actual throughput do the same, or will the high throughput expose latency as a bottleneck?  Inquiring minds want to know.

 

Of course, none of this was very practical until the latest betas with SMB2... opening one door leads to a roomful of other doors.

[Johnm]

Ugh at V-disks on an unraid... that has to be slow, especially if you are running more then one live.

 

When backing up areca box to areca box at home, I am on copper GiB.. the entire time the SMB2 (win 2008r2 boxes) network is at 100% and this is for random read-writes with a pretty much defragged filesystem.

While there will be an IO overhead, wire is my bottleneck.

 

Your next issue will be your source. if you transferring data from a single mechanical drive including the OS.. thats your new weakness.. if you copying from several at once, then you go back to the wire..

 

I would be curious to see it with dual nics. I don't have a manages switch at home to test. I would guess the array becomes the bottle neck again.. but it might not.

I would love to see both controllers in one box and the transfer speed from one to the other.

 

as far as latency.. i am sure throwing more spindles at the array makes it worse.. not better.

 

if you going for a maximum IO scenario.. toss out all arrays and build an array with 3 Raid-Z2 vdevs with your ZIL/L2ARC on SSD's for cache.. the down side is, lost drives. For example 18 3TB disks will only net you 36TB. But, depending on what drives you loose, you could loose up to 6 disks at once. then pick up some cheap Ebay FC controllers...

---

IDK my brain hurts. i didn't sleep last night.. lol

but yes.. I like the raid5/6 cache drive idea. if the SMB2 works as it should.. you will see large improvements, but at a large price tag...

 

I do this now with 2 boxes.. My win2k8 and my unraid..

it goes to the win box first. It then gets bumped down to unraid when i finish a project, when I copy stuff to a certain folder or after a certain date-stamp.

I also backup the entire win box to my unraid. so even the current projects are there too if i need them (hence the raid 5 and no hotspare).

 

you would be doing this all on one box.

[bubbaQ]

I image drives either directly to the cache drive in the server (large drives) over eSATA, or to the RevoDrive (smaller drives) in my acquisition workstation.  Imaging is of course limited to the read speed of the source drive over eSata but it will come close to saturating the existing cache drive, so nothing else can use the cache drive while imaging.

 

After large drives are imaged, I have a VM running on unRAID that does the indexing and carving.  Then I access the results from the workstation for analysis.  I only make VDIs from the images when necessary.  When I did analysis in the past, I had to pull the image down from unRAID to the local Revodrive because going over the LAN and reading/writing to protected drive on unRAID was too slow.  Drives are getting bigger now... too big for my 480GB Revo, and I'm not going to shell out 3K for a 1TB Revo drive.  Not to mention that the drive images don't compress due to encryption, so the Revo speeds are halved.  So I need to leave images on the server, and to more work over the wire.  Not to mention some new database hooks in the new versions of FTK will make the backend even more important. I've profiled everything many times, and disk busy is eating my lunch.  I can't just keep buying bigger/faster RevoDrives every 6 months.

 

In short, to be more efficient, I need disk access by the workstation to the server to be faster than local (spinning) disk on the workstation.

 

I have to get link aggregation working too, but until SMB2, the LAN was not the bottleneck.  And it won't do much good to solve the LAN bottleneck if the backend won't keep up.

[Johnm]

I did a quick stress test.

 

Areca 1222, raid5, 8x samsung F4's, NTFS, 90% full/fragmented drive (i know i need to clean it up)...

190-380 MB/sec Disk I/O simultaneous read/write (unpacking 3 blurays at once, from the array to the same array)

This was only measured with resource meter. in no way is this scientific, but you get the idea of speed.

As i recall, when the array was new i was in the upper 400's

 

I'll pm you a screen shot..