To Cache drive or not to Cache drive?

[terminaldawn]

Thanks Joe

[jw48335]

So I have a quick question about speed in my config.  Here's the config:

 

3 x 3TB green drive sata 2

3 x 1TB 7200rpm sata 2

1 x 60gb sata 3 SSD cache

beta 14, system is dual core with 8gig.

 

The sustained write speed to a share on the 3TB spindles was ~35 megabytes/second with a single write stream and no cache.  With the SSD cache drive enabled, it goes up to ~55 megabytes.  On a cached share on the 1TB spindles it's ~62MB/s.  Writing directly to the SSD cache drive share, I see ~90 megabytes. 

 

Is that about what you would expect?  Is the difference in write speed between the direct cache share and the cached array shares expected?

 

Thanks!

[Rajahal]

So I have a quick question about speed in my config.  Here's the config:

 

3 x 3TB green drive sata 2

3 x 1TB 7200rpm sata 2

1 x 60gb sata 3 SSD cache

beta 14, system is dual core with 8gig.

 

The sustained write speed to a share on the 3TB spindles was ~35 megabytes/second with a single write stream and no cache.  With the SSD cache drive enabled, it goes up to ~55 megabytes.  On a cached share on the 1TB spindles it's ~62MB/s.  Writing directly to the SSD cache drive share, I see ~90 megabytes. 

 

Is that about what you would expect?  Is the difference in write speed between the direct cache share and the cached array shares expected?

 

Thanks!

 

Actually, that seems a bit odd to me.  The write speed to a cached share should be nearly the same speed as writing directly to the cache drive.  If you are writing files larger than 60 GB, then the cache drive will be bypassed and your figures would make sense.  Likewise if the cache drive was full of other data.  However, if your cache drive is empty and you are writing, say, a 10 GB movie to your Movies share, then you should see the same speed no matter where the movie actually ends up (slower 2 TB or faster 1 TB drive) since the write is going to the cache drive first.  I expect there are other factors at play, such as other network traffic, the speed of your SATA controller cards, or the size of the files you are transferring (lots of small files will transfer slower than one large file, even if the total size is the same).  If you give us the full picture of your hardware and network set up, we may be able to figure out what is going on.  However, even the slowest speed you are seeing is plenty fast by most unRAID user's standards, so maybe this is a non-issue to begin with...

[jw48335]

This is a vanilla rig, so I've wiped everything between tests, and the files I'm generating the I/O streams with are 4gig ISOs.  I connected locally to the gig switch that it's on to generate the I/O, to rule out the house network.  I live alone anyway, though, so unless I make traffic, there isn't any.  The house network is cat 6a, so I wasn't really concerned about it.

 

Cache drive is attached to the motherboard with a certified sata 3 cable:

 

http://www.newegg.com/Product/Product.aspx?Item=N82E16813131698

http://www.newegg.com/Product/Product.aspx?Item=N82E16820227725

 

two of the sata drives are on a pci-e x1 card:

 

http://www.amazon.com/gp/product/B005B0A6ZS/ref=oh_o01_s01_i00_details

 

I'm planning on hosting my Adobe Lightroom catalog on this system, so write speed is actually very important, which is why I opted for the SSD.  If you're interested in checking out the system directly, I can host a webex.

 

Thanks,

 

Jeremy

[Rajahal]

Your motherboard's NIC (Realtek 8111E) is incompatible with unRAID 4.7, but you are using 5.0-beta-14, which should be fine.  What's the speed of your source drive?

[jw48335]

Source systems are all on SSD (Macbook pro, Linux htpc, Windows Desktop), and will easily max a 1gig link to each other.  Behavior is consistent no matter the source. 

 

I can drop in a quad port intel gigE card, but I don't think that's it due to the cache drive being so much faster direct.  I hadn't bothered, since mii-tool shows gig full.

 

root@Unraid:/var/log# mii-tool

eth0: negotiated 1000baseT-FD flow-control, link ok

 

The only item of interest in the logs is the following complaint-

 

unable to load firmware patch rtl_nic/rtl8168e-2.fw

 

But that should be harmless.  I'll test with a different gigE card as soon as i get some time.

 

-J

[jw48335]

Dropped in a couple of 1gig Intel NICs, didn't affect the problem.  I should have some more time this evening to mess with it.  It still makes no sense that cached shares are 40-50% slower than using the cache directly.

 

I'm going to move the SSD off the motherboard controller over to the pci-e.  I'll also try limiting the cached share test to a single 7200 1tb drive, and see if those changes affect anything.  It's about the only thing left to try.

 

-J

[jw48335]

If there's anyone following this thread that's using an SSD as a cache drive under 5b14, are you seeing the same speed going directly to cache as to a cached array share?

 

I've tested everything backwards and forwards, including wiping the system and starting again from scratch last evening, and I still can't find an explanation.

 

Thanks,

 

-J

[Rajahal]

I'm still stumped too, but I haven't done much testing with unRAID betas.  Maybe you've uncovered a bug?

[Alex R. Berg]

I have a vanilla 5b14 rig, and am currently running with a Vertex 3 120gb ssd cache drive (totally overkill with that level of ssd, but it was one I already had).

 

I have consistent write speeds to lying at around 100MB/s. That is regardless of whether I write to \\tower\cache or \\tower\myusershare (with caching enabled). I use Windows 7, in its task manager in networking tab, I can see network usage.

 

I have a Intel Gigabit NIC (motherboard with NIC is Asus P8B WS). Actually I have two NICs, and long for the time when unRaid will support Link Aggregation, but that's another story...

 

My read speed from the array through samba lies around 40MB a second. My read speed through ftp from array lies at 75MB/s if it is a large file lying in the first and fast part of my 5400RPM disk. My write speeds lie at 15MB/s (through samba) and 23-30MB/s through ftp (they start at a steady 40MB/s, but after few secs drops).

[loomdog32]

Of these 4 configurations, what would provide the fastest cache?

 

1TB Seagate Barracuda 7200.11 SATA 3Gb/s  or  750GB Seagate Barracuda 7200.11 SATA 3Gb/s

 

As

 

Single Drive  or  2 Drive RAID 0 via Sil3132 PCIe HBA

[Rajahal]

A RAID 0 will almost always be faster than an individual hard drive.  Whether a SIL3132 RAID 0 will work as a cache drive I don't know, I've never tried it (I'll add it to my to-do list, as I have all the hardware necessary to test it).  However, do you really need a cache drive that is faster than a single drive can perform?  If so, you might want to consider an SSD as a safer and potentially cheaper option (depending on the amount of space you need).

[loomdog32]

A RAID 0 will almost always be faster than an individual hard drive.  Whether a SIL3132 RAID 0 will work as a cache drive I don't know, I've never tried it (I'll add it to my to-do list, as I have all the hardware necessary to test it).  However, do you really need a cache drive that is faster than a single drive can perform?  If so, you might want to consider an SSD as a safer and potentially cheaper option (depending on the amount of space you need).

 

I was only thinking of using that as it is existing hardware I already have on hand.

 

I am almost tempted to pull 500GB Seagate Momentus XT Hybrid Drive from my HTPC to test as a cache disk, as I am more than happy with their performance as a boot/OS drive.

[Whomastadon]

Is there a guide that shows how to actually install and configure a cache drive. I can find multiple guides on what they are for and their purpose. But I still don't know how to add a cache drive to my new unraid array

[dikkiedirk]

Is there a guide that shows how to actually install and configure a cache drive. I can find multiple guides on what they are for and their purpose. But I still don't know how to add a cache drive to my new unraid array

 

You need a Plus or Pro license for using a cache drive. Drop it in, connect it, preclear it (don't know if neccessary), add it to the cache slot, and configure your shares to use a cache drive. Rather easy.

[fido666]

At the start of this thread Raj makes the point that "If you attempt a data transfer that is larger than the size of your cache drive, the transfer will fail.  The transfer will not automatically 'spill over' into the parity protected array ".  Does that mean if I start a transfer that is say 600GB in total (as in all selected directories/files) and my Cache drive is 500GB that the transfer will not complete or that it won't start at all?

 

Also, can I force the mover script to run mid-transfer or do I have to wait till there is no disk activity to the cache drive?

 

Sorry if this has been answered already, in a bit of a panic as my OH has started transferring files to the Tardis (my UnRAID box) and I'm not sure how large the total size of the transfer is.

 

Edit: Just looked at the disk transfer on the Windows PC and it's only 480GB so hopefully will not have any issues (my Cache drive is 500GB).

[Johnm]

If there's anyone following this thread that's using an SSD as a cache drive under 5b14, are you seeing the same speed going directly to cache as to a cached array share?

 

I've tested everything backwards and forwards, including wiping the system and starting again from scratch last evening, and I still can't find an explanation.

 

Thanks,

 

-J

 

Sorry, I don't normally look in this topic.

 

Yes. I have a 120GB Vertex3 SSD for a cache drive. I see 100+MB/s to and from the cache drive. I can even write and 100+MB/s while the mover is running. (My mover runs every 2 hours).  Usually I just see a max gigabit flat-line on my Windows box when writing to a cashed share.

 

One thing to keep in mind, there is no trim or advanced garbage collection on the OCZ-3 series drives. they will get dirty and start performing slower.

 

maybe you might want to pop it out of your unraid. quick format it on a windows box and trim it.. then put it back into the unRAID and re-test?

 

I am considering replacing my OCZ with a Corsair performance pro. for better *nix support.

see here http://lime-technology.com/forum/index.php?topic=19499.msg173163#msg173163

 

other factors could be something outside the drive. jumbo packets, cpu over taxed, bad cable, to long of a cable run (if your switch is in the basement), etc.

 

[RobJ]

At the start of this thread Raj makes the point that "If you attempt a data transfer that is larger than the size of your cache drive, the transfer will fail.  The transfer will not automatically 'spill over' into the parity protected array ".  Does that mean if I start a transfer that is say 600GB in total (as in all selected directories/files) and my Cache drive is 500GB that the transfer will not complete or that it won't start at all?

Hopefully, Raj will clarify the statements you quoted.  What I can say is that from unRAID's standpoint, transfers are file by file not groups of files.  In other words, you obviously could not transfer a 600GB file to a 500GB drive, but you could transfer most of a batch of smaller files totaling 600GB to that drive, and then the rest would be written directly to the User Share disks.

 

Also, can I force the mover script to run mid-transfer or do I have to wait till there is no disk activity to the cache drive?

You should be able to run the Mover script at any time, but it will skip any files that are in use, open, or 'mid-transfer'.

 

Sorry if this has been answered already, in a bit of a panic as my OH has started transferring files to the Tardis (my UnRAID box) and I'm not sure how large the total size of the transfer is.

 

Edit: Just looked at the disk transfer on the Windows PC and it's only 480GB so hopefully will not have any issues (my Cache drive is 500GB).

Just a comment, you recall that a 500GB drive is not really 500GB, but closer to 475GB (if I recall) of free space, so that 480GB may not all fit.

[fido666]

In other words, you obviously could not transfer a 600GB file to a 500GB drive, but you could transfer most of a batch of smaller files totaling 600GB to that drive, and then the rest would be written directly to the User Share disks.

 

That's what I wanted to clarify since Raj mentioned the transfer wouldn't spill over.

 

You should be able to run the Mover script at any time, but it will skip any files that are in use, open, or 'mid-transfer'.

 

OK, thanks for confirming.  How can you tell when the mover script has done it's thing?  Should the Cache drive then be empty?  I'm not sure if it just does a copy rather than a pure move.

 

Nevermind, answered my own question by forcing a move and waiting to see what happened to the cache (it's now empty for those wondering).

 

Just a comment, you recall that a 500GB drive is not really 500GB, but closer to 475GB (if I recall) of free space, so that 480GB may not all fit.

 

Good point, it actually appears as a 465GB disk under Windows Properties for the drive.  It was left transferring overnight and seems to have completed OK; I'm assuming the Mover script kicked in while the rest of the transfer was still going and moved the files not marked as in-use.

[jwcolby]

Any chance you could do a "to add a cache drive do this" to this thread?  I have moved back and forth through the many threads and nowhere do I find anything about actually setting up the cache drive.  Tons of "you can now add a cache drive" and " a cache drive does this" but I just am not finding the "Follow these steps to take an existing drive, clean it off, remove it from the array and reassign it as a cache drive" kind of post.

 

I mean it *must* be there somewhere but like so many things UnRaid related, there is an enormous underlying assumption that we are all Linux kinda folks.  Sadly I am not. 

 

I have my smallest drive at the end of the drive list (500 gigs, drive 15).  As it happens I had stuff on it so I dutifully moved everything off.  Boy was that fun.  I poked around until I discovered the move command and then moved disk15 to disk 14 and... hmm... it looks like I moved everything twice.  Once to a subdirectory called disk15 on disk14 and again to the root of disk14.  SIGH.  I do know that it took a long damned time, and this was using the command line in UnRaid.

 

So anyway, Disk15 is now empty.  I then removed it from the array. 

 

Unraid now tells me that it is missing and that dire things are going to happen because the array is missing a disk and please add the disk back as soon as possible.  I cannot for the life of me figure out how to tell UnRaid that I know it is missing, because *I REMOVED IT*.  SIGH.

 

I want to recalc parity to restore my system to it's normal robust state.  I do use UnMenu (or whatever it is called).  I click on Array Management and then click the "Check and correct parity".  Nothing obvious happens.  It continues to say that parity was checked yesterday and that parity is fine.  Yea, except that I just removed a disk from the array.  And UnRaid is telling me everything is NOT fine!  And I just clicked the Check Parity button.  One would assume that it would at the very least tell me that it had checked parity a few minutes ago and that parity was fine.  SIGH.

 

Do we get that UnRaid is my favorite "sits in the corner and runs" system... until I have to do anything on it and then I would rather pull my own teeth with pliers than look at this thing. 

 

Did I mention that I am not a Linux kinda guy and the underlying assumption (AFAICT) is that *everybody* is a Linux kinda guy. 

 

I just wanna do a cache drive!  It shouldn't be this hard.  And I get the distinct feeling (from reading the threads - yep, I really do, and yep I am extremely computer literate), that once I get the damned cache drive installed and recognized I am going to spend another day figuring out how to make it recognized by all my shares.  SIGH. 

 

If I sound unhappy, let me say that as long as I don't have to do anything to it, I LOVE this thing.  It is exactly what it says it is, it does exactly what it says it will and it just works.  I have 13 terabytes of disk half filled with all kinds of valuable stuff and I trust this thing with my life.  Good job Limetech.

 

Now...

 

1) how the heck do I get my array back up after removing this drive.  I mean it is up... but it scared the bejeesus out of me with that "not protected" stuff.  Is it protected?

2) How do I make the disk I just removed be used as the cache drive?

3) How do I get it used by all of my shares?  In fact I write directly to individual disks (and I have 14 of them).  So if I write to any of my disks will it just automatically go to the cache and then be distributed from there?

 

Thanks for understanding my frustrations and for any assistance you can give.

 

Just as an aside, I am a consultant, database analyst programmer in the MS world - C# and SQL Server.  I was told by a friend reading my documentation for my system that "I can't write my way out of a paper bag". 

 

I think that when we are inside, when we know the stuff too well, then we are just unable to appreciate that instructions have to be *really basic* for outsiders.  Step by excruciating (to us) step, every keystroke because the outsider is going to do exactly what they are reading and when a step is missing...

 

In this case the entire chapter is missing!  Or buried deep in the bowels of a WIKI somewhere?  I found this thread and was so excited that I was finally going to get the missing instructions...

[dgaschk]

See here: http://lime-technology.com/wiki/index.php/FAQ#How_do_I_remove_a_hard_disk_that_I_do_not_plan_on_replacing.3F

 

And then assign the drive as cache.

[boof]

Did I mention that I am not a Linux kinda guy and the underlying assumption (AFAICT) is that *everybody* is a Linux kinda guy. 

 

1) how the heck do I get my array back up after removing this drive.  I mean it is up... but it scared the bejeesus out of me with that "not protected" stuff.  Is it protected?

2) How do I make the disk I just removed be used as the cache drive?

3) How do I get it used by all of my shares?  In fact I write directly to individual disks (and I have 14 of them).  So if I write to any of my disks will it just automatically go to the cache and then be distributed from there?

 

1) See the info linked above.

 

2) Once your array is 'normal' and you just have your disk floating around spare - stop the array and assign it in the cache drive slot. You may have to format it too, but that can be done via the web interface / is generally the same procedure as just adding any data drive. The big difference is that parity doesn't need to be updated.

 

In your global share settings change 'use cache disk' to yes and, optionally, tweak the min free space parameter. But you can just leave this as the sensible default.

 

Once done you will have to configure each of your *user* shares individually and tell them also to 'use cache disk = yes'. This can be set just in the usual place

where you configure each share.

 

That's it!

 

3) Cache drive will only be used for *user* shares. Not writing direct to the disk shares. You need to have user shares enabled and, as above, tell each user share to use the cache drive individually. If you're not using user shares the cache drive won't do anything for you.

 

Sorry this is vague and not the step by step you wanted but it's off the top of my head. See how you get on.

 

Regarding not being a linux guy - all this stuff is done via the unraid web gui. The OS underneath and terminologies used (for purpose of setting up a basic cache drive setup) is completely hidden / agnostic. If unraid ran on windows you would likely have the same interface and steps to perform.

 

What I'm saying, in a roundabout way, is don't be afraid of this stuff because it's linux and you're not a linux guy. Just apply your general computer knowledge / common sense to the actual concepts (as you've suggested you do in the rest of your post) and you'll be fine. For the purposes of what you're doing here the fact it's linux is a red herring so don't let it get in your way.

 

Good luck / post back if you get stuck.

 

You are correct in that the official docs are poor. The user content is better but by it's nature not overly cohesive. Think about the problems you'd have if it didn't exist at all though - or if this forum wasn't here to help...

 

[Sean_W_Smith]

So if I have a 250G cache drive and try to copy 300G to the array the cache drive fills. Is this normal.  Why doesn't the mover start as the cache drive is filling.  This makes using the cache drive useless for me.

[UhClem]

So if I have a 250G cache drive and try to copy 300G to the array the cache drive fills. Is this normal.  Why doesn't the mover start as the cache drive is filling.  This makes using the cache drive useless for me.

Either that (ie, "... mover start ...") or stop calling it a cache drive--maybe, a staging drive.

 

[Joe L.]

Any chance you could do a "to add a cache drive do this" to this thread?  I have moved back and forth through the many threads and nowhere do I find anything about actually setting up the cache drive.  Tons of "you can now add a cache drive" and " a cache drive does this" but I just am not finding the "Follow these steps to take an existing drive, clean it off, remove it from the array and reassign it as a cache drive" kind of post.

 

I mean it *must* be there somewhere but like so many things UnRaid related, there is an enormous underlying assumption that we are all Linux kinda folks.  Sadly I am not. 

 

I have my smallest drive at the end of the drive list (500 gigs, drive 15).  As it happens I had stuff on it so I dutifully moved everything off.  Boy was that fun.  I poked around until I discovered the move command and then moved disk15 to disk 14 and... hmm... it looks like I moved everything twice.  Once to a subdirectory called disk15 on disk14 and again to the root of disk14.  SIGH.  I do know that it took a long damned time, and this was using the command line in UnRaid.

 

So anyway, Disk15 is now empty.  I then removed it from the array. 

 

Unraid now tells me that it is missing and that dire things are going to happen because the array is missing a disk and please add the disk back as soon as possible.  I cannot for the life of me figure out how to tell UnRaid that I know it is missing, because *I REMOVED IT*.  SIGH.

You must now set a new disk configuration. 

You must Stop the array first.

Then, on the 5.0 series, there is a utility to Set a new Disk configuration.  In the 4.7 and prior there is an "initconfig" command line command. (see below)

I want to recalc parity to restore my system to it's normal robust state.  I do use UnMenu (or whatever it is called).  I click on Array Management and then click the "Check and correct parity".  Nothing obvious happens.  It continues to say that parity was checked yesterday and that parity is fine.  Yea, except that I just removed a disk from the array.  And UnRaid is telling me everything is NOT fine!  And I just clicked the Check Parity button.  One would assume that it would at the very least tell me that it had checked parity a few minutes ago and that parity was fine.  SIGH.

  No, it should tell you there is no way to check parity with a missing disk.  Basically, the button does nothing if all your disks are not online.

Do we get that UnRaid is my favorite "sits in the corner and runs" system... until I have to do anything on it and then I would rather pull my own teeth with pliers than look at this thing. 

 

Did I mention that I am not a Linux kinda guy and the underlying assumption (AFAICT) is that *everybody* is a Linux kinda guy. 

 

I just wanna do a cache drive!  It shouldn't be this hard.  And I get the distinct feeling (from reading the threads - yep, I really do, and yep I am extremely computer literate), that once I get the damned cache drive installed and recognized I am going to spend another day figuring out how to make it recognized by all my shares.  SIGH. 

 

If I sound unhappy, let me say that as long as I don't have to do anything to it, I LOVE this thing.  It is exactly what it says it is, it does exactly what it says it will and it just works.  I have 13 terabytes of disk half filled with all kinds of valuable stuff and I trust this thing with my life.  Good job Limetech.

 

Now...

 

1) how the heck do I get my array back up after removing this drive.  I mean it is up... but it scared the bejeesus out of me with that "not protected" stuff.  Is it protected?

You set a new disk configuration.

See here in the WIKI

http://lime-technology.com/wiki/index.php/FAQ#How_do_I_remove_a_hard_disk_that_I_do_not_plan_on_replacing.3F'>http://lime-technology.com/wiki/index.php/FAQ#How_do_I_remove_a_hard_disk_that_I_do_not_plan_on_replacing.3F

2) How do I make the disk I just removed be used as the cache drive?

See here in the WIKI

http://lime-technology.com/wiki/index.php/Cache_disk

3) How do I get it used by all of my shares?  In fact I write directly to individual disks (and I have 14 of them).  So if I write to any of my disks will it just automatically go to the cache and then be distributed from there?

It must be enabled for each share.  Once enabled on a given user-share (On the Share-settings page), it is automatically used.

Thanks for understanding my frustrations and for any assistance you can give.

 

Just as an aside, I am a consultant, database analyst programmer in the MS world - C# and SQL Server.  I was told by a friend reading my documentation for my system that "I can't write my way out of a paper bag". 

 

I think that when we are inside, when we know the stuff too well, then we are just unable to appreciate that instructions have to be *really basic* for outsiders.  Step by excruciating (to us) step, every keystroke because the outsider is going to do exactly what they are reading and when a step is missing...

 

In this case the entire chapter is missing!  Or buried deep in the bowels of a WIKI somewhere?  I found this thread and was so excited that I was finally going to get the missing instructions...

I think the entire chapter you are missing is the WIKI.  (Especially the FAQ,user-contributed/un-official part)

http://lime-technology.com/wiki/index.php/FAQ

http://lime-technology.com/wiki/index.php/Unofficial_Documentation

 

Joe L.