To Cache drive or not to Cache drive?


Recommended Posts

  • 2 weeks later...
  • Replies 366
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Posted Images

I currently have a 1T WD Green EARS drive as my cache drive as i had it laying about the most i ever see is 24-48 (more 48)MB/s.

 

Was looking at replacing it with a 320G Western Digital Black. would i see a marked improvement in the transfer speed to the server?

 

Also as moneys a bit tight at the moment been looking around on eBay and came across

 

http://www.ebay.co.uk/itm/Western-Digital-Scorpio-Black-320-GB-Internal-7200-RPM-6-35-cm-2-5-SATA-/230863509258#vi-content

 

and the auction states that its Seller refurbished, has anyone got any idea what that means and is it worth considering refurbished drives, Never brought one before. Experiences?

 

Link to comment

I see now something strange:

 

syslog:

root@Tower2:/var/log# tail syslog
Oct 13 10:17:08 Tower2 shfs/user: cache disk full
Oct 13 10:17:39 Tower2 last message repeated 4 times
Oct 13 10:18:50 Tower2 last message repeated 3 times
Oct 13 10:19:57 Tower2 last message repeated 11 times
Oct 13 10:21:06 Tower2 last message repeated 9 times
Oct 13 10:22:09 Tower2 last message repeated 495 times
Oct 13 10:33:49 Tower2 last message repeated 3 times
Oct 13 10:35:27 Tower2 last message repeated 4 times
Oct 13 10:37:37 Tower2 last message repeated 17 times
Oct 13 10:38:38 Tower2 last message repeated 481 times

 

The copy is still running with 40-50 MByte/sec

 

attache image is the main view of unraid webinterface showing 460 GB free

 

Is this a bug ?

 

 

Update:

I found the problem, there is a minmal free space setting for the cache drive. It was set to 500 GB, so user error i guess.

cache_disk_full_small.jpg.cd4aca9b063307ca35edee6852433689.jpg

Link to comment

I'm wondering if below stats are normal:

 

all my drives are WD green 3TB. i don't have cache drive setup. i'm interested in knowing if adding a cache drive will improve this stats significantly or not.

 

/dev/sda:

Timing cached reads:  11988 MB in  2.00 seconds = 6003.55 MB/sec

Timing buffered disk reads:  56 MB in  3.07 seconds =  18.25 MB/sec

 

/dev/sdb:

Timing cached reads:  12502 MB in  2.00 seconds = 6260.80 MB/sec

Timing buffered disk reads: 380 MB in  3.01 seconds = 126.14 MB/sec

 

/dev/sdc:

Timing cached reads:  12616 MB in  2.00 seconds = 6318.38 MB/sec

Timing buffered disk reads: 372 MB in  3.01 seconds = 123.60 MB/sec

 

/dev/sdd:

Timing cached reads:  12444 MB in  2.00 seconds = 6231.35 MB/sec

Timing buffered disk reads: 378 MB in  3.00 seconds = 125.86 MB/sec

 

/dev/sde:

Timing cached reads:  12482 MB in  2.00 seconds = 6251.29 MB/sec

Timing buffered disk reads: 368 MB in  3.00 seconds = 122.64 MB/sec

 

/dev/sdf:

Timing cached reads:  12460 MB in  2.00 seconds = 6239.76 MB/sec

Timing buffered disk reads: 362 MB in  3.01 seconds = 120.17 MB/sec

 

/dev/sdg:

Timing cached reads:  12446 MB in  2.00 seconds = 6233.11 MB/sec

Timing buffered disk reads: 378 MB in  3.01 seconds = 125.51 MB/sec

 

/dev/sdh:

Timing cached reads:  12484 MB in  2.00 seconds = 6251.94 MB/sec

Timing buffered disk reads: 376 MB in  3.01 seconds = 124.71 MB/sec

Link to comment
  • 2 weeks later...

I'm wondering if below stats are normal:

 

all my drives are WD green 3TB. i don't have cache drive setup. i'm interested in knowing if adding a cache drive will improve this stats significantly or not.

The  "cache" drive improves "write" speeds to the array.  It slows down read speeds (a tiny-tiny-bit)

 

Write speeds are improved only because the process of writing to the protected array is delayed to a later time.  You files are NOT protected until copied to the protected array/

 

Read speeds are slowed because the cache drive must FIRST be scanned for any given file before scanning the protected data disks in the array.  That takes a tiny bit of time.

 

To answer your question, the stats on the physical drives when reading them will not change at all.  They have nothing to do with actual read and write speeds to the array.  Most disks will not be able to sustain 100MB/s read speeds.

 

Joe L.

Link to comment

Thanks! I just found the "Use Cache Drive" drop down on the User Shares.

I just tested it,..and now I see the data writing to only the cache drive.

 

So if I write specifically to one of the Disk directly,..then no cache.

I suppose I have to reread how to setup the User shares to only use one Disk in the case of wanting to copy data to only one disk.

Link to comment
  • 1 month later...

I like the cache idea and think it is a great implementation.  I just thought I'd add my 2 cents since I was bored and came across this thread :)

 

I do not use a cache drive simply because I do not implement writable user shares.  I write directly to disk shares and leave my user shares as read-only for my Media Centers.  I've been tempted to change the way I do things a few times, but I began this way nearly 5 years ago, so I guess old habits just die hard.

 

I have implemented one for kicks a time or two to function as a scratch drive, but I never got any real use of it and ended up removing it.  I still think it's one of the best additions in the past few years though.

Link to comment
  • 1 month later...
Edit: (3/7/2011) I'll add that LimeTech has mentioned in several threads their intention to create a 'hot spare' option for unRAID, which would allow for the use of a cache drive AND a spare drive side-by-side.  Great news!

you mentioned in your main post, that there have been rumors about this feature a while ago.

 

any updates?

Link to comment

Hi

 

I am about to install a 30 GB SSD as my cache drive and I have couple questions.

 

I want to put my Plex and XBMC database on there so I don't have to spin up other drives constantly, how do I prevent the mover script to move those folders from SSD ?

 

What happens if I try to copy a single 50 GB file over (since the cache drive is only 30) ?

 

What happens if I try to for example two 20 GB files at the same time ?

 

thanks

Link to comment
  • 5 weeks later...

You can create a folder that starts with a dot, and these are ignored by mover.  Alternatively set up, a share and set it to be restricted to the cache drive.

 

so a folder named ".Plex Library" will be ignored by the script?

 

So the wiki says, yes. Also, so the wiki states, any files that are in the root directory will also be ignored, such as:-

 

/blah.txt

/thisIsAnAwesomePiratedMovie.mkv

 

So, only things in folders that don't start with "." are moved... or so the wiki states.

Link to comment
  • 2 months later...

OK guys.  I think I am missing the forest for the trees here...

 

Finally added my cache drive (2TB like all of the other drives in my unraid server).  So say I normally move a video/TV file from my desktop (over network connection) to a "TV" folder share on unraid (all through Windows 7 network interface).  Now that the cache drive is present do I just move the file to the cache drive (not a share) and the mover script will take care of it later (currently at the default 3:40 am setting)?  If on my "TV" share on the server the TV files are each in a separate folder for each TV show title.  How will the mover script know where to move this particular file?

 

Apologies if I am missing something simple.  Or perhaps the mover script does not actually separate files and categorize for me (which is understandable)

 

Thanks in advance for any help or direction.

 

Link to comment

OK guys.  I think I am missing the forest for the trees here...

Apologies if I am missing something simple.  Or perhaps the mover script does not actually separate files and categorize for me (which is understandable)

Thanks in advance for any help or direction.

 

There's really nothing to do....just copy your files to the unRAID shares (or folders under the main share point) you've created. With the cache drive active, unRAID will drop them onto the cache temporarily and then at 3:40a move them to the corresponding folder structure you've created under your share point.

 

You don't really need to over think it. You can manually run the mover and then check the syslog to see what's been done to get a feel for what to expect. Normally, you won't ever see the files on the cache drive; just like you'll never see what's on the parity drive.

 

Best of luck with unRAID!

Link to comment

In other words:

 

just copy the files to the share where you want them, unraid will underneath copy to the cache drive instead, as far as you can tell the files are where you copied them.

 

In even simpler words:

 

1) add cache drive

2) do what you always did

3) hey... its quicker !

Link to comment

As noted above, the operation of the cache is essentially transparent (as long as the shares are "cache enabled").

 

One thing to remember, however, is that use of a cache means that files you've copied to UnRAID are NOT fault-tolerant until the mover script moves them to the actual array.    If the cache drive was to fail between the time you copied the files and they were moved, you'd lose the cached data.

 

This is a relatively minor distinction ... but Murphy has a way of causing failures at the least opportune times !!    If you're currently "Moving" your files to UnRAID, you may want to Copy them instead -- and then Move the copied files to a "Delete Tomorrow" folder on your PC  :)

 

V5 final is going to provide the ability to have a fault-tolerant cache, which will eliminate this issue.

 

Link to comment

As noted above, the operation of the cache is essentially transparent (as long as the shares are "cache enabled").

 

 

And there was what I missed.  I missed enabling it on a couple of shares.  So files were not moving to cache but just straight to the shares.

 

Thanks guys for slapping some common sense into me.  I figured I had to be missing something silly.

 

 

 

Link to comment

As noted above, the operation of the cache is essentially transparent (as long as the shares are "cache enabled").

 

One thing to remember, however, is that use of a cache means that files you've copied to UnRAID are NOT fault-tolerant until the mover script moves them to the actual array.    If the cache drive was to fail between the time you copied the files and they were moved, you'd lose the cached data.

 

This is a relatively minor distinction ... but Murphy has a way of causing failures at the least opportune times !!    If you're currently "Moving" your files to UnRAID, you may want to Copy them instead -- and then Move the copied files to a "Delete Tomorrow" folder on your PC  :)

 

V5 final is going to provide the ability to have a fault-tolerant cache, which will eliminate this issue.

 

How will the fault tolerant cache work?

Link to comment

V5 final is going to provide the ability to have a fault-tolerant cache, which will eliminate this issue.

 

How will the fault tolerant cache work?

 

You can assign 2 drives to a RAID-1 btrfs cache pool, so the cache "drive" will actually be a RAID array.    Exact details aren't available yet, but I'd expect the operation will be exactly as it is now ... i.e. the "mover" will still move the contents to the primary protected array on a schedule.  The key difference is the data in the cache isn't "at risk" (i.e. not protected) like it is now.

 

Link to comment
  • 3 weeks later...
  • 3 weeks later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.