Questions on Cache and Mover functionality

[mbezzo]

Hi,

So I'm on day 3 of my trial and have been working on getting things setup and have now moved on to some big data transfers to test things out.  It's going pretty well, but I've got a few questions on how to deal with certain situations and looking for any input/thoughts/ways to optimize.

 

First, I have 2 SSD's set up as a cache (256 and a 120GB which gives 188GB of Cache space) I've noticed that things run really great up until that size, then, of course, the transfers slow to a crawl.  I understand it's not every day that you'd be moving MORE than 188GB chunks at a time, so I'm not too worried.  What I am worried about is this:  Data mover starts on schedule, there's only a few GB of data on the cache, and THEN I start a big transfer.  This seems to leave unraid in a perpetual move as more data is being added to the cache while it's being moved off... so that initial move can never finish.  As such, there's a ton of room left on the cache that can't be taken advantage of.  Does this mean in these situations I should do something different?  Like disable the cache or something?  How do others deal with moving lots of data onto these things when they're getting setup?  

 

I presume I'm not the first to ask these particulars, but I've dug through the manuals a good bit and never found good answers.  Please point me in the right direction if this is covered elsewhere!

Thanks,
Matt

 

[JonathanM]

2 hours ago, mbezzo said:

(256 and a 120GB which gives 188GB of Cache space)

No, it doesn't. The figure on the GUI is broken. In order to have a full copy on 2 devices, you can only use the capacity of  the smaller of the two.

http://carfax.org.uk/btrfs-usage/

[mbezzo]

Thanks for that - I was scratching my head trying to understand how that was possible!   

 

Been poking around a bit more and have found a few topics covering this and it seems like the best advice for massive transfers is simply to turn off caching for the duration.  Once the big stuff is out of the way, turn it back on and resume life.

 

Thanks all!
Matt

[JonathanM]

2 hours ago, mbezzo said:

Does this mean in these situations I should do something different?  Like disable the cache or something?

Yes, for occasions like loading mass amounts of data, set the share to cache:no and enable "turbo write" (reconstruct mode in the disk settings write method).

 

That should give you speed at the expense of keeping all the drives spun up during writes instead of just the parity and the disk being written to.

[mbezzo]

Awesome, thank you.

[itimpi]

In practise since mover is normally set to run at 3:00 in the morning it would be unlikely for you to be doing a large transfer at the same time.   As was mentioned in special cases such as initial data load you can always temporarily disable caching for any share.