[SOLVED] Copy operation via Cache drive maddeningly slow

[tillkrueger]

Last week I finally got around to upgrading to the newest version of unRAID (v6.0-rc6a) and set up on of my 3TB Red drives as a Cache drive.

 

I needed to back up a bunch of bare-metal system backups from an external 4TB Red drive (connected via USB2 to the iMac i use to maintain my unRAID system, connected via 1Gb ethernet), for which i usually use ChronoSync to secure myself against the not uncommon case that the copy operation gets interrupted for some reason or another.

 

i started to back up one of the folders with two sparsebundle images (one of them 870GB, the other 690GB) and was not particularly impressed when ChronoSync showed transfer rates just under 30MB/s at first...but within 10hrs transfer rates dropped to under 10MB/s, and now - over 26hrs later -  they are at 4.1MB/s...it seems to be getting slower with every hour.

 

needless to say, 3 days for *one* 870GB file is downright unacceptable, and this doesn't even yet take into account the time it will take for the data to be moved to the actual array, which also took days when i copied my first 2TB to the cache drive last week.

 

something ain't right, it seems, but where do i start to figure out what that is? with these transfer rates i'd obviously be better off going straight to the array rather than via a cache drive.

 

i'm baffled.

[garycase]

Not sure why the rate dropped so low ... but as an observation I will note that there is NO reason to cache a write that's done the way you've outlined.    A USB v2 connection won't be able to sustain a transfer rate any higher than your system should be able to write directly to the array ... so there's NO benefit to caching the write.

 

What is your Mover schedule?    I'm wondering if perhaps the Mover started running and copying the part of the file that's already been cached ... and the consequences of this "file thrashing" (constant access to it by two different processes at different locations) was the slowdown you're seeing.

 

I'd definitely copy the 2nd image directly to your array !!

 

[tillkrueger]

good points, garycase, and you're right about the disk thrashing...i've been changing the mover schedule to stay ahead of the slow copy operation but forgot to do so last night before going to sleep, so that i am now looking at exactly this situation...i guess i'll just have to wait for the copy and move operations are done, which could be days, literally :-/

or is there a way to cancel the mover? to abort the copy operation would be no problem, as it is a single file that i could then copy directly in less time, probably, than having to wait another 2-3 days for what is going on right now (let alone the rather brutal disk-thrashing situation).

 

and i've been trying to remedy this silly USB2 situation by attempting to find a USB3 card for my server that can be purchased here in Germany (the ones i saw mentioned here on the forum i couldn't find here) and have even started a thread in the Hardware forum to look for help with finding out which chipset i'd have to look for (http://lime-technology.com/forum/index.php?topic=40726.0), but haven't gotten any advice on that yet.

 

with the help of the "Unassigned Devices" plugin i hope to be able to connect my backup drives directly to my unRAID server and then use Midnight Commander to do these file operations...but in the meantime i will need to somehow get by with what i currently have...and wait...and wait...and...

[garycase]

I don't know if you can safely abort the mover, but you could certainly cancel the copy.    But depending on how far along it is ... and also how much stuff the mover still has to move (which I don't think there's any direct way to find out, although I suspect you could look at the cached share folders on the cache drive and see what's still in them) => it may not be worth doing that ... might be net quicker to just wait it out.  Hard to say for sure.

 

As for copying from USB2 => in reality, that doesn't slow you down much at all.  A typical rate from a USB2 drive is ~ 30MB/s, which isn't much slower than you can usually write directly to your array.  It IS, of course, slower than you can write to a cache drive ... but remember that this is an "artificially faster" write => in fact, it's actually net-slower than writing directly to the array, since it still has to do that at a later time.    The benefit, of course, is that it "hides" that slow write from the user (you).

 

For a VERY large file (such as the two you're copying) I'd write them directly to the array anyway so you can avoid the VERY large move that would then have to happen.

 

[tillkrueger]

i see what you mean...i'll weigh the pluses and cons and then decide whether i should maybe just add the 3TB to my array again and enjoy the extra space over the perceived speed advantage of the cache drive...frankly, in reality i don't often copy large amounts of data to the array (save for when i go through my timelapse shooting periods), and even then, i don't mind waiting a while until the data is backed up...i'll mull it over.

 

i aborted the copy operation and rebooted the server, which was necessary anyway after my update to unRAID v6.0.0 from rc6a...the Mover operation was aborted in the process, which is just as well, as it was working on a single 860GB file, rather than a bunch of smaller files...and even of, ChronoSync has the advantage of finding and resuming partial copy operations.

 

so for now i'm good, and until i figure out my strategy regarding cache drive or not, let's consider this thread resolved...thanks garycase!