After trying SNAP and a few other things and posting etc I gave up and took the suggestion of some of the responses and just got a gigabit switch (since my unraid box and laptop have gigabit ports) and copy through the network, problem is its taking forever... through, perhaps it is normal in terms of speed. Its taking me about days (actually more as I am using teracopy and am doing a CRC check so about 5-6 days) per drive and its killing me.

 

Alot of what I have is xbmc stuff which has lots of little info and thumbnails etc (like one videos can have a thumbnail for every actor and I have over 1000 videos) which are just a few K which seem to copy much slower than the large files? I know very little about networking, is there perhaps a setting I might be missing or is this just par for the course when dealing with a drive this size on a gigabit network and USB 3.0? (I am currently in the "checking stage of the second to last drive" so still have a few days I can hopefully save).

 

Thanks!

1 day per TB is about right as a rule of thumb for mixed file sizes.  That 11.5 MB/sec.  With large files, you can get 2-3 times faster... but all them little files kill you.

 

I would:

 

1) spin up all disks and set unRAID to "mdcmd set md_write_method 1"

2) plug the USB3 dock directly into unRAID, mount the drives natively and copy.

 

 

 

what disk format is the usb disk in?

gfjardim released a tool to mount USB disks on Unraid that may help (it also installs NTFS3G)

Personally I would try and mount it on a sata controller (take drive out of the usb box) and copy directly.

 

Tony

@bubbaq

Thanks, I will try out "mdcmd set md_write_method 1"

 

@spants

My drive is formatted in NTFS (had it attached to a windows box before)

 

@bubbaQ & spants

As for mounting the drives directly, been there tried that till I ran out of options, I posted here and here regarding various ways I might get it to work but if you all have some other thoughts I and *totally* open to suggestions!

 

Have you done any speed tests copying from the USB dock direct to your local PC?  It isn't likely, but that may be your limiting factor.  That USB3 dock may be throttled down to USB2 speeds under some conditions.

Thanks. Yeah that occurred to me, its a new dock but I actually double checked it and its pretty darn fast when I copy it locally. Actually, when copying the larger files via network teracopy is telling me that I am getting about 50mb/sec but then when it has to copy a string of 5 or 20kb files it slows to a crawl (I mean a like less than 50kb/second).

 

Next I will be copying from server to server via the network (pretty much want to mirror two servers which I will then take one offsite and use syncthing from there on), I'd be open to suggestions there, maybe a way at least to skip the all *.nfo and *.tbn files that XBMC creates , which I can live with a few of them being corrupted, using rsync? (I don't know much about rsync but it seems like it could do such a thing?)

You really don't have a problem here -- you're simply copying a LOT of small files, and as already noted, this slows things down a LOT.    Doesn't matter if you're doing it over the network or locally, this will still be true.  It's caused by the directory updates ... after each file, the directory info is updated, and this requires a bunch of extra seeks and disk rotations, so the net average speed is reduced dramatically.

 

Nothing you can really do about that.  The good news is it doesn't really take any of "your" time ... it's all the computer's time.    You simply have to be patient and wait for it to finish.

 

I don't know if there's a way to force Linux to buffer all of the directory writes or not -- in Windows you could change the policies for the destination drive to force that, and things would be MUCH faster [the risk you take is if you have any power glitches or other problems before the writes are actually committed you'll end up with a corrupted disk].    If somebody knows how to force directory buffering in Linux, it would be convenient to know that ... please post the details.

 

Actually, when copying the larger files via network teracopy is telling me that I am getting about 50mb/sec but then when it has to copy a string of 5 or 20kb files it slows to a crawl (I mean a like less than 50kb/second).

 

That is completely expected.  SMB (Samba) sucks for lots of small files.

 

Considered making some big zip/rar files of the source, then copying to the server and unzip/unrar there?