Reasons for a very slow file copying

[temkins]

Hi there! I've got a couple of questions about the file copying speed, need to know if my situation is normal or should I investigate further?

I need to copy multiple external USB HDD to my Unraid server. Each drive is 4Tb with over 100k files (large or small). I'm hooking these USB drives directly to Unraid through the USB 3 port and using the Midnight Commander to copy files from mounted unassigned drive (/mnt/disks/drive_name) to (mnt/user0/share_name/disk_name)

 

My unraid is running with GUI on and monitor hooked, so that I could keep the MC open and see how it goes. Since I'm totally new to unraid (and to linux in general), I've got a few questions and hope you guys could help me:

 

1. The speed of file transfer from the external USB drive (Lacie RAID 0, HFS+) to unraid share is 20 MB/s, sometimes drops to as low as 3 MB/s, sometimes goes up to 40-50 MB/s. I believe that both unraid and external drives can do much faster through the USB 3. What should I check?

2. Is it ok to copy files the way I do -- from "/mnt/disks/drive_name" to "mnt/user0/share_name/disk_name". I'm using "user0" as I don't need those files on my cache drive. I tried using the Krusader docker, but have a weird feeling that it copies even slower.

 

Would appreciate any suggestions.

Edited September 25, 2019 by temkins

[itimpi]

Regarding point 2, this is valid but it will end up creating User Shares of ‘disk_name’ which is going to confuse the system if those are of the form disk1, disk2, etc as those normally refer to physical drives, not User Shares.

[temkins]

7 minutes ago, itimpi said:

Regarding point 2, this is valid but it will end up creating User Shares of ‘disk_name’ which is going to confuse the system if those are of the form disk1, disk2, etc as those normally refer to physical drives, not User Shares.

So does it mean that there is a better, non-confusing way to move a large amount of files to the share? Except for using windows, mounting that share as the shared drive a copying everything via network?

[itimpi]

14 minutes ago, temkins said:

So does it mean that there is a better, non-confusing way to move a large amount of files to the share? Except for using windows, mounting that share as the shared drive a copying everything via network?

Any folder name that appears after the /mnt/user0 part becomes a User Share of that name.   Just make sure the name is what you want the User Share to be called (and it should NOT be of the form 'diskX').

[temkins]

21 minutes ago, itimpi said:

Any folder name that appears after the /mnt/user0 part becomes a User Share of that name.   Just make sure the name is what you want the User Share to be called (and it should NOT be of the form 'diskX').

Oh, got that. I already created a share, so my “drive-name” are actually directories under that share. Thanks for clarifying that.

 

Still puzzled with speed. Is there any way to see if there is a hardware issues with my usb ports? Or does that speed is ok for hfs+ to unraid transfer?

[itimpi]

Speeds of 30-40MBps are typical for writing to the array.    Each ‘write’ operation actually involves a sector read from target and parity drives; a disk revolution; and then a sector write to the target and parity drives.   With small files the speed tends to be lower due to the overhead of creating the relevant directory entries.   Enabling ‘Turbo’ mode can speed up things as it eliminates the initial read from target/parity drives but at the expense of needing all drives spun up.

[Squid]

5 hours ago, temkins said:

4Tb with over 100k files

Should also be noted that when you're dealing with that many files (and at 100k files, most of them are small) that the filesystem time overhead in figuring out where to place the files, creating the entries, etc etc starts to become very noticeable.  No different than any other OS with bulk copying like that.

[temkins]

Thanks for those hints! So I figured out, that I had a couple of HUGE Final Cut libraries on my drives (my Unraid is a file server/archive). On MacOS this is a single file, while Unraid sees it as a folder with nearly 1 million of tiny files. Those fcpbundle files were causing the major copying slow down, while single files are copied in line with USB->HDD expected speeds - just as you described above.