November 9, 20169 yr Moving a bunch of files in my system using MC and performance has dropped to 4.9mb/s. Confusing... I'm doing direct drive to drive copy (MNT not Shares), am I doing something wrong? Drives are all 6mb/s...
November 9, 20169 yr Community Expert Sorry, I thought you had been using V6 for a while. Tools - Diagnostics.
November 9, 20169 yr Author Nope, new to 6.2. Diags attached... hunternas-diagnostics-20161109-1550.zip
November 9, 20169 yr Community Expert Nothing obvious in diagnostics. Is that 4.9 MBytes or MBits? 4.9 MBytes might be about right with parity. You could turn on turbo-write.
November 9, 20169 yr Nope, new to 6.2. Diags attached... Were the diagnostics taken whilst you were having issues?
November 10, 20169 yr Author No, I don't think so, I had stopped the copy. So I've taken another diagnostic while its running. Here's a screenshot, running about 7MB/s...diagnostic attached... http://my.jetscreenshot.com/12412/20161110-kwqt-93kb.jpg[/img] hunternas-diagnostics-20161109-1550_1.zip
November 11, 20169 yr Author Speed sounds about right for millions of small files. Maybe we need to come up with a copy mechanism that skips the file copy process and just does a bit for bit transfer. Probably a lot of holes in that idea, but given the OH in dealing with many (millions) of files is the problem, perhaps one of you geniuses can find a way to skip that process. Yeah, I know, files are never contiguous on the disk, but its a nice fantasy...
November 12, 20169 yr Author This is crazy, it just keeps slowing down more and more. Is there a way to diagnose this performance issue? Down to 1.59 MB/s now... http://my.jetscreenshot.com/12412/20161111-pkub-132kb.jpg[/img] I'm just moving files from one disk to another (not sure if they are on the same controller or not), so it should not be too bad, not sure why its getting so slow... Thoughts?
November 12, 20169 yr You're trying to move 16 million files. Squid & Jonathan have explained that this is slow.
November 12, 20169 yr Author Sorry to be dense - Ok, then it is what it is. I'm using MC to do the move. Is there a better tool to use to move millions of small files? RSYNC? Or is MC just using the same commands (CP?)... Would it be faster to copy from unRaid to my windows machine, then back to unRaid so I don't halve the disk channel (since I'm pretty sure both drives are on the Supermicro Controller)? Or maybe to the Cache drive (but its not big enough)...
November 12, 20169 yr Sorry to be dense - Ok, then it is what it is. I'm using MC to do the move. Is there a better tool to use to move millions of small files? RSYNC? Or is MC just using the same commands (CP?)... Would it be faster to copy from unRaid to my windows machine, then back to unRaid so I don't halve the disk channel (since I'm pretty sure both drives are on the Supermicro Controller)? Or maybe to the Cache drive (but its not big enough)... A lot of your overhead is seek time for the heads, it's not just about the raw quantity of data. I don't think you'll see any appreciable benefit with rsync and doubling the number of copies required by copying it back and forth as well as adding in network overhead certainly won't make it any quicker. You just got to ride it out.
November 12, 20169 yr Your other option is to zip the 16M files, copy the zip, then unzip it at the destination. But then the actual transfer would go super fast, but the overhead in the zip / unzip operations would pretty much add up to the same total time you're seeing now. It would however make a lot of sense to do the zip if you were transferring 16M files from a windows box to the unraid box
November 12, 20169 yr Author Third post down explains it fairly well.... Great explanation. There is a LOT going on to be sure...
November 12, 20169 yr Author Your other option is to zip the 16M files, copy the zip, then unzip it at the destination. But then the actual transfer would go super fast, but the overhead in the zip / unzip operations would pretty much add up to the same total time you're seeing now. It would however make a lot of sense to do the zip if you were transferring 16M files from a windows box to the unraid box Painful either way, thanks for the ideas!
November 12, 20169 yr Third post down explains it fairly well.... You need to link to the specific post in question (grab the permalink from the "share" link below the answer). SE answers will move around based on votes. "3rd" is fairly meaningless...
November 13, 20169 yr Third post down explains it fairly well.... You need to link the specific post in question (grab the permalink from the "share" link below the answer). SE answers will move around based on votes. "3rd" is fairly meaningless... I've spent countless hours on superuser, and I never knew that. Probably because I never feel the need to "share" anything (I assumed it meant faceache or google-) Thanks, that's actually really handy..
November 15, 20169 yr Just thought I'd show what happens on my system when copying a lot of small files. As expected, atrocious speeds.
November 15, 20169 yr Your other option is to zip the 16M files, copy the zip, then unzip it at the destination. But then the actual transfer would go super fast, but the overhead in the zip / unzip operations would pretty much add up to the same total time you're seeing now. It would however make a lot of sense to do the zip if you were transferring 16M files from a windows box to the unraid box doing a tar to tar might work better, particularly no time will be wasted with compression. tar -C /mnt/disk1/path1 -cf - * | tar -C /mnt/disk2/path2 -xvf -
November 16, 20169 yr Author Just thought I'd show what happens on my system when copying a lot of small files. As expected, atrocious speeds. Yep, I get it now... Thanks!
Archived
This topic is now archived and is closed to further replies.