(solved) Is there a plugin or tool to zip all files in a folder?


Recommended Posts

Whenever I consolidate disks with Krusader, like going from three 3TB disks to one 8TB disk (as I am doing right now), I see that very often the first 98% take - say - 12hrs, and the final 2% take 2-3 days. Looking at Krusader's Queue window, I can see that it is usually the computer sparseimage backups, the software install file(s)/disk(s) backups (ie. OS X 10.x installers), email archives (like 10'2 of thousands of .eml files), and small files that will take seemingly forever.

My theory is that if I were to archive those files in some sort of compressed format (zip,  rar, etc.), the process would be *much* faster (Krusader usually reaches around 50-100MB/s on big files, such as captured/rendered video files, big CMYK files, etc.

 

Is there another way to avoid these super slow moving operations of sometimes hundreds of thousands of small files, and if these is, how? If compressing the offending files is the way to go, is there a plugin or tool which makes this easy for someone like me, who is not comfortable with Terminal (or at least using Terminal every time I want to do such a thing). Maybe Krusader itself has a built-in method or tool, either in its standard install or as an extension/plugin?

Hoping to find a solution, as this has really bit me in the butt too many times in the many years of trying to grow my trusty ole' unRAID system. 

Link to comment

Are you doing the 3 disks in parallel or in sequence? And are you seeing the same slow down if doing cp (or mv) from console?

 

I have found Krusader to be terrible for large file moves. It starts at normal speed but it grinds to an excruciating halt after some hours. It also creates many fragmentation patches, which cp / mv doesn't do. So maybe you should try doing it with cp / mv instead.

 

And I don't think your proposal will at all help with transfer speed.

The reason it's slow is because small files are inherently slower to read and write due to latency.

Introducing compression the middle isn't going to save you time because you still have to read those same files from the old disk and write those same files to the new disk. If anything it will take even more time because you lose the smoothing effect of ram cache.

Link to comment

I am doing the 3 disks in sequence, now on the second one (it has sat at 99% for over an hour now, even all files appear to have been moved).

Since I am trying to avoid having to resort to Terminal commands, I posted about the possibility of a tool or plugin.

I understand why you would think that compression would make it even slower, but I still don't think so. See, some file formats are really just "containers", such as .img, .sparseimage, and even .app, and Krusader moves the files within them, one by one, rather than the whole container as one. A typical 400GB computer backup to a .sparseimage file (I use Carbon Copy Cloner) contains some 500,000 files or more...but my theory is that if that is zipped, the resulting zip file would be moved as *one* singular file, just like a big Photoshop .psd or QuickTime .mov file.

In my experience, Krusader has ben very consistent with copy/move speeds, but it appears to me that it always seems to move those container files last, which makes it *appear* as if it first copies/moves very quickly and slows to a crawl...because it does...not because of Krusader as such, but because of the order in which it moves files. At least that's what I think.

A quick Google search comes up with what seems like tons of solutions for batch compressing files in a folder, but what I was wondering is whether there is something that's specific to, or proven on, unRAID...like a 7zip plugin that ties into Krusader, or something like that.

Edited by tillkrueger
syntax error correction
Link to comment

I do see a "pack" and "unpack" function in Krusader...will have to experiment with how long it takes to pack a 600GB computer backup.

If anyone has some insight into how to pack or compress all files in a folder (recursively, if possible), I'd greatly appreciate hearing/reading about it. For that purpose, I suppose that going into Terminal would be fine, as this is something that wouldn't have to happen very often.

Link to comment

That's your problem for insisting on using Krusader then. I told you what's wrong with it (and that is a generic problem regardless of file type).

 

Personally I don't see what's so hard about using CA Users Script -> create new script -> put this line in the script

cp -rv /mnt/diskX/* /mnt/diskY/

-> Save -> Run in Background -> Go have a beer

 

Also I don't think you are right about those container files. Unless you mount them, the file system exposes it to Krusader (or any other tool) as a single file. Krusader won't even attempt to mount such image to read what is underneath to move the files within. Any competent coder knows it is extremely silly to do so and I trust that Krusader coders are competent.

 

PS: you asked a similar question 1.5 years ago.

 

  • Like 1
Link to comment

testdasi, please don't get me wrong...I don't "insist" on using Krusader over a solution such as the one you just gave me, which is exactly the kind of suggestion I was hoping to get. I had never known about CA Users Script, and I appreciate you letting me know about it.

if I wanted to *move* my files, would it then be "mv -rv /mnt/diskX/* /mnt/diskY/"?
is there a parameter to cause Terminal to show progress as it happens?
is this process going to be persistent, even if my Terminal session got interrupted?

and thanks for reminding me that I had already posted about this a year and a half ago...my memory fails me quite often.

I am almost done with the 3rd disk (deleted a bunch of old archives), so when I consolidate the next batch of disks to a much bigger disk, I shall try CA Users Script and hope that you might be able to help me if I run into any head-scratching, which is quite likely.

sorry if I came across as stubborn and unwilling to take advice...that really wasn't my intent.

Link to comment
  • tillkrueger changed the title to (solved) Is there a plugin or tool to zip all files in a folder?

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.