advCPaMV

Support the plugin?

This is my Advanced Copy and Move Plugin for unRAID v6.8.3 and later.

 

Please feel free to test it and report back if there are problems.

 

To install it:

 

v6.8.3 and later:

Install it using CA or in the unRAID plugin manager by installing from github:

https://raw.githubusercontent.com/SlrG/unRAID/master/Plugins/advCPaMV.plg

 

Description:

 

This plugin provides advanced cp and mv shell command alternatives named cpg and mvg that were created by applying the advcpmv-0.9-8.32.patch
from here on the coreutils version used by unRAID. The Patch is a modified version of the one available here.

 

But what does it do?

cpg and mvg are the normal cp and mv shell commands extended to support the parameters -g or --progress-bar, which will show the progress of the copy or move commands.

It will look like this while in progress:

cpg -rg temp/ temp1

2 files copied so far...                             3.0 GiB /   5.0 GiB
[================================>                              ] 60.0 %
Copying at   2.6 GiB/s (about 0h 0m 12s remaining)
temp/file2.txt                                       1.0 GiB /   1.0 GiB
[=============================================================> ] 99.9 %

And afterwards it will show a summary of the copy process:

5 files (  5.0 GiB) copied in 44.5 seconds (115.2 MiB/s).

 

THIS SOFTWARE IS PROVIDED 'AS IS' AND WITHOUT ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE AUTHOR SHALL NOT BE HELD LIABLE FOR ANY DAMAGES RESULTING FROM THE USE OF THIS SOFTWARE, EITHER DIRECTLY OR INDIRECTLY, INCLUDING, BUT NOT LIMITED TO, LOSS OF DATA OR DATA BEING RENDERED INACCURATE.

 

SlrG

Edited February 17, 20215 yr by SlrG
rename commands

Is it advisable to add 'cp' and 'mv' as aliases for 'gcp' and 'gmv'?

Well I have and I made the aliases to always use the -g switch too, as I really like to be able to view the progress of those operations. But I left that out of the plugin, so the users can decide for themselves.

 

My aliases from my .bash_profile look like this:

alias cp="gcp -g"
alias mv="gmv -g"
alias n="sudo -u nobody"
alias ncp="sudo -u nobody gcp -g"
alias nmv="sudo -u nobody gmv -g"

The last three allow me to run commands, copy or move files from a root shell as user nobody, which is the default file owner on unRAID. Otherwise the new files will have root as owner, which might cause problems later on.

Edited April 24, 20197 yr by SlrG
added aliases

On 4/24/2019 at 2:41 AM, SlrG said:

Well I have and I made the aliases to always use the -g switch too, as I really like to be able to view the progress of those operations. But I left that out of the plugin, so the users can decide for themselves.

 

My aliases from my .bash_profile look like this:

alias cp="gcp -g"
alias mv="gmv -g"
alias n="sudo -u nobody"
alias ncp="sudo -u nobody gcp -g"
alias nmv="sudo -u nobody gmv -g"

The last three allow me to run commands, copy or move files from a root shell as user nobody, which is the default file owner on unRAID. Otherwise the new files will have root as owner, which might cause problems later on.

I added the aliases to .bash_profile-

# console coloring for kool kids
PS1='\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '

# if this is an xterm set the title to user@host:dir
case "$TERM" in
xterm*|rxvt*)
    PS1="\[\e]0;\u@\h: \w\a\]$PS1"
    ;;
*)
    ;;
esac

# impersonate a user
alias user="su -ls /bin/bash"

alias v="ls -lA"

# advanced copy aliases
alias cp="gcp -g"
alias mv="gmv -g"
alias n="sudo -u nobody"
alias ncp="sudo -u nobody gcp -g"
alias nmv="sudo -u nobody gmv -g"

How do I get them to survive a reboot?

Copy the .bash_profile to your flash (/boot/config/) and add the following line to your go file:

ln -sf /boot/config/.bash_profile /root/.bash_profile

This will create a link overwriting the original file and pointing to the new one on the flash. Changing something in the original location will be put on the flash automatically and thus survive a reboot.

Thanks.

 

After my previous post I was doing a little reading about .bash_profile vs .bashrc. Just want to confirm that the aliases need to be added to .bash_profile to work in the built in terminal window?

@wgstarks

When trying just now, it works fine in the built in terminal when using .bash_profile.

Thanks again.

On 8/12/2019 at 5:04 AM, SlrG said:

Copy the .bash_profile to your flash (/boot/config/) and add the following line to your go file:

ln -sf /boot/config/.bash_profile /root/.bash_profile

This will create a link overwriting the original file and pointing to the new one on the flash. Changing something in the original location will be put on the flash automatically and thus survive a reboot.

@SlrG Thank you for this!!!

Have you tested it with *large* files?  I get overflows with large files.  See attached.

 

Sadly not with files that large. I guess it's better to withdraw the plugin then. I hope it did not corrupt files for you. Thank you for reporting the problem.

Don't be hasty.... I'm running hashes against them now to compare to backups.

 

You might just need to use long integers and recompile it.

19 minutes ago, bubbaQ said:

Don't be hasty.... I'm running hashes against them now to compare to backups.

 

You might just need to use long integers and recompile it.

I've been asked to remove it from CA as a precaution.  It can always get added back in at a later time.

Well, it might be hasty, but better save, then sorry. Until we know it does not damage data and how and if it can be changed to prevent this if it does, I wan't to minimize any risk as much as possible.

By "haste" I meant abandoning the project permanently. 

 

I completed my tests and all the hashes all match, so even on files over a TB, there was no corruption.

 

That's cool. Thank you very much for testing. I'll try to modify the patch to hopefully fix the overflows. Even if they are only graphical in nature, they could potentially unsettle people and need to be gone. I hope I'll able to fix them. It will take some time however.

You might also choose different command names. The repo suggests cpg/mvg or advcp/advmv it seems.

 

gcp clashes with the ohmyzsh alias for "git cherry-pick"

Or change the alias  

@all

Released a new version and renamed the commands from gcp and gmv to cpg and mvg. If you have shell aliases please update them accordingly!

 

@bubbaQ

If you still have those large testfiles and the freetime, could you give the new version a spin? I don't expect problems, but sometimes tests on one (my) system don't  match others. Thank you very much!

I just used it to migrate about 35TB of data from one unraid server to another without issue - usually use rsync, but it was a one time copy and didn't need the overhead I thought I'd incur.

 

No issues on my end other than some graphical oddities that were more likely related to where I was transferring the data to than the tool itself - I had three concurrent streams going in tmux (so I wasn't just hitting one disk at a time) transferring via NFS over a 10G bonded pair, and I noticed whenever it went from copying many small files to the first in a series of larger ones, there was no progress noted for anywhere from 5 to 40 or so seconds. Looking at the network stats at the time, sure enough, things were dropping off significantly...

 

Sure enough, it was due to network back pressure. If I end up having to do this kind of thing more often, I might finally have the justification needed for the time to implement RDMA, but for now, it's fine.

 

Handy little plugin, thanks!

 

 

On 2/17/2021 at 9:08 PM, SlrG said:

@all

Released a new version and renamed the commands from gcp and gmv to cpg and mvg. If you have shell aliases please update them accordingly!

Can I ask why?

@cinereus

In your last post you mentioned, that

Quote

gcp clashes with the ohmyzsh alias for "git cherry-pick"

and that the path repo suggested to use cpg and mvg as the command names.

Very nice plugin ! Thank you @SlrG

Not sure if this is a known issue, but the numbers don't add up.

 

Version: 2021.03.17

@SvbZ3r0

Yes, sadly the last change by me to solve the problems with quotes and double quotes in filenames is not perfect either and shows the shown symptoms under certain conditions. In all my test there was never data lost, but from time to time these "cosmetic" problems show up. If I have the time, I'll have a look at it again and will try to solve it. Sadly my times is very limited, so please don't expect fast results.