[Plugin] Advanced Copy and Move for unRAID v6.8.3 and later


SlrG

Recommended Posts

advCPaMV

Support the plugin?

Donate

This is my Advanced Copy and Move Plugin for unRAID v6.8.3 and later.

 

Please feel free to test it and report back if there are problems.

 

To install it:

 

v6.8.3 and later:

Install it using CA or in the unRAID plugin manager by installing from github:

https://raw.githubusercontent.com/SlrG/unRAID/master/Plugins/advCPaMV.plg

 

Description:

 

This plugin provides advanced cp and mv shell command alternatives named cpg and mvg that were created by applying the advcpmv-0.9-8.32.patch
from here on the coreutils version used by unRAID. The Patch is a modified version of the one available here.

 

But what does it do?

cpg and mvg are the normal cp and mv shell commands extended to support the parameters -g or --progress-bar, which will show the progress of the copy or move commands.


It will look like this while in progress:

cpg -rg temp/ temp1

2 files copied so far...                             3.0 GiB /   5.0 GiB
[================================>                              ] 60.0 %
Copying at   2.6 GiB/s (about 0h 0m 12s remaining)
temp/file2.txt                                       1.0 GiB /   1.0 GiB
[=============================================================> ] 99.9 %

And afterwards it will show a summary of the copy process:

5 files (  5.0 GiB) copied in 44.5 seconds (115.2 MiB/s).

 

THIS SOFTWARE IS PROVIDED 'AS IS' AND WITHOUT ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE AUTHOR SHALL NOT BE HELD LIABLE FOR ANY DAMAGES RESULTING FROM THE USE OF THIS SOFTWARE, EITHER DIRECTLY OR INDIRECTLY, INCLUDING, BUT NOT LIMITED TO, LOSS OF DATA OR DATA BEING RENDERED INACCURATE.

 

SlrG

Edited by SlrG
rename commands
  • Like 1
Link to comment

Well I have and I made the aliases to always use the -g switch too, as I really like to be able to view the progress of those operations. But I left that out of the plugin, so the users can decide for themselves.

 

My aliases from my .bash_profile look like this:

alias cp="gcp -g"
alias mv="gmv -g"
alias n="sudo -u nobody"
alias ncp="sudo -u nobody gcp -g"
alias nmv="sudo -u nobody gmv -g"

The last three allow me to run commands, copy or move files from a root shell as user nobody, which is the default file owner on unRAID. Otherwise the new files will have root as owner, which might cause problems later on.

Edited by SlrG
added aliases
  • Like 3
Link to comment
  • 3 months later...
On 4/24/2019 at 2:41 AM, SlrG said:

Well I have and I made the aliases to always use the -g switch too, as I really like to be able to view the progress of those operations. But I left that out of the plugin, so the users can decide for themselves.

 

My aliases from my .bash_profile look like this:


alias cp="gcp -g"
alias mv="gmv -g"
alias n="sudo -u nobody"
alias ncp="sudo -u nobody gcp -g"
alias nmv="sudo -u nobody gmv -g"

The last three allow me to run commands, copy or move files from a root shell as user nobody, which is the default file owner on unRAID. Otherwise the new files will have root as owner, which might cause problems later on.

I added the aliases to .bash_profile-

# console coloring for kool kids
PS1='\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '

# if this is an xterm set the title to user@host:dir
case "$TERM" in
xterm*|rxvt*)
    PS1="\[\e]0;\u@\h: \w\a\]$PS1"
    ;;
*)
    ;;
esac

# impersonate a user
alias user="su -ls /bin/bash"

alias v="ls -lA"

# advanced copy aliases
alias cp="gcp -g"
alias mv="gmv -g"
alias n="sudo -u nobody"
alias ncp="sudo -u nobody gcp -g"
alias nmv="sudo -u nobody gmv -g"

How do I get them to survive a reboot?

  • Like 1
Link to comment

Copy the .bash_profile to your flash (/boot/config/) and add the following line to your go file:

ln -sf /boot/config/.bash_profile /root/.bash_profile

This will create a link overwriting the original file and pointing to the new one on the flash. Changing something in the original location will be put on the flash automatically and thus survive a reboot.

  • Thanks 1
Link to comment
  • 1 month later...
On 8/12/2019 at 5:04 AM, SlrG said:

Copy the .bash_profile to your flash (/boot/config/) and add the following line to your go file:


ln -sf /boot/config/.bash_profile /root/.bash_profile

This will create a link overwriting the original file and pointing to the new one on the flash. Changing something in the original location will be put on the flash automatically and thus survive a reboot.

@SlrG Thank you for this!!!

  • Thanks 2
Link to comment
  • 5 months later...
19 minutes ago, bubbaQ said:

Don't be hasty.... I'm running hashes against them now to compare to backups.

 

You might just need to use long integers and recompile it.

I've been asked to remove it from CA as a precaution.  It can always get added back in at a later time.

Link to comment

That's cool. Thank you very much for testing. I'll try to modify the patch to hopefully fix the overflows. Even if they are only graphical in nature, they could potentially unsettle people and need to be gone. I hope I'll able to fix them. It will take some time however.

Link to comment
  • 1 month later...

@all

Released a new version and renamed the commands from gcp and gmv to cpg and mvg. If you have shell aliases please update them accordingly!

 

@bubbaQ

If you still have those large testfiles and the freetime, could you give the new version a spin? I don't expect problems, but sometimes tests on one (my) system don't  match others. Thank you very much!

Link to comment
  • 3 weeks later...

I just used it to migrate about 35TB of data from one unraid server to another without issue - usually use rsync, but it was a one time copy and didn't need the overhead I thought I'd incur.

 

No issues on my end other than some graphical oddities that were more likely related to where I was transferring the data to than the tool itself - I had three concurrent streams going in tmux (so I wasn't just hitting one disk at a time) transferring via NFS over a 10G bonded pair, and I noticed whenever it went from copying many small files to the first in a series of larger ones, there was no progress noted for anywhere from 5 to 40 or so seconds. Looking at the network stats at the time, sure enough, things were dropping off significantly...

 

Sure enough, it was due to network back pressure. If I end up having to do this kind of thing more often, I might finally have the justification needed for the time to implement RDMA, but for now, it's fine.

 

Handy little plugin, thanks!

 

 

Link to comment
  • 1 month later...
  • 2 weeks later...
  • 5 months later...

@SvbZ3r0

Yes, sadly the last change by me to solve the problems with quotes and double quotes in filenames is not perfect either and shows the shown symptoms under certain conditions. In all my test there was never data lost, but from time to time these "cosmetic" problems show up. If I have the time, I'll have a look at it again and will try to solve it. Sadly my times is very limited, so please don't expect fast results.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.