Help with rsync and UNASSIGNED devices

coblck · March 28, 2021

Great thanks I'm guessing approx 24 hrs

Mihle · December 5, 2021

On 12/29/2020 at 5:03 AM, Hoopster said:

You don't create anything; let rsync do it. I just formatted the destination drive as NTFS and let it run the backup script when plugged in. The first time the destination folders (same name as the shares) are created.

I only made minor modifications to the UD backup script by adding the rsync commands for backing up the additional shares.

/mnt/user/Pictures for example just becomes Pictures on the USB drive and all these folders appear just fine in Windows.

#!/bin/bash
PATH=/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin
## Available variables: 
# AVAIL      : available space
# USED       : used space
# SIZE       : partition size
# SERIAL     : disk serial number
# ACTION     : if mounting, ADD; if unmounting, UNMOUNT; if unmounted, REMOVE; if error, ERROR_MOUNT, ERROR_UNMOUNT
# MOUNTPOINT : where the partition is mounted
# FSTYPE     : partition filesystem
# LABEL      : partition label
# DEVICE     : partition device, e.g /dev/sda1
# OWNER      : "udev" if executed by UDEV, otherwise "user"
# PROG_NAME  : program name of this script
# LOGFILE    : log file for this script

case $ACTION in
  'ADD' )
    #
    # Beep that the device is plugged in.
    #
    beep  -l 200 -f 600 -n -l 200 -f 800
    sleep 2

    if [ -d $MOUNTPOINT ]
    then
      if [ $OWNER = "udev" ]
      then
        beep  -l 100 -f 2000 -n -l 150 -f 3000
        beep  -l 100 -f 2000 -n -l 150 -f 3000

        logger Started -t$PROG_NAME
        echo "Started: `date`" > $LOGFILE

        logger Pictures share -t$PROG_NAME
        rsync -a -v /mnt/user/Pictures $MOUNTPOINT/ 2>&1 >> $LOGFILE
		
	logger Videos share -t$PROG_NAME
        rsync -a -v /mnt/user/Videos $MOUNTPOINT/ 2>&1 >> $LOGFILE
		
	logger Movies share -t$PROG_NAME
        rsync -a -v /mnt/user/Movies $MOUNTPOINT/ 2>&1 >> $LOGFILE

	logger Family Videos share -t$PROG_NAME
        rsync -a -v /mnt/user/FamVideos $MOUNTPOINT/ 2>&1 >> $LOGFILE

        logger Syncing -t$PROG_NAME
        sync

        beep  -l 100 -f 2000 -n -l 150 -f 3000
        beep  -l 100 -f 2000 -n -l 150 -f 3000
        beep  -r 5 -l 100 -f 2000

        logger Unmounting Backup -t$PROG_NAME
        /usr/local/sbin/rc.unassigned umount $DEVICE

       echo "Completed: `date`" >> $LOGFILE
        logger Backup drive can be removed -t$PROG_NAME

        /usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Server Backup" -d "Backup completed" -i "normal"
    fi
    else
        logger Backup Drive Not Mounted -t$PROG_NAME
  fi
  ;;

  'REMOVE' )
    #
    # Beep that the device is unmounted.
    #
    beep  -l 200 -f 800 -n -l 200 -f 600
  ;;

  'ERROR_MOUNT' )
	/usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Server Backup" -d "Could not mount Backup" -i "normal"
  ;;

  'ERROR_UNMOUNT' )
	/usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Server Backup" -d "Could not unmount Backup" -i "normal"
  ;;
esac

How would I go about backing up Nextcloud database in a script like this where partal of the data could be on the cache? Would just selecting the hole share work? would Nextcloud have to be stopped?

trurl · December 5, 2021

5 hours ago, Mihle said:

partal of the data could be on the cache?

User shares includes data from any pools.

kilonde · January 7, 2022

On 12/29/2020 at 4:23 PM, Hoopster said:

and every 3-4 month I run an alternative version of that script that deletes from the backup anything that has been deleted from the source. I'll eventually likely do the same with the external drive, but, it's another fallback in case something gets accidentally deleted from the server (as has happened to me twice now with entire year of photos).

Is there any chance you could share your backup "pruning" script?

Mihle · March 17, 2022

On 12/5/2021 at 6:11 PM, trurl said:

User shares includes data from any pools.

So, If I used:

rsync -avr --delete /mnt/cache/Ncloud

It would only do data on the cache, if I did;

rsync -avr --delete /mnt/disk1/Ncloud

It would only do the data on the array disk, but if I do:

rsync -avr --delete /mnt/user/Ncloud

It will do everything no matter where its located?
(same with appdata)

itimpi · March 17, 2022

17 minutes ago, Mihle said:
So, If I used:
rsync -avr --delete /mnt/cache/Ncloud
It would only do data on the cache, if I did;
rsync -avr --delete /mnt/disk1/Ncloud
It would only do the data on the array disk, but if I do:
rsync -avr --delete /mnt/user/Ncloud
It will do everything no matter where its located?
(same with appdata)

yes

Mihle · March 21, 2022

This is my current scrip, havent tested the hole script at once yet:

#!/bin/bash

#Destination of the backup:
destination="/mnt/disks/OfflineBackup"
#Name of Backup disk
drive=dev1

########################################################################
#Backup with docker containers stopped:

#Stop Docker Containers
echo 'Stopping Docker Containers'
docker stop $(docker ps -a -q)


#Backup Docker Appdata via rsync
echo 'Making Backup of Appdata'
rsync -avr --delete "/mnt/user/appdata" $destination

#Backup Flash drive backup share via rsync
echo 'Making Backup of Flash drive'
rsync -avr --delete "/mnt/user/NAS Flash drive backup" $destination

#Backup Nextcloud share via rsync
echo 'Making Backup of Nextcloud'
rsync -avr --delete "/mnt/user/NCloud" $destination


#Start Docker Containers
echo 'Starting up Docker Containers'
/etc/rc.d/rc.docker start

########################################################################
#Backup of the other stuff:

#Backup Game Backup share via rsync
echo 'Making Backup Game Backup share'
rsync -avr --delete "/mnt/user/Game Backups" $destination

#Backup Photos and Videos share via rsync
echo 'Making Backup of Photos and Videos share'
rsync -avr --delete "/mnt/user/Photos and Videos" $destination

########################################################################

#Umount destination
echo 'Unmounting and spinning down Disk....'
/usr/local/sbin/rc.unassigned umount $drive
/usr/local/sbin/rc.unassigned spindown $drive

#Notify about backup complete
/usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Backup to Unassigned Disk" -d "A copy of important data has been backed up" -i "normal"
echo 'Backup Complete'

I am not sure if I should use echo or logger?

Also, in a test script, I didnt manage to get the

2>&1 >> $LOGFILE

that I see some use at the end of rsync commands to do anything, what does it do and should I try to get it to work?

for reference, I set the drive to automount, then I want to manually have to start the script, but when its done running I want to be ready to just unplug without having to press anything more. (and then take to another location until its time to update it) (offline backup)

Edited March 21, 2022 by Mihle

kalidus · July 8, 2022

On 3/21/2022 at 2:20 PM, Mihle said:

This is my current scrip, havent tested the hole script at once yet:

#!/bin/bash

#Destination of the backup:
destination="/mnt/disks/OfflineBackup"
#Name of Backup disk
drive=dev1

########################################################################
#Backup with docker containers stopped:

#Stop Docker Containers
echo 'Stopping Docker Containers'
docker stop $(docker ps -a -q)


#Backup Docker Appdata via rsync
echo 'Making Backup of Appdata'
rsync -avr --delete "/mnt/user/appdata" $destination

#Backup Flash drive backup share via rsync
echo 'Making Backup of Flash drive'
rsync -avr --delete "/mnt/user/NAS Flash drive backup" $destination

#Backup Nextcloud share via rsync
echo 'Making Backup of Nextcloud'
rsync -avr --delete "/mnt/user/NCloud" $destination


#Start Docker Containers
echo 'Starting up Docker Containers'
/etc/rc.d/rc.docker start

########################################################################
#Backup of the other stuff:

#Backup Game Backup share via rsync
echo 'Making Backup Game Backup share'
rsync -avr --delete "/mnt/user/Game Backups" $destination

#Backup Photos and Videos share via rsync
echo 'Making Backup of Photos and Videos share'
rsync -avr --delete "/mnt/user/Photos and Videos" $destination

########################################################################

#Umount destination
echo 'Unmounting and spinning down Disk....'
/usr/local/sbin/rc.unassigned umount $drive
/usr/local/sbin/rc.unassigned spindown $drive

#Notify about backup complete
/usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Backup to Unassigned Disk" -d "A copy of important data has been backed up" -i "normal"
echo 'Backup Complete'

I am not sure if I should use echo or logger?

Also, in a test script, I didnt manage to get the

2>&1 >> $LOGFILE

that I see some use at the end of rsync commands to do anything, what does it do and should I try to get it to work?

for reference, I set the drive to automount, then I want to manually have to start the script, but when its done running I want to be ready to just unplug without having to press anything more. (and then take to another location until its time to update it) (offline backup)

Correct me if i am wrong but, isn't it a bad idea to copy between user and disk folders?

Source:

Edited July 8, 2022 by kalidus

JorgeB · July 8, 2022

20 minutes ago, kalidus said:

Correct me if i am wrong but, isn't it a bad idea to copy between user and disk folders?

That's an unsigned disk, danger is user shares and disk shares for array/pool disks.

Mihle · August 13, 2022

On 7/8/2022 at 4:59 PM, JorgeB said:

That's an unsigned disk, danger is user shares and disk shares for array/pool disks.

unassigned disk dont have any user shares anyway.

BlackMamba24 · April 5, 2023

Hello I have the same failed: Operations not permitted and error code 23 despite the while transferring but rsync says otherwise. I have tried updating the UD plugin and reinstall to no luck. My destination device is an exfat format external hard drive disk. Is there any way to solve this issue?

Edited April 5, 2023 by BlackMamba24

dlandon · April 5, 2023

Use another file system. Exfat does not respect the file system attributes.

jonitfcfan · April 5, 2023

3 hours ago, BlackMamba24 said:

Hello I have the same failed: Operations not permitted and error code 23 despite the while transferring but rsync says otherwise. I have tried updating the UD plugin and reinstall to no luck. My destination device is an exfat format external hard drive disk. Is there any way to solve this issue?

If you care about retaining the Linux file permissions/ownership you can do what dlandon said (I use EXT4 myself on my external drive). Otherwise you could modify the rsync command so you don't get the above errors.

Instead of this:

rsync -aP [source dir] [destination dir]

(-a on its own is the same as -rlptgoD)

you could do this:

rsync -rltDP [source dir] [destination dir]

This removes the flags that otherwise tell rsync to retain the Linux file permissions/ownership.

BlackMamba24 · April 6, 2023

14 hours ago, dlandon said:

Use another file system. Exfat does not respect the file system attributes.

Thank you, eventually found that conclusion after some digging around

BlackMamba24 · April 6, 2023

13 hours ago, jonitfcfan said:
If you care about retaining the Linux file permissions/ownership you can do what dlandon said (I use EXT4 myself on my external drive). Otherwise you could modify the rsync command so you don't get the above errors.

Instead of this:
rsync -aP [source dir] [destination dir]

(-a on its own is the same as -rlptgoD)
you could do this:
rsync -rltDP [source dir] [destination dir]
This removes the flags that otherwise tell rsync to retain the Linux file permissions/ownership.

Thank you so much! I came this command "rsync -av --no-o --no-g --no-perms --progress --stats", which is the same as yours but just longer? This includes the stats variable and I assume --progress is the same as P if I read that correctly. Your version would be rsync -rltvDP --stats and would do the same thing? Also, does the order matter for where I put the new v variable?

jonitfcfan · April 6, 2023

6 hours ago, BlackMamba24 said:

I assume --progress is the same as P if I read that correctly

Yes

6 hours ago, BlackMamba24 said:

Your version would be rsync -rltvDP --stats and would do the same thing?

Unless you want extra output that -P or --stats doesn't already give you, you probably don't need the -v (or --verbose) flag

6 hours ago, BlackMamba24 said:

Also, does the order matter for where I put the new v variable?

I don't think the order of the flags normally matters

BlackMamba24 · April 6, 2023

10 hours ago, jonitfcfan said:

Yes

Unless you want extra output that -P or --stats doesn't already give you, you probably don't need the -v (or --verbose) flag

I don't think the order of the flags normally matters

Oh I see, was playing around and I believe the info provided from -P and _-stats are different from each other right? Seems to info I want but wanted to make sure there wasn't any overlap between them. I

I was wondering what the verbose flag actually did if you happen to know. Read the description and it wasn't very clear but it provided some extra info that seemed useful to me.

Also, does z help in any way? Saw that it compresses the data I think but I did not see any differences

Help with rsync and UNASSIGNED devices

Recommended Posts

coblck

Link to comment

Mihle

Link to comment

trurl

Link to comment

kilonde

Link to comment

Mihle

Link to comment

itimpi

Link to comment

Mihle

Link to comment

kalidus

Link to comment

JorgeB

Link to comment

Mihle

Link to comment

BlackMamba24

Link to comment

dlandon

Link to comment

jonitfcfan

Link to comment

BlackMamba24

Link to comment

BlackMamba24

Link to comment

jonitfcfan

Link to comment

BlackMamba24

Link to comment

Join the conversation