Help with rsync and UNASSIGNED devices

[adminmat]

I'm practicing some rsync commands before I set up an external USB HDD for incremental backups. 

 

I added a USB Flash drive to my USB port on my server mobo and mounted it with Unassigned Devices. The drive is working properly. I can transfer files to the drive using Krusader, rsync and remotely via windows SMB. When I transferred some files from a share on the array to the USB drive using rsync I got some errors that I believe are related to ownership. 

 

Although the files transferred I got these errors: 

 

What rsync command would you recommend that would make an easy restore source if needed? Not messing up ownership, etc. You can leave out the --delete for now as I'm more confused about the "options".

 

Also, the ball is still grey, not turning green although the drive is working. Does this matter? Thoughts, @dlandon ?

 

[dlandon]

1 hour ago, adminmat said:

I'm practicing some rsync commands before I set up an external USB HDD for incremental backups. 

 

I added a USB Flash drive to my USB port on my server mobo and mounted it with Unassigned Devices. The drive is working properly. I can transfer files to the drive using Krusader, rsync and remotely via windows SMB. When I transferred some files from a share on the array to the USB drive using rsync I got some errors that I believe are related to ownership. 

 

Although the files transferred I got these errors: 

 

What rsync command would you recommend that would make an easy restore source if needed? Not messing up ownership, etc. You can leave out the --delete for now as I'm more confused about the "options".

 

Also, the ball is still grey, not turning green although the drive is working. Does this matter? Thoughts, @dlandon ?

 

You need to update UD.  The vfat file system does not respect the Linux attributes, hence the error.

[adminmat]

2 hours ago, dlandon said:

You need to update UD.  The vfat file system does not respect the Linux attributes, hence the error.

Thanks. I updated UD and it did turn the ball green. However, I'm still getting rsync errors. I'm not sure if these errors are even important. Or if they are only showing up because I'm using a USB Flash Type drive. I want to make sure i dont screw up my backups

 

Thoughts?
 

 

 

[JorgeB]

Adding --no-perms to the rsync command should get ride of those, like mentioned FAT doesn't support permissions.

[adminmat]

9 hours ago, JorgeB said:

Adding --no-perms to the rsync command should get ride of those, like mentioned FAT doesn't support permissions.

I tried this and still got the errors. I think this exercise is fruitless using a Flash USB drive as a destination. Since I'm just testing. When I get the external USB HDD drive in the mail it will formatted to xfs. Unless you guys recommend a different file system for a local rsync backup drive. 

[trurl]

I rsync to NTFS all the time.

[adminmat]

10 minutes ago, trurl said:

I rsync to NTFS all the time.

This seems like it may be easier if I need to pull a file off from a Windows PC. Does rsync sort out that the destination is a different file system? 

[trurl]

4 minutes ago, adminmat said:

Does rsync sort out that the destination is a different file system? 

Filesystems are below the level that rsync works at so it doesn't care.

[adminmat]

On 12/15/2020 at 2:27 PM, trurl said:

I rsync to NTFS all the time.

So I mounted an NTFS external HDD via USB and am hoping to rsync some shares to it for a backup. I've made some test folders on the drive using krusader but they are not showing up when I view the share from windows. I assume this is because krusader is making linux folders so windows can't see them. 

Is it easier to format this entire drive to xfs for these backups? What is the benefit of using NTFS? I think I'm making things for complicated for myself 🙃

 

[Hoopster]

57 minutes ago, adminmat said:

What is the benefit of using NTFS?

The benefit of using NTFS for a backup is that if your unRAID server is unavailable, you can access your backed up data on a Windows machine.

 

I use a modified Unassigned Devices backup script to backup 4 shares (Pictures, Movies, FamVideos and Videos) from my unRAID array to an external 14TB USB drive formatted NTFS.  It is accessible via UD should I need to copy files from it back to the array or from Windows should my array bite the dust and I need to access the content in Windows while sorting through unRAID issues.

Edited December 29, 2020 by Hoopster

[adminmat]

12 minutes ago, Hoopster said:

The benefit of using NTFS for a backup is that if your unRAID server is unavailable, you can access your backed up data on a Windows machine.

 

I use a modified Unassigned Devices backup script to backup 4 shares (photos, movies, TV and videos) from my unRAID array to an external 14TB USB drive formatted NTFS.  It is accessible via UD should I need to copy files from it back to the array or from Windows should my array bite the dust and I need to access the content in Windows while sorting through unRAID issues.

 

I like the flexibility this gives you. But I'm confused, if your script uses rsync (for example) to copy the files from unRAID to the NTFS drive doesn't that make them unreadable in windows? And would you care to share your script? I'm hoping for similar but specifically this:
 

- /mnt/user/appdata            (to a folder on the USB drive called: appdata)

- /mnt/user/Media/photos    (to a folder on the USB drive called: photos)

- /mnt/user/nextcloud          (to a folder on the USB drive called: nextcloud)

- /mnt/user/backups/flashdrive backup      (to a folder on the USB drive called: flash backup)

- /mnt/user/system              (to a folder on the USB drive called: system)

I tried to make sense of the sample script on the first page of the unassigned Devices thread but I don't know what to change / add. 

and another question: I understand you have to copy share to share OR mountpoint to mountpoint so how do you make a destination share on an Unassigned (external USB) Device? 

 

[Hoopster]

17 minutes ago, adminmat said:

and another question: I understand you have to copy share to share OR mountpoint to mountpoint so how do you make a destination share on an Unassigned (external USB) Device? 

You don't create anything; let rsync do it.  I just formatted the destination drive as NTFS and let it run the backup script when plugged in.  The first time the destination folders (same name as the shares) are created.

 

I only made minor modifications to the UD backup script by adding the rsync commands for backing up the additional shares.

 

/mnt/user/Pictures for example just becomes Pictures on the USB drive and all these folders appear just fine in Windows.

 

#!/bin/bash
PATH=/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin
## Available variables: 
# AVAIL      : available space
# USED       : used space
# SIZE       : partition size
# SERIAL     : disk serial number
# ACTION     : if mounting, ADD; if unmounting, UNMOUNT; if unmounted, REMOVE; if error, ERROR_MOUNT, ERROR_UNMOUNT
# MOUNTPOINT : where the partition is mounted
# FSTYPE     : partition filesystem
# LABEL      : partition label
# DEVICE     : partition device, e.g /dev/sda1
# OWNER      : "udev" if executed by UDEV, otherwise "user"
# PROG_NAME  : program name of this script
# LOGFILE    : log file for this script

case $ACTION in
  'ADD' )
    #
    # Beep that the device is plugged in.
    #
    beep  -l 200 -f 600 -n -l 200 -f 800
    sleep 2

    if [ -d $MOUNTPOINT ]
    then
      if [ $OWNER = "udev" ]
      then
        beep  -l 100 -f 2000 -n -l 150 -f 3000
        beep  -l 100 -f 2000 -n -l 150 -f 3000

        logger Started -t$PROG_NAME
        echo "Started: `date`" > $LOGFILE

        logger Pictures share -t$PROG_NAME
        rsync -a -v /mnt/user/Pictures $MOUNTPOINT/ 2>&1 >> $LOGFILE
		
	logger Videos share -t$PROG_NAME
        rsync -a -v /mnt/user/Videos $MOUNTPOINT/ 2>&1 >> $LOGFILE
		
	logger Movies share -t$PROG_NAME
        rsync -a -v /mnt/user/Movies $MOUNTPOINT/ 2>&1 >> $LOGFILE

	logger Family Videos share -t$PROG_NAME
        rsync -a -v /mnt/user/FamVideos $MOUNTPOINT/ 2>&1 >> $LOGFILE

        logger Syncing -t$PROG_NAME
        sync

        beep  -l 100 -f 2000 -n -l 150 -f 3000
        beep  -l 100 -f 2000 -n -l 150 -f 3000
        beep  -r 5 -l 100 -f 2000

        logger Unmounting Backup -t$PROG_NAME
        /usr/local/sbin/rc.unassigned umount $DEVICE

       echo "Completed: `date`" >> $LOGFILE
        logger Backup drive can be removed -t$PROG_NAME

        /usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Server Backup" -d "Backup completed" -i "normal"
    fi
    else
        logger Backup Drive Not Mounted -t$PROG_NAME
  fi
  ;;

  'REMOVE' )
    #
    # Beep that the device is unmounted.
    #
    beep  -l 200 -f 800 -n -l 200 -f 600
  ;;

  'ERROR_MOUNT' )
	/usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Server Backup" -d "Could not mount Backup" -i "normal"
  ;;

  'ERROR_UNMOUNT' )
	/usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Server Backup" -d "Could not unmount Backup" -i "normal"
  ;;
esac

 

Edited December 29, 2020 by Hoopster

[adminmat]

35 minutes ago, Hoopster said:

You don't create anything; let rsync do it.  I just formatted the destination drive as NTFS and let it run the backup script when plugged in.  The first time the destination folders (same name as the shares) are created.

 

I only made minor modifications to the UD backup script by adding the rsync commands for backing up the additional shares.

 

/mnt/user/Pictures for example just becomes Pictures on the USB drive.

 

 

Thanks for this. Exactly what I was looking for. Although I have a new problem. Since I created those files on the drive with Krusader then I tried to format the drive to NTFS it's not able to be formatted. I connected it to my windows PC. Tried formatting regularly then with diskpart. Then with the WD drive utilities. All keeps failing to format.  Any ideas? 

 

[Hoopster]

12 minutes ago, adminmat said:

I connected it to my windows PC. Tried formatting regularly then with diskpart

Does Windows Disk Management see the drive or any partitions?  If so you can remove partitions and format there.

 

There are also a couple of Linux utilities for formatting a drive with NTFS; mkntfs is a command line utility and GParted is a GUI that will do it.

[adminmat]

4 minutes ago, Hoopster said:

Does Windows Disk Management see the drive or any partitions?  If so you can remove partitions and format there.

 

There are also a couple of Linux utilities for formatting a drive with NTFS; mkntfs is a command line utility and GParted is a GUI that will do it.

Yes it shows up in Disk Management. I have tried removeing the partition and formatting. But I keep getting errors:
Maybe I should try in unRAID?

 

 

 

[adminmat]

OK, update: I connected the drive to my Windows laptop and it formatted first try. Something is wrong with my Windows desktop...
 

@Hoopster thanks again for the help. I'll try the script as you recommended and will follow up 

[adminmat]

2 hours ago, Hoopster said:

You don't create anything; let rsync do it.  I just formatted the destination drive as NTFS and let it run the backup script when plugged in.  The first time the destination folders (same name as the shares) are created.

 

I only made minor modifications to the UD backup script by adding the rsync commands for backing up the additional shares.

 

/mnt/user/Pictures for example just becomes Pictures on the USB drive and all these folders appear just fine in Windows.

 

 

Success. I plug it in and it runs just as expected. Lots of entertaining beeps. I see you are not deleting the items on the backup (destination) which you have deleted from the source. 

 

I can just add "--delete" prior to the source like this? 

 

	logger Videos share -t$PROG_NAME
        rsync -a -v --delete /mnt/user/Videos $MOUNTPOINT/ 2>&1 >> $LOGFILE

 

I'll keep the source as a master copy. 

 

Thanks 

[Hoopster]

8 hours ago, adminmat said:

I can just add "--delete" prior to the source like this? 

Yes.  Do a couple of tests first. 

 

I also run an rsync backup to a second server and every 3-4 month I run an alternative version of that script that deletes from the backup anything that has been deleted from the source.  I'll eventually likely do the same with the external drive, but, it's another fallback in case something gets accidentally deleted from the server (as has happened to me twice now with entire year of photos).

[trurl]

13 hours ago, adminmat said:

I've made some test folders on the drive using krusader but they are not showing up when I view the share from windows. I assume this is because krusader is making linux folders so windows can't see them. 

Things don't work that way. It is not possible for anything to make "linux folders" on NTFS filesystem. Are you sure you mapped the Unassigned Device correctly in Krusader (r/w slave)?

[adminmat]

2 hours ago, trurl said:

Things don't work that way. It is not possible for anything to make "linux folders" on NTFS filesystem. Are you sure you mapped the Unassigned Device correctly in Krusader (r/w slave)?

I probably have something mapped incorrectly. I have krusader mapped to /mnt/disks/ for the unassigned devices. 

 

I guess it doesn't matter now as I have things sorted. I was able to format the drive.

 

@trurl I'm running some auto-mount / rsync script tests now. Do you know if it's necessary to stop the docker / VM service before running an rsync backup? I'm including the system share, appdata, flash drive, my nextcloud data share. Is there a chance I could cause a corruption issue copying the files while the services are running? Would it be better to use the backups located on the array as sources instead of the shares in use?  

 

 

Edited December 29, 2020 by adminmat

[trurl]

That is the wrong access mode for Unassigned Devices. You want Read/Write Slave mode:

 

[adminmat]

11 minutes ago, trurl said:

That is the wrong access mode for Unassigned Devices. You want Read/Write Slave mode:

 

Thanks, I made the correction. Thoughts on my other question Re: stopping services before rsync? 

[trurl]

4 hours ago, adminmat said:

use the backups located on the array as sources instead of the shares in use

 

[coblck]

@Hoopster just found this thread and wondering if ok to use your script?? have just taken delivery of new 16tb external hard drive to make much needed copy of my movies and tv shows, would you recommend moving approx 11tb of data over usb 3 connection using rysnc or is there any other way ? eventually when funds allow will build another unraid server for backups.

 

[Hoopster]

2 hours ago, coblck said:

just found this thread and wondering if ok to use your script?

Be my guest.  Use/modify it as you wish.  It is nothing more than the standard UD Backup script modified to backup some additional shares.

 

2 hours ago, coblck said:

would you recommend moving approx 11tb of data over usb 3 connection

This is fine. It will take a while but just plug it in and let it run.  I initially did about 8TB.  I can't remember exactly how long it took but it was not an unreasonable amount of time.