adminmat Posted December 14, 2020 Share Posted December 14, 2020 I'm practicing some rsync commands before I set up an external USB HDD for incremental backups. I added a USB Flash drive to my USB port on my server mobo and mounted it with Unassigned Devices. The drive is working properly. I can transfer files to the drive using Krusader, rsync and remotely via windows SMB. When I transferred some files from a share on the array to the USB drive using rsync I got some errors that I believe are related to ownership. Although the files transferred I got these errors: What rsync command would you recommend that would make an easy restore source if needed? Not messing up ownership, etc. You can leave out the --delete for now as I'm more confused about the "options". Also, the ball is still grey, not turning green although the drive is working. Does this matter? Thoughts, @dlandon ? Quote Link to comment
dlandon Posted December 15, 2020 Share Posted December 15, 2020 1 hour ago, adminmat said: I'm practicing some rsync commands before I set up an external USB HDD for incremental backups. I added a USB Flash drive to my USB port on my server mobo and mounted it with Unassigned Devices. The drive is working properly. I can transfer files to the drive using Krusader, rsync and remotely via windows SMB. When I transferred some files from a share on the array to the USB drive using rsync I got some errors that I believe are related to ownership. Although the files transferred I got these errors: What rsync command would you recommend that would make an easy restore source if needed? Not messing up ownership, etc. You can leave out the --delete for now as I'm more confused about the "options". Also, the ball is still grey, not turning green although the drive is working. Does this matter? Thoughts, @dlandon ? You need to update UD. The vfat file system does not respect the Linux attributes, hence the error. Quote Link to comment
adminmat Posted December 15, 2020 Author Share Posted December 15, 2020 2 hours ago, dlandon said: You need to update UD. The vfat file system does not respect the Linux attributes, hence the error. Thanks. I updated UD and it did turn the ball green. However, I'm still getting rsync errors. I'm not sure if these errors are even important. Or if they are only showing up because I'm using a USB Flash Type drive. I want to make sure i dont screw up my backups Thoughts? Quote Link to comment
JorgeB Posted December 15, 2020 Share Posted December 15, 2020 Adding --no-perms to the rsync command should get ride of those, like mentioned FAT doesn't support permissions. Quote Link to comment
adminmat Posted December 15, 2020 Author Share Posted December 15, 2020 9 hours ago, JorgeB said: Adding --no-perms to the rsync command should get ride of those, like mentioned FAT doesn't support permissions. I tried this and still got the errors. I think this exercise is fruitless using a Flash USB drive as a destination. Since I'm just testing. When I get the external USB HDD drive in the mail it will formatted to xfs. Unless you guys recommend a different file system for a local rsync backup drive. Quote Link to comment
trurl Posted December 15, 2020 Share Posted December 15, 2020 I rsync to NTFS all the time. Quote Link to comment
adminmat Posted December 15, 2020 Author Share Posted December 15, 2020 10 minutes ago, trurl said: I rsync to NTFS all the time. This seems like it may be easier if I need to pull a file off from a Windows PC. Does rsync sort out that the destination is a different file system? Quote Link to comment
trurl Posted December 15, 2020 Share Posted December 15, 2020 4 minutes ago, adminmat said: Does rsync sort out that the destination is a different file system? Filesystems are below the level that rsync works at so it doesn't care. Quote Link to comment
adminmat Posted December 29, 2020 Author Share Posted December 29, 2020 On 12/15/2020 at 2:27 PM, trurl said: I rsync to NTFS all the time. So I mounted an NTFS external HDD via USB and am hoping to rsync some shares to it for a backup. I've made some test folders on the drive using krusader but they are not showing up when I view the share from windows. I assume this is because krusader is making linux folders so windows can't see them. Is it easier to format this entire drive to xfs for these backups? What is the benefit of using NTFS? I think I'm making things for complicated for myself 🙃 Quote Link to comment
Hoopster Posted December 29, 2020 Share Posted December 29, 2020 (edited) 57 minutes ago, adminmat said: What is the benefit of using NTFS? The benefit of using NTFS for a backup is that if your unRAID server is unavailable, you can access your backed up data on a Windows machine. I use a modified Unassigned Devices backup script to backup 4 shares (Pictures, Movies, FamVideos and Videos) from my unRAID array to an external 14TB USB drive formatted NTFS. It is accessible via UD should I need to copy files from it back to the array or from Windows should my array bite the dust and I need to access the content in Windows while sorting through unRAID issues. Edited December 29, 2020 by Hoopster Quote Link to comment
adminmat Posted December 29, 2020 Author Share Posted December 29, 2020 12 minutes ago, Hoopster said: The benefit of using NTFS for a backup is that if your unRAID server is unavailable, you can access your backed up data on a Windows machine. I use a modified Unassigned Devices backup script to backup 4 shares (photos, movies, TV and videos) from my unRAID array to an external 14TB USB drive formatted NTFS. It is accessible via UD should I need to copy files from it back to the array or from Windows should my array bite the dust and I need to access the content in Windows while sorting through unRAID issues. I like the flexibility this gives you. But I'm confused, if your script uses rsync (for example) to copy the files from unRAID to the NTFS drive doesn't that make them unreadable in windows? And would you care to share your script? I'm hoping for similar but specifically this: - /mnt/user/appdata (to a folder on the USB drive called: appdata) - /mnt/user/Media/photos (to a folder on the USB drive called: photos) - /mnt/user/nextcloud (to a folder on the USB drive called: nextcloud) - /mnt/user/backups/flashdrive backup (to a folder on the USB drive called: flash backup) - /mnt/user/system (to a folder on the USB drive called: system) I tried to make sense of the sample script on the first page of the unassigned Devices thread but I don't know what to change / add. and another question: I understand you have to copy share to share OR mountpoint to mountpoint so how do you make a destination share on an Unassigned (external USB) Device? Quote Link to comment
Hoopster Posted December 29, 2020 Share Posted December 29, 2020 (edited) 17 minutes ago, adminmat said: and another question: I understand you have to copy share to share OR mountpoint to mountpoint so how do you make a destination share on an Unassigned (external USB) Device? You don't create anything; let rsync do it. I just formatted the destination drive as NTFS and let it run the backup script when plugged in. The first time the destination folders (same name as the shares) are created. I only made minor modifications to the UD backup script by adding the rsync commands for backing up the additional shares. /mnt/user/Pictures for example just becomes Pictures on the USB drive and all these folders appear just fine in Windows. #!/bin/bash PATH=/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin ## Available variables: # AVAIL : available space # USED : used space # SIZE : partition size # SERIAL : disk serial number # ACTION : if mounting, ADD; if unmounting, UNMOUNT; if unmounted, REMOVE; if error, ERROR_MOUNT, ERROR_UNMOUNT # MOUNTPOINT : where the partition is mounted # FSTYPE : partition filesystem # LABEL : partition label # DEVICE : partition device, e.g /dev/sda1 # OWNER : "udev" if executed by UDEV, otherwise "user" # PROG_NAME : program name of this script # LOGFILE : log file for this script case $ACTION in 'ADD' ) # # Beep that the device is plugged in. # beep -l 200 -f 600 -n -l 200 -f 800 sleep 2 if [ -d $MOUNTPOINT ] then if [ $OWNER = "udev" ] then beep -l 100 -f 2000 -n -l 150 -f 3000 beep -l 100 -f 2000 -n -l 150 -f 3000 logger Started -t$PROG_NAME echo "Started: `date`" > $LOGFILE logger Pictures share -t$PROG_NAME rsync -a -v /mnt/user/Pictures $MOUNTPOINT/ 2>&1 >> $LOGFILE logger Videos share -t$PROG_NAME rsync -a -v /mnt/user/Videos $MOUNTPOINT/ 2>&1 >> $LOGFILE logger Movies share -t$PROG_NAME rsync -a -v /mnt/user/Movies $MOUNTPOINT/ 2>&1 >> $LOGFILE logger Family Videos share -t$PROG_NAME rsync -a -v /mnt/user/FamVideos $MOUNTPOINT/ 2>&1 >> $LOGFILE logger Syncing -t$PROG_NAME sync beep -l 100 -f 2000 -n -l 150 -f 3000 beep -l 100 -f 2000 -n -l 150 -f 3000 beep -r 5 -l 100 -f 2000 logger Unmounting Backup -t$PROG_NAME /usr/local/sbin/rc.unassigned umount $DEVICE echo "Completed: `date`" >> $LOGFILE logger Backup drive can be removed -t$PROG_NAME /usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Server Backup" -d "Backup completed" -i "normal" fi else logger Backup Drive Not Mounted -t$PROG_NAME fi ;; 'REMOVE' ) # # Beep that the device is unmounted. # beep -l 200 -f 800 -n -l 200 -f 600 ;; 'ERROR_MOUNT' ) /usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Server Backup" -d "Could not mount Backup" -i "normal" ;; 'ERROR_UNMOUNT' ) /usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Server Backup" -d "Could not unmount Backup" -i "normal" ;; esac Edited December 29, 2020 by Hoopster 1 Quote Link to comment
adminmat Posted December 29, 2020 Author Share Posted December 29, 2020 35 minutes ago, Hoopster said: You don't create anything; let rsync do it. I just formatted the destination drive as NTFS and let it run the backup script when plugged in. The first time the destination folders (same name as the shares) are created. I only made minor modifications to the UD backup script by adding the rsync commands for backing up the additional shares. /mnt/user/Pictures for example just becomes Pictures on the USB drive. Thanks for this. Exactly what I was looking for. Although I have a new problem. Since I created those files on the drive with Krusader then I tried to format the drive to NTFS it's not able to be formatted. I connected it to my windows PC. Tried formatting regularly then with diskpart. Then with the WD drive utilities. All keeps failing to format. Any ideas? Quote Link to comment
Hoopster Posted December 29, 2020 Share Posted December 29, 2020 12 minutes ago, adminmat said: I connected it to my windows PC. Tried formatting regularly then with diskpart Does Windows Disk Management see the drive or any partitions? If so you can remove partitions and format there. There are also a couple of Linux utilities for formatting a drive with NTFS; mkntfs is a command line utility and GParted is a GUI that will do it. Quote Link to comment
adminmat Posted December 29, 2020 Author Share Posted December 29, 2020 4 minutes ago, Hoopster said: Does Windows Disk Management see the drive or any partitions? If so you can remove partitions and format there. There are also a couple of Linux utilities for formatting a drive with NTFS; mkntfs is a command line utility and GParted is a GUI that will do it. Yes it shows up in Disk Management. I have tried removeing the partition and formatting. But I keep getting errors: Maybe I should try in unRAID? Quote Link to comment
adminmat Posted December 29, 2020 Author Share Posted December 29, 2020 OK, update: I connected the drive to my Windows laptop and it formatted first try. Something is wrong with my Windows desktop... @Hoopster thanks again for the help. I'll try the script as you recommended and will follow up Quote Link to comment
adminmat Posted December 29, 2020 Author Share Posted December 29, 2020 2 hours ago, Hoopster said: You don't create anything; let rsync do it. I just formatted the destination drive as NTFS and let it run the backup script when plugged in. The first time the destination folders (same name as the shares) are created. I only made minor modifications to the UD backup script by adding the rsync commands for backing up the additional shares. /mnt/user/Pictures for example just becomes Pictures on the USB drive and all these folders appear just fine in Windows. Success. I plug it in and it runs just as expected. Lots of entertaining beeps. I see you are not deleting the items on the backup (destination) which you have deleted from the source. I can just add "--delete" prior to the source like this? logger Videos share -t$PROG_NAME rsync -a -v --delete /mnt/user/Videos $MOUNTPOINT/ 2>&1 >> $LOGFILE I'll keep the source as a master copy. Thanks Quote Link to comment
Hoopster Posted December 29, 2020 Share Posted December 29, 2020 8 hours ago, adminmat said: I can just add "--delete" prior to the source like this? Yes. Do a couple of tests first. I also run an rsync backup to a second server and every 3-4 month I run an alternative version of that script that deletes from the backup anything that has been deleted from the source. I'll eventually likely do the same with the external drive, but, it's another fallback in case something gets accidentally deleted from the server (as has happened to me twice now with entire year of photos). Quote Link to comment
trurl Posted December 29, 2020 Share Posted December 29, 2020 13 hours ago, adminmat said: I've made some test folders on the drive using krusader but they are not showing up when I view the share from windows. I assume this is because krusader is making linux folders so windows can't see them. Things don't work that way. It is not possible for anything to make "linux folders" on NTFS filesystem. Are you sure you mapped the Unassigned Device correctly in Krusader (r/w slave)? Quote Link to comment
adminmat Posted December 29, 2020 Author Share Posted December 29, 2020 (edited) 2 hours ago, trurl said: Things don't work that way. It is not possible for anything to make "linux folders" on NTFS filesystem. Are you sure you mapped the Unassigned Device correctly in Krusader (r/w slave)? I probably have something mapped incorrectly. I have krusader mapped to /mnt/disks/ for the unassigned devices. I guess it doesn't matter now as I have things sorted. I was able to format the drive. @trurl I'm running some auto-mount / rsync script tests now. Do you know if it's necessary to stop the docker / VM service before running an rsync backup? I'm including the system share, appdata, flash drive, my nextcloud data share. Is there a chance I could cause a corruption issue copying the files while the services are running? Would it be better to use the backups located on the array as sources instead of the shares in use? Edited December 29, 2020 by adminmat Quote Link to comment
trurl Posted December 29, 2020 Share Posted December 29, 2020 That is the wrong access mode for Unassigned Devices. You want Read/Write Slave mode: Quote Link to comment
adminmat Posted December 29, 2020 Author Share Posted December 29, 2020 11 minutes ago, trurl said: That is the wrong access mode for Unassigned Devices. You want Read/Write Slave mode: Thanks, I made the correction. Thoughts on my other question Re: stopping services before rsync? Quote Link to comment
trurl Posted December 29, 2020 Share Posted December 29, 2020 4 hours ago, adminmat said: use the backups located on the array as sources instead of the shares in use 1 Quote Link to comment
coblck Posted March 28, 2021 Share Posted March 28, 2021 @Hoopster just found this thread and wondering if ok to use your script?? have just taken delivery of new 16tb external hard drive to make much needed copy of my movies and tv shows, would you recommend moving approx 11tb of data over usb 3 connection using rysnc or is there any other way ? eventually when funds allow will build another unraid server for backups. Quote Link to comment
Hoopster Posted March 28, 2021 Share Posted March 28, 2021 2 hours ago, coblck said: just found this thread and wondering if ok to use your script? Be my guest. Use/modify it as you wish. It is nothing more than the standard UD Backup script modified to backup some additional shares. 2 hours ago, coblck said: would you recommend moving approx 11tb of data over usb 3 connection This is fine. It will take a while but just plug it in and let it run. I initially did about 8TB. I can't remember exactly how long it took but it was not an unreasonable amount of time. 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.