2 hours ago, mgutt said:

Does not really make sense as you said you used -a, which already includes -o and -g on your first sync?! Another strange thing: If you use rsync though the terminal, it is executed as root, so if -o and -g wouldn't be active, the files should get root:root and not nobody:users.

 

Hey,

 

I actually did the copying using a Krusader docker.  When I first started using Unraid, I watched a lot of SpaceInvader One's guides - one of which was about moving data around. I needed to move data from my old NAS to the new one, and the method SIO explained was using Krusader, and it has worked fine whenever I needed to move data.  Since it was so much data I was moving, I wanted a way of being able to see what is happening in the terminal window. As I understand it, if I had run the command in a shell window from Unraid GUI, I would have had to keep that window open for it to keep going.  Using Krusader just seemed like a more stable way of doing it.

 

I am making an educated guess here: In order for the Krusader docker to be able to copy the files while maintaining user and group, the docker needs root access, which is not enabled by default...and was not enabled when I copied the da;ta over. So I would guess that is why the files copied, but the user and group was not.  Would likely also explain why it was nobody:users?  

 

When I ran "rsync -avhH" WITHOUT root access (as per the instruction from one of SIO's videoes) - everything copied with no errors. When I tried "rsync -avhogH" without access, there were loads of errors saying it didn't have permission to do something (I can't recall the exact wording) - trying the same command with root access enabled it worked fine and backups seem to be working spot on now.

 

I'll be honest, I find it confusing understanding what a lot of the arguments for command line applications mean. I had no idea that "a" would essentially include o and g with it.  Since it didn't error when copying the data, I assume that copying the user:group is an optional thing?  Like "try and copy the user and group if you can but if you can't just skip it and give it a default user:group" type deal?

 

Thanks for taking the time to read.

 

One other thing: is there anything one can add to the end of your script that once it's finished it will change the owner of the .log files the script generates?  They are owned by "root" as default and so am unable to open the log files over the network.  Hopefully just a simple chown command for the log file(s) at the end of the script?

 

Thanks!

1 hour ago, Stupot said:

Would likely also explain why it was nobody:users?  

Yes, this is the reason.

 

1 hour ago, Stupot said:

trying the same command with root access enabled it worked fine

Then -o and -g is not needed, but it does not cause problems to use it in combination with -a. Rsync does not warn you, if you use options twice.

 

1 hour ago, Stupot said:

As I understand it, if I had run the command in a shell window from Unraid GUI, I would have had to keep that window open for it to keep going.  Using Krusader just seemed like a more stable way of doing it.

Two hints:

- The File Explorer Plugin has a move feature included, acts as root and works even if the browser window has been closed. But I'm not sure if Hardlinks are preserved 🤷

Downside: It checks for each file if there is access, which makes it a little bit slower than rsync alone. But I'm not sure r

- You can execute rsync as follows to be able to close the Terminal:

rsync -aH src/ dst & disown

Downside: You need to use kill <pid> to stop rsync.

 

On 10/1/2024 at 10:08 AM, mgutt said:

Yes, this is the reason.

 

Then -o and -g is not needed, but it does not cause problems to use it in combination with -a. Rsync does not warn you, if you use options twice.

 

Two hints:

- The File Explorer Plugin has a move feature included, acts as root and works even if the browser window has been closed. But I'm not sure if Hardlinks are preserved 🤷

Downside: It checks for each file if there is access, which makes it a little bit slower than rsync alone. But I'm not sure r

- You can execute rsync as follows to be able to close the Terminal:

rsync -aH src/ dst & disown

Downside: You need to use kill <pid> to stop rsync.

 

 

Thanks, everything makes sense now haha!

 

Thank you for the tips too!  I do use the file manager plugin for smaller transfers - but using rsync in the Krusader docker's terminal feels way more reliable for larger transfers.  Now I've got the root access issue sorted hopefully will be my go to!  Though will defo keep your "disown" trick in mind for future!

Thank you!

Hello,
I have the problem that the Docker containers do not start again after the backup has been performed. What could this be?

11 minutes ago, Jensus11 said:

Hello,
I have the problem that the Docker containers do not start again after the backup has been performed. What could this be?

If I am not mistaking, there is a log create with the backup. Have you checked the content ? It could point you in the right direction.

I looked at the log file. Unfortunately I can't find any information about stopping or starting.

This morning the same thing happened again. After the backup all containers were shut down and I had to start them again manually.

Everything is logged: Stopping, Snapshot Creation and Starting the containers. So check the logs.

I'm trying to backup to a external ssd and I get the message saying hardlinks are not supported.  How would I fix that?  The script works when backing up to my array just not to the portable ssd.  

Thanks

On 11/2/2024 at 7:45 PM, bozzio said:

I'm trying to backup to a external ssd and I get the message saying hardlinks are not supported.  How would I fix that?  The script works when backing up to my array just not to the portable ssd.  

Thanks

What Filesystem is the external SSD formatted? 

Have a look at the "Warnings" section on the first post on the first page:

 

Quote

Do not use NTFS or other partition formats, which do not support Hardlinks and/or Linux permissions. Format external USB drives with BTRFS and install WinBTRFS, if you want to access your backups through Windows.

 

3 hours ago, b_tob said:

What Filesystem is the external SSD formatted? 

Have a look at the "Warnings" section on the first post on the first page:

 

 

Ah thanks man.  It was xfs which I thought would be okay. But formatted to btrfs and it worked. Thanks so much!

So, after endless attempts, I finally got it done.

 

For everyone who also wants to use the script in connection with the rsync server, the lines must look like this:

 

# user-defined rsync command
alias rsync='rsync -e "ssh -p5020 -o StrictHostKeyChecking=no"'

# user-defined ssh command
alias ssh='ssh -p5020 -o StrictHostKeyChecking=no'

 

I have chosen port 5020 as an example.

Hi Marc,

 

thanks for the great script. I am still struggling with some of my messy folders from various (backup) locations from my pre-unraid past.

However, your script is helping a lot, to maintain simple and accessible backups.

 

Is there an option or a flag to pin or keep a specific backup to ensure it won't be pruned during the cleanup phase?

 

Cheers, c

I use the script to back up my data to an external Synology. For this, I have created a customized script there (based on the template from here). Now, I would like to limit the bandwidth using --bwlimit=1M . Where should I insert it?

@mguttTo avoid having a named list of several directories and each directory versioned separately I wanted to use a directory where softlinks point to the directories, i.e.

 

# backup source to destination

backup_jobs=(
  # source                          # destination
  "/mnt/user/backup-links/"          "/mnt/disks/EXTERNAL_3TB_A/tower-backup"

 

whereas in the backup-links directory I have softlinks to all the directories I'd like to backup

 

root@Tower:/mnt/user/backup-links# ll
lrwxrwxrwx 1 root root 18 Mar 16 18:17 appdata -> /mnt/cache/appdata/
lrwxrwxrwx 1 root root 16 Mar 16 18:18 archiv -> /mnt/user/archiv/
lrwxrwxrwx 1 root root  5 Mar 16 18:17 boot -> /boot/
lrwxrwxrwx 1 root root 23 Mar 19 18:01 docker-volumes -> /var/lib/docker/volumes/
lrwxrwxrwx 1 root root 17 Mar 16 18:18 domains -> /mnt/user/domains/
lrwxrwxrwx 1 root root 18 Mar 16 18:18 download -> /mnt/user/download/
drwxrwxrwx 2 root root 53 Mar 16 18:19 multimedia/
lrwxrwxrwx 1 root root 18 Mar 19 18:01 pdfocred -> /mnt/user/pdfocred/

 

which unfortunately creates

 

root@Tower:/mnt/disks/EXTERNAL_3TB_A/tower-backup/20250319_230326# ll
total 32
-rw------- 1 root root 1300 Mar 19 23:03 20250319_230326.log
lrwxrwxrwx 2 root root   18 Mar 16 18:17 appdata -> /mnt/cache/appdata/
lrwxrwxrwx 2 root root   16 Mar 16 18:18 archiv -> /mnt/user/archiv/
lrwxrwxrwx 2 root root    5 Mar 16 18:17 boot -> /boot/
lrwxrwxrwx 2 root root   23 Mar 19 18:01 docker-volumes -> /var/lib/docker/volumes/
lrwxrwxrwx 2 root root   17 Mar 16 18:18 domains -> /mnt/user/domains/
lrwxrwxrwx 2 root root   18 Mar 16 18:18 download -> /mnt/user/download/
drwxrwxrwx 1 root root   46 Mar 16 18:19 multimedia/
lrwxrwxrwx 2 root root   18 Mar 19 18:01 pdfocred -> /mnt/user/pdfocred/

 

is there any way to avoid / work around this?

Edited March 19, 20251 yr by Moerky
formatting

To be protected agains Ransomware I execute this awesome script from a second unraid server, which has read-only access to the files on the main server. So the backupserver "pulls" the files.

Due to some changes in my config I'd like to only use my main server with one or more external hdds.

 

But if the server has 24/7 access to the backup destination where the files are pushed to, there's a higher risk of ransomware attacks affecting also the backup.

 

So can this script be modified to manually mount the external usb disk, do the backup and unmount the disk when finished?

 

That might solve the problem.

 

Best wishes!

Quick question, I searched the thread but didn't find any mention of it. I assume it doesn't work with the Docker Compose plugin, right?

@mgutt How to set owner and permissions for log file? It is always created with: root -rw-------

On 5/11/2025 at 3:39 PM, jon_of_us said:

Quick question, I searched the thread but didn't find any mention of it. I assume it doesn't work with the Docker Compose plugin, right?

I tested it today and it seems to work.

My test scenario was:

1. I created an immich stack via Docker Compose Manager

2. Immich ran via docker compose

3. Starting the backup script stops all running containers including the ones started via compose up.

4. After snapshot creation by the script all containers started up again, also the compose ones

I have a specific share on my array which serves as the destination of several backups made by this script. Now I want to move the whole share to another disk on the array because the old disk is running out of space. How can I do that without losing all the hardlinks the script creates? When moving around huge shares I usually use the unbalanced plugin, but it seems like it's unstable in handling hardlinks properly. Can someone point me in the right direction here?

4 hours ago, ArdNsc said:

Can someone point me in the right direction here?

Option A)

Ask the dev of the unbalanced plugin to add hard link support.

Option B)

Use rsync:

rsync -aH --remove-source-files --numeric-ids /mnt/disk3/sharename/backups/ /mnt/disk5/sharename/backups & disown

• the final "/" of the source path is important or it will create an additional source "backups" folder in the destination path.

• the "& disown" makes it a background process, so you can close the terminal.

• The "-H" keeps hardlinks

• Use "/disk" paths only (as in my example), not "/user" paths (avoids overhead and avoids accidentally overwriting of files, which can happen if "/user" points to the same disk)

To finally delete the empty source dirs:

find /mnt/disk3/sharename/backups/ -type d -empty -delete

On 9/16/2025 at 3:37 PM, mgutt said:

Option B)

I did that with the first folder now. While I got around 1TB free space on the old drive on the new target drive it took around 2TB of space, regarding to the used/free columns in the unraid UI. What I should have mentioned is that the fs on the "old" drive ("disk5") is btrfs and on the "new" one ("disk3") is zfs. Is that a problem?

Edited September 24, 2025Sep 24 by ArdNsc

Trying to get this to work for backup to a remote Unraid machine. I have passwordless SSH working to the remote machine on a custom port using ssh rsa id. How do I specify this custom port in the script?

Right now, when I run the script I get the error:

Could not obtain last backup! (/boot)

Description: Error: ()!

Not sure how to proceed from here. I have path set up as follows:

backup_jobs=(

# source # destination

"/boot" "root@<external ip>:/mnt/user/barba-rsync-bck/boot"

34 minutes ago, lococola said:

How do I specify this custom port in the script?

According to Google (haven't tried it myself as I don't use a custom SSH port in my rsync backup script):

To specify a custom SSH port number in an

rsync backup script, use the -e or --rsh option followed by the SSH command including the port number specified with the -p flag, all enclosed in quotes. 

Command Line Examples

The general syntax for rsync with a custom SSH port is:

bash

rsync [OPTIONS] -e 'ssh -p PORT_NUMBER' [SOURCE] [USER]@[HOST]:[DESTINATION]

Example:

To back up a local directory /path/to/source to a remote server remotehost using username user and SSH port 2222:

bash

rsync -avz -e 'ssh -p 2222' /path/to/source user@remotehost:/path/to/destinatio

Thanks, but I don't know how to incorporate it into this script specifically. But for now I have removed the custom port and set it back to 22 on the target. But now I get a different error when I run the script:

created directory /mnt/user/barba-rsync-bck/boot/link_dest
--link-dest arg does not exist: /mnt/user/barba-rsync-bck/boot/link_dest
removed '/tmp/_tmp_user.scripts_tmpScripts_rsync incremental backup remote_script/empty.file'
cd+++++++++ ./
Error: Your destination root@<external ip>:/mnt/user/barba-rsync-bck/boot does not support hardlinks!

Not sure what's going wrong. You can use hardlinks over SSH, right? Or maybe it's not making an SSH connection for some reason?

Here is the destination path I set up:

backup_jobs=(

# source # destination

"/boot" "root@<external ip>:/mnt/user/barba-rsync-bck/boot"

What could be the issue?