strike

Do you have a public IP address? If not you're behind CGNAT and that's why you can't get a direct connection. If you do have a public IP try opening a port in your firewall. Check out tailscale's help page for direct connections.

Are you behind CGNAT?

Make sure you disable "use tailscale dns"

That is indeed far simpler. ☺️ Glad you found something that works.

To apply permissions and ownership only to the newly transferred files, you can modify your script to specifically target only those files. One way to achieve this is to include the --itemize-changes command and log the files transfered which can help you apply the chmod and chown commands to just the most recently transferred files after your rsync operation. You can make the following changes to your script wherever you handle the permissions: Use rsync --itemize-changes to identify which files were just transferred. Process only those files for changing permissions and ownership. Here’s how you can adjust your script: #!/bin/bash # Remote and local directories REMOTE_SERVER="user@remoteserver" # replace with your remote server SSH user and address REMOTE_FOLDER1="/home/remoteserver/media/folder1/" REMOTE_FOLDER2="/home/remoteserver/downloads/folder2/" LOCAL_FOLDER1="/data1/folder1/" LOCAL_FOLDER2="/data1/documents/folder2/" # Path to the SSH key (if it's not the default) SSH_KEY_PATH="/path/to/your/private/key" # Replace with your private key path, if different from default (~/.ssh/id_rsa) # Function to rsync and clear remote folder sync_and_clear() { local remote_folder=$1 local local_folder=$2 echo "Syncing from $remote_folder to $local_folder..." # Use --itemize-changes with --log-file to capture detailed info about the transfer rsync --protocol=31 -avz --itemize-changes -e "ssh -v" --remove-source-files "$REMOTE_SERVER:$remote_folder" "$local_folder" --log-file=/tmp/rsync_log.txt if [ $? -eq 0 ]; then echo "Sync completed successfully. Clearing remote folder $remote_folder..." ssh -i "$SSH_KEY_PATH" "$REMOTE_SERVER" "find '$remote_folder' -type f -delete" # Delete files from the remote folder # Process the log to apply permissions and ownership only to the newly transferred files echo "Changing permissions and ownership for newly transferred files..." while read -r line; do # Extracting the transferred file path if [[ $line =~ ^[<][^ ] ]]; then new_file=$(echo $line | awk '{print $2}') # Ensure it starts with the local folder path if [[ $new_file == $local_folder* ]]; then chmod 770 "$new_file" chown nobody:users "$new_file" fi fi done < /tmp/rsync_log.txt # Clean up rm /tmp/rsync_log.txt echo "Contents of $remote_folder cleared and permissions applied." else echo "Error during syncing. Remote folder contents were not cleared." fi } # Sync and clear folder1 sync_and_clear "$REMOTE_FOLDER1" "$LOCAL_FOLDER1" # Sync and clear folder2 sync_and_clear "$REMOTE_FOLDER2" "$LOCAL_FOLDER2"Key Changes:I added the --itemize-changes option to the rsync command, which provides detailed output about changes made during the transfer process. The log file specified with --log-file will reflect only the transferred files. Processing the Log: After transferring files, the script reads the /tmp/rsync_log.txt file. The while read -r line; do loop processes each line of the log. It checks if the line starts with a <, which indicates a file that was transferred. It extracts the new file path and applies chmod and chown only to that file. Delete the Log: After processing the log file, the script removes it with rm /tmp/rsync_log.txt to keep the environment clean. Important Note:Make sure to check the paths and adjust based on the actual transfer and any potential patterns that rsync logs. This approach assumes a certain log format and might need tweaking based on the actual output. This will ensure that you apply permissions and ownership only to files that were just transferred, rather than the entire local folder. How It Works:Files that were newly transferred will begin with a < character in this output when --itemize-changes is combined with rsync. The script reads through the log file, checks for newly transferred entries, and applies the desired permission/ownership changes only to those files. This should help you isolate newly transferred files for permission adjustments in a more effective manner.

Also, maybe try this: https://serverfault.com/questions/447699/rsync-error-unexplained-error-code-255-at-io-c

Comment out this part if your ssh key is in the default ~/.ssh/id_rsa location: # Path to the SSH key (if it's not the default) SSH_KEY_PATH="/path/to/your/private/key" # Replace with your private key path, if different from default (~/.ssh/id_rsa)The --remove-source-files option will only remove files that have successfully been transferred. If the connection is dropped before any files are transferred, that can lead to the error you are seeing. You might try removing that option temporarily to see if the sync goes through, which can help narrow down issues. If you're still facing issues, you can add the -v option to ssh in your rsync command to get more verbose output, which may help in troubleshooting: rsync -avz -e "ssh -i $SSH_KEY_PATH -v" "$remote_folder" "$local_folder"This should provide more context about the connection issue. You can also add -vvv for even more verbosity to see all debug messages.

I assume you have ssh access too and not just sftp? Just ssh into the box and use rsync. Set up a script with the user script plugin to rsync all the data over then delete on the remote server. Run the script on whatever schedule you like. The script can maybe be something like this: #!/bin/bash # Remote and local directories REMOTE_SERVER="user@remoteserver" # replace with your remote server SSH user and address REMOTE_FOLDER1="/home/remoteserver/media/folder1/" REMOTE_FOLDER2="/home/remoteserver/downloads/folder2/" LOCAL_FOLDER1="/data1/folder1/" LOCAL_FOLDER2="/data1/documents/folder2/" # Path to the SSH key (if it's not the default) SSH_KEY_PATH="/path/to/your/private/key" # Replace with your private key path, if different from default (~/.ssh/id_rsa) # Make sure the SSH key has proper permissions chmod 600 "$SSH_KEY_PATH" # Function to rsync and clear remote folder sync_and_clear() { local remote_folder=$1 local local_folder=$2 echo "Syncing from $remote_folder to $local_folder..." rsync -avz -e "ssh -i $SSH_KEY_PATH" --remove-source-files "$remote_folder" "$local_folder" if [ $? -eq 0 ]; then echo "Sync completed successfully. Clearing remote folder $remote_folder..." ssh -i "$SSH_KEY_PATH" "$REMOTE_SERVER" "find '$remote_folder' -type f -delete" # Delete files from the remote folder echo "Contents of $remote_folder cleared." else echo "Error during syncing. Remote folder contents were not cleared." fi } # Sync and clear folder1 sync_and_clear "$REMOTE_FOLDER1" "$LOCAL_FOLDER1" # Sync and clear folder2 sync_and_clear "$REMOTE_FOLDER2" "$LOCAL_FOLDER2"

It's 99% misconfigured volume mapping/not using the correct mapped path in one or more of your containers. It could be a simple typo, Movies vs movies. Linux is case sensitive, so mnt/user/movies is not the same path as mnt/user/Movies. Start by posting the docker run command of the containers which has volume mappings that contains the most data. i.e plex, jellyfin,radarr,sonarr,sab,immich,Metube and maybe firefox. Then go to the settings for each container where you have set up the mapped volumes and post a screenshot of that too.

Yes, the array will operate as normal when upgrading drives. If you want you can upgrade both drives at the same time, BUT not recommended since you're unprotected if a drive fails during the rebuild. So upgrade one at the time. You'll have to do two rebuilds, but as mentioned the array is operational during the rebuild so there's really no downtime. It could be wise to minimize writes to the array tho, as writes to the array will slow down the rebuild. But if it were me I wouldn't care about it and just run the server as normal. It just takes a bit longer, but hey, who are we racing anyway?? :P You have extra protection in the second parity drive so no need to rush it. And preclear won't help when upgrading parity drives, it only helps on data disks.

Glad you found a solution. I still think SMB will be slower than SFTP, but give it a go :)

You can use tailscale or wireguard with sftp. Make sure you have a direct connection in tailscale.How did you connect in filezilla? With the tailscale ip or hostname? If you want to NOT use tailscale or wireguard you have to set up ssh keys and forward port 2022 on the remote router. It's pretty secure with ssh keys,but you can also set up whitelist in sftpgo so only your IP can connect and block everything else. But there shouldn't really be much overhead using tailscale (which also uses wireguard btw), IF you have a direct connection. If not you'll be routed through slow derp servers. So make sure you have a direct connection.

SMB has a massive performance hit with high latency. You should really set up the SFTPGo docker container and use that instead. Especially if you're going to transfer large files. For working with documents or photos SMB are fine but you really want to use SFTP instead. You could try to disable bandwidth throttling on high latency network (HKLM\System\CurrentControlSet\Services\LanmanServer\Parameters\DisableBandwidthThrottling) this will probably help a bit but it'll still be slower then FTP.

Set it to custom and use this to find the correct crontab value: https://crontab.guru/#00_00_*_*_1 Or change the daily time in settings-Scheduler-Fixed Schedules. This will change the time for all tasks (except custom) that run daily., if any.

I understand, but deleting the image won't delete any of your data or settings. All the data/settings lives in your appdata on your cache drive and not in the image. Here you can see the process in the manual: https://docs.unraid.net/unraid-os/manual/docker-management/#:~:text=Go to Settings → Docker,file manually if you prefer). In some cases recreating the docker image can solve a variety of problems. But I'm not sure it will help in your case. it can't hurt to try it tho. I don't use pi-hole so I can't say it was because of that either, but probably not if you deleted it and put your DNS servers back to what it was in your router settings.

Ah,Okey. Then it's probably not the same issue. Deleting the docker image will delete all your apps,but it's super easy to install them again from apps-previous apps. And you won't loose any settings or data. To delete the docker image just go to docker,enable advanced view,disable docker then choose to delete the image. It will be recreated once you enable docker again.

Are you having the same issues I was having here?: https://forums.unraid.net/topic/164058-docker-issues-how-severe-was-the-docker-vulnerability-that-was-just-patched-in-the-recent-unraid-versions-thinking-about-downgrading/ If so, try to delete the docker image and put it on a different pool if you have.

Login to your tailnet admin dashboard and disable key expiry

I did something similar 6 months ago as I was not happy with the fan control I was able to get with my supermicro MB. I went from intel xeon E-2288G on supermicro X11SCA-F to i7-12700k on a z690 edge MB with 64GB on-die ECC DDR5 RAM. Now I'm quite happy as my fans runs a lot quieter. It's more power efficient and the igpu on the 12700k runs circles around the igpu on 2288G. Best upgrade I did for my server in a while. Go for it!

Have you tried the steps listed in the faq Q21? https://github.com/binhex/documentation/blob/master/docker/faq/vpn.md

Which unraid version are you running? Have you changed permission on you appdata folder recently, maybe ran the new (non safe docker) permission tool lately? There's some permission issues with iptables causing deluge not to abe to start. If you've not changed permissions on your appdata folder recently I suggest remove the deluge container, delete the appdata for it and install it again. If you're on an old unraid version maybe try to update.

They were just shot. I was thinking about using them in a pool for backup. But after testing them both in my main server and moving them to my backup server with the same issue, I gave up. It was weird tho because the speed was normal for a few minutes (probably because it was still in RAM), but then it just got slower and slower and I realized I couldn't use them. One thing I didn't test tho was just having them in a single 1 drive pool. It could be that just one of them is shot and the other one is fine.

I just ran some write tests to the cache pool which confirmed the drives was the culprit. They were probably thrashed due to trim not working. Tried to secure erase them and put them back in a new pool which sometimes can help,but it didn't this time.

Correct

Yes, you just won't have any parity protection. You'll have to do tools->new config and reassign all the drives except the parity to remove it from the array.