maxse

Guys I need some help please. I found another issue. I am running rclone crypt to a minio bucket on my unraid server. Both servers are local on my network and when the backup is done I will move the minio bucket server offsite. I am having an issue where the file transfer just stops and minio on the backup server reports that the filename is too long. I don't understand why this happens because the OS (unraid) is the same on both servers. I was really hoping that this would be a great solution for me and it took a very long time to figure out this type of a backup solution. Could someone help me out please? This is the error I got on the backup server running minio in docker "Error: file name too long (cmd.StorageErr):

folks I have a question. I got this working backing up my unraid to a remote unraid running minio. I am backing up a share that's 50tb, however, after running rclone for about a week it says in the logs that the size of the share I am backing up and hence it's data as 12TB, and then it slowly goes up to 12.212, 12.213, etc... It seems to go up rather slowly, is it going to ultimately reach 50tb before the upload is finished? I am concerned because my other smaller shares around 1-2TB had no issues when running the script and during the process rclone would show the correct size of the share as it's total size.... Is this something to be concerned about? Is there share just too big for rclone to know it's size? According to what I'm reading 50tb shouldn't be an issue. Would appreciate some help

Thank you sooo much! Looks complex though, but I think I figured out what I will do. I am just going to conitnue running minio docker with letsencrypt reverse proxy, and use rclone. There aren't any limits on file count sfor minio as far as I understand. The biggest issue then becomes the versioning that synology would provide via the snapshots, in case of ransomware getting accidentally synced, etc... I guess I'll have to be extra careful and then hopefully soon unraid will have the ability to take snapshots.... I just didn't want to have to rely on another program like minio in case I need to recover and something crashes with its database, I won't be able to read the files... unfortunately, it seems too difficult to set up sftp on the unraid, and also again, no versioning. I would just get the 12bay synology already but it's soo $$$

Do guys know if I could set this entire thing up while the unraid server is in a remote location? I have access to the machine through VPN on an rpi... Would I be able to set this entire thing up without having physical access to the server the remote location? Thhank you so much @Baskedk I want to get this going with snapshots on a remote machine and back up to it via sftp with snapshots, since I don't want to rely on minio. But the machine is in a remote location right now and I'm wondering if I could get this all going remotely while connected through the VPN, or if I have to have physical contact with the machine during certain steps? I will read through this thread again, I did twice already before and I believe the bootloader is linked within this thread? But just in case, which one should I use? This will be strictly for a remote sftp server with snapshots that I will rclone too. Supermicro 8XSIL board with an LSI controller and HP port expander. Node 804 has capicity for 12 drives which is all I want to run in this. It currently had 8, not includng the unraid cache drive and I want to be able to use all 12 for xpenology *EDIT* BTW, I plan to set up sftp on it, and use rclone from my main unraid to back up to the xpenology/synology using sftp on a different port. other than 22. It seems that it's just a matter of checking off some boxes on the xpenology to allow this. Will this automatically make it secure and an encrypted transfer, or will I also need to set up letsencrypt, like I have been doing with the minio intance? Or is that all built-in to the synology/xpenology all I need to do is forward a port and check of some boxes in the DSM?

thank you

wow that's amazing. I have been reading about it all day, this is truly amazing. I was a bit hesistant to store this data in the cloud even with encryption... but everything that I've read seems to be it's not an issue if things are enrypted... I did just see that gdrive limited daily bandwidth to 2TB which includes the 750GB upload. Any idea on how this will be effected? Not sure how much bandwidth would be used. I have 100Mbit symmetrical. Not sure how many clients could stream a 1080p file for 1.25tb? Any ideas? @mgutt oh man, I was reading all these scripts with so many characters that I missed that. I will try it again, thank you so much!

BTW, hypothetically, has anyone ever been able to get this to work with possibly radarr or the like? So that no files are stored locally? I'm thinking that radarr would put the finished file in a dummy folder, than that file is moved to the cloud. Then plex could be pointed to the seperate mounted cloud drive... But then what would happen to radarr, wouldn't it then think that the file is "missing" and try to fetch it again? I understand you wouldn't want radarr to be pointing to the mounted cloud drive because you shouldn't use the mount to write files, and only read.... Anyone have a set up like this that works?

right I understand. I thought you meant to use the the fact that google keeps multiple versions of a file without using --back-dir. I see now it's important because that folder would be easy to navigate to especially in the even of accidental deletion or change in file etc... I am really hoping that unraid comes up with snapshots soon. I am strongly looking at running xpenology because of the snapshots, at least trying it out in a VM, but I digress. Could you pls help me out with the script though. I can't get that to work BTW this is the error that I get Failed to make fs for --backup-dir "remote:backups/2020-10-09": didn't find section in config file

Thanks bud. I did read it. Of course it's possible that ransomeware will take control of gdrive and overwrite files 100 times. But with the -backup-dir command, The original files would still keep nesting in the backup directory, unless this ransomware runs a delete command for the drive? I'm also not planning on mounting it so hopefully the chances of that happening are much less. I actually saw that exact post and added it to your script rclone copy /mnt/user/backups secure:backups --backup-dir remote:backups/`date -I` --create-empty-src-dirs --ignore-checksum --bwlimit 8M --checkers 2 --transfers 1 -vv --stats 10s That's what I wrote exactly and I get an error and it won't run at all Also, out of curiosity, since google drive does record previous versions, since it's an encrypted remote, how would I know which files to restore to their previous versions, since it's all encrypted when browsing directly from the drive... Thank you so much!

Folks, I almost got it! I am using the script that @mgutt posted and it is working great! rclone copy /mnt/user/abc secure:abc --create-empty-src-dirs --ignore-checksum --bwlimit 8M --checkers 2 --transfers 1 -vv --stats 10s My only issue that I am having, (and I have been reading all day and can't figure it out) is how to have rclone create a s subfolder in that directory with the date and time stamp attached to the folder name, and place the file that was either modified or deleted from the source into that folder. I understand I have to use --backup-dir somewhere but I don't know how to get it to add the date and time stamp to it... This would be helpful if ransomware encrypts files and it starts to sync before I have the chance to catch it, or in the case of accidental deletion... I checked the restore from the trash with google drive directly, however, with an encrypted drive it's hard to figure out which files I would need to restore... Would be best to create this kind of folder, so in the case of diaster I can just mount and drag and drop and restore the original unmodified files

thank you all, yes I was aware of that. Thank you all

I understand what you're saying but all I cold find on here are how to mount, play files from the mount etc... I was looking for help if you folks had any suggestions like to limit file transfer speeds for google drive to not get banned, I remember reading something about it here a while ago but can't find it anymore... There is also a switch --backupdir or something like that, that could basically place the modified files into a directory with that date instead of overwriting them. Was hoping someone here had the script for something like this already that they could help. Thanks

anyone? There is so much to learn and read and I'm not sure what script to use for the purposes I mentioned above...

Hey folks, Can someone please help me out. I had issues with using this in the past but I'm committed to making it work now I am using google drive I am not interested in mounting the drive, but just to run a daily script. I want something that will be as close to a true backup as possible to preotect from a possible crypto or accidental deletion. What would be the best script for this? I want to be able to easily restore if a crypto comes along and I don't catch it say for a while, I need to just be able to pull down the files from before crypto hit.... I already have the user scripts plugin installed. Thank you so much! With encryption of course. Thanks again

solved

So quick question. I have a supermicro 846 SAS2 chasis with an Asus board and LSI controller currently running unraid with an i5. Would it be possible to run a stable version of Xpenology virtualized in in unraid and make the adjustments in Xpenology to see and format all the 24 drives in Synology? That would be amazing because I'm thinking I could use then transfer the VM to an X8SIL board with the same chasis and have it be a huge synology hyper back up vault set up! Can anyone help with this or think it's a bad idea? I am currently testing out minio with a vm of cloudberry which took me forever to get working and it still errors out from time to time and I can't get a another off site synology to back up to it with s3 because it just won't work with the minio S3 So this would be a great solution!

That's interesting, but I am using an LSI controller. So it would end up passing all of the drives then? Can anyone else help with this please? I am thinking of using XPENology on my remoe server to simply have it as the target for Hyper Backup Vault, and to easily be able to back up with sftp instead of running minio.

Are yuo folks doing this mainly to use the few apps like the camera app, or are you using synology for any actual storage and its DSM features? I am interested if I could just run the synolgy VM and use its DSM and SHR-2 have it set as a hyperbackup vault for another synology off site and just use the synology features? I'm just confused because Im still a noob to VMs. Would it be possible to pass through the actual drives not the unraid shares, and have the xpenology VM format the drives as it needs for their own hybrid raid and btrfs file system? I wan to then be able to utilize the snapshot feature in DSM... Would appreciate some help on this

Resurrecting an old thread here but I have question. I'm looking for an simple way to backup my unraid to a remote location. I want to purchase an 8-bay synology NAS to place at a friend's place. Would it be possible to run the XPEnology VM on my unraid at home, and use their hyperbackup to backup the unraid shares to the remote synology? Would I also be able to create snapshots of the shared folder on my home unraid machine? That would be amazing and just go this thought. I am currently attempting to use cludberry to an ofiste unraid running minio and it's just been such as hassle. *EDIT* Hmmm now that I think about. I wouldn't mind even purchasing a small 2-bay synology NAS if this could be done. I can't seen to find any info though. Do you guys know if I could the Synology Hyperbackup on the network shares from unraid, and than back those shares up to the remote Synology?

@SpaceInvaderOne hope all is well! Any chance you could update the video the simpler way you mentioned? Or anyone know if this could be easier to set up with a windows VM? I can have my vpn app running on a windows VM, I just don't know how to set everything up to actually turn it into a virtual router The issue that I'm having is my VPN provider changed servers, and now I don't remember any more how I set this up and have to relearn everything as it has been months and I don't know how to copy a configuration file easily into the ubuntu VM from my computer. Any advice?

Both servers are local, but I am using the external IP by duckdns. Both servers are on the same local network for now. I'm not sure if by using the external IP makes the data go outside/back inside, or if the router understands that they are still on the local network and just uses my local network... However, switching to the actual local IP address 192.168... doesn't have the same issue and speed is as expected... But anyway, I am trying duplicacy right now and it seems to be working well with great speeds.

it is super slow with rclone and minio. I am talking about 200KB/s! no issues when connecting directly to minio on internal network. Any suggestions?

Hey guys, sonive been pulling my hair out all week. I have this set up with minio and I can access it no problem. I am trying to use this with rclone as an off site backup. Both surveys are on the local network for now it seems that when I use the https addres in rclone the speed becomes unusable basically 400 KB/s. And sometimes I get a 403 error in rclone. when I set the local IP address for minio in rclone, the speed is back to normal. I have a 100/100 connection. So it seems like it has something to do with letsencrypt because when the local IP address is used for minio the transfer speed is fine. I even tried to use a VPN and transfer from my MacBook and same issue when using https duckdns ip. Inset it up using spaceinvader one’s video with some changes for minio. I can access the minio webui without any issues using https... would really appreciate some help guys Below is my config file ---------------------- server { listen 443 ssl; listen [::]:443 ssl; server_name name of my subdomain form duck dns.*; include /config/nginx/ssl.conf; client_max_body_size 0; location / { include /config/nginx/proxy.conf; resolver 127.0.0.11 valid=30s; proxy_max_temp_file_size 2048m; # Below is where you add the local IP and port that minio runs on proxy_pass http://192.168.1.55:9768; } }

okay so I tried just entering the local ip address as the endpoint and the speeds were fast! Then went back to https:// and even tried to enter the duckdns ip without the https prefix and the speed was pretty much non-existent at like 100k/s. Do you guys think it's because both servers are on the same network and will speed up once I move the backup unraid offsite? Or is this some kind of other issue? I am not specifying the port with the duck dns address, since I am using letsencrypt to forward that to the minio docker. I have no issues accessing the minio docker in my web browser by going to the https:// link *EDIT* So I just installed rclone on my macbook and enabled VPN, I am still getting 200k/s speed when using the duckdns address. Could this be an issue with letsencrypt and how it processes rclone?

guys please help me out. This has been a frustrating week. I went frmo creating VMs for Arq Backup, to duplicacy, and finally decided to use rclone and I'm still hacing issues with this. I am running minio with a letsencrypt docker on a backup unraid server Whe I run rclone it's rediculously slow at like 400K/s, and then errors out. I'm not sure what's happening. Both servers are on my local network for now, but I have the location set at the https:// endpoint in rclone. These are the errors that I am getting. Wold really appreciate your help. MultipartUpload: upload multipart failed upload id: 26d9ce71-9b32-4ff1-8670-c20a00d05cae caused by: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method. status code: 403, request id: 15E44E9A5F3FEDAE, host id: 2019/12/27 13:43:14 ERROR : Encrypted drive 'secure:': not deleting files as there were IO errors 2019/12/27 13:43:14 ERROR : Encrypted drive 'secure:': not deleting directories as there were IO errors 2019/12/27 13:43:14 ERROR : Attempt 2/3 failed with 3 errors and: MultipartUpload: upload multipart failed upload id: 26d9ce71-9b32-4ff1-8670-c20a00d05cae caused by: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method. status code: 403, request id: 15E44E9A5F3FEDAE, host id: 2019/12/27 13:43:14 INFO : Encrypted drive 'secure:': Waiting for checks to finish 2019/12/27 13:43:14 INFO : Encrypted drive 'secure:': Waiting for transfers to finish 2019/12/27 13:44:12 INFO : Transferred: 240M / 4.507 GBytes, 5%, 455.269 kBytes/s, ETA 2h44m1s Errors: 0 Checks: 3 / 3, 100% Transferred: 0 / 1, 0% Elapsed time: 8m59.8s