BLKMGK

Yup, Amazon! I've just done a bit of hack and slash with a bulk renamer so we'll see what fails this round. IF all of the lengths check out it will still fail however since it will be unable to avoid trying to backup the changing log file. Has anyone got excludes working properly? I've got a work directory or two I'd like to skip and that log file for sure! Deleting the extra crap would sure be nice and that won't occur without a full run without I/O errors it says... Edit: 4 more and counting! Some of these I've edited 2&3 times now and have names shorter than others. The cryto must occasionally get a wild card and screw them up!

Here's an example of one file that's just failed. I'm still getting over 100 failures despite having modified every file that was claimed to fail on the last run! BlackHat/Defcon/DEF CON 9/DEF CON 9 audio/DEF CON 9 Hacking Conference Presentation By Daniel Burroughs - Applying Information Warfare Theory to Generate a Higher Level of Knowledge from Current IDS - Audio.m4b Another BlackHat/Defcon/DEF CON 19/DEF CON 19 slides/DEF CON 19 Hacking Conference Presentation By - Kenneth Geers - Strategic Cyber Security An Evaluation of Nation-State Cyber Attack Mitigation Strategies - Slides.m4v These don't show up with a find as short as 225 chars so I'm a bit frustrated and am wondering if there's something in the name of the file itself that's tripping these up except I see little in common. The error is: u0027 failed to satisfy constraint: Member must have length less than or equal to 280

Finally have my entire backup nearly done! However while the backup has been occurring I've been adding files, moving files and doing what gets done with NAS like unRAID. I've also run into issues with files having pathnames too long. For folks who may run into this the following command will find files that are too long so you can rename them. Run it from a terminal session or SSH in the /mnt/user directory and it'll return the names of files that exceed 230. find -regextype posix-extended -regex '.{230,}' Note that max allowed length is 280 but for some reason quite a few extra characters seem to get tacked on and I've had to search for lengths longer than 230. I had over 130 instances of this, no fun to clean up! I have one last issue - exclusions! I run this using UserScripts with the following CustomScript: rclone --log-file /mnt/user/work/rclone.log --max-size=40G --transfers=2 --bwlimit=8.8M sync /mnt/user/ crypt: --exclude="work/HB" --exclude="work/rclone.log" --exclude="work"--checkers=10 "work" is a shared mount under /mnt/user and I'd like to exclude it. When I'm compressing video or doing random downloads this is where that winds up - it's also where I have a running log for rclone being created. Because the rclone log is constantly being updated when rclone tries to upload it the file changes and I get an error. If rclone finds an I/O error like this it refuses to do deletions on the target. Since I've had some pretty hefty files get uploaded from my Work directory that I don't want hanging around I need that full synch to occur. So, how best to exclude "work"? I've tried "/mnt/work/" and I've tried what's above. I'm going to try "/mnt/work/" again as this seems like it ought to work and rclone has revved a few times since I last tried it. I've got it running now with just one error found so far - the darned log file! All of my name changes seem to have taken - whew! So close to having a good baseline but so far - I'd appreciate a pointer from anyone who's gotten exclusions to work. Edit: Ugh, even 230 wasn't short enough - I've still got at least 20 files with issues!

Yeah, I can see where that could be an issue I hadn't thought of. I currently transfer 9 files concurrent and have 15 threads checking files as I understand it. I do have lots of files now but once initial upload is done I think it'll be mostly large files from media and backups. It'll be 20 or 30 days before I get there but I'll try to report any bizarre behavior when I do!

Your stop script looks better than what I'm doing. UserScripts terminates the top level script but not the rest of the rclone scripts which I have to kill by hand. I expect I'll be using your's soon lol. I'm still doing my initial upload and am at 10tb but when complete I'd like to schedule it too, I like what you've done!

An observation, Amazon supposedly has a 50gig per file limit. Some of my backups were far larger so I set them to split at 49gig and rclone to stop at 50gig. I observed 3 of these files proceed to 100%, hang, and then start over again at 0%! I've now reduced the rclone max to 40gig and will try lowering the split size further to say 10gig as it takes ages at the higher numbers and failures kill throughout

There's a new version out but I've not been able to upgrade yet, anything significant? I e got a couple of 50gig files that get as far as 70% complete and then something like my router falls over and breaks the connection or I need to stop things to put in a new drive. Picked up a ton of drives on BF and slowly putting them in, at this rate these files are going to take forever lol

I track progress by knowing roughly how much I need to backup and visiting the ACD web page to see how much has been pushed up (6tb of 32). To monitor what it's up to I output to a log and then SSH in to tail the log. Using this I've noted errors with file lengths and issues when case of file names change which is pointed out but not uploaded. I've had issues killing the process occasionally but pulling up HTOP I can kill it once I identify the main process. Oh, if you have the log file in the path it's backing up it'll error

Yup, sure is. That's what I suspect may be adding characters actually. Looked at Duplicati? Seems to do the same things as rclone except for mounting and might be more user friendly overall. I say that 4TB into an rclone backup lol.

I understand the character limit is an Amazon limit, however I *think* my files aren't as long as that to being with. Is it possible that rclone changes the filename length? I'll try to track down the files when the run is done but I'm up to over 95 files now which seems a bit extreme. What's the filename limit in unRAID? EXT4 ends at 255 and Amazon is claiming 280, hence my question...

Seeing some errors from rclone, from the looks of it files with long paths trip up ACD? Limit of file path appears to be 270? I cannot tell if rclone makes longer paths when it encrypts but I think that might be the case? I'm going to wait until my entire job is done before investigating but so far I appear to have run into this about 75+ times. Anyone else seeing similar? Failed to copy: HTTP code 400: "400 Bad Request": response body: "{\"logref\":\"8051344f-b3fa-11e6-9046-fd50d35e6b20\",\"message\":\"1 validation error detected: Value \\u0027te0v5i0g8ro7ft8uqhor5072ft4ei4bi7543loomvfpc5v76s0uaeoo563ev87chlhua9f0q5fo5d53vp6p5l5u91athf57motjfm01upet75li1vhc67v93ahli3p95jfrgvevon6lhabn4jcuukb7ti78hd7qrvj5cjb00bjd6h88pad79rdmmjtltnj2tv236sttpdjkfntmdk1piubcud9qd290dn9bvmuf4mf2gkv0dmh249qon08nm2v8t9gq5ojjls5r0urmau7j5os9p2s1r0ffklonjerih6hjve1cr5ir0\\u0027 at \\u0027name\\u0027 failed to satisfy constraint: Member must have length less than or equal to 280\",\"code\":\"\"}"

So, without stopping the mount script just run the unmount script? Or stop the first then run unmount? I have gotten errors terminating one then running the other and try not to do it. It's a new feature in rclone with cautions around it so I'm not too worried about it. I've mounted the ACD from another machine t monitor things rather than risking the NAS Have hit about a TB, another 30+ to go lol.

I've been using HTOP to find the process and kill it after the first time. Stopping the server kills it as well without a full reboot. User Scripts doesn't properly kill the script, but it's not hard to find. Honestly how are they expecting you to properly stop a mount with rclone? The Fuse unmount command won't stop it if it's running will it? Must break or kill the process then use Fuse? Seems clunky if that's right but I know it's still under development. Comment on the rclone plugin. If I edit a script and apply to save it, I get taken back to the top config script (from memory). To have it moved I have to drop back down to the script I just saved and move it. It might be better to apply and not goto the top so I can just move it? Or have I missed something? P.S. If you're moving large multigig files use the --acd-upload-wait-per-gb switch. This can be used to set a wait state that kicks in after each gig of data. This is used because Amazon can take up to 30 seconds for the data to be posted. If rclone doesn't see it and this doesn't make it wait rclone will start uploading it again and spiral itself into the a circle. Need to add this to mine lol Edit: Looking at the notes for 1.34 released recently it *looks* like the wait per gb has a default already and can scale by size of file so this might not need to be set unless you see errors.

I'm not mounting to it. Actually have my mount in a bad state right now, both rclone and fusor are telling me the device or resource is busy and if I attempt to LS the mounted directory I'm informed that the transport endpoint isn't connected. I'll cycle the box and clear that when I don't have anyone else trying to access it This happened after breaking a mount command with rclone, then unmounting, then trying a mount script via User scripts. Not sure what occurred but I see an update to UserScripts that just came down fixes some PHP errors so I'll blame it on that Truly appreciate the clarification on user and user0, whew! I'd have gotten to looking it up eventually. I like the idea of adding a Read-Only Amazon share. I've gone ahead and used your code in the Extras section. I assume this will come into play upon a reboot? I won't mess with it for now but doing it R/O seems a smart way to go. When I fix the issue with my current mount I'll move to using this one - thanks! For anyone who cares here's the rclone command I'll be using to backup things for now, examples always seem to be helpful for me. My target name is crypt and the configuration will create a subdirectory on the target for me. I have a 100megabit connection, I'm reserving 6megaBYTES for upload. I'll be generating a log file that I'll try to tail in a shell. I'm limiting file size to 50Gigabytes as that's ACD max file size and my backups exceed that (stupidly) rclone --log-file /mnt/user/work/rclone.log --max-size=50G --bwlimit=6M sync /mnt/user/ crypt: Edit: Bah, the state of the mount prevents me from running. Server shutdown is going poorly too, the endpoint being in a user share likely isn't helping. May have a lengthy parity check in my future. Okay finally fusermount -u worked and sure enough the server stopped fine, whew.

The "disks" folder is normally created by the Unassigned Devices plugin. Setting the mount point to be within /mnt/disks is really only required if you need the mounts created by this plugin to be utilized by a docker container. If you don't require that, then it doesn't matter where the mount is located. Perhaps it would be a good idea for this plugin to automatically create the disks folder upon installation to eliminate any confusion. I went ahead and installed the Unassigned Devices plugin, I now have a Disks folder. However it's completely empty and I see no way to get to it from SMB etc. so I'm not sure what good it would do for me. Assigning mount points under an existing share works well for being able to see the cleartext progress of what's being loaded to ACD. One interesting issue I'm puzzled about though that I somehow hoped that the Disks share might solve is that I appear to have two main subdirectories that I need to backup: user and user0. At first glance the data in them appeared to be the same but then I began noticing that there were files in each of them that didn't exist in the other - or the sorting was whacky I've tried specifying two different targets for rclone but this appears to be a no-go. I'm not certain what to do - two different jobs run back to back maybe? Anyone else gotten around this? I could jigger mount symlinks or something but i don't want to accidently recurse something or end up moving data 2x. I've got enough data that doing just user will take a month or so, hopefully a solution presents itself or I'm stupid and missed something lol

Yup, found it! Honestly? I need to find a damn GUI to build the commandline for this sucker. I posted a few useful flags above!

What I'm trying to tell you is there's no such directory as "disks" on my system and apparently on the systems of others too. Do I need to have Docker required in order for that to appear? I'm not using it and I don't know but please be aware that not everyone has that directory and it can be confusing when others mention it. I have an ESX server that runs many of my external sorts of programs in VMs but I'll get around to Docker and KVM eventually I look forward to seeing rclone as a daemon. Still puzzling it out but have run a successful test of both it and Syncovery, ACD is rocking for me! Close to a half TB went up no problem. How long does rclone need to scan a large datastore? I'm wondering if I should do this all in one pass or not - it'll take "awhile" lol. No problems starting where it left off I wish it could tweak files ala rsync. Lots to learn but really appreciate the plugin to access the scripts more easily! P.S. Looks to be nothing native in rclone to throttle it's bandwidth? I can do this at my firewall but it's a shame it's not native as it certainly has the ability to be a hog! Edit: I was wrong, there is a Bandwidth limit. It's --bwlimit. Looks like there's also a 50Gig limit on ACD. Using --max-size=5100M should solve that. I've got a ton of system backups with 500gig files There's also a dry run flag, -n but it can still take ages to return. I am sending output to a log file with --log-file but cannot access it via Windows as it's permissions are whacked since I run as Root so I need to sort that.

Good thread! I too ran into the issues with mount points being abandoned and getting stuck, the fusor command to unmount cleared it right up. I found that the user script plugin had issues with the comments in the script for some reason, removing them and leaving the bash line cleared it. I do notice folks talking about a /mnt/disks directory. Under /mnt I too have my disks listed and two user directories. User and User0. Do others have something named disks? I've been creating amount point under one of the directories in /mnt/user/ and it seems to work fine. Still figuring out how best to run this in the background and schedule it but this seems viable to Syncovery which I'd been looking at. Have folks figured out exactly what's needed to be stored offsite to make recovery easiest? Is it simply the config file from rclone? Is anyone encrypting that file? Storing that in the ACD in a zip with password might be a good way, any reason to include rclone with it maybe? I'd hope they won't break compatibility along the line.