overbyrn

Hi all, I put together a simple docker container as a wrapper around Bulk Downloader for Reddit, the tool for downloading submissions. The container also has options to run several supporting tools; detox to clean filenames, rdfind to delete dupes or convert to symbolic links, clean messy symlnks & create relative versions. I'm currently using it as a personal docker template. I'd prefer not to support it as a fully fledged CA app. If there's a dev out there that wants to take over, please go for it You can find it here: Github | Dockerhub Regards, overbyrn

Awesome, thanks Dan. That'll be a huge help. Now, if you ever feel the desire to give UD the option to have per-device SMB and NFS share permissions, you can have my first born. Well, no... no you can't, it's a bit late for that. But you'll have my undying gratitude Big overhaul to the code there though, so do understand if you don't have the time or inclination. Thanks once again, overbyrn

I'd like to be able to set NFS to private on a per UD device basis. eg.

Hi Dan, Is it possible with UD as it stands to export a mount as NFS but with specific parameters, much like can be done with regular shares when exporting as NFS? I'm guessing not, but figured I'd ask. Admittedly it's a bit of an edge-case scenario, but would be very helpful to me at least if it was possible to tweak the NFS export parameters, rather than standard Public with read/write access. I guess also that would mean having to change NFS export from a global setting to per-mount, as I notice that /etc/exports- get's populated with all mounted devices once NFS export is set to enabled. I was going to take a look under the hood at your code and see if I could find where /etc/exports- gets written. Before using UD, I used to have a manually created exports- at /boot/config which got auto-added to /etc/export when exportfs was called. Is there something similar perhaps, whereby you store a copy of exports- or do you generate during plugin run only? I could go out an manually change /etc/exports- myself, but that's messy for all kinds of reasons. Would appreciate your thoughts. Regards, overbyrn

It's okay, you can name the original person. He won't mind I'm glad someone took up the mantle to make the plugins work for later unRAID versions. Good job!

Phaze (and others creating v6/plgman compatible plugins), don't bother creating and downloading png files. Instead generate them from inline code and save directly to your plugin directory underneath /usr/local/emhttp/plugins eg. Instead of <!-- Download application and web GUI icons --> <FILE Name="/boot/config/plugins/&name;/images/information.png"> <URL>--no-check-certificate https://raw.githubusercontent.com/PhAzE-Variance/AppSupport/master/information.png</URL> </FILE> do <FILE Name="/usr/local/emhttp/plugins/&name;/information.png" Type="base64"> <INLINE> iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAAK/INwWK6QAAABl0RVh0U29mdHdhcmUAQWRvYmUgSW1hZ2VSZWFkeXHJZTwAAAKcSURBVDjLpZPLa9RXHMU/d0ysZEwmMQqZiTaP0agoaKGJUiwIxU0hUjtUQaIuXHSVbRVc+R8ICj5WvrCldJquhVqalIbOohuZxjDVxDSP0RgzyST9zdzvvffrQkh8tBs9yy9fPhw45xhV5X1U8+Yhc3U0LcEdVxdOVq20OA0ooQjhpnfhzuDZTx6++m9edfDFlZGMtXKxI6HJnrZGGtauAWAhcgwVnnB/enkGo/25859l3wIcvpzP2EhuHNpWF9/dWs/UnKW4EOGDkqhbQyqxjsKzMgM/P1ymhlO5C4ezK4DeS/c7RdzQoa3x1PaWenJjJZwT9rQ1gSp/js1jYoZdyfX8M1/mp7uFaTR8mrt29FEMQILr62jQ1I5kA8OF59jIItVA78dJertTiBNs1ZKfLNG+MUHX1oaURtIHEAOw3p/Y197MWHEJEUGCxwfHj8MTZIcnsGKxzrIURYzPLnJgbxvG2hMrKdjItjbV11CYKeG8R7ygIdB3sBMFhkem0RAAQ3Fuka7UZtRHrasOqhYNilOwrkrwnhCU/ON5/q04vHV48ThxOCuoAbxnBQB+am65QnO8FqMxNCjBe14mpHhxBBGCWBLxD3iyWMaYMLUKsO7WYH6Stk1xCAGccmR/Ozs/bKJuXS39R/YgIjgROloSDA39Deit1SZWotsjD8pfp5ONqZ6uTfyWn+T7X0f59t5fqDhUA4ry0fYtjJcWeZQvTBu4/VqRuk9/l9Fy5cbnX+6Od26s58HjWWaflwkusKGxjm1bmhkvLXHvh1+WMbWncgPfZN+qcvex6xnUXkzvSiYP7EvTvH4toDxdqDD4+ygT+cKMMbH+3MCZ7H9uAaDnqytpVX8cDScJlRY0YIwpAjcNcuePgXP/P6Z30QuoP4J7WbYhuQAAAABJRU5ErkJggg== </INLINE> </FILE>

Yes, I get what you meant. What I was meaning is that within the 'plugin' script, the general methodology looks to be that a check method is used to perform a check of the local version against remote version, the update method is used to download the later version to tmp and then subsequently calls the install method to perform the install (again). So if you build your install method to ensure it tears down the areas you need refreshing, then calling the update method will perform an installation, creating new versions of the files where required. You can also call multiple methods, allowing the same bash script to be used across more than one method; <!-- The 'install' script. --> <FILE Name="/tmp/plugins/tmp" Run="/bin/bash" Method="install update" Type="text"> <INLINE> if [ ! -d /tmp/plugins/builtin/&name; ]; then mkdir -p /tmp/plugins/builtin mv /usr/local/emhttp/plugins/&name; /tmp/plugins/builtin else rm -r /usr/local/emhttp/plugins/&name; fi tar -xf /boot/config/plugins/&name;/&name;-&version;.tar.gz -C /usr/local/emhttp/plugins mv /usr/local/emhttp/plugins/&name;-&version; /usr/local/emhttp/plugins/&name; </INLINE> </FILE>

I have figured out how to do this, but I wonder if an 'update' method would be appropriate. I used a combination of check, update and install methods from /usr/local/emhttp/plugins/plgMan/plugin.

Tom, thanks for the explanation. I've previously used info gleaned from the code in plgman to help understand the new plugin structure (install/remove/check, anon scripts etc). I appreciate you're working on plugin documentation and I imagine it's low priority, but I'd be interested in anything you can share as I'm currently going through all my plugins (dropbox, ssh, btsync, beets, airvideo, lms, denyhosts, nzbget) and making them v6 compliant. It'd be great to get them right first time. Regards, overbyrn

FYI, back in May I re-wrote my ssh plugin to take advantage of the v6 plugin structure, including the install, remove and check methods. Copy attached here in case it's helpful. Also can be installed by entering https://raw.githubusercontent.com/overbyrn/unraid-v6-plugins/master/ssh.plg into the Install Extension section of the v6 Extentions WebGui. ssh.plg

Just chipping in on an old message... Re: OpenSSH installing openssl-solibs. I can assure you openssh_overbyrn.plg does NOT install solibs or any SSL package. Just OpenSSH. Can't speak for alternative methods (unmenu?) of installing SSH.

Will drop you a PM as I don't want to pollute this thread any further.

This has been fixed. http://lime-technology.com/forum/index.php?topic=18310.msg256480#msg256480 Get the latest version from my github page.

If you have time mrow, a write up of the steps involved including links to the dev kit, command line args for vmware-mount.exe etc, would probably benefit a ton of people. Maybe BetaQuasi could include it on the first post or link it to a wiki location? I guess the truth is, if you're running this type of config then the implication is you'd know how to do it, but well you know...

theone, thanks for the plugin. Been meaning for a long time to tinker with running virtualbox on unraid and this might make the process a lot more straightforward. I've just installed the plugin onto a dev box (no other plugins installed at this time). Noticed upon first entry into virtualbox setting page, there is an error regarding vms_session.cfg not found. I've only briefly skimmed your code, but I'm assuming that's because there are no running vms as yet? I think there may be a mismatch between where this file gets created vs where the php var looks for it. From the php and rc scripts: rc.virtualbox: VBOX_PLUGIN_PATH="/boot/config/plugins/virtualbox" vboxmanage list runningvms > ${VBOX_PLUGIN_PATH}/vms_session.cfg virtualbox.php $vm_session_cnt = count(file("/boot/config/plugins/vms_session.cfg", FILE_IGNORE_NEW_LINES)); May have misread the code, so apologies if this is by design but I think otherwise it may be looking in the wrong place. That being said, on first load of the gui page there isn't a vms_session.cfg in either location likely due to the reason mentioned above. Regards, overbyrn.

How can I check the validity of a downloaded file to check after download? If it is possible I will add it to the code. Calc an MD5 sum. Use /usr/bin/md5sum to compare against a known good hash.

Looks good. Welcome to plugin writing. Not installed yet, just skipped through the code. One very minor point; device_status.png, new_config.png & information.png are never downloaded. If this were the only plugin on a system, they'd not exist in /boot/config/plugins/images. Regards, overbyrn

Glad to hear its working for you now. I don't think it's likely to make a one-size-fits-all rsync script, so please feel free to modify mine for how you need things. I really only ever made this thread to semi-document my own rsync experience in the hope it might help others. As for swapping include with exclude, don't forget you can also include and exclude from the file you specifiy either in the the include-from= or exclude-from= parameters. Precede each line with plus or minus and this'll tell rsync what you want included/excluded. For instance, in my include from file, I don't want a subdir of sabnzbd so my "rsync_include_list" looks like this; Services/ Services/airvideo/*** Services/couchpotatoserver/*** Services/denyhosts/*** Services/dropbox/*** Services/iTunes/*** Services/lms/*** Services/MediaMonkey/*** Services/playlists/*** - Services/sabnzbd/Downloads/incomplete Services/sabnzbd/*** Services/sickbeard/*** Apps/*** Dropbox/*** Music/*** Music\ Videos/*** Music2/*** Pictures/*** Stuff/*** The lines without + or - could just as easily have + at the start, but it's kind of implied in that it's the include-from parameter which is reading this file and not exclude-from. Regards, overbyrn

dalben, I've rarely used rsync across remote systems so I'm a bit rusty on what to do. How does your example above connect to the remote rsyncd process? I seem to recall when I last set it up, it was between two Unraid systems, where one of these was running the rsync daemon. Mostly setup like WeeboTech outlines in post 3 of this thread: http://lime-technology.com/forum/index.php?topic=3159 But the syntax of that rsync command looks nothing like yours so I'm assuming perhaps you're doing it over ssh? Happy to help if you can bring me up to speed on how your setup is configured. For what it's worth I did some thinking and realised my script is going to fail at the first point where it tests for target directory presence. No good when the location is remote! But that got me thinking in fact I don't need the test in place at all anymore. Not since I rejigged the script as regardless of the reason it fails, a log will be produced and errors reported if configured. I setup an rsync daemon on a test server and using the script have been able to remotely backup to this location. Again, this is basically configured the way WeeboTech outlined. So for the moment in my script for target I have target=rsync://tower/mnt/disk1/testbackup Here's the copy of the script I've used : https://dl.dropbox.com/u/572553/UnRAID/v3.sh Regards, overbyrn

trurl, I've had another hack at the script and made some more changes (for the better I hope); Temp rsync log and final log file overhaul. It wasn't great was it I wanted a way to run the script multiple times during the same day and not have a ton of logs to show for it. So what I've done is have rsync create its log to /tmp using a combination of the script name + script PID. That gives me a unique log per script invocation. It then creates the final log as a compressed file (.tar.gz) to the logdir and logname of your choosing. This is pretty much the same as before, just that I've hardcoded the fact it's going to be a .tar.gz file. (makes the next step easier) For each subsequent time the script is run, it'll add that rsync log to the final compressed log archive file. The assumption is that the log name will always be a composite of static text + date. So with luck you should only end up with one .tar.gz log file in any given day, which contains one or more rsync logs. I've tested it here and it seems to work ok. So far every time I run the script, the temp rsync logs that are created in /tmp are being deleted once the script completes. I'm really hoping I've squashed that issue where files in /tmp were getting left behind. Sounds like you've got some real funnies where unicode files are concerned. Not sure I can say much more on that side of things I'm afraid. Previous dropbox link contains latest script amendments. Regards, overbyrn

rsync exit codes: http://bluebones.net/2007/06/rsync-exit-codes/ 23 Partial transfer due to error Sounds right given the problem you're seeing with unicode utf-8 characters. This seems less about rsync and more about how unicode characters are being handled in general. I did a couple of tests: 1. created test file "12 ~ Bourée.mp3" via Windows PC on unaid array disk. set putty to utf-8. from command line, listed the file: 2. rsync above file to external ntfs-3g mounted drive using same putty command line: What version of unRAID are you using? What is your unRAID locale set to? (issue "locale" from command line) What is the method/command you're using to mount your Offsite disk to /mnt/user/Backups/Offsite? Update: trurl, grab a fresh copy of the script on the dropbox location. I noticed I had an error where dryrun would always run even if set to blank. I've also specifically set "LANG=en_US.utf8" right before the rsync takes place. Perhaps that'll help.

trurl, Thanks for giving it a try and letting me know how you got on. Not immediately sure why rsync exited with a non-zero exit code. I must admit I was fairly lazy and simply made the assumption if rsync didn't exit with return code 0, then "something" went wrong. I don't really care what the something is other than to get an email saying it failed so I know to follow it up. We need to know what the return code was before being able to troubleshoot it further. See below... Currently, the script only creates a log if rsync completes successfully (eg. return code = 0). That's something to consider as I suppose there's value in having the rsync log even if it fails. I've made some modifcations to the script. Try the copy at this link: http://dl.dropbox.com/u/572553/UnRAID/rsync_backup_to_external.sh I've simplified the temp log creation as I was trying to be too clever for my own good and it was leaving files behind in /tmp. I've added three config options: 1) verbose - when set to true, it'll log to syslog when run as cron or alternatively you get the same log messageson the screen if running from command line. Changed the error logging to include the rsync return code so we'll see what is happening there. 2) dryrun - when set to true, it'll do everything but perform file transfers. Useful for testing. 3) email - turns on / off the emailing if there's an error IMPORTANT CHANGE: I added --delete rsync parameter. I had wrongly left this out as I used to have it enabled my end. Be aware, this means any files at the target location no longer at the source location will be DELETED. You may want to review this in case it doesn't suit your needs. Subtle change to the tar command used to create the final log file. Means the file extension (.tgz, bz2) on the logname will dictate what compression is used when creating the tar. In the script on the link above, dryrun is enabled, as is verbose logging & email. You just need to re-enter the config data for your setup. I suggest you run the script from command line as that'll help understand what's happening. Give it a go and let me know how you get on. Regards, overbyrn

trurl, I was interested to see if I could replicate your issue but it seems to be working ok for me. As a test I created a subdir on my cache disk [/mnt/cache/test]. From a windows PC, I browsed to this directory, right clicked, created a new file using ALT+146 on the keypad to create character "Æ". So now I have a file called "Æ.txt" From command line I performed an rsync from this location to my target which is an NTFS formatted USB external disk, mounted via SNAP. Checked the target and it definitely copied the file across. The log file says the same: For what it's worth, I'm using ntfs-3g-2010.3.6-i486-1.txz which I think originally came from Unmenu.

Update (10/02/13): just a note to say email functionality is dependent on your system already having this configured, such as installing simplefeatures email notify plugin. Update (14/12/12): thanks to trurl and dalben for their assistance in testing the script. Thanks to them the script is probably in a decent enough form to be used as-is by others. If someone does use this script and find issues or can help add to it, please post your thoughts. Latest version of the script attached to this post. I've been using rsync for some time to perform localised backups and although it's been working well I thought I'd have a stab at making it more robust as recently I discovered my external USB drive (where I backup to) had become detached and since I didn't have any error logging it went unnoticed for a few days. I set out to create a script with a small amount of error checking so that if the target location doesn't exist, it'll log this to the syslog and send me an email of the failure. Likewise it'll do the same if rsync comes back with a non-successful return code. Before I could proceed, I needed to consolidate an earlier shell script which contained a large number of separate rsync jobs into one. So I needed to figure out the include / exclude rsync parameters - which by the way I now hate with a passion. Once I'd figured this out, I was able to create a file which contained all the directories I wanted backed up and allowed me to work on a more generic backup script. My unRAID config is fairly simple... I have a bunch of top level user shares (Pictures, Dropbox, Music, Apps etc) that I want to backup and in addition to this I have a "Services" user share which contains a number of sub-directories where my apps are such as Sab, Sickbeard, Couchpotato. So everything I need can be referenced from /mnt/user I started off by creating a file of directories that I want backed up under the /mnt/user directory; rsync_include_list: Services/ Services/airvideo/*** Services/couchpotatoserver/*** Services/denyhosts/*** Services/dropbox/*** Services/iTunes/*** Services/lms/*** Services/MediaMonkey/*** Services/playlists/*** - Services/sabnzbd/Downloads/incomplete Services/sabnzbd/*** Services/sickbeard/*** Apps/*** Dropbox/*** Music/*** Music\ Videos/*** Music2/*** Pictures/*** Stuff/*** NB: you can create comments in the include file when starting a line with #. Helpful if you want to test with only a subset of your total directories or to put in some reminders to self. I'm not going to explain the format as it pretty much speaks for itself. The "***" tells rsync to backup the directory, any files it finds in that directory as well as any sub-directories & files. Basically, everything. Two things to point out: 1. You need to specify parent dirs if you only intend rsync to backup a sub-set of directories. So for example, in my include list, directories like Apps, Pictures are easy as I want everything. But for Services, I have a number of directories I don't want, so for that area I needed to specify just the ones I needed. But had I not explicitly included the parent dir (the line "Services/") then rsync wouldn't touch any of the Services sub-directories. 2. On the subject of directories you may not want. rsync lets you precede each directory with "+" or "-". So even though this is an include list, you can also exclude directories. In the above example, I specifically tell rsync not to backup the incomplete directory under my sabnzbd directory. The other piece of the puzzle lies in the rsync command within the script below. There's an "--exclude=*" parameter which excludes everything else as otherwise rsync will happily process every sub-directory in /mnt/user, regardless of whether it's in your include list or not. I've tried to keep the script reasonably generic so that it could be reusable without too many changes other than changing the configurable options. Regards, overbyrn rsync_backup_to_external.sh.txt