[Plug-In] Community Applications

[Squid]

in auto update section in settings i get the following text at the top of the page.

 

Warning: file_get_contents(/usr/local/emhttp/plugins//README.md): failed to open stream: No such file or directory in /usr/local/emhttp/plugins/dynamix/include/DefaultPageLayout.php(292) : eval()'d code on line 34 Warning: file_get_contents(/usr/local/emhttp/plugins//README.md): failed to open stream: No such file or directory in /usr/local/emhttp/plugins/dynamix/include/DefaultPageLayout.php(292) : eval()'d code on line 34 Warning: file_get_contents(/usr/local/emhttp/plugins//README.md): failed to open stream: No such file or directory in /usr/local/emhttp/plugins/dynamix/include/DefaultPageLayout.php(292) : eval()'d code on line 34 Warning: file_get_contents(/usr/local/emhttp/plugins//README.md): failed to open stream: No such file or directory in /usr/local/emhttp/plugins/dynamix/include/DefaultPageLayout.php(292) : eval()'d code on line 34 Warning: file_get_contents(/usr/local/emhttp/plugins//README.md): failed to open stream: No such file or directory in /usr/local/emhttp/plugins/dynamix/include/DefaultPageLayout.php(292) : eval()'d code on line 34 Warning: file_get_contents(/usr/local/emhttp/plugins//README.md): failed to open stream: No such file or directory in /usr/local/emhttp/plugins/dynamix/include/DefaultPageLayout.php(292) : eval()'d code on line 34

 

I also have old, deleted ._plugins showing up. I have rebooted to see if it's what's stored in memory, but they are still there. I've deleted the previously installed plugins folder on my flash drive (which shows the offending plugins).

Can you show me the contents of your flash drive (config/plugins)

 

EDIT: What this module was doing was basically looking at /config/plugins/*.plg to see what was installed.

 

I've now switched it to look at /var/log/plugins instead which should only show the plugins which have been successfully installed.

[Squid]

 

Destination Share This is whatever share you want the backup to go to.  DO NOT use a share that already contains any other files, as part of the process of the backup is keeping everything in sync, which ultimately means that if the file/folder does not exist in the appdata share, CA will delete it from the destination share.  In other words, if you decide to backup the appdata to your "Movies" share, I will be the first to laugh.

Maybe a little warning that you're gonna delete all the data in the target directory with the backup option. I had a bunch of other stuff in my backup folder that's gone now. Easy enough to fix the rsync, but didn't notice it at first.

You were warned.

Right above the apply button

 

NOTE: All files within your chosen destination share will be deleted when the backup runs. Only select a share created SPECIFICALLY for appdata backup purposes

 

And it's not easy enough to fix the r sync because if the source and destinations do not have the exact same contents then it's not a backup

 

Sent from my SM-T560NU using Tapatalk

 

Ugg yeah been staring at the screen too long today.  Wouldn't a better solution be to use btrfs snapshots?

Sure.  Except that

• Nothing says you have to have your cache drive formatted as btrfs
  
• Nothing says you have to have your array formmatted as btrfs
  

[trurl]

take it - it is not possible to backup to a folder under a share?  I have another script I run that is \\Tower\Backups\Cache - but I also have my crashplan in there as well under \\Tower\Backups\Crashplan

 

Wanted to keep all backups together - nice and tidy etc....

 

Myk

Gimme an update or 2 to accomplish this.

 

You really don't know what features people want / need until you throw it out there. 

I already have a .cron running a script for this but intend to convert over to your method. I am currently putting my backup in a subfolder also so if you plan to implement this I may wait for it or else I can move my existing backup to a new share. Thinking the "initial backup" should be pretty quick if I already have an "initial backup".

[Squid]

take it - it is not possible to backup to a folder under a share?  I have another script I run that is \\Tower\Backups\Cache - but I also have my crashplan in there as well under \\Tower\Backups\Crashplan

 

Wanted to keep all backups together - nice and tidy etc....

 

Myk

Gimme an update or 2 to accomplish this.

 

You really don't know what features people want / need until you throw it out there. 

I already have a .cron running a script for this but intend to convert over to your method. I am currently putting my backup in a subfolder also so if you plan to implement this I may wait for it or else I can move my existing backup to a new share. Thinking the "initial backup" should be pretty quick if I already have an "initial backup".

Should be by tomorrow (if not today).

 

- its a lot easier to implement than the exclusions which needs many many test runs (backup and restore) which all depends upon how much TV the wife wants to watch

[Squid]

 

- Added more warnings on the fact that the destination folder is going to be overwritten

- Added ability to set backup destinations to a subfolder

- Enhanced script / share selection

- Added ability to skip docker.img on backups

- Fixed autoupdate of applications would not always display only installed plugins

 

* Personally, I do not see any need at all to backup the docker.img file if you are storing it within the appdata share.  It is very easily recreated in case of a cache drive failure, and the apps are easily reinstalled using CA's previous applications section.  In my mind, it is a complete waste of hard drive space to back it up.  But, its entirely up to you.  Either way, the option defaults to excluded.

 

Wife is now watching sitcoms, which is seriously hampering my ability to develop and test excluded folders from the source, so it probably won't be released for another day or two.

 

[jeffreywhunter]

What is the name of the App?  Can't find anything in the CA...

[Bjonness406]

What is the name of the App?  Can't find anything in the CA...

Which app? 

[jeffreywhunter]

Sorry, I'm a bit vague today...   The Cache backup docker app that Squid created...

[Bjonness406]

Sorry, I'm a bit vague today...   The Cache backup docker app that Squid created...

It is not an own docker app, it is a part of CA.

First update CA to the latest version, then from the unRAID GUI go to settings, and then "Backup Appdata" under Community Applications, then you can set up a backup for appdata.

Should look like this:

 

https://i46.photobucket.com/albums/f109/squidaz/Untitled_zpsgayo2aji.png[/img]

[Squid]

Sorry, I'm a bit vague today...   The Cache backup docker app that Squid created...

It is not an own docker app, it is a part of CA.

First update CA to the latest version, then from the unRAID GUI go to settings, and then "Backup Appdata" under Community Applications, then you can set up a backup for appdata.

Should look like this:

 

https://i46.photobucket.com/albums/f109/squidaz/Untitled_zpsgayo2aji.png[/img]

And its a simple appdata backup utility, not a general purpose cache backup per se

[trurl]

take it - it is not possible to backup to a folder under a share?  I have another script I run that is \\Tower\Backups\Cache - but I also have my crashplan in there as well under \\Tower\Backups\Crashplan

 

Wanted to keep all backups together - nice and tidy etc....

 

Myk

Gimme an update or 2 to accomplish this.

 

You really don't know what features people want / need until you throw it out there. 

I already have a .cron running a script for this but intend to convert over to your method. I am currently putting my backup in a subfolder also so if you plan to implement this I may wait for it or else I can move my existing backup to a new share. Thinking the "initial backup" should be pretty quick if I already have an "initial backup".

Should be by tomorrow (if not today).

 

- its a lot easier to implement than the exclusions which needs many many test runs (backup and restore) which all depends upon how much TV the wife wants to watch

Thanks for the subfolder feature. I wound up keeping my .cron and script because it was also backing up other cache-only shares, but I modified the script rsync to exclude appdata. I have setup the appdata backup using CA for nightly and my other script is only running weekly.

 

My "initial backup" only took a little over a minute since most of it was already there.

 

One thing I would prefer is a notification only on completion with success/fail, instead of a notification on start and another on completion. I love getting emails from my server in the middle of the night but only so I can check them the next morning to see that everything went OK. No point in finding out about the start of the backup several hours after it completed.

[Squid]

take it - it is not possible to backup to a folder under a share?  I have another script I run that is \\Tower\Backups\Cache - but I also have my crashplan in there as well under \\Tower\Backups\Crashplan

 

Wanted to keep all backups together - nice and tidy etc....

 

Myk

Gimme an update or 2 to accomplish this.

 

You really don't know what features people want / need until you throw it out there. 

I already have a .cron running a script for this but intend to convert over to your method. I am currently putting my backup in a subfolder also so if you plan to implement this I may wait for it or else I can move my existing backup to a new share. Thinking the "initial backup" should be pretty quick if I already have an "initial backup".

Should be by tomorrow (if not today).

 

- its a lot easier to implement than the exclusions which needs many many test runs (backup and restore) which all depends upon how much TV the wife wants to watch

Thanks for the subfolder feature. I wound up keeping my .cron and script because it was also backing up other cache-only shares, but I modified the script rsync to exclude appdata. I have setup the appdata backup using CA for nightly and my other script is only running weekly.

 

My "initial backup" only took a little over a minute since most of it was already there.

 

One thing I would prefer is a notification only on completion with success/fail, instead of a notification on start and another on completion. I love getting emails from my server in the middle of the night but only so I can check them the next morning to see that everything went OK. No point in finding out about the start of the backup several hours after it completed.

Got no problems with that (already ahead of you actually), but what I've put in a feature req with bonienl for a low priority notification, and on my system, I've actually set all of the normal notifications to be browser only, as with CA now capable of handling plugin updates, I don't particularly see much use in normal notifications via email much anymore.

 

The excludes are actually easy to implement via a simple text box, but I'm shooting for something a hair more friendly, and HTML is not my specialty.  But I'll get it in somehow.

[jeffreywhunter]

Sorry, I'm a bit vague today...   The Cache backup docker app that Squid created...

It is not an own docker app, it is a part of CA.

First update CA to the latest version, then from the unRAID GUI go to settings, and then "Backup Appdata" under Community Applications, then you can set up a backup for appdata.

 

Ugh, hadn't thought it would be in Settings.  Found it, installed and working, thanks.

 

BTW, it does not like path names with spaces in it...(yes, I'm from a windows world...)...but I can grep the reason...

http://my.jetscreenshot.com/12412/20160501-wjyi-27kb.jpg[/img]

[jeffreywhunter]

Sorry, I'm a bit vague today...   The Cache backup docker app that Squid created...

It is not an own docker app, it is a part of CA.

First update CA to the latest version, then from the unRAID GUI go to settings, and then "Backup Appdata" under Community Applications, then you can set up a backup for appdata.

 

Ugh, hadn't thought it would be in Settings.  Found it, installed and working, thanks.

 

BTW, it does not like path names with spaces in it...(yes, I'm from a windows world...)...but I can grep the reason...

http://my.jetscreenshot.com/12412/20160501-wjyi-27kb.jpg[/img]

 

Well...perhaps I don't grep it after all...  Why would we need to specify BOTH a destination share (we mean a real share right?) AND a Disk?  I chose the wrong disk but the right share and it got confused.  Shouldn't the share not care about which disks are involved?  Choosing both the correct share and the correct disk works (its doing the backup), but seems like you really on need the share?  Help me grep it...

 

http://my.jetscreenshot.com/12412/20160501-pyuh-89kb.jpg[/img]

[Bjonness406]

Sorry, I'm a bit vague today...   The Cache backup docker app that Squid created...

It is not an own docker app, it is a part of CA.

First update CA to the latest version, then from the unRAID GUI go to settings, and then "Backup Appdata" under Community Applications, then you can set up a backup for appdata.

 

Ugh, hadn't thought it would be in Settings.  Found it, installed and working, thanks.

 

BTW, it does not like path names with spaces in it...(yes, I'm from a windows world...)...but I can grep the reason...

http://my.jetscreenshot.com/12412/20160501-wjyi-27kb.jpg[/img]

 

Well...perhaps I don't grep it after all...  Why would we need to specify BOTH a destination share (we mean a real share right?) AND a Disk?  I chose the wrong disk but the right share and it got confused.  Shouldn't the share not care about which disks are involved?  Choosing both the correct share and the correct disk works (its doing the backup), but seems like you really on need the share?  Help me grep it...

 

http://my.jetscreenshot.com/12412/20160501-pyuh-89kb.jpg[/img]

 

 

https://lime-technology.com/forum/index.php?topic=40262.msg467501#msg467501

Save To Disk Because most docker applications use hardlinks / symlinks in their appdata, the destination share HAS to be written directly to a disk share in order for the backup / restore process to work.  Set this in conjunction with the Destination Share.  Note that if you happen to set any split levels, include/exclude disks, etc they are automatically overridden by this setting

[Squid]

 

Well...perhaps I don't grep it after all...  Why would we need to specify BOTH a destination share (we mean a real share right?) AND a Disk?  I chose the wrong disk but the right share and it got confused.  Shouldn't the share not care about which disks are involved?  Choosing both the correct share and the correct disk works (its doing the backup), but seems like you really on need the share?  Help me grep it...

As Bjonness406 stated, the reason for having to specify both a disk and a share is all because of symlinks.  They only work correctly when reading / writing directly to a disk.  (This is why some apps do not work if you specify /mnt/user/appdata/appname, but do work if you specify /mnt/cache/appdata/appname).  I've now removed the "/mnt/user/" from the displayed name.

 

Yes its confusing, but it is the only way to guarantee that a backup is successful.  (Basically, I'm using the share list to present the user with a GUI to specify a folder)

 

Your other problem with rsync returning a mkdir error is either the disk you specified as a destination is either 100% full, corrupted and set to read-only mode, etc.

 

Writing to a disk share completely overrides any and all settings that you may have set in share settings.  eg: If you have your backup share constrained to disk1 in share settings, and then set this module to write to disk 3, the backup will succeed.

 

Technically, you don't even need to create a share with the GUI.  I just typed in on my system "/mnt/user/blahblahblah" (which doesn't exist at all) and the backup is happily proceeding.

 

You do have some underlying problem, which your diagnostics should tell the story on.

 

On the positive side, after seeing your screenshots, I realized that the rsync error was only going to be displaying within the warning notification that was sent, so I've now added that error to also show up in the syslog

 

Incidentally, the reason the one screen shot shows Disk8 as the destination, but rsync tried disk5 is because you ran the backup to disk5 (and it failed) and then changed the destination drive (which disabled the backup button until you apply the changes) and then took a screen shot. 

[Squid]

When two vegetarians have a rivalry, is it still called a beef?

 

Added: configurable notifications on appdata backup (disabled, start and stop, completion only, errors only)

Added: Excluded folders to appdata backup

Updated: CA manual

Added: rsync errors now logged to syslog

Lowered memory footprint of program (and download size) by ~20%

 

Excluded folders are limited to the first level of your appdata folder.  If you require more control (exclude subfolders of subfolders), then modify the rsync options accordingly

[MyKroFt]

Love the work!

 

A view log button on the backup screen would be nice

 

Myk

 

[Squid]

Love the work!

 

A view log button on the backup screen would be nice

 

Myk

To view the log as it progresses?  Logs are available (in DOS format) on the flash drive upon completion.

[MyKroFt]

Love the work!

 

A view log button on the backup screen would be nice

 

Myk

To view the log as it progresses?  Logs are available (in DOS format) on the flash drive upon completion.

 

as in a button on the backup screen to execute:  http://Tower/logging.htm?title=Log%20Information&cmd=/webGui/scripts/tail_log&arg1=community.applications.backup.log

 

Even if /var/log/community.applications.backup.log is a symlink to the log on the flash

 

?

 

As it is now - the backup.log will keep filling up the flash drive without manual intervention.....

 

Thanks

Myk

 

[Squid]

Because of the extreme size of the logs (especially on initial runs) they are not saved from run to run (either on the flash drive - written at completion, or within the docker.img file)  although I am considering adding an option to append the current log to the existing one on the flash at completion.

 

The button is no effort to implement.  My concern is that users are going to start an initial, hit the button, watch the progress and promptly have their browser bring the desktop to a standstill as it struggles to display a page that has 200,000 lines in it.  Hence why I've got that little 2 line status display.  (And everything else that the module does is logged into syslog)

 

Beyond that, there's actually nothing to see.  It's just a running list of files.  Rsync will abort on an error and the error will appear in the syslog, the log on the status display, and the log on the flash.

 

 

Edit: thought I would confirm something here too.  The log on the flash is only updated once at the end, never during the process, as continually logging to the flash would very quickly wear it out due to the 200000 ppen, write, close processes vs a single one at the end.

Sent from my SM-T560NU using Tapatalk

Edit 2:  because of the sizes of the logs nothing is stored in /var/log as that would require most users to have to expand their memory available to it from the default 128meg.  Hence why I store the temp log in the docker.img file

[MyKroFt]

ok, no problem - will add something to my win desktop monitor to do what I need

 

Thanks

Mtyk

 

[Squid]

The path to the log is /var/lib/docker/unraid/community.applications-datastore/appdata.backup.log  (filename might be wrong can't remember right now)

 

Sent from my SM-T560NU using Tapatalk

 

 

[John_M]

This appdata backup feature is very nice. I agree that there isn't a lot of point in backing up the docker.img file but I thought I'd try it anyway! It didn't work, though - presumably because it didn't find my docker.img file. Is it expecting to find it within the appdata folder?

 

With the 6.2.0-betas there's a specific place where the docker.img file is expected to be by default, namely /mnt/cache/system/docker/docker.img (see Tom's post under the heading Automatic system shares). That's where mine is. Previously, with 6.1.x, its location was /mnt/cache/docker.img so I've never actually kept it inside the appdata folder.

[Squid]

This appdata backup feature is very nice. I agree that there isn't a lot of point in backing up the docker.img file but I thought I'd try it anyway! It didn't work, though - presumably because it didn't find my docker.img file. Is it expecting to find it within the appdata folder?

 

With the 6.2.0-betas there's a specific place where the docker.img file is expected to be by default, namely /mnt/cache/system/docker/docker.img (see Tom's post under the heading Automatic system shares). That's where mine is. Previously, with 6.1.x, its location was /mnt/cache/docker.img so I've never actually kept it inside the appdata folder.

There is no specific place the docker image has to be.  Within the system share is merely a default location.  Not a required or even suggested location.  My docker.ing has always been stored in the root of the cache.

 

That option does not select to back up the image file or not.  Rather it excludes it if it's in appdata.

 

I guess I can make that clearer within the settings / manual

 

But we both agree that it is pointless to back it up anyways

 

Sent from my LG-D852 using Tapatalk