[DOCKER] Websync

[danioj]

Has been any movement on this in the linuxserver.io group? Interesting that the need for a reliable backup solution for unRAID (which doesn't require the user to drop to the command line, or instantiate a VM etc) seems to be discussed almost yearly on the dot.

 

Come 2016, come another year of discussions. Always in a seperate thread and always with a slightly different use case but the essentials are the same:

 

I have unRAID and I want to back it up. That can be to either a drive, another unRAID box, another NAS, an internal network location, an external network location or even a set of drives in the same physical box.

 

Whatever. The core requirement is the same. This Websync seemed like it was going to offer a nice front end to rsync and control the options somewhat through inputs through a GUI.

 

When I glance at Github there doesn't appear to have been any work since October 15 which suggests to me those at linuxserver.io haven't picked up development actively yet?

 

if the application developer themselves aren't developing, there is nothing for us (linuxserver) to develop.

 

My Bad. I had completely misunderstood where the development effort for this App was coming from. I had (quite incorrectly) assumed that it was yourself along with perhaps others that were developing the front end yourself - I didn't realised you had just created a container for an exisiting project.

 

I just did a quick Google and found the Git page: https://github.com/furier/websync

 

Shame that it has been over a year since there has been any movement. I guess this is a Docker for the dustbin. Pointless having it constantly stuck in BETA if it is not being maintained or developed further.

 

Sorry again for the misunderstanding.

 

Docker creation is pretty much all packaging other applications.  Linuxserver.io actually create their own containers, as do most of the other docker authors, which gives some control over things. 

 

There are a few containers available via Community Applications that are not packaged by anyone from the Unraid community and a user has just created an XML file to "template" it for use here.  One of the fundamental problems with this approach is that the Unraid member has no control over the underlying docker container, so I try as far as possible not to use such containers.

 

I understand the concept. I was just mistaken over who had control of the Application - hence why I said "My Bad" and "Sorry for the Misunderstanding".

 

 

"I didn't realised you had just created a container for an exisiting project." 

 

"Shame that it has been over a year since there has been any movement. I guess this is a Docker for the dustbin. Pointless having it constantly stuck in BETA if it is not being maintained or developed further.

 

Sorry again for the misunderstanding."

 

I assume that when we release WebSync that you won't be interested then?  Of course if you're able to find or write a better alternative then please pop over to our IRC room and contribute.....

 

Personally, even if development has ceased, if it works and does what it sets out to do, then I don't really see a problem.

 

Rsync is actually pretty simple to use and I do so at home, and I think anything that is simple to use is less likely to be automated that things that are more complicated.

 

For example. I can (and do) use Rysnc, as do many people on here, and I'm sure any of us will be willing to help if you decided to actually try and implement this, after all backups are important as you've already stated.  I cannot however implement dual parity and or virtual machines on Unraid....

 

If backing up is that important to you then surely it would be worth investing a little time and effort in?

 

I think the fact that an application with promise is not being worked on or improved is a shame whatever it is or whomever is developing it. Moreover I feel it is a shame for this community because I feel this makes it is less likely to be integrated into unRAID IMHO or even used by the community en-mass. Integration for me would be the ideal. However, as far as I am concerned it would be a mistake to release an Application to the community that users of unRAID might become dependant on to do such a critical function like Backing up their data. Especially if it is not even being maintained let alone developed. It is for that reason I don't think I would use it - so you assume correctly.

 

As for it doing what it needs to do. It "might" do what it needs to do now BUT If every man and his dog started using a docker with this application (given a feature like this application delivers IMHO is so heavily sought, recommended) and in the future a vulnerability was found / maintenance required / enhancement needed and suddenly all those who rely on it to backup their data had to cease using it because these couldn't be done.... Well, it could get messy. Not a desirable situation IMHO.

 

As for contribution - I do not have the skills to contribute to that project - it is (I don't mind saying) way beyond me. If I could, I may. Clearly no-one here in this community has been able to write anything better than this application written by furier (whomever that may be) as I am not aware of anything that comes close to this. YET it continues to be one of the most sought after features of those who use unRAID. I would even go as far as to say that it is almost the most common advice given from Mods / experienced users to new users. BACK-UP your data!!

 

As for your use of Rsync. Good for you. I am happy for you that you have the skills to drop to the command line and use the tool and it is simple for you. The fact is, for many unRAID users - it is not easy. The fact is unRAID is marketed as a tool that does not require a user to drop to the command line to perform basic and critical tasks (such as backing up their data). I feel LT still see dropping to the command line as something they expect a Power / SOHO (or whatever term you want to use) user to do. I see more and more that effort is being made to ensure that dropping to the command line is not require with more features being migrated to the GUI - I assume for this very reason. The great work done on the preclear plugin is an example. It appears to me that dropping to the command line is actively something that LT are trying to discourage. I don't believe that they want their users on the command line at all.

 

In addition, I do not agree with your statement that simple things are less likely to be automated. Once again for the purposes of example I reference the use of the preclear script. This is a simple ~1 command activity a user can do from the command line. The fact is also that almost EVERY user of unRAID would do it too. It is even recommended. I believe it was for that reason (and the command line access reasons I discuss above) I believe it was automated.

 

As for backing up being important to me. Yes it is. And it should be (IMHO the most important consideration) to every unRAID user. unRAID's primary use is still to allow a user to have "ultimate control over your storage" and IMHO ultimately the integrity and protection of a users data. Yes we now have the great benefit of being able to get the most out of our other Hardware BUT lets not forget the reason why most of us use unRAID to being with.

 

And Yes, as I noted I have a solution in place which I have spent a great deal of time and effort implementing (with the help and patience of a few Mod's and the wider community). No, the solution does not utilise Rsync and it does not, for a reason (and that is because I don't want to have to mess with the command line). So I have a working alternative. That however doesn't stop me wanting a more integrated solution from within the unRAID / Web interface and why should it?

 

[OmgImAlexis]

Since this looks interesting I may have a look at updating it. What needs to be fixed, etc?

 

If someone wants to get this running well on Docker I can work on the app itself.

Here's my Github and the repo.

[sparklyballs]

Since this looks interesting I may have a look at updating it. What needs to be fixed, etc?

 

If someone wants to get this running well on Docker I can work on the app itself.

Here's my Github and the repo.

 

here's the latest iteration of the dockerfile from my personal repository, took it out of the linuxserver repo for now while development is currently stalled from the developer.

 

 

https://github.com/sparklyballs/docker-websync

[sparklyballs]

https://gist.github.com/sparklyballs/e11a9db2ffa8b305d88a

 

it's failing to build at the moment, but it might be because i'm using too new a version of node and the dependencies have moved on.

 

last successful build was about a week ago with the same dockerfile.

[OmgImAlexis]

https://gist.github.com/sparklyballs/e11a9db2ffa8b305d88a

 

it's failing to build at the moment, but it might be because i'm using too new a version of node and the dependencies have moved on.

 

last successful build was about a week ago with the same dockerfile.

 

That definitely looks like a old node issue. I'll see if I can get it working outside of docker and then hopefully you can use my updated version with your container. I'm usually on IRC if you need me.

[page3]

I think the fact that an application with promise is not being worked on or improved is a shame whatever it is or whomever is developing it. Moreover I feel it is a shame for this community because I feel this makes it is less likely to be integrated into unRAID IMHO or even used by the community en-mass. Integration for me would be the ideal. However, as far as I am concerned it would be a mistake to release an Application to the community that users of unRAID might become dependant on to do such a critical function like Backing up their data. Especially if it is not even being maintained let alone developed. It is for that reason I don't think I would use it - so you assume correctly.

 

As for it doing what it needs to do. It "might" do what it needs to do now BUT If every man and his dog started using a docker with this application (given a feature like this application delivers IMHO is so heavily sought, recommended) and in the future a vulnerability was found / maintenance required / enhancement needed and suddenly all those who rely on it to backup their data had to cease using it because these couldn't be done.... Well, it could get messy. Not a desirable situation IMHO.

 

As for contribution - I do not have the skills to contribute to that project - it is (I don't mind saying) way beyond me. If I could, I may. Clearly no-one here in this community has been able to write anything better than this application written by furier (whomever that may be) as I am not aware of anything that comes close to this. YET it continues to be one of the most sought after features of those who use unRAID. I would even go as far as to say that it is almost the most common advice given from Mods / experienced users to new users. BACK-UP your data!!

 

As for your use of Rsync. Good for you. I am happy for you that you have the skills to drop to the command line and use the tool and it is simple for you. The fact is, for many unRAID users - it is not easy. The fact is unRAID is marketed as a tool that does not require a user to drop to the command line to perform basic and critical tasks (such as backing up their data). I feel LT still see dropping to the command line as something they expect a Power / SOHO (or whatever term you want to use) user to do. I see more and more that effort is being made to ensure that dropping to the command line is not require with more features being migrated to the GUI - I assume for this very reason. The great work done on the preclear plugin is an example. It appears to me that dropping to the command line is actively something that LT are trying to discourage. I don't believe that they want their users on the command line at all.

 

In addition, I do not agree with your statement that simple things are less likely to be automated. Once again for the purposes of example I reference the use of the preclear script. This is a simple ~1 command activity a user can do from the command line. The fact is also that almost EVERY user of unRAID would do it too. It is even recommended. I believe it was for that reason (and the command line access reasons I discuss above) I believe it was automated.

 

As for backing up being important to me. Yes it is. And it should be (IMHO the most important consideration) to every unRAID user. unRAID's primary use is still to allow a user to have "ultimate control over your storage" and IMHO ultimately the integrity and protection of a users data. Yes we now have the great benefit of being able to get the most out of our other Hardware BUT lets not forget the reason why most of us use unRAID to being with.

 

And Yes, as I noted I have a solution in place which I have spent a great deal of time and effort implementing (with the help and patience of a few Mod's and the wider community). No, the solution does not utilise Rsync and it does not, for a reason (and that is because I don't want to have to mess with the command line). So I have a working alternative. That however doesn't stop me wanting a more integrated solution from within the unRAID / Web interface and why should it?

I know that it's (probably) slightly off-topic but I didn't want such a detailed post pass by without saying how much I agree with it.

 

I simply don't have the time (I have an autistic child to look after) or the skill to contribute to this sort of project. Yes, I'm sure you get this all the time, but there you go. I try to contribute financially and answer queries when I can.

 

I absolutely know that backup is one of the most important features of any modern system and that UnRAID is severely lacking in this area. There is an interesting talk given by Apple when they introduced their "Time Machine" backup solution that gives various statistics on how many people backup and their reasons for not doing so. It's clear that ease of use is right up there as a reason (along with security, speed, etc). Apple went to the effort of implementing something in to their core OS to tackle this, the result of which is that even my 75 year old mother does backups now.

 

Again, this clearly isn't the right thread, but I really hope someone from limetech already has this in hand, as a priority. It really is one of the areas where UnRAID distances itself from the mainstream and become the property of only those with tech know-how which is a shame for a product that deserves a broader market.

[CHBMB]

I think the fact that an application with promise is not being worked on or improved is a shame whatever it is or whomever is developing it. Moreover I feel it is a shame for this community because I feel this makes it is less likely to be integrated into unRAID IMHO or even used by the community en-mass. Integration for me would be the ideal. However, as far as I am concerned it would be a mistake to release an Application to the community that users of unRAID might become dependant on to do such a critical function like Backing up their data. Especially if it is not even being maintained let alone developed. It is for that reason I don't think I would use it - so you assume correctly.

 

As for it doing what it needs to do. It "might" do what it needs to do now BUT If every man and his dog started using a docker with this application (given a feature like this application delivers IMHO is so heavily sought, recommended) and in the future a vulnerability was found / maintenance required / enhancement needed and suddenly all those who rely on it to backup their data had to cease using it because these couldn't be done.... Well, it could get messy. Not a desirable situation IMHO.

 

As for contribution - I do not have the skills to contribute to that project - it is (I don't mind saying) way beyond me. If I could, I may. Clearly no-one here in this community has been able to write anything better than this application written by furier (whomever that may be) as I am not aware of anything that comes close to this. YET it continues to be one of the most sought after features of those who use unRAID. I would even go as far as to say that it is almost the most common advice given from Mods / experienced users to new users. BACK-UP your data!!

 

As for your use of Rsync. Good for you. I am happy for you that you have the skills to drop to the command line and use the tool and it is simple for you. The fact is, for many unRAID users - it is not easy. The fact is unRAID is marketed as a tool that does not require a user to drop to the command line to perform basic and critical tasks (such as backing up their data). I feel LT still see dropping to the command line as something they expect a Power / SOHO (or whatever term you want to use) user to do. I see more and more that effort is being made to ensure that dropping to the command line is not require with more features being migrated to the GUI - I assume for this very reason. The great work done on the preclear plugin is an example. It appears to me that dropping to the command line is actively something that LT are trying to discourage. I don't believe that they want their users on the command line at all.

 

In addition, I do not agree with your statement that simple things are less likely to be automated. Once again for the purposes of example I reference the use of the preclear script. This is a simple ~1 command activity a user can do from the command line. The fact is also that almost EVERY user of unRAID would do it too. It is even recommended. I believe it was for that reason (and the command line access reasons I discuss above) I believe it was automated.

 

As for backing up being important to me. Yes it is. And it should be (IMHO the most important consideration) to every unRAID user. unRAID's primary use is still to allow a user to have "ultimate control over your storage" and IMHO ultimately the integrity and protection of a users data. Yes we now have the great benefit of being able to get the most out of our other Hardware BUT lets not forget the reason why most of us use unRAID to being with.

 

And Yes, as I noted I have a solution in place which I have spent a great deal of time and effort implementing (with the help and patience of a few Mod's and the wider community). No, the solution does not utilise Rsync and it does not, for a reason (and that is because I don't want to have to mess with the command line). So I have a working alternative. That however doesn't stop me wanting a more integrated solution from within the unRAID / Web interface and why should it?

I know that it's (probably) slightly off-topic but I didn't want such a detailed post pass by without saying how much I agree with it.

 

I simply don't have the time (I have an autistic child to look after) or the skill to contribute to this sort of project. Yes, I'm sure you get this all the time, but there you go. I try to contribute financially and answer queries when I can.

 

I absolutely know that backup is one of the most important features of any modern system and that UnRAID is severely lacking in this area. There is an interesting talk given by Apple when they introduced their "Time Machine" backup solution that gives various statistics on how many people backup and their reasons for not doing so. It's clear that ease of use is right up there as a reason (along with security, speed, etc). Apple went to the effort of implementing something in to their core OS to tackle this, the result of which is that even my 75 year old mother does backups now.

 

Again, this clearly isn't the right thread, but I really hope someone from limetech already has this in hand, as a priority. It really is one of the areas where UnRAID distances itself from the mainstream and become the property of only those with tech know-how which is a shame for a product that deserves a broader market.

 

I think one thing we need to remember is that Unraid has come from a base of users who fell into two groups.  Those that just used it as a plain old NAS completely through the GUI, and those that tinkered with it, with plugins, scripts, cron jobs etc to get it to do things that it was never really designed to do.

 

V6 has only been out less than a year and we've seen dramatic improvements in usability and the ability to do things that were once impossible primarily docker and KVM. 

 

Most of this functionality is provided by community members remember.  KVM management was the work of dmacias originally.  The vast majority of docker containers are provided by the Unraid community (Plex & BTSync are as far as I know the only LT ones), all the plugins, written by the Unraid community, Community Applications being one mammoth project by Squid being a prime example.  The vast amount of support, provided by the community.  All unpaid and volunteering their own time, and as such development and continued support of any of these could disappear at any moment.  Doesn't mean that we don't use them though....

 

The net result I think, is that we are in a somewhat transitional period, LT is three guys remember, not Apple...

 

In the meantime we are dependant on community members to provide certain functionality.....

 

 

 

[RussellinSacto]

Just a friendly nudge...  I'm another non-command-line user of Unraid... and I've been needing a solution to sync my two Unraid systems for a couple of YEARS now...  specifically a solution that just moves changes (instead of full scans every night, which burn up all my drives).

 

I understand rsync might be one of the only solutions that can do this.  (I've tried a ton of stuff on Windows to monitor changes on Unraid and they never seem to notice changes until they scan everything.  Please advise if there is a solution for this.)

 

Thanks for all the work you've done on this so far.  :-)

 

Russell

[David Bott]

Hi...

 

Typing to use this to make a backup of my Docker AppData directory from cache.  It seems to look like it will work, but I am running into permission issues.  Example...

 

"send_files failed to open "/Source/cache/appdata/couchpotato/data/cache/python/fcdc768d3cdfdb11034d3d67843355f2": Permission denied (13)"

 

"opendir "/Source/cache/appdata/mysql/databases/performance_schema" failed: Permission denied (13)"

 

 

It was running...

 

rsync -rau --progress --rsh=ssh /Source/cache/appdata/ /Backups

 

Not sure what this runs under, but surely it does not have the permissions it needs.

 

Oh, where is the "delete" option to remove files that are no longer in the source directory?  I guess I am just trying to keep a current copy incase my cache drive fails.  Or is there an option to remove all the files before it copies?

 

Also, it would be nice to be able to have it run commands before and then after.  Like stopping and starting Docker to be safer. But maybe something else can do that right before???

 

 

Thank for the help.

 

 

[David Bott]

Any help here with the above? Thanks

[krone6]

I'm confused on a couple parameters when setting this docker up. Currently I only have one unraid server with a DVR virtual machine for a few cameras. My only goal currently is to perform an rsync for daily/monthly backups on certain folders to another hard drive so I know exactly what's on it.

 

1: What'd the /Source be? Does that just point to my data share where all data resides on the unraid machine

2: Where would the /Backups reside? The actual directory all rsync jobs complete in?

 

Thank you. I'm not an expert in rsync as you can tell.

[liquid_static]

WebGUI is broken on the latest version.

[dockerPolice]

WebGUI is broken on the latest version.

The alleged perp sparklyballs has joined linuxserver.io and all development / fixes on his original personal repositories has ceased and they can all be termed now as they work if they work.  Since webSync is no longer functional, it has earned this:

http://www.financialhelper.co.uk/wp-content/uploads/2014/03/Credit-Blacklist.jpg[/img]