[DOCKER] Websync

[sparklyballs]

This is a really nice docker.

 

So far, by far and away the best simple backup solution to backup my UnRAID server to an external HDD.

 

A few suggestions:

 

1. notifications, if scheduled. An email perhaps?

2. 'marcos' - instead of having to set up rsync options one-by-one, could we have some 'tags' that add common used functions (e.g.: backup new files only)

3. persistence! Probably the most import one. If i stop/start the docker all my configurations are lost. *** this seems to be working now ***

4. Easy way to view previous scheduled runs

 

I really hope you continue with this. UnRAID is crying out for a simple, light, easy to use backup solution.

 

i'm not the app author, i only put it in a docker, lol.

 

check out their github repo though.

 

https://github.com/furier/websync/

[page3]

i'm not the app author, i only put it in a docker, lol.

Oops. I did wonder how you had time to sleep 

 

Actually, you may be able to help with point 3? Sometimes when stopping /starting the docker it wipes all my config, sometimes it seems to keep them. Other dockers have a mapping to /config which stores stuff outside the container. Are you able to do similar for this?

[Monkeyair]

I'd like to try this docker to backup my important user shares to a HDD outside of the array (The HDD is mounted using the Unassigned Devices Plugin).  I've not had much joy so far...

 

Could somebody give me a steer with regards to setup and config please?

[sixarm]

Anyone getting this issue when trying to add a host?

http://puu.sh/iCRyQ/d96ebf8929.png

 

I'm seeing the same issue on a brand new unraid setup.  I can enter a bash shell in the container and setup my ssh keyswaps just fine.  Via the UI, no luck. 

[hernandito]

Hi Sparkly.... this worked like a charm at one time for me. I used it to backup my appdata folder from my cache drive into the array. It was one of those set it and forget it situations. I have noticed that after a reboot of the server, or a re-install of the software, none of the settings come back.

 

Is there a way for the configuration to be saved into the appdata folder? There is no folder mapped to the config in the Docker.

 

Many thanks!

 

H.

[krone6]

rsync?? haha you're playing with fire my friend. No matter how big that red disclaimer is, people will still come at you with pitchforks

 

I'm not touching that with a ten foot pole 

 

Isn't rsync the better way to transfer data on Linux? I'm still working my way through my certificates and didn't get to the more specific topics yet, mainly the basics for now.

[tr0910]

Rsync is the best way. But it's a very powerful monster.  Once tamed, it's awesome.  Problem is, taming rsync is a long process.

 

Websync is a gui front end to the command line rsync. Not everyone agrees that websync is smart enough to tame rsync.

 

[hernandito]

Hi Sparkly.... this worked like a charm at one time for me. I used it to backup my appdata folder from my cache drive into the array. It was one of those set it and forget it situations. I have noticed that after a reboot of the server, or a re-install of the software, none of the settings come back.

 

Is there a way for the configuration to be saved into the appdata folder? There is no folder mapped to the config in the Docker.

 

Many thanks!

 

H.

 

Hi Sparkly, not sure if you saw my post above....

 

Thanks,

 

H.

[sparklyballs]

Hi Sparkly.... this worked like a charm at one time for me. I used it to backup my appdata folder from my cache drive into the array. It was one of those set it and forget it situations. I have noticed that after a reboot of the server, or a re-install of the software, none of the settings come back.

 

Is there a way for the configuration to be saved into the appdata folder? There is no folder mapped to the config in the Docker.

 

Many thanks!

 

H.

 

Hi Sparkly, not sure if you saw my post above....

 

Thanks,

 

H.

 

not at the present time..... it won't rebuild in a docker for some reason and at the beta stage i got it to first time around there was no mappings.

 

[foo_fighter]

There's no delete option.....too dangerous?

[randall526]

Looks like this is setup on other linux systems to use ssh-copy-id, and ssh-agent on it's own and generate it's own ssh authentication keys.

 

I have no idea if it's going to regenerate the keys or start ssh-agent every reboot on it's own.  Sounds to me a fair amount of work in your go file might be needed to set this up

 

The  home directory is not persistent on unraid I believe and I keep getting no ssh identities found and thus none of my rsync jobs can start.

 

Anyone work around this yet?

[matthewjt]

I'm getting the following error when trying to create a host: /usr/bin/ssh-copy-id: ERROR: No identities found

 

I am trying to SSH using a user which is not root.  I did generate the ssh identities for both root and the other user and setup some copy/permissions in the go script to persist these identities across reboot.

 

I am also trying to SSH to another unRAID box with the same configuration.

 

The .ssh/id_rsa.pub permissions have +r for all.

 

I also have the RSA key authentication working properly between both boxes.

 

Any ideas?

 

---

How I setup the SSH identities to persist through reboot

 

Log in as root

Create your SSH identity using the command ssh-keygen  ( Your files will generate into /root/.ssh )

If you want to do RSA or PublicKey auth, use ssh-copy-id to/from the server you want to auth to.  I have two unraid boxes, so I used this command to send the key to the client machine.  You'll get both the authorized_keys and known_hosts files created in your .ssh directory.

Create the folders in /boot/config: /boot/config/user_home

Create the folders in /boot/config/user_home: /boot/config/user_home/root

Copy the files in /root/.ssh/ to /boot/config/user_home/root/.ssh

Add the following to the bottom of the /boot/config/go script:

    cp -r /boot/config/user_home/root/.ssh /root/

    chmod 500 /root/.ssh/*

    chmod 700 /root/.ssh

 

For non-root users we can copy saved user home directories in their entirely:

    cp -r /boot/config/user_home/<user> /home/

    chown -R <user> /home/<user>

    chmod 500 /home/<user>/.ssh/*

    chmod 700 /home/<user>/.ssh

 

Also, in the /boot/config/ssh/sshd_config file, enable the following two lines:

    RSAAuthentication yes

    PubkeyAuthentication yes

    (I don't think you need RSAauthentication enabled, but i enabled both)

 

If you have issues, just add the -v flag when you're making an ssh connection ( ssh -v <user>@<host> ).  If you're using keys to authenticate and it still prompts you for a password, log in using the password and see what message the server returns to you.  I had an issue with the .ssh directory not being set to 700 causing the password prompt.  If you setup a password in your key, you'll still be asked for that when logging in.

[randall526]

The cp commands in your go file is pretty much what I expected to get this to be persistent across reboots.  The SSH identities have to be loaded in to ssh-agent first generally when setting up passwordless authentication.  The ssh-agent will keep the cert decrypted after providing the password to the agent one time and a location to the encrypted private key.  Some additional go file work is likely needed to both start the ssh-agent and load the private key "identity" on start up.  I also believe you need to set environmental variables to tell other apps were the identity is located when starting the ssh-agent if I recall.

 

I haven't gotten this sorted for myself yet or I would provide the exact setup instructions.  I was looking for this myself.

 

 

 

 

[randall526]

I've setup a fair amount of ssh certificate authentications on systems before.  For some reason this is still prompting me for a password even when testing it outside of websync

 

 

root@unRaidPrime:~/.ssh# ssh-agent $SHELL

root@unRaidPrime:~/.ssh# ssh-add -l

The agent has no identities.

root@unRaidPrime:~/.ssh# pwd

/root/.ssh

root@unRaidPrime:~/.ssh# ls -l

total 12

-rw-r--r-- 1 root root 753 Oct  5 17:36 known_hosts

-rwxrwxrwx 1 root root 975 Oct  7 19:32 ssh_host_key

-rwxrwxrwx 1 root root 640 Oct  7 19:32 ssh_host_key.pub

root@unRaidPrime:~/.ssh# ssh-copy-id -i ./ssh_host_key.pub 192.168.2.85

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

[email protected]'s password:

 

Number of key(s) added: 1

 

Now try logging into the machine, with:  "ssh '192.168.2.85'"

and check to make sure that only the key(s) you wanted were added.

 

root@unRaidPrime:~/.ssh# ssh-add ./ssh_host_key

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@        WARNING: UNPROTECTED PRIVATE KEY FILE!          @

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Permissions 0740 for './ssh_host_key' are too open.

It is recommended that your private key files are NOT accessible by others.

This private key will be ignored.

root@unRaidPrime:~/.ssh# chmod 700 ssh_host_key

root@unRaidPrime:~/.ssh# ssh-add ./ssh_host_key

Identity added: ./ssh_host_key (root@Tower)

 

root@unRaidPrime:~/.ssh# ssh 192.168.2.85

[email protected]'s password:              <=============  shouldn't prompt me for a password

 

 

 

 

root@unRAIDbackup:~# pwd

/root

root@unRAIDbackup:~# ls -al

total 84

drwx--x---  3 root root  140 Oct  7 17:36 ./

drwxrwxrwx 18 root root  380 Apr 11 17:54 ../

-rw-------  1 root root    33 Oct  6 19:50 .bash_history

-rwxr-xr-x  1 root root  156 Sep  8 22:21 .bash_profile*

drwx------  2 root root    60 Oct  7 17:36 .ssh/

-rw-rw-rw-  1 root root 76959 Oct  7 04:18 dead.letter

lrwxrwxrwx  1 root root    21 Oct  5 15:57 mdcmd -> /usr/local/sbin/mdcmd*

root@unRAIDbackup:~# cd .ssh

root@unRAIDbackup:~/.ssh# ls

authorized_keys

root@unRAIDbackup:~/.ssh# cat authorized_keys    <====== pub key is obviously there?

2048 65537 27206387622761444171596716438419633343156121986918147844279913215751926557996788199575949359364456499983862429909518408365284250359811454524514937562114748878985106247243783015054505688327197706091609494528379410635864253925304068083203325961724109868689646689862223420787173726451127329577469924166221867192223071134872829221232323155984582003337019859275675628498760929417918693406208243596609109159438061248913960880865799005303525284731545514245534996760642341001594643317043848276185418647401771190405114454662113702683723512559998108589604347373429261047336344905192016576820489549119930742058433779016838633493

 

 

 

 

[randall526]

After reviewing my post I noticed the key I cated out did not look right so I used another pair of keys using the exact same commands and boom the command line worked for no password prompts.

 

Still getting this in websync though....

 

Host key verification failed.

Connection unexpectedly closed (0 bytes received so far) [sender]

[sp0t]

Firstly! Good job on the docker, a few tweaks and we're there. There may even be probably a better way of doing this, but my thoughts below.

 

To persist the configuration, follow this screenshot, adjusting your local file as required (Since the config is in the root folder of the app, we must create 'wsdata.json' by hand locally, leaving it empty)

 

See attached screenshot.

 

Still trying to wrangle the SSH issue, I've tried to passthrough the SSH as seen in the screenshot above, but this brings with it a whole host of SSH permission issues which can't be worked around without going in and adjusting manually (Which aren't persisted on docker run). Ultimately, these are my solutions :

 

- Manually go in after boot and adjust (ew)

- On run in services have a predefined mount point with ssh files (id_rsa / id_rsa.pub) copied directly to /root/.ssh and have the permissions changed on the COPIED file, so it is no longer directly tied to the docker host. (My favourite so far)

- On docker start, generate a new set of SSH keys (bloated authorised keys anyone?)

 

This would require sparklyballs to update his docker file.

 

Secondary Failures :

 

I haven't managed to get the current installation of the web sync's source files to successfully connect over SSH even with my 'wrangling'. Opting to git checkout v1.0.4 and running the subsequent npm / bower / gulp commands

[sparklyballs]

i'm no longer developing this in my own repo, it's currently being worked on as part of my membership of the linuxserver.io team.

 

https://github.com/linuxserver/docker-websync

[danioj]

Has been any movement on this in the linuxserver.io group? Interesting that the need for a reliable backup solution for unRAID (which doesn't require the user to drop to the command line, or instantiate a VM etc) seems to be discussed almost yearly on the dot.

 

Come 2016, come another year of discussions. Always in a seperate thread and always with a slightly different use case but the essentials are the same:

 

I have unRAID and I want to back it up. That can be to either a drive, another unRAID box, another NAS, an internal network location, an external network location or even a set of drives in the same physical box.

 

Whatever. The core requirement is the same. This Websync seemed like it was going to offer a nice front end to rsync and control the options somewhat through inputs through a GUI.

 

When I glance at Github there doesn't appear to have been any work since October 15 which suggests to me those at linuxserver.io haven't picked up development actively yet?

[page3]

Has been any movement on this in the linuxserver.io group? Interesting that the need for a reliable backup solution for unRAID (which doesn't require the user to drop to the command line, or instantiate a VM etc) seems to be discussed almost yearly on the dot.

 

Come 2016, come another year of discussions. Always in a seperate thread and always with a slightly different use case but the essentials are the same:

 

I have unRAID and I want to back it up. That can be to either a drive, another unRAID box, another NAS, an internal network location, an external network location or even a set of drives in the same physical box.

Amen.

 

My opinion is that this is the single most important failing of UnRAID and the top reason it is difficult to recommend to others.

 

We keep seeing possible new backup solutions, none of which are ever complete and all of which are command line and have fairly high learning curves.

 

This really needs collaboration at the highest UnRAID level to provide this functionality, but I just don't see any urgency or drive to do so.

 

[sparklyballs]

Has been any movement on this in the linuxserver.io group? Interesting that the need for a reliable backup solution for unRAID (which doesn't require the user to drop to the command line, or instantiate a VM etc) seems to be discussed almost yearly on the dot.

 

Come 2016, come another year of discussions. Always in a seperate thread and always with a slightly different use case but the essentials are the same:

 

I have unRAID and I want to back it up. That can be to either a drive, another unRAID box, another NAS, an internal network location, an external network location or even a set of drives in the same physical box.

 

Whatever. The core requirement is the same. This Websync seemed like it was going to offer a nice front end to rsync and control the options somewhat through inputs through a GUI.

 

When I glance at Github there doesn't appear to have been any work since October 15 which suggests to me those at linuxserver.io haven't picked up development actively yet?

 

if the application developer themselves aren't developing, there is nothing for us (linuxserver) to develop.

[ironicbadger]

Has been any movement on this in the linuxserver.io group? Interesting that the need for a reliable backup solution for unRAID (which doesn't require the user to drop to the command line, or instantiate a VM etc) seems to be discussed almost yearly on the dot.

 

Come 2016, come another year of discussions. Always in a seperate thread and always with a slightly different use case but the essentials are the same:

 

I have unRAID and I want to back it up. That can be to either a drive, another unRAID box, another NAS, an internal network location, an external network location or even a set of drives in the same physical box.

Amen.

 

My opinion is that this is the single most important failing of UnRAID and the top reason it is difficult to recommend to others.

 

We keep seeing possible new backup solutions, none of which are ever complete and all of which are command line and have fairly high learning curves.

 

This really needs collaboration at the highest UnRAID level to provide this functionality, but I just don't see any urgency or drive to do so.

 

 

There are many roll your own backup solutions. You could always contribute to open source yourselves and package these backup methods in an easy to consume method (such as websync). You're always welcome to contribute, however little you know we'll help if you're willing.

 

Some examples of Linux tools almost as old as I am in the mean time...

 

http://linux.die.net/man/1/rsync

http://rsnapshot.org/

[ironicbadger]

Problem is, taming rsync is a long process.

 

Is it?

 

rsync -avzP -e ssh localdir/ user@host:backup_path/dir/

 

Backup via ssh to a remote box.

[danioj]

Has been any movement on this in the linuxserver.io group? Interesting that the need for a reliable backup solution for unRAID (which doesn't require the user to drop to the command line, or instantiate a VM etc) seems to be discussed almost yearly on the dot.

 

Come 2016, come another year of discussions. Always in a seperate thread and always with a slightly different use case but the essentials are the same:

 

I have unRAID and I want to back it up. That can be to either a drive, another unRAID box, another NAS, an internal network location, an external network location or even a set of drives in the same physical box.

 

Whatever. The core requirement is the same. This Websync seemed like it was going to offer a nice front end to rsync and control the options somewhat through inputs through a GUI.

 

When I glance at Github there doesn't appear to have been any work since October 15 which suggests to me those at linuxserver.io haven't picked up development actively yet?

 

if the application developer themselves aren't developing, there is nothing for us (linuxserver) to develop.

 

My Bad. I had completely misunderstood where the development effort for this App was coming from. I had (quite incorrectly) assumed that it was yourself along with perhaps others that were developing the front end yourself - I didn't realised you had just created a container for an exisiting project.

 

I just did a quick Google and found the Git page: https://github.com/furier/websync

 

Shame that it has been over a year since there has been any movement. I guess this is a Docker for the dustbin. Pointless having it constantly stuck in BETA if it is not being maintained or developed further.

 

Sorry again for the misunderstanding.

[danioj]

Has been any movement on this in the linuxserver.io group? Interesting that the need for a reliable backup solution for unRAID (which doesn't require the user to drop to the command line, or instantiate a VM etc) seems to be discussed almost yearly on the dot.

 

Come 2016, come another year of discussions. Always in a seperate thread and always with a slightly different use case but the essentials are the same:

 

I have unRAID and I want to back it up. That can be to either a drive, another unRAID box, another NAS, an internal network location, an external network location or even a set of drives in the same physical box.

Amen.

 

My opinion is that this is the single most important failing of UnRAID and the top reason it is difficult to recommend to others.

 

We keep seeing possible new backup solutions, none of which are ever complete and all of which are command line and have fairly high learning curves.

 

This really needs collaboration at the highest UnRAID level to provide this functionality, but I just don't see any urgency or drive to do so.

 

 

There are many roll your own backup solutions. You could always contribute to open source yourselves and package these backup methods in an easy to consume method (such as websync). You're always welcome to contribute, however little you know we'll help if you're willing.

 

Some examples of Linux tools almost as old as I am in the mean time...

 

http://linux.die.net/man/1/rsync

http://rsnapshot.org/

 

I think this is a little beyond me.

 

I agree I understand that there are tools out there that meet the Backup needs of most. I myself use a Windows program called Syncback via a Windows 10 VM and it meets the majority of my Backup needs.

 

What I am hoping for however is a plugin / docker / integrated feature that will allow a user to access the unRAID GUI, specify some locations some other parameters and execute / schedule an unattended backup WITHOUT having to drop to the command line.

 

What is quite amazing to me thought is when a question like this is asked it is often responded with a simple, the tool exists, just use Rsync, its quite easy ....

 

Fact is, it clearly can't be that simple. If it was the feature would exist. Being able to back up your array to a specific location (locally or remotely) from the unRAID GUI is the most basic of requirements of a nas owner even before the likes of Dual Parity, Docker, VM's etc etc IMHO.

 

Anyway, back on topic - clearly whomever was developing this software has given up the game. Shame.

[CHBMB]

Has been any movement on this in the linuxserver.io group? Interesting that the need for a reliable backup solution for unRAID (which doesn't require the user to drop to the command line, or instantiate a VM etc) seems to be discussed almost yearly on the dot.

 

Come 2016, come another year of discussions. Always in a seperate thread and always with a slightly different use case but the essentials are the same:

 

I have unRAID and I want to back it up. That can be to either a drive, another unRAID box, another NAS, an internal network location, an external network location or even a set of drives in the same physical box.

 

Whatever. The core requirement is the same. This Websync seemed like it was going to offer a nice front end to rsync and control the options somewhat through inputs through a GUI.

 

When I glance at Github there doesn't appear to have been any work since October 15 which suggests to me those at linuxserver.io haven't picked up development actively yet?

 

if the application developer themselves aren't developing, there is nothing for us (linuxserver) to develop.

 

My Bad. I had completely misunderstood where the development effort for this App was coming from. I had (quite incorrectly) assumed that it was yourself along with perhaps others that were developing the front end yourself - I didn't realised you had just created a container for an exisiting project.

 

I just did a quick Google and found the Git page: https://github.com/furier/websync

 

Shame that it has been over a year since there has been any movement. I guess this is a Docker for the dustbin. Pointless having it constantly stuck in BETA if it is not being maintained or developed further.

 

Sorry again for the misunderstanding.

 

Docker creation is pretty much all packaging other applications.  Linuxserver.io actually create their own containers, as do most of the other docker authors, which gives some control over things. 

 

There are a few containers available via Community Applications that are not packaged by anyone from the Unraid community and a user has just created an XML file to "template" it for use here.  One of the fundamental problems with this approach is that the Unraid member has no control over the underlying docker container, so I try as far as possible not to use such containers.

 

"I didn't realised you had just created a container for an exisiting project." 

 

"Shame that it has been over a year since there has been any movement. I guess this is a Docker for the dustbin. Pointless having it constantly stuck in BETA if it is not being maintained or developed further.

 

Sorry again for the misunderstanding."

 

I assume that when we release WebSync that you won't be interested then?  Of course if you're able to find or write a better alternative then please pop over to our IRC room and contribute.....

 

Personally, even if development has ceased, if it works and does what it sets out to do, then I don't really see a problem.

 

Rsync is actually pretty simple to use and I do so at home, and I think anything that is simple to use is less likely to be automated that things that are more complicated.

 

For example. I can (and do) use Rysnc, as do many people on here, and I'm sure any of us will be willing to help if you decided to actually try and implement this, after all backups are important as you've already stated.  I cannot however implement dual parity and or virtual machines on Unraid....

 

If backing up is that important to you then surely it would be worth investing a little time and effort in?