[Support] ich777 - Application Dockers

[ich777]

51 minutes ago, TexasUnraid said:

Here is a link to the details if you want to take a look, I can post the screenshots of the errors.

Already looked at it but they don't tell much...

 

Can't remember but have we talked about tubesync or MeTube in the past? These should be similar applications that have actually a real WebGUI.

[TexasUnraid]

7 hours ago, ich777 said:

Already looked at it but they don't tell much...

 

Can't remember but have we talked about tubesync or MeTube in the past? These should be similar applications that have actually a real WebGUI.

 

Yeah, the lack of info in the errors is why it took me so long to narrow it down.

 

We had not talked about those, looked into them briefly just now though.

 

Metube looks like a 1 time download client, very useful and I could see a use case for it on my server (when the kids want to download a single video/channel for example) but not what I am aiming for in this particular use case.

 

I have a set of youtube channels that I like to keep a local copy of, so it will automatically download all videos on the channel and any new ones that are posted.

 

Which is where tubesync comes in, this looks very interesting although still in beta.

 

There are a few features that I am not sure if they are possible with it though, such as the ability to control the resolution downloaded (I limit a lot of channels to 480p to save space, the kids don't need HD vids).

 

Also the ability to download from other sites is something that is quite handy.

 

Doesn't look like it supports downloading through a proxy as well and I have had issues routing containers through a VPN container.

 

Once fully fleshed out and a full release I could easily see it being a good option.

[JonathanM]

1 hour ago, TexasUnraid said:

 

I have a set of youtube channels that I like to keep a local copy of, so it will automatically download all videos on the channel and any new ones that are posted.

Have you tried youtubedl-material recently? You are describing exactly what I use.

[TexasUnraid]

2 minutes ago, JonathanM said:

Have you tried youtubedl-material recently? You are describing exactly what I use.

I tried it last year when I ended up settling on tartube.

 

There was some kind of usage breaking issue with material but forgot what it was, think it might of been the lack of 480p quality setting / proxy IIRC. Just know I tried it and could not make it work then for what I need.

 

Might of also been an issue with large channels, some of them have 10,000's++ videos.

 

Has anything changed in the last ~9 months?

[wambo]

Hi there,

I've managed to get LuckyBackup going with my destination mounted as smb share with UD via VPN.
I wanted to skip the mounting, and use rsync via ssh on destination, but I've run into some issues:

I copied a (ed25519) key I made for this connection into the appdata of luckybackup, so I could select it there.
(The public one got to the destination via ssh-copy-id) 

But something seems to be going wrong when I start a dry-run. I get asked to verify the fingerprint (the message says yes/no/[fingerprint]) I choose OK (or cancel, doesnt matter)- and it fails. It also asks me every time for the fingerprint.

 

Host key verification failed.

rsync connection unexpectedly closed (0 bytes by receiver [...]) rsync error: unexplained error (code 255) at io.c(288)

I went into the terminal, and ssh'd into the destination and verified the fingerprint there, but the GUI still asks me.
I tried to follow this guide:
https://unraid.net/blog/unraid-server-backups-with-luckybackup
But in that terminal, I run into some things that confuse me:
-  ls /root  doesnt show the .ssh directory, I can cd into it, but it again doesnt list the files

- similar thing with ls /luckybackup - its empty 

In the guide, they chose the private key from /root/.ssh in the GUI, but the GUI doesnt show anything in /root/, not the files I cannot list in the terminal, nor the files I can list in the terminal (cause I grabbed them via scp from the host)

I'm also trying to create a config file in the .ssh, but I'm really confused where it needs to go 😕

Edited February 1, 2022 by wambo

[ich777]

1 hour ago, wambo said:

ls /root  doesnt show the .ssh directory, I can cd into it, but it again doesnt list the files

Do you run the container as root (it's a setting in the template)? The directory is not listed because files/folders who are starting with a dot are hidden in Linux and this directory needs to be hidden.

 

1 hour ago, wambo said:

similar thing with ls /luckybackup - its empty 

If you want to see all files/directories you have to do 'ls -la /luckybackup' or 'ls -la /root'.

 

1 hour ago, wambo said:

But something seems to be going wrong when I start a dry-run. I get asked to verify the fingerprint (the message says yes/no/[fingerprint]) I choose OK (or cancel, doesnt matter)- and it fails. It also asks me every time for the fingerprint.

If I understand that right you copy from luckyBackup (source) to another server (destination), in this case I would recommend that you simply do it like in the attached screenshot and inject the '/luckybackup/.ssh/ssh_host_ed25519_key.pub' from the container to the destination server in the 'authorized_keys' file and restart the sshd service on the destination server afterwards (the restart is not really necessary but only to be sure).

 

Please keep in mind depending if you are running the container as root or not the path from above and in the screenshot is different, if you run it as non-root then it is like in the example '/luckybackup/...' and if you are running it as root it is '/root/...'

 

 

1 hour ago, wambo said:

I copied a (ed25519) key I made for this connection into the appdata of luckybackup, so I could select it there.

You don't have to create certificates this does the container for you.

 

1 hour ago, wambo said:

I'm also trying to create a config file in the .ssh, but I'm really confused where it needs to go 😕

I don't understand that completely... what config file and why?

[Commerzpunk]

Hello,

i like to use the krusader docker image to view pictures and photos while i browse thru directories to verify what this file is.

 

Is there an option to make it availible?

[alturismo]

29 minutes ago, Commerzpunk said:

Is there an option to make it availible?

 

may a little offtopic now  but the sense of a docker like krusader is to have file operation "lightweight" and easy to handle, when you start blowing up krusader with all extras like image viewer, video previews, ... you have a full linux distro running sooner or later so why not just use the client you use anyway to open the VNC Connection to krusader and browse your Pictures and so on ... wouldnt this be easier and faster and more comfortable ?

 

sorry for a little bit offtopic

[Commerzpunk]

Because of security and comfort reasons. I don’t want to connect all the shares permanently to my window PC, because a theoretical ransomware attack could easily reach there.

And, because it’s secure, I don’t want to daily re-enter the credentials.

 

[alturismo]

26 minutes ago, Commerzpunk said:

Because of security and comfort reasons.

 

ok, may consider to get an sep docker for this like Nextcloud, Photoprism, ... where you really woul dhave the benefit for this kinda actions.

 

lets see if the author add the addons like KrViewer (F3 to view or so when i remember correctly)

[Commerzpunk]

Yes, thats correct too. I use both of these, NC und PP.

But thats not the point.

Krusader is a nice software to browse, archive, cleanup, organize, move, delete and sort all of my shares.

Thats whats its made for and thats not possible via NC and PP, its much more uncomfortable and only a few shares are linked into these containers.

 

Do you get what I mean?

I am investigating some of my old archive folders and delete or move files from there. I often see pictues or photos there, and like to see them to know to which location to move them (or delete them).

 

BTW, i found CloudCommander which has this option build in. Maybe thats just better for me? I wil check both.

[ich777]

2 hours ago, Commerzpunk said:

Is there an option to make it availible?

Please go to the Docker page, turn on "Advanced View" on the top right corner and click on at Krusader (please don't forget to turn off "Advanced View" after you've did this because Advanced View actually produces a little load on your server).

 

You can double click now images to show them and if they are to big for the screen you can drag them around, you have now a right click menu and you can simply close them with "ESC".

 

Hope that helps.

[Commerzpunk]

Oh, wow, yes that helps, thanks!

Its just too quick and easy to be true. 😀

 

One more question now: the new option has a right click context menu for resize, zoom and other options.

I cant find the way to make the setting persistant, I have to redo them for every image.

[wambo]

7 hours ago, ich777 said:

Do you run the container as root (it's a setting in the template)? The directory is not listed because files/folders who are starting with a dot are hidden in Linux and this directory needs to be hidden.

If you want to see all files/directories you have to do 'ls -la /luckybackup' or 'ls -la /root'.

Not root. But now I think I'm understanding.
I wouldve sworn I tried the -a but of course, I must've mistyped or sth 😕 A bit embarrassing.
 

 

7 hours ago, ich777 said:

I don't understand that completely... what config file and why?

Its probably not necessary, but I tend to run a .ssh/config where I point specific Hosts or Users to corresponding key files.
I wanted to try this, since I couldn't point the GUI to the root's key (which I don't need to, as I understood now)

 

 

7 hours ago, ich777 said:

If I understand that right you copy from luckyBackup (source) to another server (destination), in this case I would recommend that you simply do it like in the attached screenshot and inject the '/luckybackup/.ssh/ssh_host_ed25519_key.pub' from the container to the destination server in the 'authorized_keys' file and restart the sshd service on the destination server afterwards (the restart is not really necessary but only to be sure).

 

7 hours ago, ich777 said:

 

Please keep in mind depending if you are running the container as root or not the path from above and in the screenshot is different, if you run it as non-root then it is like in the example '/luckybackup/...' and if you are running it as root it is '/root/...'

 

 

You don't have to create certificates this does the container for you.

 

I should be able to just assign a different private key in the GUI then, because I already placed the public key on the destination.

But even though I'm not getting the fingerprint window anymore, I still do get the error:

 

I can use the command:
ssh -v -p 2211 -i '~/.ssh/id_internas_ed [email protected]' from host and it works
So I dont know where the error lies 😕


 

[ich777]

1 hour ago, Commerzpunk said:

One more question now: the new option has a right click context menu for resize, zoom and other options.

I cant find the way to make the setting persistant, I have to redo them for every image.

Can't do anything about that...

 

Maybe take a look at CloudComander if that suits your needs better.

[ich777]

24 minutes ago, wambo said:

Its probably not necessary, but I tend to run a .ssh/config where I point specific Hosts or Users to corresponding key files.

But why, if you put a authorized_key file in the .ssh directory from a user with a key in it it logs on as that user usually.

Have you restarted the SSH daemon on the destination after putting the key on it? If you are connecting as root to the destination make sure you have also enabled logon with root and certificates, depending on the distribution you are using that's maybe not enabled by default.

 

28 minutes ago, wambo said:

I should be able to just assign a different private key in the GUI then, because I already placed the public key on the destination.

Yes, you should but why would you do that? The container creates a individual key file for you, I really don't get why you do that manually when there is are already key files in place, if you don't trust me and you think that I've made the key file everywhere the same look at the source code, the container generates theses files on the first start or if they aren't found.

 

32 minutes ago, wambo said:

But even though I'm not getting the fingerprint window anymore, I still do get the error:

From what I see from the screenshot you are using a non standard port but the container tries to connect to port 22 even if you have set it to 2211.

On my system a non default port (8022) works just fine. Are you sure that you have saved the sync before you clicked Run?

 

Maybe try to restart the container once more, something seems wrong with the port on your system.

[Commerzpunk]

1 hour ago, ich777 said:

Can't do anything about that...

 

Maybe take a look at CloudComander if that suits your needs better.

Ahm, no, I’ll stick with your container - it’s pretty close to perfection.

[wambo]

3 hours ago, ich777 said:

But why, if you put a authorized_key file in the .ssh directory from a user with a key in it it logs on as that user usually.

Have you restarted the SSH daemon on the destination after putting the key on it? If you are connecting as root to the destination make sure you have also enabled logon with root and certificates, depending on the distribution you are using that's maybe not enabled by default.

The destination is working (used that keypair before). I even just testet as "luckybackup" user from the docker terminal and I could get in with the key and port specified. (its not root, synology admin group though, because synology limits ssh to admin users, but I restricted access via visudo and DSM)

 

3 hours ago, ich777 said:

 

Yes, you should but why would you do that? The container creates a individual key file for you, I really don't get why you do that manually when there is are already key files in place, if you don't trust me and you think that I've made the key file everywhere the same look at the source code, the container generates theses files on the first start or if they aren't found.

Its definitely not mistrust, I just already "had it in place" on the destination before I found that there were keys in luckybackup, and I have disabled password login for that user already, so it was easier to move the private key into the container for me.
Its just how I organize myself, I have like 1 key pair for work related things, 1 for private network things.
I ran into some issues with having too many private keys.. but I guess many authorized_keys wouldnt turn into an issue...

 

3 hours ago, ich777 said:

 

From what I see from the screenshot you are using a non standard port but the container tries to connect to port 22 even if you have set it to 2211.

On my system a non default port (8022) works just fine. Are you sure that you have saved the sync before you clicked Run?

 

Maybe try to restart the container once more, something seems wrong with the port on your system.

Restarted the container (after quitting the GUI to save)  - same thing.

When I dont do a dry run I can find logs, but its no new information.
I tried removing my custom commands (bwlimit and append)...
Gonna try with one of the container-made keys now ....

 

 

Edit:

I just switched the SSH port on destination to default (22), switched it in luckybackup, restarted, same result. I can still  open the docker console, 'su luckybackup' and 'ssh -i <keyfile> user@host' and that works.
So it must be something about "grabbing the selected identityFile / private key".
Maybe a permissions thing (but why can I use it after 'su luckybackup' then?)

I created the ssh user config, and by now the command 'ssh <user>@<host>' works without any additional options, but executing it via luckybackup gets me a "permission denied, please try again" (the first line is varying it seems)

I also manually copied the command that luckybackup gives to verify and typed it into the console, but I do receive the same error. 

Edit2:
Okay, so I've tried to rsync from Unraid root manually, and i get the same error, I believe its a problem on the destination, I thought ssh access would be enough if I dont use modules - although I did activate rsync tried every access related to it that i could find mention of 😕
Damn synology... what are they doing?

 

Edited February 2, 2022 by wambo

[ich777]

7 hours ago, wambo said:

Restarted the container (after quitting the GUI to save)  - same thing.

When I dont do a dry run I can find logs, but its no new information.

Does it also try to connect to port 22 instead of port 2211 like in the screenshot above?

 

7 hours ago, wambo said:

su luckybackup

Just FYI you can also do:

su $USER

in all of my container to become the main user under which the applications runns.

 

7 hours ago, wambo said:

I just switched the SSH port on destination to default (22)

Can you try to switch it to something completely different, I just finished a Backup on port 8022.

 

7 hours ago, wambo said:

Damn synology... what are they doing?

You could also try to install a SSH container on your Synology and try to connect to this container and see if that is working, if yes you could mount a directory on your Synology to the SSH container and sync it that way <- this is just a really wild workaround but it should work well...

[wambo]

2 hours ago, ich777 said:

Does it also try to connect to port 22 instead of port 2211 like in the screenshot above?

Can you try to switch it to something completely different, I just finished a Backup on port 8022.

I think I didn't get the 22 connection refused again. Will try on another port again --> yeah,  I'm on "permission denied" always now (when I type the right port, mustve hit 22 for testing and postet the wrong error, although I think the other 2 lines never changed)😕
 

 

 

2 hours ago, ich777 said:

 

You could also try to install a SSH container on your Synology and try to connect to this container and see if that is working, if yes you could mount a directory on your Synology to the SSH container and sync it that way <- this is just a really wild workaround but it should work well...

Sadly my diskstation doesnt support containers (by default), and to hack docker for ARM on it, I'd like to evade.
I've seen other ppl with issues on DSM 7 and rsync. Maybe Synology will tweak that a bit more. Will also post in their forums.

I really wanted to run all my "mirroring" from one central point, but I might have to set it up on the other side... or stay with smb mount. 😕

[ich777]

27 minutes ago, wambo said:

I really wanted to run all my "mirroring" from one central point, but I might have to set it up on the other side... or stay with smb mount. 😕

If there are issues with DSM and rsync I would recommend to stick with the SMB solution that you have now, maybe Synology will update their SSH stack and it will work somewhere in the future...

[wambo]

On 2/2/2022 at 10:05 AM, ich777 said:

If there are issues with DSM and rsync I would recommend to stick with the SMB solution that you have now, maybe Synology will update their SSH stack and it will work somewhere in the future...

I've done some digging, and managed to run a rsync command for the daemon on the diskstation. 
This documentation states that 
"If you perform rsync backup from a non-Synology NAS client, you have to create rsync accounts on the destination Synology NAS so that the rsync users can back up data to your Synology NAS."; "DSM users for encrypted rsync, or rsync accounts for unencrypted rsync" - but I can't really assign a publickey to the rsync accounts. So yeah, I might be stuck with smb.  That also means no compression? (And other features?)

I asked in the synology forums, how others handle it.

Edited February 3, 2022 by wambo

[ich777]

4 hours ago, wambo said:

I asked in the forums there how others handle it.

Have you yet tried it with a password instead, if the Synology is on your local network?

I'm not too familiar with Synology...

[wambo]

On 2/3/2022 at 6:43 AM, ich777 said:

Have you yet tried it with a password instead, if the Synology is on your local network?

I'm not too familiar with Synology...

I managed to run an rsync command from  unraid host onto a module ( with the :: ) with the rsync user and password. (So a different password but same username as "normal" user / the user I use for ssh... *security feature* -.-)
But I thought that wouldnt be an option, cause LuckyBackup doesnt have the password field (accessible) ?
Also can LuckyBackup handle the module thingies, what I got, that is talking to the rsync deamon  (might need other options / commands? not sure tbh)

Edited February 4, 2022 by wambo

[ich777]

7 hours ago, wambo said:

But I thought that wouldnt be an option, cause LuckyBackup doesnt have the password field (accessible) ?

You should be able to create a variable in the template with the key: "RSYNC_PASSWORD" and as value enter your password or you create a rsync password file and take this as the authentication method.

 

7 hours ago, wambo said:

Also can LuckyBackup handle the module thingies, what I got, that is talking to the rsync deamon  (might need other options / commands? not sure tbh)

No, that's not possible in luckyBackup.