[Support] lnxd - github-backup


Recommended Posts

Overview: Support thread for lnxd/github-backup in CA.

Application: https://github.com/abusesa/github-backup

Docker Hub: https://hub.docker.com/r/lnxd/github-backup

GitHub: https://github.com/lnxd/docker-github-backup

 

 

This container contains a script, backup.py, for backing up GitHub repositories.

 

The script requires a GitHub token and a destination directory. It then uses the token to populate the destination directory with clones of all the repositories the token can access. 

 

It is possible to set it to run on a schedule, and repeated runs only update the already existing backups and add new repositories, if any.

 

 

Instructions:

Installation can be completed via CA. All you need to do is grab a token from here and fill out the template.

 

 

Feel free to comment here if you need any assistance. 

Edited by lnxd
Link to comment

Hi Op.

Great docker.  I am new to github, but I have a project running right now that I want to back up.

It does not appear to be working.

 

In addition to adding my token (blocked out here) the only thing I changed in the config is the location of the backups and the backups to run every 24 hours.

 

Here is my log:

 

sed: ab*********************************************\",: No such file or directory
Traceback (most recent call last):
File "backup.py", line 103, in <module>
main()
File "backup.py", line 79, in main
config = json.loads(f.read())
File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.6/json/decoder.py", line 355, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid control character at: line 2 column 15 (char 16)

 

 

Link to comment
6 hours ago, eds said:

Hi Op.

Great docker.  I am new to github, but I have a project running right now that I want to back up.

It does not appear to be working.

 

In addition to adding my token (blocked out here) the only thing I changed in the config is the location of the backups and the backups to run every 24 hours.

 

Here is my log:

 


sed: ab*********************************************\",: No such file or directory
Traceback (most recent call last):
File "backup.py", line 103, in <module>
main()
File "backup.py", line 79, in main
config = json.loads(f.read())
File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.6/json/decoder.py", line 355, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid control character at: line 2 column 15 (char 16)

 

Hey @eds, thanks!

 

Can I please grab your GitHub username? If you have any symbols in there it might not be handled correctly.

Link to comment
Posted (edited)
50 minutes ago, eds said:

Yeah, I have a dash in the username.

ESmall-Project

 

@eds can you please visit this page and delete the token.

 

Then jump into a terminal via webUI or SSH and run:

docker exec github-backup cat /home/docker/github-backup/config.json

 

I'll get you to please copy and paste the output here in code tags or a text file.

 

Also the output of:

cat /boot/config/plugins/dockerMan/templates-user/my-github-backup.xml

 

Both will have your token in it, but as long as you delete it from the GitHub website first it's useless (you can see a few of my former tokens in the repo). If you don't feel confident you can always PM me the output directly.

 

Thanks!

Edited by lnxd
Link to comment
23 minutes ago, lnxd said:

@eds can you please visit this page and delete the token.

 

Then jump into a terminal via webUI or SSH and run:


docker exec github-backup cat /home/docker/github-backup/config.json

 

 

Here you go.  The output of docker exec github-backup cat /home/docker/github-backup/config.json:

{
   "token" : "
    "directory": "/home/docker/backups"
}

 

I will pm you the output of the second section.   Thank you.

 

Link to comment
31 minutes ago, eds said:

 

 

Here you go.  The output of docker exec github-backup cat /home/docker/github-backup/config.json:


{
   "token" : "
    "directory": "/home/docker/backups"
}

 

I will pm you the output of the second section.   Thank you.

 

Thanks,

If anyone else gets the same error as @eds, my code doesn't handle whitespace in the Token field so you need to make sure there are no spaces before or after your token. I'll patch this in the next update.

Link to comment
16 minutes ago, lnxd said:

Thanks,

If anyone else gets the same error as @eds, my code doesn't handle whitespace in the Token field so you need to make sure there are no spaces before or after your token. I'll patch this in the next update.

 

Thanks op.  The answer was so obvious, I cannot understand why it did not occur to me.   Works for me now.

 

One thing I am wondering -- it looks like the files downloaded are owned by root.  Is that to be expected?

Edited by eds
question.
  • Like 1
Link to comment
Posted (edited)
4 hours ago, eds said:

Thanks op.  The answer was so obvious, I cannot understand why it did not occur to me.   Works for me now.

 

One thing I am wondering -- it looks like the files downloaded are owned by root.  Is that to be expected?

Haha no worries.

 

Thanks for pointing that out. It's expected, I didn't think anything of it. It could cause problems for some people so I'll update that too, you can expect a fix within the next 12 hours or so and I'll tag you when it's ready. Thanks for the feedback!

 

EDIT: With the latest update the data is now owned by the user nobody which is default for Unraid. Anyone who wants to change ownership on the data that's already there just run:

 

sudo chown -R nobody /mnt/user/backups/github

 

In your case @eds replace /mnt/user/backups/github in that command with /mnt/user/github as I believe that's where you store it.

Edited by lnxd
Link to comment
11 hours ago, lnxd said:

EDIT: With the latest update the data is now owned by the user nobody which is default for Unraid. Anyone who wants to change ownership on the data that's already there just run:

 

 

In your case @eds replace /mnt/user/backups/github in that command with /mnt/user/github as I believe that's where you store it.

Brilliant. Thanks for the update!

  • Like 1
Link to comment

This looks excellent. I have been meaning to set up something similar myself recently and just stumbled across this app which looks like it will do exactly what I need!

 

However, I am running into what appears to be an issue - the script ran through and indexed all of my repos, but it only seemed to download metadata - not the actual content of the repo. Is this intended? Or do I just have something configured incorrectly? I checked that my access token has the correct permissions and the container logs don't give any indication that anything went wrong.

 

Here's a couple images showing what I see in the backup data... it looks quite similar to the .git folder of my downloaded repo.

backup_data.png

git_metadata.png

Link to comment
1 hour ago, MrBurrito said:

This looks excellent. I have been meaning to set up something similar myself recently and just stumbled across this app which looks like it will do exactly what I need!

 

However, I am running into what appears to be an issue - the script ran through and indexed all of my repos, but it only seemed to download metadata - not the actual content of the repo. Is this intended? Or do I just have something configured incorrectly? I checked that my access token has the correct permissions and the container logs don't give any indication that anything went wrong.

 

Here's a couple images showing what I see in the backup data... it looks quite similar to the .git folder of my downloaded repo.

backup_data.png

 

 

I'm seeing the same thing on my end.  I hadn't realized this until I read this thread. I recently increased the permissions on my access token to see if that could be the issue as well.

Link to comment
Posted (edited)
1 hour ago, MrBurrito said:

This looks excellent. I have been meaning to set up something similar myself recently and just stumbled across this app which looks like it will do exactly what I need!

Awesome to hear!
 

1 hour ago, MrBurrito said:

However, I am running into what appears to be an issue - the script ran through and indexed all of my repos, but it only seemed to download metadata - not the actual content of the repo. Is this intended? Or do I just have something configured incorrectly? I checked that my access token has the correct permissions and the container logs don't give any indication that anything went wrong.

All good, this is how it should look. This is similar to how your data is stored on Github's servers, and contains the entire history of your repos. You need to use the git fetch and then git checkout commands if you want to access a particular version of the data from the folders you can see in the screenshots. If you're used to GUI, it's also possible (although arguably less efficient) to do it using GitKraken and GitHub Desktop for example.

 

In addition to that, yes. If you clone a git repository without specifying a depth it will backup the same data which you can see in the .git folder. However what you're probably used to is that it also checks out a specific branch (by default at the latest commit) simultaneously. 

The point of this container is just so that you have a local backup of all of the repositories you have on GitHub. Although you could possibly use it with rsync + Gitea/GitHub Enterprise/GitLab CE to host your own local mirror of your GitHub account, that's well outside of what I can support here 😅

Edited by lnxd
Link to comment
12 minutes ago, lnxd said:

 

All good, this is how it should look. This is similar to how your data is stored on Github's servers, and contains the entire history of your repos. You need to use the git fetch and then git checkout commands if you want to access a particular version of the data from the folders you can see in the screenshots. If you're used to GUI, it's also possible (although arguably less efficient) to do it using GitKraken and GitHub Desktop for example.

 

 

Thanks for the additional information, @lnxd.

 

Just to be clear, say github server was down, so you could not access the files from repository.  How would a user access their files using the docker you created?

 

 

 

 

Link to comment
Posted (edited)
27 minutes ago, eds said:

 

Thanks for the additional information, @lnxd.

 

Just to be clear, say github server was down, so you could not access the files from repository.  How would a user access their files using the docker you created?

 

 

 

 

Something like this, assuming the following:

 

Backup Path: /mnt/user/backups/github

Working Directory (Share): /mnt/user/projects

Github Username: lnxd

Repo Name: docker-alpine

Current Branch: master

 

SSH into your Unraid server or open a terminal from the WebUI and run:

# Make the project directory in your working path
# Note that if you do not change this to a valid path it will make a share on your server called projects
mkdir -p /mnt/user/projects/docker-alpine

# Copy the backed up data to the hidden .git directory inside the project directory
cp -r /mnt/user/backup/github/lnxd/docker-alpine /mnt/user/projects/docker-alpine/.git

# Change your present working directory to the project directory you created
cd /mnt/user/projects/docker-alpine

# Fetch the contents of .git
git fetch

# (Optional) Check the name of the branch, usually either called master or main
# Press q when done as it starts an interactive session
git branch

# Checkout the branch you want
git --work-tree=/mnt/user/projects/docker-alpine checkout master

 

Edited by lnxd
Link to comment
1 hour ago, eds said:

 

I'm seeing the same thing on my end.  I hadn't realized this until I read this thread. I recently increased the permissions on my access token to see if that could be the issue as well.

Glad to see it wasn't just me that was a bit confused.

 

22 minutes ago, lnxd said:

Something like this, assuming the following:

 

Backup Path: /mnt/user/backups/github

Working Directory (Share): /mnt/user/projects

Github Username: lnxd

Repo Name: docker-alpine

Current Branch: master

 

SSH into your Unraid server or open a terminal from the WebUI and run:


# Make the project directory in your working path
# Note that if you do not change this to a valid path it will make a share on your server called projects
mkdir -p /mnt/user/projects/docker-alpine

# Copy the backed up data to the hidden .git directory inside the project directory
cp -r /mnt/user/backup/github/lnxd/docker-alpine /mnt/user/projects/docker-alpine/.git

# Change your present working directory to the project directory you created
cd /mnt/user/projects/docker-alpine

# Fetch the contents of .git
git fetch

# (Optional) Check the name of the branch, usually either called master or main
# Press q when done as it starts an interactive session
git branch

# Checkout the branch you want
git --work-tree=/mnt/user/projects/docker-alpine checkout master

 

Excellent. Thanks for the clarification and the example here. This makes a lot of sense and I was able to test and confirm this does indeed work for me. 

  • Thanks 2
Link to comment

Haha all good, you guys made me think. I'd never actually tried using the data before, I just knew the theory was sound. 

 

3 minutes ago, MrBurrito said:

Excellent. Thanks for the clarification and the example here. This makes a lot of sense and I was able to test and confirm this does indeed work for me. 

Perfect! No worries. I'm also pleased I didn't make any mistakes in my commands 😅

Link to comment
  • 2 months later...

Hello all. Whenever I run this docker container I get the following error: "chown: /home/docker/backups: Operation not permitted". I"m fairly sure this is because I'm trying to save all the github files to a harddrive that is not apart of the main Unraid array, but I'm not sure how to fix the permission error. It is specifically failing on line 20 (chown -R nobody /home/docker/backups) of the backup.sh script. Is there a way to give the container the permission it needs to save the files?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.