Overview: Support thread for lnxd/github-backup in CA.

Application: https://github.com/abusesa/github-backup

Docker Hub: https://hub.docker.com/r/lnxd/github-backup

GitHub: https://github.com/lnxd/docker-github-backup

 

 

This container contains a script, backup.py, for backing up GitHub repositories.

 

The script requires a GitHub token and a destination directory. It then uses the token to populate the destination directory with clones of all the repositories the token can access. 

 

It is possible to set it to run on a schedule, and repeated runs only update the already existing backups and add new repositories, if any.

 

 

Instructions:

Installation can be completed via CA. All you need to do is grab a token from here and fill out the template.

 

 

Feel free to comment here if you need any assistance. 

Edited March 26, 20215 yr by lnxd

Hi Op.

Great docker.  I am new to github, but I have a project running right now that I want to back up.

It does not appear to be working.

 

In addition to adding my token (blocked out here) the only thing I changed in the config is the location of the backups and the backups to run every 24 hours.

 

Here is my log:

 

sed: ab*********************************************\",: No such file or directory
Traceback (most recent call last):
File "backup.py", line 103, in <module>
main()
File "backup.py", line 79, in main
config = json.loads(f.read())
File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.6/json/decoder.py", line 355, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid control character at: line 2 column 15 (char 16)

 

 

6 hours ago, eds said:

Hi Op.

Great docker.  I am new to github, but I have a project running right now that I want to back up.

It does not appear to be working.

 

In addition to adding my token (blocked out here) the only thing I changed in the config is the location of the backups and the backups to run every 24 hours.

 

Here is my log:

 

sed: ab*********************************************\",: No such file or directory
Traceback (most recent call last):
File "backup.py", line 103, in <module>
main()
File "backup.py", line 79, in main
config = json.loads(f.read())
File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.6/json/decoder.py", line 355, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid control character at: line 2 column 15 (char 16)

 

Hey @eds, thanks!

 

Can I please grab your GitHub username? If you have any symbols in there it might not be handled correctly.

1 hour ago, lnxd said:

Hey @eds, thanks!

 

Can I please grab your GitHub username? If you have any symbols in there it might not be handled correctly.

Yeah, I have a dash in the username.

ESmall-Project

 

50 minutes ago, eds said:

Yeah, I have a dash in the username.

ESmall-Project

 

@eds can you please visit this page and delete the token.

 

Then jump into a terminal via webUI or SSH and run:

docker exec github-backup cat /home/docker/github-backup/config.json

 

I'll get you to please copy and paste the output here in code tags or a text file.

 

Also the output of:

cat /boot/config/plugins/dockerMan/templates-user/my-github-backup.xml

 

Both will have your token in it, but as long as you delete it from the GitHub website first it's useless (you can see a few of my former tokens in the repo). If you don't feel confident you can always PM me the output directly.

 

Thanks!

Edited April 1, 20215 yr by lnxd

23 minutes ago, lnxd said:

@eds can you please visit this page and delete the token.

 

Then jump into a terminal via webUI or SSH and run:

docker exec github-backup cat /home/docker/github-backup/config.json

 

 

Here you go.  The output of docker exec github-backup cat /home/docker/github-backup/config.json:

{
   "token" : "
    "directory": "/home/docker/backups"
}

 

I will pm you the output of the second section.   Thank you.

 

31 minutes ago, eds said:

 

 

Here you go.  The output of docker exec github-backup cat /home/docker/github-backup/config.json:

{
   "token" : "
    "directory": "/home/docker/backups"
}

 

I will pm you the output of the second section.   Thank you.

 

Thanks,

If anyone else gets the same error as @eds, my code doesn't handle whitespace in the Token field so you need to make sure there are no spaces before or after your token. I'll patch this in the next update.

16 minutes ago, lnxd said:

Thanks,

If anyone else gets the same error as @eds, my code doesn't handle whitespace in the Token field so you need to make sure there are no spaces before or after your token. I'll patch this in the next update.

 

Thanks op.  The answer was so obvious, I cannot understand why it did not occur to me.   Works for me now.

 

One thing I am wondering -- it looks like the files downloaded are owned by root.  Is that to be expected?

Edited April 1, 20215 yr by eds
question.

4 hours ago, eds said:

Thanks op.  The answer was so obvious, I cannot understand why it did not occur to me.   Works for me now.

 

One thing I am wondering -- it looks like the files downloaded are owned by root.  Is that to be expected?

Haha no worries.

 

Thanks for pointing that out. It's expected, I didn't think anything of it. It could cause problems for some people so I'll update that too, you can expect a fix within the next 12 hours or so and I'll tag you when it's ready. Thanks for the feedback!

 

EDIT: With the latest update the data is now owned by the user nobody which is default for Unraid. Anyone who wants to change ownership on the data that's already there just run:

 

sudo chown -R nobody /mnt/user/backups/github

 

In your case @eds replace /mnt/user/backups/github in that command with /mnt/user/github as I believe that's where you store it.

Edited April 1, 20215 yr by lnxd

11 hours ago, lnxd said:

EDIT: With the latest update the data is now owned by the user nobody which is default for Unraid. Anyone who wants to change ownership on the data that's already there just run:

 

 

In your case @eds replace /mnt/user/backups/github in that command with /mnt/user/github as I believe that's where you store it.

Brilliant. Thanks for the update!

This looks excellent. I have been meaning to set up something similar myself recently and just stumbled across this app which looks like it will do exactly what I need!

 

However, I am running into what appears to be an issue - the script ran through and indexed all of my repos, but it only seemed to download metadata - not the actual content of the repo. Is this intended? Or do I just have something configured incorrectly? I checked that my access token has the correct permissions and the container logs don't give any indication that anything went wrong.

 

Here's a couple images showing what I see in the backup data... it looks quite similar to the .git folder of my downloaded repo.

1 hour ago, MrBurrito said:

This looks excellent. I have been meaning to set up something similar myself recently and just stumbled across this app which looks like it will do exactly what I need!

 

However, I am running into what appears to be an issue - the script ran through and indexed all of my repos, but it only seemed to download metadata - not the actual content of the repo. Is this intended? Or do I just have something configured incorrectly? I checked that my access token has the correct permissions and the container logs don't give any indication that anything went wrong.

 

Here's a couple images showing what I see in the backup data... it looks quite similar to the .git folder of my downloaded repo.

 

 

I'm seeing the same thing on my end.  I hadn't realized this until I read this thread. I recently increased the permissions on my access token to see if that could be the issue as well.

1 hour ago, MrBurrito said:

This looks excellent. I have been meaning to set up something similar myself recently and just stumbled across this app which looks like it will do exactly what I need!

Awesome to hear!
 

1 hour ago, MrBurrito said:

However, I am running into what appears to be an issue - the script ran through and indexed all of my repos, but it only seemed to download metadata - not the actual content of the repo. Is this intended? Or do I just have something configured incorrectly? I checked that my access token has the correct permissions and the container logs don't give any indication that anything went wrong.

All good, this is how it should look. This is similar to how your data is stored on Github's servers, and contains the entire history of your repos. You need to use the git fetch and then git checkout commands if you want to access a particular version of the data from the folders you can see in the screenshots. If you're used to GUI, it's also possible (although arguably less efficient) to do it using GitKraken and GitHub Desktop for example.

 

In addition to that, yes. If you clone a git repository without specifying a depth it will backup the same data which you can see in the .git folder. However what you're probably used to is that it also checks out a specific branch (by default at the latest commit) simultaneously. 

The point of this container is just so that you have a local backup of all of the repositories you have on GitHub. Although you could possibly use it with rsync + Gitea/GitHub Enterprise/GitLab CE to host your own local mirror of your GitHub account, that's well outside of what I can support here 😅

Edited April 7, 20215 yr by lnxd

12 minutes ago, lnxd said:

 

All good, this is how it should look. This is similar to how your data is stored on Github's servers, and contains the entire history of your repos. You need to use the git fetch and then git checkout commands if you want to access a particular version of the data from the folders you can see in the screenshots. If you're used to GUI, it's also possible (although arguably less efficient) to do it using GitKraken and GitHub Desktop for example.

 

 

Thanks for the additional information, @lnxd.

 

Just to be clear, say github server was down, so you could not access the files from repository.  How would a user access their files using the docker you created?

 

 

 

 

27 minutes ago, eds said:

 

Thanks for the additional information, @lnxd.

 

Just to be clear, say github server was down, so you could not access the files from repository.  How would a user access their files using the docker you created?

 

 

 

 

Something like this, assuming the following:

 

Backup Path: /mnt/user/backups/github

Working Directory (Share): /mnt/user/projects

Github Username: lnxd

Repo Name: docker-alpine

Current Branch: master

 

SSH into your Unraid server or open a terminal from the WebUI and run:

# Make the project directory in your working path
# Note that if you do not change this to a valid path it will make a share on your server called projects
mkdir -p /mnt/user/projects/docker-alpine

# Copy the backed up data to the hidden .git directory inside the project directory
cp -r /mnt/user/backup/github/lnxd/docker-alpine /mnt/user/projects/docker-alpine/.git

# Change your present working directory to the project directory you created
cd /mnt/user/projects/docker-alpine

# Fetch the contents of .git
git fetch

# (Optional) Check the name of the branch, usually either called master or main
# Press q when done as it starts an interactive session
git branch

# Checkout the branch you want
git --work-tree=/mnt/user/projects/docker-alpine checkout master

 

Edited April 7, 20215 yr by lnxd

1 hour ago, eds said:

 

I'm seeing the same thing on my end.  I hadn't realized this until I read this thread. I recently increased the permissions on my access token to see if that could be the issue as well.

Glad to see it wasn't just me that was a bit confused.

 

22 minutes ago, lnxd said:

Something like this, assuming the following:

 

Backup Path: /mnt/user/backups/github

Working Directory (Share): /mnt/user/projects

Github Username: lnxd

Repo Name: docker-alpine

Current Branch: master

 

SSH into your Unraid server or open a terminal from the WebUI and run:

# Make the project directory in your working path
# Note that if you do not change this to a valid path it will make a share on your server called projects
mkdir -p /mnt/user/projects/docker-alpine

# Copy the backed up data to the hidden .git directory inside the project directory
cp -r /mnt/user/backup/github/lnxd/docker-alpine /mnt/user/projects/docker-alpine/.git

# Change your present working directory to the project directory you created
cd /mnt/user/projects/docker-alpine

# Fetch the contents of .git
git fetch

# (Optional) Check the name of the branch, usually either called master or main
# Press q when done as it starts an interactive session
git branch

# Checkout the branch you want
git --work-tree=/mnt/user/projects/docker-alpine checkout master

 

Excellent. Thanks for the clarification and the example here. This makes a lot of sense and I was able to test and confirm this does indeed work for me. 

Haha all good, you guys made me think. I'd never actually tried using the data before, I just knew the theory was sound. 

 

3 minutes ago, MrBurrito said:

Excellent. Thanks for the clarification and the example here. This makes a lot of sense and I was able to test and confirm this does indeed work for me. 

Perfect! No worries. I'm also pleased I didn't make any mistakes in my commands 😅

Hello all. Whenever I run this docker container I get the following error: "chown: /home/docker/backups: Operation not permitted". I"m fairly sure this is because I'm trying to save all the github files to a harddrive that is not apart of the main Unraid array, but I'm not sure how to fix the permission error. It is specifically failing on line 20 (chown -R nobody /home/docker/backups) of the backup.sh script. Is there a way to give the container the permission it needs to save the files?

Hi, i am wondering if it would be possible to tweak this container to work for GHES repo's.
From what im reading though, it may be 1000X harder with GHES as it requires ssh and a few others things.

 

But figured id ask anyway.

On 6/15/2021 at 11:38 AM, Nilok3 said:

Hello all. Whenever I run this docker container I get the following error: "chown: /home/docker/backups: Operation not permitted". I"m fairly sure this is because I'm trying to save all the github files to a harddrive that is not apart of the main Unraid array, but I'm not sure how to fix the permission error. It is specifically failing on line 20 (chown -R nobody /home/docker/backups) of the backup.sh script. Is there a way to give the container the permission it needs to save the files?

Did you ever figure out this issue? I'm having the same problem, but not sure why. @lnxd have you run into this issue before?

On 3/31/2021 at 9:03 PM, lnxd said:

EDIT: With the latest update the data is now owned by the user nobody which is default for Unraid. Anyone who wants to change ownership on the data that's already there just run:

sudo chown -R nobody /mnt/user/backups/github

 

Not sure if this is changed or something weird on my end, but I just installed this and by default my GitHub folder is owned by "65534". The folders inside it are part of the group "nobody" so I think something went weird, like an overflow.

 

 

I ran

sudo chown -R nobody:users -R GitHub/

 

But after restarting the container it just went back to 65534:users

 

I just saw this docker today in the community apps spotlight but it looks like it hasn't been updated in a year and no responses here. Is this project abandoned? Do you need someone to take it over?

Can you add the owners config as part of the container config?  If you happen to work for a company using Github, this script will backup any organizations you are a member of and not just personal repos.

 

"owners": ["username"]

 

For me its shows owner and group as nobody. Which meant i cannot access the folders via SMB. I think the group is supposed to be "users". After changing the group it worked for me.

I also tried to set the PGID in the container but it seems it doesnt make a difference.

while true
 do python3 github-backup.py /home/docker/github-backup/config/config.json
 chown -R nobody /home/docker/backups
 sleep $SCHEDULE
done

https://github.com/lnxd/docker-github-backup/blob/master/backup.sh

Seems to set user to nobody. Maybe we should set group as well. And maybe use IDs instead of words?

Hi there, is there a way to backup any projects that you have starred to zip files? Please.