lnxd Posted March 25, 2021 Share Posted March 25, 2021 (edited) Overview: Support thread for lnxd/github-backup in CA. Application: https://github.com/abusesa/github-backup Docker Hub: https://hub.docker.com/r/lnxd/github-backup GitHub: https://github.com/lnxd/docker-github-backup This container contains a script, backup.py, for backing up GitHub repositories. The script requires a GitHub token and a destination directory. It then uses the token to populate the destination directory with clones of all the repositories the token can access. It is possible to set it to run on a schedule, and repeated runs only update the already existing backups and add new repositories, if any. Instructions: Installation can be completed via CA. All you need to do is grab a token from here and fill out the template. Feel free to comment here if you need any assistance. Edited March 26, 2021 by lnxd 1 Quote Link to comment
eds Posted March 31, 2021 Share Posted March 31, 2021 Hi Op. Great docker. I am new to github, but I have a project running right now that I want to back up. It does not appear to be working. In addition to adding my token (blocked out here) the only thing I changed in the config is the location of the backups and the backups to run every 24 hours. Here is my log: sed: ab*********************************************\",: No such file or directory Traceback (most recent call last): File "backup.py", line 103, in <module> main() File "backup.py", line 79, in main config = json.loads(f.read()) File "/usr/lib/python3.6/json/__init__.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.6/json/decoder.py", line 355, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Invalid control character at: line 2 column 15 (char 16) Quote Link to comment
lnxd Posted March 31, 2021 Author Share Posted March 31, 2021 6 hours ago, eds said: Hi Op. Great docker. I am new to github, but I have a project running right now that I want to back up. It does not appear to be working. In addition to adding my token (blocked out here) the only thing I changed in the config is the location of the backups and the backups to run every 24 hours. Here is my log: sed: ab*********************************************\",: No such file or directory Traceback (most recent call last): File "backup.py", line 103, in <module> main() File "backup.py", line 79, in main config = json.loads(f.read()) File "/usr/lib/python3.6/json/__init__.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.6/json/decoder.py", line 355, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Invalid control character at: line 2 column 15 (char 16) Hey @eds, thanks! Can I please grab your GitHub username? If you have any symbols in there it might not be handled correctly. Quote Link to comment
eds Posted March 31, 2021 Share Posted March 31, 2021 1 hour ago, lnxd said: Hey @eds, thanks! Can I please grab your GitHub username? If you have any symbols in there it might not be handled correctly. Yeah, I have a dash in the username. ESmall-Project Quote Link to comment
lnxd Posted March 31, 2021 Author Share Posted March 31, 2021 (edited) 50 minutes ago, eds said: Yeah, I have a dash in the username. ESmall-Project @eds can you please visit this page and delete the token. Then jump into a terminal via webUI or SSH and run: docker exec github-backup cat /home/docker/github-backup/config.json I'll get you to please copy and paste the output here in code tags or a text file. Also the output of: cat /boot/config/plugins/dockerMan/templates-user/my-github-backup.xml Both will have your token in it, but as long as you delete it from the GitHub website first it's useless (you can see a few of my former tokens in the repo). If you don't feel confident you can always PM me the output directly. Thanks! Edited April 1, 2021 by lnxd Quote Link to comment
eds Posted April 1, 2021 Share Posted April 1, 2021 23 minutes ago, lnxd said: @eds can you please visit this page and delete the token. Then jump into a terminal via webUI or SSH and run: docker exec github-backup cat /home/docker/github-backup/config.json Here you go. The output of docker exec github-backup cat /home/docker/github-backup/config.json: { "token" : " "directory": "/home/docker/backups" } I will pm you the output of the second section. Thank you. Quote Link to comment
lnxd Posted April 1, 2021 Author Share Posted April 1, 2021 31 minutes ago, eds said: Here you go. The output of docker exec github-backup cat /home/docker/github-backup/config.json: { "token" : " "directory": "/home/docker/backups" } I will pm you the output of the second section. Thank you. Thanks, If anyone else gets the same error as @eds, my code doesn't handle whitespace in the Token field so you need to make sure there are no spaces before or after your token. I'll patch this in the next update. Quote Link to comment
eds Posted April 1, 2021 Share Posted April 1, 2021 (edited) 16 minutes ago, lnxd said: Thanks, If anyone else gets the same error as @eds, my code doesn't handle whitespace in the Token field so you need to make sure there are no spaces before or after your token. I'll patch this in the next update. Thanks op. The answer was so obvious, I cannot understand why it did not occur to me. Works for me now. One thing I am wondering -- it looks like the files downloaded are owned by root. Is that to be expected? Edited April 1, 2021 by eds question. 1 Quote Link to comment
lnxd Posted April 1, 2021 Author Share Posted April 1, 2021 (edited) 4 hours ago, eds said: Thanks op. The answer was so obvious, I cannot understand why it did not occur to me. Works for me now. One thing I am wondering -- it looks like the files downloaded are owned by root. Is that to be expected? Haha no worries. Thanks for pointing that out. It's expected, I didn't think anything of it. It could cause problems for some people so I'll update that too, you can expect a fix within the next 12 hours or so and I'll tag you when it's ready. Thanks for the feedback! EDIT: With the latest update the data is now owned by the user nobody which is default for Unraid. Anyone who wants to change ownership on the data that's already there just run: sudo chown -R nobody /mnt/user/backups/github In your case @eds replace /mnt/user/backups/github in that command with /mnt/user/github as I believe that's where you store it. Edited April 1, 2021 by lnxd Quote Link to comment
eds Posted April 1, 2021 Share Posted April 1, 2021 11 hours ago, lnxd said: EDIT: With the latest update the data is now owned by the user nobody which is default for Unraid. Anyone who wants to change ownership on the data that's already there just run: In your case @eds replace /mnt/user/backups/github in that command with /mnt/user/github as I believe that's where you store it. Brilliant. Thanks for the update! 1 Quote Link to comment
MrBurrito Posted April 7, 2021 Share Posted April 7, 2021 This looks excellent. I have been meaning to set up something similar myself recently and just stumbled across this app which looks like it will do exactly what I need! However, I am running into what appears to be an issue - the script ran through and indexed all of my repos, but it only seemed to download metadata - not the actual content of the repo. Is this intended? Or do I just have something configured incorrectly? I checked that my access token has the correct permissions and the container logs don't give any indication that anything went wrong. Here's a couple images showing what I see in the backup data... it looks quite similar to the .git folder of my downloaded repo. Quote Link to comment
eds Posted April 7, 2021 Share Posted April 7, 2021 1 hour ago, MrBurrito said: This looks excellent. I have been meaning to set up something similar myself recently and just stumbled across this app which looks like it will do exactly what I need! However, I am running into what appears to be an issue - the script ran through and indexed all of my repos, but it only seemed to download metadata - not the actual content of the repo. Is this intended? Or do I just have something configured incorrectly? I checked that my access token has the correct permissions and the container logs don't give any indication that anything went wrong. Here's a couple images showing what I see in the backup data... it looks quite similar to the .git folder of my downloaded repo. I'm seeing the same thing on my end. I hadn't realized this until I read this thread. I recently increased the permissions on my access token to see if that could be the issue as well. Quote Link to comment
lnxd Posted April 7, 2021 Author Share Posted April 7, 2021 (edited) 1 hour ago, MrBurrito said: This looks excellent. I have been meaning to set up something similar myself recently and just stumbled across this app which looks like it will do exactly what I need! Awesome to hear! 1 hour ago, MrBurrito said: However, I am running into what appears to be an issue - the script ran through and indexed all of my repos, but it only seemed to download metadata - not the actual content of the repo. Is this intended? Or do I just have something configured incorrectly? I checked that my access token has the correct permissions and the container logs don't give any indication that anything went wrong. All good, this is how it should look. This is similar to how your data is stored on Github's servers, and contains the entire history of your repos. You need to use the git fetch and then git checkout commands if you want to access a particular version of the data from the folders you can see in the screenshots. If you're used to GUI, it's also possible (although arguably less efficient) to do it using GitKraken and GitHub Desktop for example. In addition to that, yes. If you clone a git repository without specifying a depth it will backup the same data which you can see in the .git folder. However what you're probably used to is that it also checks out a specific branch (by default at the latest commit) simultaneously. The point of this container is just so that you have a local backup of all of the repositories you have on GitHub. Although you could possibly use it with rsync + Gitea/GitHub Enterprise/GitLab CE to host your own local mirror of your GitHub account, that's well outside of what I can support here 😅 Edited April 7, 2021 by lnxd Quote Link to comment
eds Posted April 7, 2021 Share Posted April 7, 2021 12 minutes ago, lnxd said: All good, this is how it should look. This is similar to how your data is stored on Github's servers, and contains the entire history of your repos. You need to use the git fetch and then git checkout commands if you want to access a particular version of the data from the folders you can see in the screenshots. If you're used to GUI, it's also possible (although arguably less efficient) to do it using GitKraken and GitHub Desktop for example. Thanks for the additional information, @lnxd. Just to be clear, say github server was down, so you could not access the files from repository. How would a user access their files using the docker you created? Quote Link to comment
lnxd Posted April 7, 2021 Author Share Posted April 7, 2021 (edited) 27 minutes ago, eds said: Thanks for the additional information, @lnxd. Just to be clear, say github server was down, so you could not access the files from repository. How would a user access their files using the docker you created? Something like this, assuming the following: Backup Path: /mnt/user/backups/github Working Directory (Share): /mnt/user/projects Github Username: lnxd Repo Name: docker-alpine Current Branch: master SSH into your Unraid server or open a terminal from the WebUI and run: # Make the project directory in your working path # Note that if you do not change this to a valid path it will make a share on your server called projects mkdir -p /mnt/user/projects/docker-alpine # Copy the backed up data to the hidden .git directory inside the project directory cp -r /mnt/user/backup/github/lnxd/docker-alpine /mnt/user/projects/docker-alpine/.git # Change your present working directory to the project directory you created cd /mnt/user/projects/docker-alpine # Fetch the contents of .git git fetch # (Optional) Check the name of the branch, usually either called master or main # Press q when done as it starts an interactive session git branch # Checkout the branch you want git --work-tree=/mnt/user/projects/docker-alpine checkout master Edited April 7, 2021 by lnxd Quote Link to comment
MrBurrito Posted April 7, 2021 Share Posted April 7, 2021 1 hour ago, eds said: I'm seeing the same thing on my end. I hadn't realized this until I read this thread. I recently increased the permissions on my access token to see if that could be the issue as well. Glad to see it wasn't just me that was a bit confused. 22 minutes ago, lnxd said: Something like this, assuming the following: Backup Path: /mnt/user/backups/github Working Directory (Share): /mnt/user/projects Github Username: lnxd Repo Name: docker-alpine Current Branch: master SSH into your Unraid server or open a terminal from the WebUI and run: # Make the project directory in your working path # Note that if you do not change this to a valid path it will make a share on your server called projects mkdir -p /mnt/user/projects/docker-alpine # Copy the backed up data to the hidden .git directory inside the project directory cp -r /mnt/user/backup/github/lnxd/docker-alpine /mnt/user/projects/docker-alpine/.git # Change your present working directory to the project directory you created cd /mnt/user/projects/docker-alpine # Fetch the contents of .git git fetch # (Optional) Check the name of the branch, usually either called master or main # Press q when done as it starts an interactive session git branch # Checkout the branch you want git --work-tree=/mnt/user/projects/docker-alpine checkout master Excellent. Thanks for the clarification and the example here. This makes a lot of sense and I was able to test and confirm this does indeed work for me. 2 Quote Link to comment
lnxd Posted April 7, 2021 Author Share Posted April 7, 2021 Haha all good, you guys made me think. I'd never actually tried using the data before, I just knew the theory was sound. 3 minutes ago, MrBurrito said: Excellent. Thanks for the clarification and the example here. This makes a lot of sense and I was able to test and confirm this does indeed work for me. Perfect! No worries. I'm also pleased I didn't make any mistakes in my commands 😅 Quote Link to comment
Nilok3 Posted June 15, 2021 Share Posted June 15, 2021 Hello all. Whenever I run this docker container I get the following error: "chown: /home/docker/backups: Operation not permitted". I"m fairly sure this is because I'm trying to save all the github files to a harddrive that is not apart of the main Unraid array, but I'm not sure how to fix the permission error. It is specifically failing on line 20 (chown -R nobody /home/docker/backups) of the backup.sh script. Is there a way to give the container the permission it needs to save the files? Quote Link to comment
Natcoso9955 Posted November 25, 2021 Share Posted November 25, 2021 Hi, i am wondering if it would be possible to tweak this container to work for GHES repo's. From what im reading though, it may be 1000X harder with GHES as it requires ssh and a few others things. But figured id ask anyway. Quote Link to comment
xthursdayx Posted February 21, 2022 Share Posted February 21, 2022 On 6/15/2021 at 11:38 AM, Nilok3 said: Hello all. Whenever I run this docker container I get the following error: "chown: /home/docker/backups: Operation not permitted". I"m fairly sure this is because I'm trying to save all the github files to a harddrive that is not apart of the main Unraid array, but I'm not sure how to fix the permission error. It is specifically failing on line 20 (chown -R nobody /home/docker/backups) of the backup.sh script. Is there a way to give the container the permission it needs to save the files? Did you ever figure out this issue? I'm having the same problem, but not sure why. @lnxd have you run into this issue before? Quote Link to comment
s449 Posted March 11, 2022 Share Posted March 11, 2022 On 3/31/2021 at 9:03 PM, lnxd said: EDIT: With the latest update the data is now owned by the user nobody which is default for Unraid. Anyone who wants to change ownership on the data that's already there just run: sudo chown -R nobody /mnt/user/backups/github Not sure if this is changed or something weird on my end, but I just installed this and by default my GitHub folder is owned by "65534". The folders inside it are part of the group "nobody" so I think something went weird, like an overflow. I ran sudo chown -R nobody:users -R GitHub/ But after restarting the container it just went back to 65534:users I just saw this docker today in the community apps spotlight but it looks like it hasn't been updated in a year and no responses here. Is this project abandoned? Do you need someone to take it over? Quote Link to comment
Goobaroo Posted March 30, 2022 Share Posted March 30, 2022 Can you add the owners config as part of the container config? If you happen to work for a company using Github, this script will backup any organizations you are a member of and not just personal repos. "owners": ["username"] 1 Quote Link to comment
Zapfmeister Posted March 31, 2022 Share Posted March 31, 2022 For me its shows owner and group as nobody. Which meant i cannot access the folders via SMB. I think the group is supposed to be "users". After changing the group it worked for me. I also tried to set the PGID in the container but it seems it doesnt make a difference. Quote Link to comment
Zapfmeister Posted March 31, 2022 Share Posted March 31, 2022 while true do python3 github-backup.py /home/docker/github-backup/config/config.json chown -R nobody /home/docker/backups sleep $SCHEDULE done https://github.com/lnxd/docker-github-backup/blob/master/backup.sh Seems to set user to nobody. Maybe we should set group as well. And maybe use IDs instead of words? Quote Link to comment
Goldmaster Posted July 21, 2022 Share Posted July 21, 2022 Hi there, is there a way to backup any projects that you have starred to zip files? Please. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.