[Support] Josh.5 Docker Templates


Recommended Posts

I finally figured out why the "hit" and "miss" have disappeared from the access log entries.

I spotted 2 mistakes i think, both corrected below -

 

Typo in "overlay\hooks\supervisord-pre.d\90-nginx_config.sh"

"sed -i -e "s|access_log .*$|error_log /var/log/nginx/access.log;|"  /etc/nginx/sites-available/10_generic.conf;"

should be 

"sed -i -e "s|access_log .*$|access_log /var/log/nginx/access.log  cachelog;|"  /etc/nginx/sites-available/10_generic.conf;"

 

I'm guessing this line is only used to format the log, with the switch in file names I'm not sure how all the info still ended up in the access.log file with this typo. The " cachelog" bit is also needed for the "hit" and "miss" to show up, not sure how that works, but hey.

If you take a look at https://github.com/lancachenet/generic/blob/master/overlay/etc/nginx/sites-available/10_generic.conf from the original repo, there it is, cachelog is also mentioned in 10_log_format.conf, hence my assumption above.

 

This is what it looks like now, hopefully you agree and can make the changes.

[steam] 192.168.2.81 / - - - [06/Jul/2020:23:34:02 +0100] "GET /depot/468801/chunk/58fe09c302f87a52e82d56f294fcddcdce9256db HTTP/1.1" 200 346048 "-" "Valve/Steam HTTP Client 1.0" "HIT" "cache32-lhr1.steamcontent.com" "-"

[steam] 192.168.2.81 / - - - [06/Jul/2020:23:34:03 +0100] "GET /depot/468801/chunk/d3397140b02f7be618df6f135c55a9f92ef8b700 HTTP/1.1" 200 257232 "-" "Valve/Steam HTTP Client 1.0" "HIT" "cache34-lhr1.steamcontent.com" "-"

[steam] 192.168.2.81 / - - - [06/Jul/2020:23:34:03 +0100] "GET /depot/468801/chunk/166a3a8d57fa0cae1b889756b8282cc6188d44a6 HTTP/1.1" 200 180368 "-" "Valve/Steam HTTP Client 1.0" "HIT" "cache29-lhr1.steamcontent.com" "-"

[steam] 192.168.2.81 / - - - [06/Jul/2020:23:34:03 +0100] "GET /depot/468801/chunk/7cf49b9291e42c51fb708274b680698e048b95e5 HTTP/1.1" 200 110304 "-" "Valve/Steam HTTP Client 1.0" "HIT" "cache29-lhr1.steamcontent.com" "-"

[steam] 192.168.2.81 / - - - [06/Jul/2020:23:34:03 +0100] "GET /depot/468801/chunk/6155e37cd60d85844512ef68ac24858d748e8e77 HTTP/1.1" 200 222608 "-" "Valve/Steam HTTP Client 1.0" "HIT" "cache29-lhr1.steamcontent.com" "-"

[steam] 192.168.2.81 / - - - [06/Jul/2020:23:34:03 +0100] "GET /depot/468801/chunk/977085c8be56693dc10855b96fc5cdbb6d0780c8 HTTP/1.1" 200 248016 "-" "Valve/Steam HTTP Client 1.0" "HIT" "cache34-lhr1.steamcontent.com" "-"

[steam] 192.168.2.81 / - - - [06/Jul/2020:23:34:40 +0100] "GET /depot/402/manifest/5794932327719644273/5 HTTP/1.1" 200 23959 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache29-lhr1.steamcontent.com" "-"

[steam] 192.168.2.81 / - - - [06/Jul/2020:23:34:41 +0100] "GET /depot/409/manifest/5129103634722163296/5 HTTP/1.1" 200 261690 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache34-lhr1.steamcontent.com" "-"

[steam] 192.168.2.81 / - - - [06/Jul/2020:23:34:41 +0100] "GET /depot/410/manifest/106212190778449766/5 HTTP/1.1" 200 40659 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache29-lhr1.steamcontent.com" "-"

[steam] 192.168.2.81 / - - - [06/Jul/2020:23:34:42 +0100] "GET /depot/401/manifest/4679553557812664192/5 HTTP/1.1" 200 278270 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache34-lhr1.steamcontent.com" "-"

[steam] 192.168.2.81 / - - - [06/Jul/2020:23:34:42 +0100] "GET /depot/402/chunk/d46be629cb9745ba144b304ecc994d28fda75b16 HTTP/1.1" 200 58640 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache29-lhr1.steamcontent.com" "-"

[steam] 192.168.2.81 / - - - [06/Jul/2020:23:34:42 +0100] "GET /depot/402/chunk/7d5fed3423cae872d24583fe959bea6093069bb2 HTTP/1.1" 200 25936 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache29-lhr1.steamcontent.com" "-"

[steam] 192.168.2.81 / - - - [06/Jul/2020:23:34:42 +0100] "GET /depot/402/chunk/fc5e18833bc253175264cf6945727ac7aee23ef9 HTTP/1.1" 200 102000 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache34-lhr1.steamcontent.com" "-"

[steam] 192.168.2.81 / - - - [06/Jul/2020:23:34:42 +0100] "GET /depot/402/chunk/562bd66f337d5221d92795c0535af5b404d5477f HTTP/1.1" 200 71552 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache32-lhr1.steamcontent.com" "-"

[steam] 192.168.2.81 / - - - [06/Jul/2020:23:34:43 +0100] "GET /depot/402/chunk/de379329a45140c382ccce900ad7e6945c4e437a HTTP/1.1" 200 340832 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache29-lhr1.steamcontent.com" "-"

[steam] 192.168.2.81 / - - - [06/Jul/2020:23:34:43 +0100] "GET /depot/402/chunk/c2e252f0bc5b73a09e325abba621f4e4c5389ce8 HTTP/1.1" 200 354240 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache34-lhr1.steamcontent.com" "-"

[steam] 192.168.2.81 / - - - [06/Jul/2020:23:34:43 +0100] "GET /depot/402/chunk/7be35ebc6a89189381f435b4cffd547a75c72a73 HTTP/1.1" 

 

Edited by Supermillhouse
Link to comment
4 hours ago, Supermillhouse said:

Typo in "overlay\hooks\supervisord-pre.d\90-nginx_config.sh"

"sed -i -e "s|access_log .*$|error_log /var/log/nginx/access.log;|"  /etc/nginx/sites-available/10_generic.conf;"

should be 

"sed -i -e "s|access_log .*$|access_log /var/log/nginx/access.log  cachelog;|"  /etc/nginx/sites-available/10_generic.conf;"

 

 

EXCELLENT FIND!

 

Just as a note, I also had to "fix" the /etc/nginx/sites-available/10_generic.conf you referenced in the link, as mine had the error_log listed twice.  I too am now seeing proper "HIT" and "MISS"!  Thanks for the info!

Link to comment
7 hours ago, jzawacki said:

EXCELLENT FIND!

 

Just as a note, I also had to "fix" the /etc/nginx/sites-available/10_generic.conf you referenced in the link, as mine had the error_log listed twice.  I too am now seeing proper "HIT" and "MISS"!  Thanks for the info!

Yea, from what i understand the file i have identified, edits the file you mention, and the typo causes the access_log line to be replaced with error_log so you end up with 2 error_log lines. :) Because the way docker works you are correct, you have to edit the 10_generic.conf file until josh.5 changes the template. Did you have to restart nginx for it to pick up the change if you were editing inside the running docker, i didn't test that far as i was editing docker files on the outside?

 

Also, @Josh.5 does the "root" folder in the git contribute to the docker image or is it just for your records?

Edited by Supermillhouse
Link to comment
21 hours ago, Supermillhouse said:

Also, @Josh.5 does the "root" folder in the git contribute to the docker image or is it just for your records?

The root folder needs to be removed. As much as it pains me to say it, this docker is not very well put together. I made it with the intent of just spending an hour or 2 getting it working and then leaving it there. So its very slapdash. Sorry. 

 

I committed this change to master this morning. So it should be available to you all now.

  • Thanks 2
Link to comment

Hello all,

 

I'm having issue's running lancache... I cannot seem to get any connection to the lancache container.

My UnRaid server is running on 192.168.102.200.

I have given lancache a dedicated IP (192.168.102.202) in my LAN range (192.168.102.0/23), but connecting isn't possible.

Network type is set to Custom: br0

If I ping the IP from a random system on my network I never receive a reply, but when checking with arp -a I do see the mac address beloning to the container (checked with ifconfig from within the container).

Is there anything I should check (firewall on the unraid server or something like that)? I didn't configure any firewall rules myself (if there is a firewall on Unraid as I don't see anything about it in the menu.

 

Thank you for your time.

Link to comment
On 7/8/2020 at 6:49 AM, Josh.5 said:

The root folder needs to be removed. As much as it pains me to say it, this docker is not very well put together. I made it with the intent of just spending an hour or 2 getting it working and then leaving it there. So its very slapdash. Sorry. 

 

I committed this change to master this morning. So it should be available to you all now.

Thanks for keeping things up to date, but... Since the update i'm not able to login for Steam or Epic, and games won't download/update for origin and uplay. :S 
Windows update cache seems to be working, I have no other services in use to test other pakages.

Also; previous downloaded cache seems to be (partly?) erased, although files where only 3 months old (MAX_AGE was set to 700 days).

Is there a way to downgrade to the older version?


Greetings

Joris

Edit:

The problem I had was with a single windows pc on the network, which network settings are broken,... even though dns is set to Lancache, it does not use lancache when downloading games.

To be embarrasing truthfull: I only tried another pc after my girlfriend pitched that idea... I down know why I didnt try that, I was so sure that that could not be the problem, as the settings where correct... but windows... 

Edited by NeoJoris
Unrelated Problem
Link to comment
  • 2 weeks later...

Hello everyone. 

 

I apologise in advance for what may be a simplistic question having just set up LanCache. 

 

I was hoping to ask for a recommendation based on users' experiences. I have been looking at LanCache as my solution to move Steam files off a spinner HDD and upgrade to a smaller SSD. I thought this was the solution, but being a single user in the household I realise that it may be excessive. Is LanCache suitable or would using the built in mover functionality in Steam better serve the purpose. The aim is to off-load as much as possible and reinstall if required. 

 

I do appreciate that LanCache can work with multiple gaming platforms so there is the further appeal. 

 

Finally, is there a way to populate the LanCache folder from existing installations without having to manually download each game again. 

 

-Alex

Link to comment
On 6/20/2020 at 7:10 PM, dansonamission said:

Have install the lancache-bundle this evening, DNS is working fine.

 

When we try and download a game on Steam, all cpu cores go straight to 100% on unraid and nothing actually downloads.

 

The cache directory is empty and steam says downloading with no progress. Cancelling the download, the cpu goes back to idle.

I'm having similar issue. I cached Fallout 4 on Steam. I can download it at approx. 8.4MB/s when not cached. It's averaging 15MB/s when cached, but it pegs all CPU cores at 100%.

 

It's a weak CPU (AMD FX8150), but even so this seems disproportionate. Unraid 6.7.2.

 

EDIT:

 

OK, so this was quite a journey to resolve.

 

1. Hypothesised that the cache slice size of 1MB was likely too low, resulting in lots of read requests saturating CPU.

2. Added a custom variable to the lancache-bundle docker in unRAID GUI. name = CACHE_SLICE_SIZE, key = CACHE_SLICE_SIZE, value = 25m

3. Delete CONFIGHASH file in the configured data folder.

 

Ran into problems here. Running Steam kept throwing errors about not being able to log me in. Managed to get logged in after some time and a reboot.

 

Tried downloading games. Very slow downloads. Cache not being created. Lots of cache misses in the log.

 

Reboot. Check DNS. Try more downloads. Curse. Change CACHE_SLICE_SIZE to 8m in case 25m was just too large for some reason. Reboot. Reboot server.

 

After some time, I noted that the folder structure in the actual cache data folder has actually now appeared (was previously empty).

 

Great success. Able to download Cuphead at full speed from internet. Uninstall. Install. Download now >100MB (on my 1Gb LAN).

 

 

Edited by DigitalStefan
Link to comment

(not a support request)

 

I made a fork of this lancache repo, and added the ability to define custom DNS entries in the docker config.

Its simple and super jank, but If you really want to for example give your NAS a local DNS name, but don't want to spin up a separate DNS server, you can use this 2 in 1 and save resources.

 

Sidenote: I didn't make a pull request since I don't think this is a feature that people would usually want, also the implementation is kinda lazy

 

https://github.com/permissionBRICK/lancache-bundle

Link to comment

I spoke too soon. Lancache has gone back to pegging CPU during Steam installs, which now top out at around 24MB/s. I forgot to mention in previous post that I did also upgrade to unRAID 6.8.3 near the start of the troubleshooting process.

 

What's interesting / frustrating is that whilst the web UI shows CPU usage on all cores being very high, htop from a terminal does not and the CPU usage stats on the web UI docker tab also shows very low usage by the lancache-bundle docker instance.

 

There's very little else running on the server: Plex, PiHole (which lancache-bundle gets DNS from) and lancache-bundle. No VM's running.

 

I'm going to see what happens if I set lancache-bundle to pull DNS directly from my router instead.

Edited by DigitalStefan
Link to comment

First test after setting upstream DNS to my router resulted in no spike in CPU usage and the download hit 80MB/s (over a 2GB game download).

 

I then uninstalled the game, set the upstream DNS back to the PiHole docker instance and re-downloaded the same game, which again did not spike the server CPU and the download again topped 80MB/s.

 

Really not sure what's going on.

Edited by DigitalStefan
Link to comment

I am having issues with the docker honoring the CACHE_SLICE_SIZE setting.

 

I added a setting to the Docker app for this CACHE_SLICE_SIZE variable, setting it to 8m. As expected, the docker refused to start since it detected a change to the CONFIGHASH file. I removed it from the lancache directory and the Docker started up, recreating the CONFIGHASH with:

GENERICCACHE_VERSION=2;CACHE_MODE=monolithic;CACHE_SLICE_SIZE=8m;CACHE_KEY=$cacheidentifier$uri$slice_range;

 

However, all of my cache slices are still being limited to 1m. Any ideas?

Link to comment

So, I think I've encountered an issue with the Lancache-bundle.  If the computer I download Steam games on (and presumably other things) is connected to Private Internet Access VPN, then the Lancache doesn't seem to work.

 

It looks like Lancache gets the files in the Cache folder even with the VPN, but when downloading again it doesn't get the files from the cache.

 

I monitored Terminal (docker container stats), and it doesn't work with the VPN active, but it works with out the VPN active.


With the VPN off, CPU cycles on Lancache-bundle increase as well as IO, so it's working with the VPN off.

Link to comment
8 hours ago, c0rrre said:

has someone managed to get some statistics working for the cache?

 

Something like https://github.com/zeropingheroes/lancache-elk

I have a modified docker container that I have crow bared filebeat into. I cant guarantee I will maintain it but it is there in docker hub under my user account if you feel brave enough to find and use it. It should autobuild any changes to base docker but may also brake. I currently use it so for the time being I will try to keep it working, but no promises. Statistics use the elk link you have linked.

 

You will probably need to add these additional configs to unraid to get filebeats working, change as applicable -

 

Variable - LOGSTASH_HOST - 192.168.X.X:5044 (IP to Elk VM)

Path - /mnt/user/appdata/lancache-bundle/log/reg (directory used so on a docker restart it doesnt re transmit all off the log file again)

 

Good luck

josh5, if you want me to delete this post just let me know.

Edited by Supermillhouse
Link to comment
  • 2 weeks later...
On 8/12/2020 at 4:37 PM, Supermillhouse said:

I have a modified docker container that I have crow bared filebeat into. I cant guarantee I will maintain it but it is there in docker hub under my user account if you feel brave enough to find and use it. It should autobuild any changes to base docker but may also brake. I currently use it so for the time being I will try to keep it working, but no promises. Statistics use the elk link you have linked.

 

You will probably need to add these additional configs to unraid to get filebeats working, change as applicable -

 

Variable - LOGSTASH_HOST - 192.168.X.X:5044 (IP to Elk VM)

Path - /mnt/user/appdata/lancache-bundle/log/reg (directory used so on a docker restart it doesnt re transmit all off the log file again)

 

Good luck

josh5, if you want me to delete this post just let me know.

wait how do you install that 

i open a terminal but during step 3 i don't know what to do 

Edited by James_Darkness
Link to comment
  • 1 month later...

Just set this up and have a strange issue

 

If i use lancache im getting sub 2MB/s downloads in battle.net launcher but if i bypass it i get my usual 15-20MB/s

Any suggestions?

 

quick test in steam does not seem to have the same issues

 

thanks!

 

 

edit: looks like its recommended to change the slice size for blizzard - where is the config files for this docker?

Edited by macmanluke
Link to comment
  • 2 weeks later...
On 10/18/2020 at 5:01 AM, zzsprade said:

I have found that the sub-folders to the Cache folder have permissions that are preventing me from viewing their properties or deleting them.

 

Is there any advice on the process on how to delete these folders?

 

-Alex

On UNRAID's Tools page, run "New Permissions" against the lancache share. You'll need to repeat this process if any new files are created by the app. Alternately, you can telnet in and run MC.

Edited by jbartlett
Link to comment
On 10/20/2020 at 3:28 AM, jbartlett said:

On UNRAID's Tools page, run "New Permissions" against the lancache share. You'll need to repeat this process if any new files are created by the app. Alternately, you can telnet in and run MC.

Thank you John. A well detailed and easy solution. I was not successful with the New Permissions tool (it continued to be loading), but Telnet and MC did the trick beautifully. Thank you again. 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.