Jump to content
cheesemarathon

[Support] cheesemarathons repo

493 posts in this topic Last Reply

Recommended Posts

Hi @humanjhawkins Im glad your interested!

I hope my answers help

 

  1. Yes, the cache will be persistent across docker stops, provided the cache is mounted outside of the container and remounted in the same location. If you set the container up in unRAID, this is taken care for you.
  2. As far as I am aware, this value can be set as large as you like. Do note though that CACHE_DISK_SIZE will still limit what is cached. I'm not sure if it will just stop caching when the max size is reached or if it starts to delete old files. So make sure that you set CACHE_DISK_SIZE appropriately.
  3. It is certainly possible to copy between instances of steamcache. As for copying from a different caching service, I do not know. I would give it a try and see if it serves up your cached data.

I hope that helps. Have a look over at the github pages for the steamcache project as the info there applies here too. I didn't create steamcache you see. I just maintain it here for unRAID. mlebjerg did however combine all the required docker containers into one image for easier deployment.

Share this post


Link to post
10 hours ago, cheesemarathon said:

Hi @humanjhawkins Im glad your interested!

<cut>

   2. Do note though that CACHE_DISK_SIZE will still limit what is cached. I'm not sure if it will just stop caching when the max size is reached or if it starts to delete old files. So make sure that you set CACHE_DISK_SIZE appropriately.

<cut>

FYI on the above, info at https://www.nginx.com/blog/nginx-caching-guide/ says: 

  • max_size sets the upper limit of the size of the cache<cut>. It is optional; not specifying a value allows the cache to grow to use all available disk space. When the cache size reaches the limit, a process called the cache manager removes the files that were least recently used to bring the cache size back under the limit.

NOTE (for others... Guessing cheesemarathon knows this. Ha!) : I believe SteamCacheBundle docker sets the Nginx max_size param based on the input of MAX_CACHE_SIZE. Although max_size is optional in Nginx, MAX_CACHE_SIZE may not be optional in this Docker implementation.

Thanks!

Share this post


Link to post
1 hour ago, humanjhawkins said:

FYI on the above, info at https://www.nginx.com/blog/nginx-caching-guide/ says: 

  • max_size sets the upper limit of the size of the cache<cut>. It is optional; not specifying a value allows the cache to grow to use all available disk space. When the cache size reaches the limit, a process called the cache manager removes the files that were least recently used to bring the cache size back under the limit.

NOTE (for others... Guessing cheesemarathon knows this. Ha!) : I believe SteamCacheBundle docker sets the Nginx max_size param based on the input of MAX_CACHE_SIZE. Although max_size is optional in Nginx, MAX_CACHE_SIZE may not be optional in this Docker implementation.

Thanks!

 

You sure do! The bootstrap file which is run during startup sets the input of the variable, but if there is none it will look like this:

 

sed -i "s/CACHE_DISK_SIZE/${CACHE_DISK_SIZE}/" /etc/nginx/sites-available/generic.conf (Bootstrap)

--->

proxy_cache_valid 200 206 CACHE_MAX_AGE; (generic.conf)

Which nginx will NOT like! ;) because it will either be null or CACHE_MAX_AGE

 

You can find all the .conf files used on github :) 

 

Edited by mlebjerg

Share this post


Link to post

Hey guys,

 

Having a little trouble getting this working.

 

I have SteamCache configured to use the host's IP (192.168.1.100). I've tried setting a fixed IP for the container itself as well, that doesn't seem to chance things.

 

When I run bootstrap.sh I'm shown:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
2018/07/29 18:38:35 [emerg] 107#107: bind() to 0.0.0.0:80 failed (98: Address in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use)
2018/07/29 18:38:35 [emerg] 107#107: bind() to 0.0.0.0:80 failed (98: Address in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use)
2018/07/29 18:38:35 [emerg] 107#107: bind() to 0.0.0.0:80 failed (98: Address in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use)
2018/07/29 18:38:35 [emerg] 107#107: bind() to 0.0.0.0:80 failed (98: Address in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use)
2018/07/29 18:38:35 [emerg] 107#107: bind() to 0.0.0.0:80 failed (98: Address in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use)
2018/07/29 18:38:35 [emerg] 107#107: still could not bind()
nginx: [emerg] still could not bind()
/scripts #

 

Running: 

sudo lsof -i -P -n | grep LISTEN

I don't see port 80 in use anywhere. 

 

I've made sure to run the unRAID GUI off 81/444 as well. I've double-checked all my containers and VMs; nothing should be using that port. Is there something stupid I'm overlooking here?

Share this post


Link to post
On 7/29/2018 at 6:41 PM, -Daedalus said:

Hey guys,

 

Having a little trouble getting this working.

 

I have SteamCache configured to use the host's IP (192.168.1.100). I've tried setting a fixed IP for the container itself as well, that doesn't seem to chance things.

 

When I run bootstrap.sh I'm shown:


nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
2018/07/29 18:38:35 [emerg] 107#107: bind() to 0.0.0.0:80 failed (98: Address in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use)
2018/07/29 18:38:35 [emerg] 107#107: bind() to 0.0.0.0:80 failed (98: Address in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use)
2018/07/29 18:38:35 [emerg] 107#107: bind() to 0.0.0.0:80 failed (98: Address in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use)
2018/07/29 18:38:35 [emerg] 107#107: bind() to 0.0.0.0:80 failed (98: Address in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use)
2018/07/29 18:38:35 [emerg] 107#107: bind() to 0.0.0.0:80 failed (98: Address in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use)
2018/07/29 18:38:35 [emerg] 107#107: still could not bind()
nginx: [emerg] still could not bind()
/scripts #

 

Running: 


sudo lsof -i -P -n | grep LISTEN

I don't see port 80 in use anywhere. 

 

I've made sure to run the unRAID GUI off 81/444 as well. I've double-checked all my containers and VMs; nothing should be using that port. Is there something stupid I'm overlooking here?

 

Hey @-Daedalus

Sorry it's taken me ages to reply.

 

It looks like you have checked all the obvious stuff so ill get @mlebjerg to chime in here as well.

When you say you run bootstrap.sh, is that when you start the container or are you manually running bootstrap.sh. Bootstrap.sh runs when the container starts so there is no need to manually run it. Running it again would probably create a similar output to above.

If that's not the case, can we see a screenshot of your config for SteamCache? Also I'd suggest stopping all your VM's and restarting SteamCache, just in case there was something running your unaware of.

Share this post


Link to post

Hey cheese, thanks for the reply!

 

The bootstrap output from before was me manually running it when the container was started. Didn't realise it was run anyway.

 

I changed nothing, restarted the container (for the hundredth time) and it the DNS test in PowerShell works!

... but now when I change a machine's DNS to point to SteamCache, I get a no internet error when trying to download a game. My machine still has internet access though, so I think it's just the Steam servers themselves that can't resolve. I can't login to Steam in this state either.

 

Container config attached. I also tried changing LANCACHE_IP to match the container at x.x.x.110 (the wording was a little unclear which one I should use) but everything's the same.

 

Any help would be great!

 

steamcache_config.png

Share this post


Link to post

@-Daedalus You definitely need to set LANCACHE_IP to 192.168.1.110

I would also remove all the "true" values you entered to disable some of the caching services for now to make sure they are not causing issues. You can set them back to what you like once it is all working. DO NOT remove the "true" value for USE_GENERIC_CACHE. That will break it ?

Make sure to set your DNS settings on your gaming machine to 192.168.1.110. Also make sure that that is the only IP you set. Try that and see how it go's. If that doesn't work let me know and post the output log from the container (best to use paste bin if it's long).

Share this post


Link to post

I have recently setup the VNCWebbrowser container and it works great but the only issue is that there is no persistence of settings etc between container restarts. I presume that as it does not create a directory in appdata this is the intended design?

Share this post


Link to post

Hi @bwelsher!

I'm afraid that persistence of settings is not an option with the current build of the docker image. I'm also not the author of the image. I just created the template for use with unRAID. If you require persistent settings, I suggest that you open an issue with the author of the image here. Let me know if you do open an issue so I can keep an eye on it.

Share this post


Link to post
17 minutes ago, bwelsher said:

@cheesemarathon ah sorry I hadn't realised that was the setup here. I've created an issue on the github page so let's see if anything comes of that. Cheers.

That's ok. Alot of devs on the forums here produce their own images. I have a couple on here. VNCbrowser just happens to not be one of them. Thanks for the issue, I'll keep an eye on it. 

Share this post


Link to post

Hey cheese, thanks for the help! I did try .110 this morning before work for the steamcache_ip, and it didn't seem to change things, but trying it again now worked. I'm getting "MISS / HIT" output in the log (also for a Windows update that got pulled down at the same time).

 

The only odd thing is that it doesn't seem to keep everything.

I install a 1GB game, and I only have 600MB of cached files. I install another 1GB game, and the cache grows to 1.2GB. I'm not even sure how much of it is Steam and how much is from the Windows update. Any idea what might be happening? Looking for content on a different endpoint, maybe?

 

Anyway, it's working, so that's a plus. I'll start re-enabling bits tomorrow to see if it still works with the 'true' flags set (no reason why it wouldn't, but hey).

Share this post


Link to post

Anyone else with comcast still having their caches bypassed?

 

[2018-08-07 21:12:48] HTTP (CDN,64) - edge.steam-dns.top.comcast.net (69.241.88.210:443 / 69.241.88.210:443, host: edge.steam-dns.top.comcast.net): Closing connection
[2018-08-07 21:12:48] HTTP (CDN,64) - edge.steam-dns.top.comcast.net (69.241.88.210:443 / 69.241.88.210:443, host: edge.steam-dns.top.comcast.net): Closing connection
[2018-08-07 21:12:48] HTTP (CDN,64) - edge.steam-dns.top.comcast.net (69.241.88.210:443 / 69.241.88.210:443, host: edge.steam-dns.top.comcast.net): Closing connection
[2018-08-07 21:12:48] HTTP (CDN,64) - edge.steam-dns-2.top.comcast.net (69.241.32.70:443 / 69.241.32.70:443, host: edge.steam-dns-2.top.comcast.net): Closing connection

 

Share this post


Link to post

@cheesemarathon I just tried your bitwarden docker...after installing I could not get the webui to come up.  I checked the logs then the port mappings... everything looked good...then I opened the advanced settings in the docker and it looks like you have a typo I the port number for the webui....once I corrected this to match your mapping all is well.

Share this post


Link to post

Hi

 

I am having some issues getting the steam cache bundle going.

Steps to get where I am at now:

- set unraid ports to something else than 80 and 443

- install steamcachebundle from community applications

- cleared download cache in steam

- in windows manually set the dns ip to '192.168.1.90' (no secondary server)

- in windows disable ipv6

 

config:

Capture.thumb.PNG.ee19ccac9bc142fcc186ddc5e6367d31.PNG

 

 

Unraid version: Version 6.5.3 2018-06-12

 

Powershell output:

PS C:\WINDOWS\system32> Resolve-DnsName steamcache.cs.steampowered.com -server 192.168.1.90

Name                                           Type   TTL   Section    IPAddress
----                                           ----   ---   -------    ---------
steamcache.cs.steampowered.com                 A      600   Answer     192.168.1.90

Name      : cs.steampowered.com
QueryType : NS
TTL       : 600
Section   : Authority
NameHost  : ns1.cs.steampowered.com

ns1.cs.steampowered.com                        A      600   Additional 192.168.1.90

 

 

I do not get any hits in the /data/logs/access.log.

 

I appreciate any help :) 

Share this post


Link to post

How u try if it works? Install some games? Did you restartet?

 

If the powershell outputs the correct ip, then problem can be only on steams (client) side.

 

Are you sure your data folder for steam isnt increasing?

Edited by nuhll

Share this post


Link to post

Hi there!

As @nuhll said, there if powershell gives the correct IP there should't be a problem :)

 

Sometimes it takes some time for the client to get the hint. How long have you waited?

 

Also if the steamcache IP is different from UnRaid's IP there is no reason to change its ports ;) 

Share this post


Link to post

yeah, also true.

 

btw, i mean the cache folder from steamcache, if that is increasing or not.

 

Because not all will get cached, for me it happend too often that only 1/4 of a game was cache (able) , thats why i removed it.

Share this post


Link to post

I cannot get bitwarden to load at all. I've deleted and reinstalled multiple times and i'm only using default settings, so not sure what exactly is going on. Any assistance would be much apprecaited. The Docker never starts and this is all I see in the logs:

 

First run:

JWT keys don't exist, checking if OpenSSL is available...
OpenSSL detected, creating keys...
Keys created correctly.
thread 'main' panicked at 'Can't run migrations: QueryError(DatabaseError(__Unknown, "disk I/O error"))', libcore/result.rs:945:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.

 

Subsequent Runs:

thread 'main' panicked at 'Failed to turn on WAL: DatabaseError(__Unknown, "disk I/O error")', libcore/result.rs:945:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.
thread 'main' panicked at 'Failed to turn on WAL: DatabaseError(__Unknown, "disk I/O error")', libcore/result.rs:945:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.
thread 'main' panicked at 'Failed to turn on WAL: DatabaseError(__Unknown, "disk I/O error")', libcore/result.rs:945:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.

Edited by Tweak3D

Share this post


Link to post

Hi @nuhll, @mlebjerg

 

Thanks for the tips!

I think i have given it half an hour or something, i'll leave it on while i'm sleeping to see if it changes something.

 

I have tried to be as thorough as possible when testing it:

windows:
1. check the hosts file, nothing strange there
2. set dns ip to 192.168.1.90
3. flushdns
4. clear download cache on steam
5. clear the steam content log



unraid:
1. compute steamcache share: 0B
2. make sure that steamcache share is not using the cache drive
3. clear the logfiles
4. open 2 command windows that tail the /data/logs/access.log and error.log



windows:
1. check dns in powershell --> gives me 192.168.1.90
2. start a download on steam

unraid:
1. check the steamcache share size: 0 B



windows:
1. uninstall game
2. reinstall game
3. try another game



dd-wrt router:
1. check if there is traffic going outside of the LAN: yes



unraid:

1. observe the tails that i had started on error.log and access.log. they are empty

2. double check on the docker container, logs are empty

 

 

I have added the steam content log, maybe that can help.

 

Are there other ways I can test if it is working properly? 

 

Thanks for the help!

 

 

this is something i found in the steam connection log:


[2018-08-21 23:51:35] [0,0] Existing WebSocket ping data (cm-03-ams1.cm.steampowered.com:27020) results: 52.0000ms + load 66.0000 (757 seconds ago)
[2018-08-21 23:51:35] [0,0] Existing WebSocket ping data (cm-04-ams1.cm.steampowered.com:27020) results: 52.0000ms + load 63.0000 (757 seconds ago)
[2018-08-21 23:51:35] [0,0] Existing WebSocket ping data (CM02-LUX.cm.steampowered.com:27021) results: 50.0000ms + load 61.0000 (719 seconds ago)
[2018-08-21 23:51:35] [0,0] Existing WebSocket ping data (CM01-FRA.cm.steampowered.com:27021) results: 49.0000ms + load 63.0000 (719 seconds ago)
[2018-08-21 23:51:35] [0,0] Existing WebSocket ping data (CM02-STO.cm.steampowered.com:27021) results: 49.0000ms + load 61.0000 (751 seconds ago)
[2018-08-21 23:51:35] [0,0] Existing WebSocket ping data (cm-02-ams1.cm.steampowered.com:443) results: 39.0000ms + load 62.0000 (756 seconds ago)
[2018-08-21 23:51:35] [0,0] Existing WebSocket ping data (cm-02-ams1.cm.steampowered.com:27021) results: 39.0000ms + load 63.0000 (756 seconds ago)
[2018-08-21 23:51:35] [0,0] Existing WebSocket ping data (cm-02-ams1.cm.steampowered.com:27020) results: 32.0000ms + load 63.0000 (756 seconds ago)
[2018-08-21 23:51:35] [0,0] PingWebSocketCM() (cm-03-ams1.cm.steampowered.com:443) starting...
[2018-08-21 23:51:35] [0,0] PingWebSocketCM() (CM03-STO.cm.steampowered.com:443) starting...
[2018-08-21 23:51:35] [0,0] PingWebSocketCM() (CM02-FRA.cm.steampowered.com:27021) starting...
[2018-08-21 23:51:35] [0,0] PingWebSocketCM() (CM06-IAD.cm.steampowered.com:27021) starting...
[2018-08-21 23:51:36] [0,0] PingWebSocketCM() (cm-03-ams1.cm.steampowered.com:443) failed talking to cm (timeout/neterror - OK)
[2018-08-21 23:51:36] [0,0] PingWebSocketCM() (CM06-IAD.cm.steampowered.com:27021) failed talking to cm (timeout/neterror - Invalid)
[2018-08-21 23:51:36] [0,0] PingWebSocketCM() (CM02-FRA.cm.steampowered.com:27021) failed talking to cm (timeout/neterror - Invalid)
[2018-08-21 23:51:36] [0,0] PingWebSocketCM() (CM03-STO.cm.steampowered.com:443) failed talking to cm (timeout/neterror - Invalid)
[2018-08-21 23:51:36] [1,3] Connect() starting connection (eNetQOSLevelHigh, cm-02-ams1.cm.steampowered.com:27020, WebSocket)

 

 

content_log.txt

Share this post


Link to post

@mrcake

Weird! What does the docker logs say, is everything running?

 

Just to make sure, have you tried changing the dns on your router? Or are you just testing it in windows?

It could help since there probably are some DNSSec settings on the router that could interfere.

And lastly are you running any VPN on the windows client?

 

Share this post


Link to post

@mlebjerg

 

I had an adblocker running on dd wrt, disabled that (did not help)

Steam is failing its connections now, so that is better i think?

 

No vpns active or used on my network.

 

Here is the docker log:

root@Tower:~# docker logs SteamCacheBundle
Running bootstrap.sh...
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
Bind9 {
Enabling cache for Blizzard
2018-08-20 14:31:57 Failed to set file handle limit: Operation not permitted
Enabling cache for Frontier
Enabling cache for Origin
Enabling cache for Riot
Enabling cache for Steam
Enabling cache for Uplay
Enabling cache for Windows
Enabling custom DNS forwarder
bootstrap finished.
checking Bind9 config
Running Bind9
Running bootstrap.sh...
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
Bind9 {
Enabling cache for Blizzard
2018-08-20 15:24:04 Failed to set file handle limit: Operation not permitted
Enabling cache for Frontier
Enabling cache for Origin
Enabling cache for Riot
Enabling cache for Steam
Enabling cache for Uplay
Enabling cache for Windows
Enabling custom DNS forwarder
bootstrap finished.
checking Bind9 config
Running Bind9
Running bootstrap.sh...
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
Bind9 {
Enabling cache for Blizzard
2018-08-21 08:39:09 Failed to set file handle limit: Operation not permitted
Enabling cache for Frontier
Enabling cache for Origin
Enabling cache for Riot
Enabling cache for Steam
Enabling cache for Uplay
Enabling cache for Windows
Enabling custom DNS forwarder
bootstrap finished.
checking Bind9 config
Running Bind9
Running bootstrap.sh...
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
Bind9 {
Enabling cache for Blizzard
2018-08-21 13:31:56 Failed to set file handle limit: Operation not permitted
Enabling cache for Frontier
Enabling cache for Origin
Enabling cache for Riot
Enabling cache for Steam
Enabling cache for Uplay
Enabling cache for Windows
Enabling custom DNS forwarder
bootstrap finished.
checking Bind9 config
Running Bind9
Running bootstrap.sh...
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
Bind9 {
Enabling cache for Blizzard
2018-08-21 15:31:52 Failed to set file handle limit: Operation not permitted
Enabling cache for Frontier
Enabling cache for Origin
Enabling cache for Riot
Enabling cache for Steam
Enabling cache for Uplay
Enabling cache for Windows
Enabling custom DNS forwarder
bootstrap finished.
checking Bind9 config
Running Bind9
Running bootstrap.sh...
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
Bind9 {
Enabling cache for Blizzard
2018-08-21 15:33:55 Failed to set file handle limit: Operation not permitted
Enabling cache for Frontier
Enabling cache for Origin
Enabling cache for Riot
Enabling cache for Steam
Enabling cache for Uplay
Enabling cache for Windows
Enabling custom DNS forwarder
bootstrap finished.
checking Bind9 config
Running Bind9

Share this post


Link to post

Again, how u test its not working?


Do you download a game in steam client? (does it download?)

Edited by nuhll

Share this post


Link to post

It is working now.

 

What i did now is disable the adblock on dd-wrt, after that steam had some trouble connecting and I left the pc on while I was asleep.

In the morning the downloads started and I am now observing that the share is growing in size, access log is showing the misses.

and now tried to reinstall the game and that hit the cache.

 

Thanks for the help

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.