cheesemarathon Posted July 24, 2018 Author Share Posted July 24, 2018 Hi @humanjhawkins Im glad your interested! I hope my answers help Yes, the cache will be persistent across docker stops, provided the cache is mounted outside of the container and remounted in the same location. If you set the container up in unRAID, this is taken care for you. As far as I am aware, this value can be set as large as you like. Do note though that CACHE_DISK_SIZE will still limit what is cached. I'm not sure if it will just stop caching when the max size is reached or if it starts to delete old files. So make sure that you set CACHE_DISK_SIZE appropriately. It is certainly possible to copy between instances of steamcache. As for copying from a different caching service, I do not know. I would give it a try and see if it serves up your cached data. I hope that helps. Have a look over at the github pages for the steamcache project as the info there applies here too. I didn't create steamcache you see. I just maintain it here for unRAID. mlebjerg did however combine all the required docker containers into one image for easier deployment. 1 Quote Link to comment
humanjhawkins Posted July 24, 2018 Share Posted July 24, 2018 10 hours ago, cheesemarathon said: Hi @humanjhawkins Im glad your interested! <cut> 2. Do note though that CACHE_DISK_SIZE will still limit what is cached. I'm not sure if it will just stop caching when the max size is reached or if it starts to delete old files. So make sure that you set CACHE_DISK_SIZE appropriately. <cut> FYI on the above, info at https://www.nginx.com/blog/nginx-caching-guide/ says: max_size sets the upper limit of the size of the cache<cut>. It is optional; not specifying a value allows the cache to grow to use all available disk space. When the cache size reaches the limit, a process called the cache manager removes the files that were least recently used to bring the cache size back under the limit. NOTE (for others... Guessing cheesemarathon knows this. Ha!) : I believe SteamCacheBundle docker sets the Nginx max_size param based on the input of MAX_CACHE_SIZE. Although max_size is optional in Nginx, MAX_CACHE_SIZE may not be optional in this Docker implementation. Thanks! Quote Link to comment
mlebjerg Posted July 24, 2018 Share Posted July 24, 2018 (edited) 1 hour ago, humanjhawkins said: FYI on the above, info at https://www.nginx.com/blog/nginx-caching-guide/ says: max_size sets the upper limit of the size of the cache<cut>. It is optional; not specifying a value allows the cache to grow to use all available disk space. When the cache size reaches the limit, a process called the cache manager removes the files that were least recently used to bring the cache size back under the limit. NOTE (for others... Guessing cheesemarathon knows this. Ha!) : I believe SteamCacheBundle docker sets the Nginx max_size param based on the input of MAX_CACHE_SIZE. Although max_size is optional in Nginx, MAX_CACHE_SIZE may not be optional in this Docker implementation. Thanks! You sure do! The bootstrap file which is run during startup sets the input of the variable, but if there is none it will look like this: sed -i "s/CACHE_DISK_SIZE/${CACHE_DISK_SIZE}/" /etc/nginx/sites-available/generic.conf (Bootstrap) ---> proxy_cache_valid 200 206 CACHE_MAX_AGE; (generic.conf) Which nginx will NOT like! because it will either be null or CACHE_MAX_AGE You can find all the .conf files used on github Edited July 24, 2018 by mlebjerg 1 Quote Link to comment
-Daedalus Posted July 29, 2018 Share Posted July 29, 2018 Hey guys, Having a little trouble getting this working. I have SteamCache configured to use the host's IP (192.168.1.100). I've tried setting a fixed IP for the container itself as well, that doesn't seem to chance things. When I run bootstrap.sh I'm shown: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful 2018/07/29 18:38:35 [emerg] 107#107: bind() to 0.0.0.0:80 failed (98: Address in use) nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use) 2018/07/29 18:38:35 [emerg] 107#107: bind() to 0.0.0.0:80 failed (98: Address in use) nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use) 2018/07/29 18:38:35 [emerg] 107#107: bind() to 0.0.0.0:80 failed (98: Address in use) nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use) 2018/07/29 18:38:35 [emerg] 107#107: bind() to 0.0.0.0:80 failed (98: Address in use) nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use) 2018/07/29 18:38:35 [emerg] 107#107: bind() to 0.0.0.0:80 failed (98: Address in use) nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use) 2018/07/29 18:38:35 [emerg] 107#107: still could not bind() nginx: [emerg] still could not bind() /scripts # Running: sudo lsof -i -P -n | grep LISTEN I don't see port 80 in use anywhere. I've made sure to run the unRAID GUI off 81/444 as well. I've double-checked all my containers and VMs; nothing should be using that port. Is there something stupid I'm overlooking here? Quote Link to comment
cheesemarathon Posted August 6, 2018 Author Share Posted August 6, 2018 On 7/29/2018 at 6:41 PM, -Daedalus said: Hey guys, Having a little trouble getting this working. I have SteamCache configured to use the host's IP (192.168.1.100). I've tried setting a fixed IP for the container itself as well, that doesn't seem to chance things. When I run bootstrap.sh I'm shown: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful 2018/07/29 18:38:35 [emerg] 107#107: bind() to 0.0.0.0:80 failed (98: Address in use) nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use) 2018/07/29 18:38:35 [emerg] 107#107: bind() to 0.0.0.0:80 failed (98: Address in use) nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use) 2018/07/29 18:38:35 [emerg] 107#107: bind() to 0.0.0.0:80 failed (98: Address in use) nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use) 2018/07/29 18:38:35 [emerg] 107#107: bind() to 0.0.0.0:80 failed (98: Address in use) nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use) 2018/07/29 18:38:35 [emerg] 107#107: bind() to 0.0.0.0:80 failed (98: Address in use) nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address in use) 2018/07/29 18:38:35 [emerg] 107#107: still could not bind() nginx: [emerg] still could not bind() /scripts # Running: sudo lsof -i -P -n | grep LISTEN I don't see port 80 in use anywhere. I've made sure to run the unRAID GUI off 81/444 as well. I've double-checked all my containers and VMs; nothing should be using that port. Is there something stupid I'm overlooking here? Hey @-Daedalus Sorry it's taken me ages to reply. It looks like you have checked all the obvious stuff so ill get @mlebjerg to chime in here as well. When you say you run bootstrap.sh, is that when you start the container or are you manually running bootstrap.sh. Bootstrap.sh runs when the container starts so there is no need to manually run it. Running it again would probably create a similar output to above. If that's not the case, can we see a screenshot of your config for SteamCache? Also I'd suggest stopping all your VM's and restarting SteamCache, just in case there was something running your unaware of. Quote Link to comment
-Daedalus Posted August 7, 2018 Share Posted August 7, 2018 Hey cheese, thanks for the reply! The bootstrap output from before was me manually running it when the container was started. Didn't realise it was run anyway. I changed nothing, restarted the container (for the hundredth time) and it the DNS test in PowerShell works! ... but now when I change a machine's DNS to point to SteamCache, I get a no internet error when trying to download a game. My machine still has internet access though, so I think it's just the Steam servers themselves that can't resolve. I can't login to Steam in this state either. Container config attached. I also tried changing LANCACHE_IP to match the container at x.x.x.110 (the wording was a little unclear which one I should use) but everything's the same. Any help would be great! Quote Link to comment
cheesemarathon Posted August 7, 2018 Author Share Posted August 7, 2018 @-Daedalus You definitely need to set LANCACHE_IP to 192.168.1.110 I would also remove all the "true" values you entered to disable some of the caching services for now to make sure they are not causing issues. You can set them back to what you like once it is all working. DO NOT remove the "true" value for USE_GENERIC_CACHE. That will break it ? Make sure to set your DNS settings on your gaming machine to 192.168.1.110. Also make sure that that is the only IP you set. Try that and see how it go's. If that doesn't work let me know and post the output log from the container (best to use paste bin if it's long). Quote Link to comment
bwelsher Posted August 7, 2018 Share Posted August 7, 2018 I have recently setup the VNCWebbrowser container and it works great but the only issue is that there is no persistence of settings etc between container restarts. I presume that as it does not create a directory in appdata this is the intended design? Quote Link to comment
cheesemarathon Posted August 7, 2018 Author Share Posted August 7, 2018 Hi @bwelsher! I'm afraid that persistence of settings is not an option with the current build of the docker image. I'm also not the author of the image. I just created the template for use with unRAID. If you require persistent settings, I suggest that you open an issue with the author of the image here. Let me know if you do open an issue so I can keep an eye on it. Quote Link to comment
bwelsher Posted August 7, 2018 Share Posted August 7, 2018 @cheesemarathon ah sorry I hadn't realised that was the setup here. I've created an issue on the github page so let's see if anything comes of that. Cheers. Quote Link to comment
cheesemarathon Posted August 7, 2018 Author Share Posted August 7, 2018 17 minutes ago, bwelsher said: @cheesemarathon ah sorry I hadn't realised that was the setup here. I've created an issue on the github page so let's see if anything comes of that. Cheers. That's ok. Alot of devs on the forums here produce their own images. I have a couple on here. VNCbrowser just happens to not be one of them. Thanks for the issue, I'll keep an eye on it. Quote Link to comment
-Daedalus Posted August 7, 2018 Share Posted August 7, 2018 Hey cheese, thanks for the help! I did try .110 this morning before work for the steamcache_ip, and it didn't seem to change things, but trying it again now worked. I'm getting "MISS / HIT" output in the log (also for a Windows update that got pulled down at the same time). The only odd thing is that it doesn't seem to keep everything. I install a 1GB game, and I only have 600MB of cached files. I install another 1GB game, and the cache grows to 1.2GB. I'm not even sure how much of it is Steam and how much is from the Windows update. Any idea what might be happening? Looking for content on a different endpoint, maybe? Anyway, it's working, so that's a plus. I'll start re-enabling bits tomorrow to see if it still works with the 'true' flags set (no reason why it wouldn't, but hey). Quote Link to comment
defiant Posted August 8, 2018 Share Posted August 8, 2018 Anyone else with comcast still having their caches bypassed? [2018-08-07 21:12:48] HTTP (CDN,64) - edge.steam-dns.top.comcast.net (69.241.88.210:443 / 69.241.88.210:443, host: edge.steam-dns.top.comcast.net): Closing connection [2018-08-07 21:12:48] HTTP (CDN,64) - edge.steam-dns.top.comcast.net (69.241.88.210:443 / 69.241.88.210:443, host: edge.steam-dns.top.comcast.net): Closing connection [2018-08-07 21:12:48] HTTP (CDN,64) - edge.steam-dns.top.comcast.net (69.241.88.210:443 / 69.241.88.210:443, host: edge.steam-dns.top.comcast.net): Closing connection [2018-08-07 21:12:48] HTTP (CDN,64) - edge.steam-dns-2.top.comcast.net (69.241.32.70:443 / 69.241.32.70:443, host: edge.steam-dns-2.top.comcast.net): Closing connection Quote Link to comment
goinsnoopin Posted August 8, 2018 Share Posted August 8, 2018 @cheesemarathon I just tried your bitwarden docker...after installing I could not get the webui to come up. I checked the logs then the port mappings... everything looked good...then I opened the advanced settings in the docker and it looks like you have a typo I the port number for the webui....once I corrected this to match your mapping all is well. Quote Link to comment
cheesemarathon Posted August 12, 2018 Author Share Posted August 12, 2018 Thanks @goinsnoopin! Fixed now! Quote Link to comment
mrcake Posted August 21, 2018 Share Posted August 21, 2018 Hi I am having some issues getting the steam cache bundle going. Steps to get where I am at now: - set unraid ports to something else than 80 and 443 - install steamcachebundle from community applications - cleared download cache in steam - in windows manually set the dns ip to '192.168.1.90' (no secondary server) - in windows disable ipv6 config: Unraid version: Version 6.5.3 2018-06-12 Powershell output: PS C:\WINDOWS\system32> Resolve-DnsName steamcache.cs.steampowered.com -server 192.168.1.90 Name Type TTL Section IPAddress ---- ---- --- ------- --------- steamcache.cs.steampowered.com A 600 Answer 192.168.1.90 Name : cs.steampowered.com QueryType : NS TTL : 600 Section : Authority NameHost : ns1.cs.steampowered.com ns1.cs.steampowered.com A 600 Additional 192.168.1.90 I do not get any hits in the /data/logs/access.log. I appreciate any help Quote Link to comment
NewDisplayName Posted August 21, 2018 Share Posted August 21, 2018 (edited) How u try if it works? Install some games? Did you restartet? If the powershell outputs the correct ip, then problem can be only on steams (client) side. Are you sure your data folder for steam isnt increasing? Edited August 21, 2018 by nuhll 1 Quote Link to comment
mlebjerg Posted August 21, 2018 Share Posted August 21, 2018 Hi there! As @nuhll said, there if powershell gives the correct IP there should't be a problem Sometimes it takes some time for the client to get the hint. How long have you waited? Also if the steamcache IP is different from UnRaid's IP there is no reason to change its ports Quote Link to comment
NewDisplayName Posted August 21, 2018 Share Posted August 21, 2018 yeah, also true. btw, i mean the cache folder from steamcache, if that is increasing or not. Because not all will get cached, for me it happend too often that only 1/4 of a game was cache (able) , thats why i removed it. Quote Link to comment
Tweak3D Posted August 21, 2018 Share Posted August 21, 2018 (edited) I cannot get bitwarden to load at all. I've deleted and reinstalled multiple times and i'm only using default settings, so not sure what exactly is going on. Any assistance would be much apprecaited. The Docker never starts and this is all I see in the logs: First run: JWT keys don't exist, checking if OpenSSL is available...OpenSSL detected, creating keys...Keys created correctly.thread 'main' panicked at 'Can't run migrations: QueryError(DatabaseError(__Unknown, "disk I/O error"))', libcore/result.rs:945:5note: Run with `RUST_BACKTRACE=1` for a backtrace. Subsequent Runs: thread 'main' panicked at 'Failed to turn on WAL: DatabaseError(__Unknown, "disk I/O error")', libcore/result.rs:945:5note: Run with `RUST_BACKTRACE=1` for a backtrace.thread 'main' panicked at 'Failed to turn on WAL: DatabaseError(__Unknown, "disk I/O error")', libcore/result.rs:945:5note: Run with `RUST_BACKTRACE=1` for a backtrace.thread 'main' panicked at 'Failed to turn on WAL: DatabaseError(__Unknown, "disk I/O error")', libcore/result.rs:945:5note: Run with `RUST_BACKTRACE=1` for a backtrace. Edited August 21, 2018 by Tweak3D Quote Link to comment
mrcake Posted August 21, 2018 Share Posted August 21, 2018 Hi @nuhll, @mlebjerg Thanks for the tips! I think i have given it half an hour or something, i'll leave it on while i'm sleeping to see if it changes something. I have tried to be as thorough as possible when testing it: windows: 1. check the hosts file, nothing strange there 2. set dns ip to 192.168.1.90 3. flushdns 4. clear download cache on steam 5. clear the steam content log unraid: 1. compute steamcache share: 0B 2. make sure that steamcache share is not using the cache drive 3. clear the logfiles 4. open 2 command windows that tail the /data/logs/access.log and error.log windows: 1. check dns in powershell --> gives me 192.168.1.90 2. start a download on steam unraid: 1. check the steamcache share size: 0 B windows: 1. uninstall game 2. reinstall game 3. try another game dd-wrt router: 1. check if there is traffic going outside of the LAN: yes unraid: 1. observe the tails that i had started on error.log and access.log. they are empty 2. double check on the docker container, logs are empty I have added the steam content log, maybe that can help. Are there other ways I can test if it is working properly? Thanks for the help! this is something i found in the steam connection log: [2018-08-21 23:51:35] [0,0] Existing WebSocket ping data (cm-03-ams1.cm.steampowered.com:27020) results: 52.0000ms + load 66.0000 (757 seconds ago) [2018-08-21 23:51:35] [0,0] Existing WebSocket ping data (cm-04-ams1.cm.steampowered.com:27020) results: 52.0000ms + load 63.0000 (757 seconds ago) [2018-08-21 23:51:35] [0,0] Existing WebSocket ping data (CM02-LUX.cm.steampowered.com:27021) results: 50.0000ms + load 61.0000 (719 seconds ago) [2018-08-21 23:51:35] [0,0] Existing WebSocket ping data (CM01-FRA.cm.steampowered.com:27021) results: 49.0000ms + load 63.0000 (719 seconds ago) [2018-08-21 23:51:35] [0,0] Existing WebSocket ping data (CM02-STO.cm.steampowered.com:27021) results: 49.0000ms + load 61.0000 (751 seconds ago) [2018-08-21 23:51:35] [0,0] Existing WebSocket ping data (cm-02-ams1.cm.steampowered.com:443) results: 39.0000ms + load 62.0000 (756 seconds ago) [2018-08-21 23:51:35] [0,0] Existing WebSocket ping data (cm-02-ams1.cm.steampowered.com:27021) results: 39.0000ms + load 63.0000 (756 seconds ago) [2018-08-21 23:51:35] [0,0] Existing WebSocket ping data (cm-02-ams1.cm.steampowered.com:27020) results: 32.0000ms + load 63.0000 (756 seconds ago) [2018-08-21 23:51:35] [0,0] PingWebSocketCM() (cm-03-ams1.cm.steampowered.com:443) starting... [2018-08-21 23:51:35] [0,0] PingWebSocketCM() (CM03-STO.cm.steampowered.com:443) starting... [2018-08-21 23:51:35] [0,0] PingWebSocketCM() (CM02-FRA.cm.steampowered.com:27021) starting... [2018-08-21 23:51:35] [0,0] PingWebSocketCM() (CM06-IAD.cm.steampowered.com:27021) starting... [2018-08-21 23:51:36] [0,0] PingWebSocketCM() (cm-03-ams1.cm.steampowered.com:443) failed talking to cm (timeout/neterror - OK) [2018-08-21 23:51:36] [0,0] PingWebSocketCM() (CM06-IAD.cm.steampowered.com:27021) failed talking to cm (timeout/neterror - Invalid) [2018-08-21 23:51:36] [0,0] PingWebSocketCM() (CM02-FRA.cm.steampowered.com:27021) failed talking to cm (timeout/neterror - Invalid) [2018-08-21 23:51:36] [0,0] PingWebSocketCM() (CM03-STO.cm.steampowered.com:443) failed talking to cm (timeout/neterror - Invalid) [2018-08-21 23:51:36] [1,3] Connect() starting connection (eNetQOSLevelHigh, cm-02-ams1.cm.steampowered.com:27020, WebSocket) content_log.txt Quote Link to comment
mlebjerg Posted August 21, 2018 Share Posted August 21, 2018 @mrcake Weird! What does the docker logs say, is everything running? Just to make sure, have you tried changing the dns on your router? Or are you just testing it in windows? It could help since there probably are some DNSSec settings on the router that could interfere. And lastly are you running any VPN on the windows client? 1 Quote Link to comment
mrcake Posted August 21, 2018 Share Posted August 21, 2018 @mlebjerg I had an adblocker running on dd wrt, disabled that (did not help) Steam is failing its connections now, so that is better i think? No vpns active or used on my network. Here is the docker log: root@Tower:~# docker logs SteamCacheBundle Running bootstrap.sh... nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful Bind9 { Enabling cache for Blizzard 2018-08-20 14:31:57 Failed to set file handle limit: Operation not permitted Enabling cache for Frontier Enabling cache for Origin Enabling cache for Riot Enabling cache for Steam Enabling cache for Uplay Enabling cache for Windows Enabling custom DNS forwarder bootstrap finished. checking Bind9 config Running Bind9 Running bootstrap.sh... nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful Bind9 { Enabling cache for Blizzard 2018-08-20 15:24:04 Failed to set file handle limit: Operation not permitted Enabling cache for Frontier Enabling cache for Origin Enabling cache for Riot Enabling cache for Steam Enabling cache for Uplay Enabling cache for Windows Enabling custom DNS forwarder bootstrap finished. checking Bind9 config Running Bind9 Running bootstrap.sh... nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful Bind9 { Enabling cache for Blizzard 2018-08-21 08:39:09 Failed to set file handle limit: Operation not permitted Enabling cache for Frontier Enabling cache for Origin Enabling cache for Riot Enabling cache for Steam Enabling cache for Uplay Enabling cache for Windows Enabling custom DNS forwarder bootstrap finished. checking Bind9 config Running Bind9 Running bootstrap.sh... nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful Bind9 { Enabling cache for Blizzard 2018-08-21 13:31:56 Failed to set file handle limit: Operation not permitted Enabling cache for Frontier Enabling cache for Origin Enabling cache for Riot Enabling cache for Steam Enabling cache for Uplay Enabling cache for Windows Enabling custom DNS forwarder bootstrap finished. checking Bind9 config Running Bind9 Running bootstrap.sh... nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful Bind9 { Enabling cache for Blizzard 2018-08-21 15:31:52 Failed to set file handle limit: Operation not permitted Enabling cache for Frontier Enabling cache for Origin Enabling cache for Riot Enabling cache for Steam Enabling cache for Uplay Enabling cache for Windows Enabling custom DNS forwarder bootstrap finished. checking Bind9 config Running Bind9 Running bootstrap.sh... nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful Bind9 { Enabling cache for Blizzard 2018-08-21 15:33:55 Failed to set file handle limit: Operation not permitted Enabling cache for Frontier Enabling cache for Origin Enabling cache for Riot Enabling cache for Steam Enabling cache for Uplay Enabling cache for Windows Enabling custom DNS forwarder bootstrap finished. checking Bind9 config Running Bind9 Quote Link to comment
NewDisplayName Posted August 22, 2018 Share Posted August 22, 2018 (edited) Again, how u test its not working? Do you download a game in steam client? (does it download?) Edited August 22, 2018 by nuhll Quote Link to comment
mrcake Posted August 22, 2018 Share Posted August 22, 2018 It is working now. What i did now is disable the adblock on dd-wrt, after that steam had some trouble connecting and I left the pc on while I was asleep. In the morning the downloads started and I am now observing that the share is growing in size, access log is showing the misses. and now tried to reinstall the game and that hit the cache. Thanks for the help Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.