web ui loads frame but not content

[Cull2ArcaHeresy]

i've had this issue for months now (think it started early 2022 but might have been at some point in 2021). When server boots everything is fine, and for a while after it is fine. But after a few days to a couple weeks it will start having this issue. The "frame" of the web ui loads, but the content does not. Sometimes stuff will load...a few times a day if i'm lucky. Behaves the same on dedicated 10g nic (172...) and local ip (192...), but used the 172 screenshot because dark reader messes with 192 differently. Also works the same on chrome, ff, ie, edge, browser in unraid gui, and i feel like i might have tried another browser too (plus mobile chrome and ff do same).

 

The error below screenshot is what it gives after the page sits with the frame but not content loaded for a few minutes.

Dec 17 08:40:30 Raza nginx: 2022/12/17 08:40:30 [error] 35647#35647: *7626036 upstream timed out (110: Connection timed out) while reading upstream, client: 172.16.16.2, server: , request: "GET /Dashboard HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "172.16.16.1", referrer: "http://172.16.16.1/Dashboard"

 

SSH and vms (via other vnc access) and dockers and file serving all work fine, just the content of page fails to load. All the other answers i've seen have to do with nothing loading, unless i missed this issue being answered.

 

 

raza-diagnostics-20221217-0850.zip

[Squid]

You do have a bit to look at here

Dec 11 04:57:28 Raza root: Fix Common Problems: Warning: Deprecated plugin ca.backup2.plg
Dec 11 04:57:30 Raza root: Fix Common Problems: Error: Blacklisted plugin speedtest.plg
Dec 11 04:57:32 Raza root: Fix Common Problems: Error: Unable to write to disk14
Dec 11 04:57:32 Raza root: Fix Common Problems: Error: Unable to write to disk15
Dec 11 04:57:32 Raza root: Fix Common Problems: Error: Unable to write to disk19
Dec 11 04:57:32 Raza root: Fix Common Problems: Error: Unable to write to tv_pool ** Ignored

The disks are full, but I'm wondering if it's not due to misconfiguration as it would be rare (not impossible) to have 20K free space on any random drive

[Squid]

But that's not going to be the problem anyways.  You should reboot and then try it again and if it still fails then post a clean set of diagnostics

[Cull2ArcaHeresy]

on the plugins i have meant to look into that for a while (grab history stuff before removing), also i think i am a couple updates behind on base os. Total free space (if you include cache) is ~1.5tb, and it complains about different disks over time as stuff gets moved around. The tv_pool is full, but acts more as a level 3 cache (when there is room i change share setting to move from array to that cache and then after mover runs change it back to move from ssd cache to array). I'm hoping to add at least 16tb in the next month and then can rebalance to make it happy. I do have a script that am gonna finish writing in next week to check for duplicate files, but optimizing it to where it will have a thread hashing per drive/pool to speed it up (rough estimate of 1 to 5 tb to clear so can redistribute some and make it happy then too).

 

This problem has been happening for a while now and a reboot fixes it for some time. Just to double check, 1) reboot, 2) grab diagnostics, 3) when issue happens again grab diagnostics and post both here? After that i can look into updating os.

[Cull2ArcaHeresy]

On 12/17/2022 at 9:22 AM, Squid said:

But that's not going to be the problem anyways.  You should reboot and then try it again and if it still fails then post a clean set of diagnostics

 

Rebooted and generated a copy of diagnostics from web ui. At some point between 1 and 2 days the issue happened again (does seem to have happened faster this time). Generated new copy of diagnostics via cli shortly before making this post.

 

After reboot i saw there were updates to some plugins and dockers, but for consistency did not update them yet. When back home after holiday trip i can reboot, update plugins and dockers, reboot, and wait for issue to happen again. Also having the ui back (for a short bit) can look back into what prevented me from updating the os and after resolving that will update os (unless you say not to until this issue is resolved).

 

Also a non-optimized version of my dupe finder is running on a selection of files to "quickly" free up some space to be able to redistribute space between the drives to make sure that isn't somehow causing a problem (will probably be done tomorrow and have 3 to 4 tb to free up).

rightafterreboot_raza-diagnostics-20221221-0834.zip issuehappened_raza-diagnostics-20221223-1354.zip

[Cull2ArcaHeresy]

updating to 6.11.5 seems to have fixed this, but not marking as solved till server is up for a month without it happening again