vw-kombi

And now - about 30 minutes later, another lockup - this time with this call trace on the screen - i stopped the parity check this time:

And, onto the next issue in this seemingly unstable system - I was using the system, then lost all containers. no ping to server. This time, not a lockup, and not a reboot, but I got a kernel panic on the display I have connected all the time. Here is is - any idea ? Diags attached. Nothing in the syslog from the failure. tower-diagnostics-20240312-1342.zip

I did wonder if I somehow caused that as I was messing with smart home plugs around that time - I really hope I did. Either way, I have resigned to the fact of a new MOBO, CPU, RAM and Powersupply at some stage in the future. Hit all the parts at the same time. Those costs were not on the radar so I have to plan for it a bit.

It had a good run - but today I noticed a parity check running - checked logs and it has rebooted itself (rather than a lockup). Nothing in the logs - whatever is happening is just restarting the server with no time up update anything.

Meter say 112W, Unraid UPS says 63W..........

No-one offered any support on this. I am moving these USBs back to a rasberry Pi now - and no longer connected to unraid, hence will no longer by an issue..... Should be fixed however if you ask me.

No more issues, so start to put things back - but I dont plan on going back to docker for home assistant, as that is somehting that also changed in January, along with two USB connections from inverter and current cost - so they are no longer connected - moving back to Rasberry Pi for them. So - I have something very strange to report........ I was adding my stuff back, - APC UPS cable - so I plugged that in, and re-enabled the UPS in unraid. The strange thing - the UPS load is now showing just 63W. This would normally be between 93W and 120W. There are 5 disks running also......... I checked my other things connected, and they are still running - I can see 12W being delivered to the POE devices, and there is a few other low power stuff that would add up to a small bit extra - maybe another 15W. So - on paper it would seem my server is now using under 40W - which I doubt......... I will get a meter on it later today and update here.

Great News! I got a clean parity check for the first time in what seems like ages....... So - question is - which of these things was the cause : 1 - reset bios, disable Cstates, clean PC and CPU fans (note C states were always enabled before for years). 2 - removal of the USB connection for the solar inverter, and removal of home assistant (which gets randon flooding on that port) 3 - combination of the above I will plan for another parity check with full dockers running as normal. I am going to move home assistant and the USB connecters back to the raspberry pi until I get a small mini pc to run that, and maybe a few frigate cameras on it.

So I have been really babysitting this while the parity check is running. Strangely, I notice disk 4 was spun down - which I thought should never happen in a parity check. I clicked to spin up, and also changes the disk setting to never spin down. Notice the reads on the disks - almost all the same except that one - and also - after spinning up - the reads are not changing. Is this some sort of bug/issue ? Is it because that disk is so much smaller than the others ?

Shut it down, reset bios, (bios is up to date for my CPU), disabled C states, no overclocking or anything, gave it a big clean out - got dust of CPU fan etc etc. Disconnected UPS, solar and current cost USB's and stopped home assistant docker also. New parity check running now. Hope to get through one on 6.12.x......

woke up to a crash again overnight. I got the emails from the nightly ZFS sync so this crash was after 4:30. I am running another memtest to see if anything different (as the kids did it remotely for me last time). I see so many reports on redit and here about crashes on this later 6.12.x releases. Could it be an OS compatibility and I am wasting all my time on hardware for nothing ? As I said, I have had many years of stability and all these issues are on 6.12.4 starting in Jan for me. My 6.11.5 stayed up from the moment it was upgraded until the moment it went to 6.12.4 - surely that is not a coincidence with all the other posting of hardware lockups. Would I be wasting my time and effort in going back to 6.11.5 ? I dont have the money for a new CPU and motherboard so I will have to dust of my backup emby server that was de-commissioned in 2019 and get all that updated to latest and slowly move the drives over to it one by one (as they are all old ones).

That's the first ever auto restart like that. One thing that is new (only yesterday) is the connection of the unraid server to my solar inverter via a USB cable so I could add the monitoring of that to the home assistant docker. That's the once change since I moved from a VM for home assistant to a docker. I am monitoring temps and there is nothing suspect on the CPU, or on the disks. The power supply is also reasonably new as of January - so I cant discount that as being part / the cause of the issue at this stage. It is a Corsair 750W 80+ Bronze Power Supply. Its supplying an AMD Ryzen 7 2700 Eight-Core @ 3200 MHz, and 6 drives (4x10TB, 1x8TB and 1x4TB). There is only a basic graphics card for the system to post. With tips and tweaks, Turbo boost is off, and the system is set to power save. UPS load with the parity check running (all disk) is currently showing as 126W, but that includes a router, three switches, rasberri pi, isp gateway and two unifi access points - so I suspect the system is drawing under 80W.

And.....had to do this again after the system restarted itself (thats on another thread)!

Oh - Got a new thing - I lost comms. Ran to the server and I could see it rebooting..... Its not done that before....... I have a UPS so no reason for that. Off we go with a parity check again!!!!!!!syslog-192.168.1.7.log Nothing in the syslog that gives anything new

I noticed a number of ffmpeg oom's - not sure if related - I have stopped the frigate docker due to this. I only have emby, cloudflared, home assistant, tvheadend, mosquitto and unifi controller running now. Ram usage is 27% of the 32GB now. Normally it sits around 49%. Parity check says it will complete in 13 hours ish - maybe I will be awake for this one...... sync errors corrected shows 1 so far!!!!

syslog-192.168.1.7.logsyslog-192.168.1.7.log.1syslog-192.168.1.7.log.2

I found a solution for this online - BUT IT IS A BUG - PLEASE FIX. The Host Access to Custom Networks does not survive a reboot often. The solution - stop docker, change host access to custom networks disabled, click apply, change back to enabled, click apply, start docker.

I have shutdown all bar essential containers. No VM's running (have not for a while now), in the hope of a full parity check.

Did that - writing to a share called syslog now - just woke up to another crash. The monitor was attached in gui mode and had the logon screen there - but after user/pwd, nothing happened. Had to give it the finger again. Parity running again - which I expect to fail while I am sleeping again. It seems to be where I would expect the parity to complete based on timeline - but I am not in country, or awake when it has happened to date. Diags attached. tower-diagnostics-20240306-0611.zip

Further to this, I have installed the apcupsd app on windows and it can connect to the apcupsd service on unraid so the unraid host side is good - just the docker container cant connect. And it did work until last reboot.

For ages, my home assistant used an integration for APC and was connecting to the unraid hosts apcupsD and reporting all that as sensors. Then, with the upgrade to 6.12.4 that 'comms' seemed to be stopped - I ended up buying a power monitor for the UPS instead that home assistant can use. Then I upgraded to 6.12.8 yesterday, and it all started working again. But I just rebooted unraid and the comms is busted again - I had repurposed my energy monitor too........ Anyone have any ideas ?

Well - that was short lived. My unraid system locked up again - and after a reboot, this can not longer connect again from the home assistant docker container to the APCUPSD running on the unraid host

Re the link to the ryzon stuff - I have had a stable system for many years - since when it was first 'upgraded' to ryzen in 2019. So I cant see that as being relevant after all this time. I have had no bios updates or anything - the system is the same as it has always been - so why does the ryzon matter now - can only be Unraid O/S related Had two more system lockups this week. I was overseas for the other four lockups. I am home now, so this time I am able to see the monitor screen - and its blank. Have no idea what has made this so unstable lately - since OS update to 6.12.5 it seems. memtest is all clear. Maybe another hardware issues - mobo/cpu ? I had 6.9.2 running for a record 6 months straight I think at one stage - only planned outages. Went to the mid 6.10's also with no lockups. Went to mid 6.11's also with no lockups - and I never had the macvlan issues that people kept on about. The upgraded to 6.12.4 - and did the special macvlan instructions - as I really need that for my unifi controller to report IP addresses per each mac. Seemed stable on that release too. I suspect 6.12.6 done in Jan was when this instability for me came in. I'm on 6.12.8 now - as of yesterday - and just had the first crash on that release also. Considering getting a fingerbot for home assistant to push the button remotely at this stage!!!!!!

The unraid connect install has fixed the errors. I wont uninstall it for now - just wont enable the remote access / port forward (I have tailscale and wg for that). May be handy to have another flash backup floating around rather than the manual ones I keep.

Rebooted - same issues are seen. I have removed this folder in the plugins also before the reboot - /boot/config/plugins/dynamix.my.servers. Mar 4 12:31:51 Tower nginx: 2024/03/04 12:31:51 [error] 10889#10889: *335 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 192.168.1.10, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.1.7", referrer: "http://192.168.1.7/Dashboard" Mar 4 12:31:56 Tower nginx: 2024/03/04 12:31:56 [error] 10889#10889: *335 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 192.168.1.10, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.1.7", referrer: "http://192.168.1.7/Dashboard" Mar 4 12:32:01 Tower nginx: 2024/03/04 12:32:01 [error] 10889#10889: *335 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 192.168.1.10, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.1.7", referrer: "http://192.168.1.7/Dashboard" Mar 4 12:32:07 Tower nginx: 2024/03/04 12:32:07 [error] 10889#10889: *335 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 192.168.1.10, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.1.7", referrer: "http://192.168.1.7/Dashboard" Mar 4 12:32:12 Tower nginx: 2024/03/04 12:32:12 [error] 10889#10889: *335 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 192.168.1.10, server: , request: "POST /plugins/dynamix.my.servers/include/unraid-api.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.1.7", referrer: "http://192.168.1.7/Dashboard"