I attached my syslog from the remote syslog server I set up on my personal computer. I set up a remote syslog last night, while i was still able to access my webGUI. This morning after waking up and checking to see if i still have access to the webGUI, I can reach the page to enter my credentials, but when I click log in the page just loads indefinitely. My solution so far has just been to hard restart my server, as when I run a reboot command via ssh I'm never able to get my server to come back online, or sometimes it seems like my system just ignores the command and never reboots as I'm able to access the webGUIs of my docker containers long after running the command.

syslog

Edited March 25Mar 25 by minnesotaonion

Now that we have that syslog, can you reboot and get us diagnostics while still able to access webUI?

Here are those diagnostics. I was able to reboot -f which successfully rebooted the server and allowed me to get into the webUI.

cayde-6-diagnostics-20260325-1107.zip

Edited March 25Mar 25 by minnesotaonion

Normal syslog starts over after every boot, enable the syslog server and post that the next time it happens.

28 minutes ago, JorgeB said:

Normal syslog starts over after every boot, enable the syslog server and post that the next time it happens.

yes this is from my syslog server that is running on my personal machine.

The persistent syslog is not included with the diags, you will need to upload it separately. It will be in the share you set in the syslog server settings.

Ah okay. I had my local syslog folder set but for some reason it is not there. I did manage to get this from my remote syslog from the reboot though.

syslog

Multiple call traces logged:

192.168.5.61 Mar 25 01:00:32 cayde-6 kern crit kernel kernel BUG at fs/inode.c:668!

192.168.5.61 Mar 25 01:00:32 cayde-6 kern warning kernel Oops: invalid opcode: 0000 [#3] PREEMPT SMP NOPTI

192.168.5.61 Mar 25 13:00:23 cayde-6 kern alert kernel BUG: kernel NULL pointer dereference, address: 000000000000007c

192.168.5.61 Mar 25 13:00:23 cayde-6 kern alert kernel #PF: supervisor write access in kernel mode

192.168.5.61 Mar 25 13:00:23 cayde-6 kern alert kernel #PF: error_code(0x0002) - not-present page

The fact that you have two different types of crashes (a kernel BUG and a NULL pointer dereference) in the same general area of the kernel code points strongly toward a hardware issue rather than a specific software bug.

My main suspect would be bad RAM. Start by running Memtest, or because memtest is only definitive if it finds errors, if you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.

will do. I have been fearing RAM may be the issue and have been trying to put off having to run Memtest but i'll give it a shot.

I'm continuing to see odd behavior in my syslog as regular tasks are running on my server, are the docker "entering disabled state" and "entering forwarding state" messages anything to be concerned about as well?

syslog

1 hour ago, minnesotaonion said:

I have been fearing RAM may be the issue and have been trying to put off having to run Memtest

You should never put off running memtest. You must do it immediately!

You must never attempt to run any computer unless RAM is working perfectly. Everything goes through RAM. The OS and other executable code. Your DATA!. EVERYTHING! The CPU can't do anything with anything until it is loaded into RAM.

I've only ever had to run memtest once on a personal machine many years ago. I've read online that 3 passes should be enough for pretty conclusive findings, but I've also seen people recommending no less than 5 or 6 passes or to just let it run for an entire day. What is the best suggestion?

You can quit when it gets even a single error, or let it run overnight.

2 hours ago, minnesotaonion said:

are the docker "entering disabled state" and "entering forwarding state" messages anything to be concerned about as well

You see these whenever a docker container starts up. If you are getting them continuously then you have one or more containers crashing and restarting.

Well, one of my memory sticks is bad. 17 errors in the first minute of the memtest. Ran a few different configurations and am pretty confidently able to say that it's the stick not the slot. Now when booting with the good stick I'm getting an error when trying to boot that says

umount: /: not mounted

bzroot checksum error - press ENTER key to reboot...

I'm fairly certain I have encountered this before and found that the solution is to basically replace the bz* files on my USB?

24 minutes ago, minnesotaonion said:

replace the bz* files on my USB?

Worth a try.

success. My server is booted and so far haven't seen any errors in the syslog. thank you everyone for helping get to the bottom of this.

Since you had bad RAM your data is somewhat suspect. You might consider check filesystem all around and a parity check. No parity?