Good morning,

 

I've been having some intermittent stability issues where either my torrent container (both Deluge and qBitTorrent after migrating from Deluge) are unable to be stopped, my server becomes completely unresponsive and no longer shows as being online in my router, or the webUI crashes and the Docker are still available via IP:PORT address. I've recreated the docker.img file to no success, I have disabled C states on my mobo as well. This issue seems to have started after upgrading to ver. 6.11.5 of unRAID. I thought maybe it was related to the LibTorrent 2.X issues that were called out in the Bug forums, but I don't believe that is the case because the the BUG line in my syslog does not match the standard symptoms of that issue. What I always see before a crash in the logs is:

 

Feb 20 23:53:30 Deathstar kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000

 

Honestly, I'm a bit at a loss right now, haha. I can't seem to pin-point what/why is causing my server to crash like this. Any help would be greatly appreciated

syslog_deathstar.log deathstar-diagnostics-20230221-0949.zip

btrfs csum errors often indicate bad RAM

Hmm, I wonder if the same issue I was having back in March '22 has reared its ugly head again.  I'll fire up memtest and see if anything gets flagged. I'll report back in 24 hours. Thanks trurl!

A little early on the 24hr mark, but my memory passed 6 sets of the 10 tests in memtest86+ ver 6.10. Prior to kicking off the tests, I did reseat the 4 modules just to be sure they were properly secured in each slot.

 

What do you think about these btrfs errors binge caused by the mover trying to move files from my cache to array? I had a user script set up to stop my torrent docker at 3:55am each day then the mover is scheduled to move at 4am. I'm wondering if there's an issue with the script failing to stop the docker and the mover then executing on files that are in use by qBitTorrent.

1 hour ago, prymordial said:

btrfs errors binge caused by the mover

File operations can't cause these unless something else is affecting the data as it's being transferred. Everything goes through RAM.

Hmm ok. So given that the memtest ran without any errors in almost 24 hours, could I be looking at a false negative with this test? I'm just at a loss at trying to figure out what the issue could be at this point. The server was completely stable when I was on 6.9.4 and really only started crashing when I upgraded to 6.11.5.

Are you overclocking?

 

17 minutes ago, trurl said:

Are you overclocking?

I am not. I went into my BIOS and set the XMP profile to Auto when I posted about some data corruption back in 2022 and set the memory speed to 1866 per AMD's recommendation

On 2/21/2023 at 10:57 AM, trurl said:

btrfs csum errors often indicate bad RAM

Those could be leftover from some previous problem and nobody noticed.

 

Shouldn't cause crashing though.

2 hours ago, trurl said:

Shouldn't cause crashing though.

That's been the strangest part about this whole thing. It will be fine for a few weeks and then one day the server is completely inaccessible in one of a few different ways:

 

1. The server goes completely offline from a software standpoint. Containers are no longer accessible, web UI can't be reached, SSH is DOA, but the server still has power.

2. The web UI goes offline, but the containers are still accessible.

3. qBitTorrent shows as running in the console, but is unreachable. When I try to stop the container using the context menu, it just throws a "server exception."

 

The more I dig into this, the more I think my issue is either A. a piece of hardware is dying (HBA, mobo, CPU, add-in NIC) or B. there is some compatibility issue with my hardware and the current kernel version. There's been a few other posts that mention the same symptoms that I just described with no direct resolution. This is one of the posts that mention trying 6.12:

 

 

So this is where I'm at currently; I removed 2 sticks of memory and set the XMP config in the BIOS to Auto for everything. I have stopped every docker and disabled every script schedule with the exception of Plex. I'm going to monitor syslog for any other btrfs errors to see if I can narrow down the issue further.

Checking in after 1 week - After rolling back to version 6.9.2 (not 6.9.4 like I previously mentioned), my server seems to be much more stable. I'll continue to monitor and plan on adding the two sticks of memory back in over the weekend.

 

I'm hoping this is simply an issue with the kernel and my hardware having some sort of bug.