Kernel Panic during btrfs (cache) balance.

[CorneliousJD]

So I was doing the unassign/replace method of fixing the high cache writes in 6.9.0 stable and I had removed the first of two disks, rebalance was fine, then I added it back, and during a rebalance I got a kernel panic.

 

IPMI shows this.

Anonymized diags attached. 

 

Does anyone know what actually happend here.

 

Also, miraciously, I'm able to boot back up and paritcy check AND another balance are both running.

 

Docker containers all seem to be starting up, appdata is all accessible still. 

I thought for sure the cache pool would have suffered badly from this but it seems like it may be correcting itself without too much fuss.

Fingers crossed...

 

Thanks in advance!

server-diagnostics-20210305-0126.zip

[CorneliousJD]

HomeAssistant is acctually the only container that wont start, all other (50-some of them) have started fine.

 

When trying to run HomeAssistant I get this "Execution Error" Server error with no other information.

This containr is in br0 mode running on a a different IP Address, but so is PiHole and that's started without issues.

 

I pulled up server log and I see a few references to eth0 stuff, and IPV6addresses, but not sure what to make of it, everything else in server log is related to BTRFS balance going on.

 

Mar 5 01:45:03 Server kernel: docker0: port 48(veth013765a) entered blocking state
Mar 5 01:45:03 Server kernel: docker0: port 48(veth013765a) entered disabled state
Mar 5 01:45:03 Server kernel: device veth013765a entered promiscuous mode
Mar 5 01:45:08 Server kernel: eth0: renamed from vethd164395
Mar 5 01:45:09 Server kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth013765a: link becomes ready
Mar 5 01:45:09 Server kernel: docker0: port 48(veth013765a) entered blocking state
Mar 5 01:45:09 Server kernel: docker0: port 48(veth013765a) entered forwarding state
Mar 5 01:45:10 Server avahi-daemon[10338]: Joining mDNS multicast group on interface veth013765a.IPv6 with address fe80::e894:abff:fe00:467f.
Mar 5 01:45:10 Server avahi-daemon[10338]: New relevant interface veth013765a.IPv6 for mDNS.
Mar 5 01:45:10 Server avahi-daemon[10338]: Registering new address record for fe80::e894:abff:fe00:467f on veth013765a.*.

 

[JorgeB]

Diags shows balance still running, did it finish now?

[CorneliousJD]

2 hours ago, JorgeB said:

Diags shows balance still running, did it finish now?

I called it a night and went to bed but yes, it's finished now "no balance found on /mnt/cache/"

 

I still need to remove the other drive, balance, and add it back and balance again now, but parity check is running so I'll wait for that.

 

Is there any way to glean more info on what caused the kernel panic? And also what the execution error of that container is? 

 

Thanks in advance, much appreciated!

[JorgeB]

10 minutes ago, CorneliousJD said:

Is there any way to glean more info on what caused the kernel panic? And also what the execution error of that container is? 

Sorry, no.

[CorneliousJD]

8 hours ago, JorgeB said:

Sorry, no.

 

I was able to narrow down my HomeAssistant docker, unrelated to everything else that happened, a USB device wasn't passing through properly.

 

I found that out by just starting the container from the command line and got this error then I fixed it and started it again and it worked.

 

root@Server:~# docker start HomeAssistant
Error response from daemon: error gathering device information while adding custom device "/dev/wyzesense": no such file or directory
Error: failed to start containers: HomeAssistant
root@Server:~# docker start HomeAssistant
HomeAssistant
root@Server:~# 

 

So now I just need to wait for parity check to complete, stop array, remove 2nd cache drive, start and rebalance, stop, re-add the disk, start and rebalance again and then I should be good on the 1MiB partiton on the SSDs and be back to normal.

 

Fingers crossed!