I had issues this afternoon where my docker were going unresponsive. So I shutdown all VMs and dockers and tried to restart my server but it hung up on shutdown so I had to force it off. I then tried restarting my server just for the IP/ GUI to never appear. So I connected a basic video card I use for troubleshooting and saw I had a panic kernal which typically means flash drive failed.

 

So I downloaded my backup and restored to a new flash drive but couldn't get my mobo to boot the flashdrive automatically. So I upgraded the bios since it seemed like a bug and I was able to get everything to work as expected until I started the array.

 

My cache now reads "Unmountable: No file system". I tried following these steps "Link" but am afraid of breaking something and losing a lot of photos I forgot I stored on my cache (NextCloud sync).

 

I assume something is corrupt according to these disk logs.
Jan 3 21:58:30 Server kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
Jan 3 21:58:30 Server kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0

 

Attached is my diagnostic report. Any help to try and save some data would be greatly appreciated!!!

server-diagnostics-20230103-2200.zip

3 hours ago, Kuchar09 said:

So I shutdown all VMs and dockers and tried to restart my server but it hung

Because you didn't wait long enough?... one of your VMs didn't want to quit because it was doing something?...

 

3 hours ago, Kuchar09 said:

So I upgraded the bios

So... Now the flash/boot drive has a new bios to deal with?...

 

3 hours ago, Kuchar09 said:

My cache now reads "Unmountable: No file system".

This is why we backup. You use CA Backup?... please say you do because I think your photos are there if you do?

 

Someone smarter than I will come along and give better news, I hope.

 

MrGrey.

If the problem is only the log tree it might help to zero it:

btrfs rescue zero-log /dev/nvme0n1p1

 

If that doesn't help see here for some recovery options, but suggest upgrading to v6.11.5 first.

13 hours ago, JorgeB said:

If the problem is only the log tree it might help to zero it:

btrfs rescue zero-log /dev/nvme0n1p1

 

If that doesn't help see here for some recovery options, but suggest upgrading to v6.11.5 first.

Thanks for the quick reply!

 

So I ran "btrfs rescue zero-log /dev/nvme0n1p1" and "btrfs rescue zero-log /dev/nvme1n1p1" and that appears to have let my cache mount without any issues.

 

I then upgraded to v6.11.5

 

Interesting enough I'm still having issues with Dockers randomly not responding. Then once I stop them and restart them I recieve an error "Execution error Error code 403" then eventually the server just locks up. This was my original issue that forced me to hard reboot yesterday making me thing my flash drive was dying.

I took a diagnostic report (server-diagnostics-20230104-1501.zip) right after the 403 error before the lockup and another after a hard reboot (server-diagnostics-20230104-1512.zip).

 

I'm currently going to just let it sit without my 2 main VMs running and see if it survives the night. I have a feeling something got corrupt or some hardware is dying.

 

EDIT: And my server just went unresponsive again within minutes without even doing anything. Can't ping it or anything.... Would it be worth trying to move everything off the cache drive, reformating it, then try putting the appdata back on it?

server-diagnostics-20230104-1501.zip server-diagnostics-20230104-1512.zip

Edited January 4, 20233 yr by Kuchar09

Jan  4 14:54:30 Server kernel: BTRFS error (device nvme0n1p1): block=11621850218496 write time tree block corruption detected

 

This usually means bad RAM, note that Ryzen with overclocked RAM like you have is known to corrupt data, see here.

 

12 hours ago, Kuchar09 said:

So I ran "btrfs rescue zero-log /dev/nvme0n1p1" and "btrfs rescue zero-log /dev/nvme1n1p1"

You only need to run it for one of the pool devices, it will automatically find all members.