As far as I can tell I have errors on my ZFS RAID1 2-disk cache pool which leads to a corrupt docker.img and crashing docker containers and unresponsive system or outright crashes.
When everything seems to run "fine" i get CRC errors on nearly everything I download and unrar and eventually the server seems to self-destruct and I need to hard reboot.
Also my Home Assistant regularly get's a broken db and starts a new one. Guess it's because it too get's corrupted from what ever is going on.

 

I attached a few screenshots with errors I'm getting.
The first few is the "crashed" state where I can still access the webGUI, navigate it but nothing I do in there does anything meaningful.

The fourth is when trying to backup the docker.img

I had bought new hardware and saw this as a sign to finally install the new mainboard and CPU. I also installed new SATA cables out of caution.
I also ran memtest for ~4h and nothing came up.
Nothing changed, but I didn't get my hopes up anyway, as the cache and/or docker.img are in a broken/corrupt state.

I found what to do individually for a corrupt docker.img or pool, but am unsure as to how to proceed.

I mounted the pool in readonly mode and backed up all I could (docker.img wouldn't copy but will be recreated anyway).
I've got one new SSD on hand and will be ordering a second one.

So far I've manually copied /appdata and docker.img from the pool to a usb drive.
While copying them (via cp) I got a bunch of errors that it can't copy symlinks. (last screenshot) I could force it via -L but then when moving stuff back I would have lost the linking and just have hard copies. I guess I have to tackle the problem when recreating the docker.img and just don't copy any linked files right?

 

 

What steps do I take to get to a running server with a fresh docker.img and a new pool with only the new SSD (the second one will be added once it arrives).

Do I just

• remove both SSDs
• install the single new SSD
• create a new pool
• manually move the files back
• delete the docker.img and recreate it and the Apps
• ???? 
• profit

 

Keep in mind I have the 6 disk license and am already at the cap.

 

Thanks in advance

tower-diagnostics-20240501-1653.zip

Both btrfs and zfs are detecting data corruption, and Ryzen with overclocked RAM like you have is known to corrupt data, after fixing that, it would still be a good idea to run memtest.

I turned the RAM speed down to 3200MHz.
Hopefully that is enough, as I can't find any official metrics for my 5600G. But the page of the 4600G says 3200MHz.

Running memtest now and will let it run through the night.

 

If all is well are the next steps I wrote above the correct way to go about fixing it?

See if no more cheksum errors are detected, you were also having issues with a second flash drive, so check also that, or disconnect if not needed.

The memtest ran for 16 hours without issues at 3200 MHz.

 

I noticed that the backup I tried to do from the readonly zfs pool to the second flash drive wasn't successful.
Only a few folders are on the flash drive.
Curiously while copying the flash drive seemed to fill and showed ~80GB used (even if the cache pool only has ~55GB of data).

Looking at it now there are only 3.9GB used and merely three folders from the appdata copied.

This would be the plethora of errors in the syslog.

 

Is there any way to get all the data from the pool?

As far as I can tell when I configure app data to move cache -> array it stops at the same file in the ESPHOME folder as if I would the files manually.
I guess this would be one of the broken files mentioned below.
 

The GUI shows there are 12 corrupted files on the pool, but running `zpool status -v` only shows this

 

 

Pool appears to have serious metadata corruption, so some data may fail to copy, you will need to restore from a backup

I think I got it fixed.

 

Here's what I did in the end:

• (disable every autostart function. Array, docker service and VMs)
• copy all important files from `appdata` and everything in `system` to a backup folder on the array via rsync
  • Curiously I only got one error while syncing. If any copied files are still broken, that's the price I have to pay for not having a recent backup ¯\_(ツ)_/¯
  • I included the -a option so I wouldn't have problems with wrong permissions for the docker containers in the end. Not that I ever went through this before...
• shrink the cache pool to one drive by unassigning the second drive (because I was at the drive limit)
• remove the unused SSD
• install the new SSD and create a new pool (`cachetwo`) with it
• copy the data from step 1 to the new cache pool
• swap every relevant share from `cache` to `cachetwo`
  • probably best to disable mover before this step, but I knew it wasn't time for it to run
  • Fix common problems will tell you that there is data for the shares on the old pool. While true, it is irrelevant in this instance
• unassign the remaining drive from the original cache (this way I could remount it if I did something wrong)
• delete the docker.img and rebuild it via `docker -> add container` and selecting
• (add a new second pool drive when it arrives today)

This way I also swapped out the pool drives for peace of mind.
So the only hardware left from before are the array drives, PSU and the RAM (which is now at 3200MHz).

For now everything seems to be stable and so far no CRC errors while downloading and unpacking .... stuff

 

Did I do anything that could bite me in the ass later?

 

Thank you very much for helping! I only would have stumbled across the Ryzen RAM issue after nights of furiously googling and ripping my hair out. If ever