Docker error 403 / Docker Service failed to start.

[odrel]

I woke up this morning to Netdata shooting me the following alerts :

 

 

Dockers were unresponsive and I couldn't start them (Error Code 403).

 

After rebooting, I'm getting a "Docker Service failed to start."

 

Diagnostics enclosed below.

 

EDIT : found the following errors in the syslog, they seem related to a filesystem error. I'm not sure what to do next. 

Quote

Jun  7 09:03:29 Antares avahi-daemon[14809]: Server startup complete. Host name is Antares.local. Local service cookie is 2287378824.
Jun  7 09:03:29 Antares kernel: BTRFS error (device nvme0n1p1): parent transid verify failed on logical 97455603712 mirror 1 wanted 2305843009214987099 found 1293147
Jun  7 09:03:29 Antares kernel: BTRFS error (device nvme0n1p1): parent transid verify failed on logical 97455603712 mirror 2 wanted 2305843009214987099 found 1293147
Jun  7 09:03:29 Antares kernel: BTRFS: error (device nvme0n1p1: state A) in __btrfs_free_extent:3076: errno=-5 IO failure
Jun  7 09:03:29 Antares kernel: BTRFS info (device nvme0n1p1: state EA): forced readonly
Jun  7 09:03:29 Antares kernel: BTRFS error (device nvme0n1p1: state EA): failed to run delayed ref for logical 97314553856 num_bytes 16384 type 176 action 2 ref_mod 1: -5
Jun  7 09:03:29 Antares kernel: BTRFS: error (device nvme0n1p1: state EA) in btrfs_run_delayed_refs:2150: errno=-5 IO failure
Jun  7 09:03:29 Antares root: truncate: cannot open '/mnt/fastnvme/system/docker/docker.img' for writing: Read-only file system
Jun  7 09:03:29 Antares kernel: loop2: detected capacity change from 0 to 146800640
Jun  7 09:03:29 Antares kernel: BTRFS: device fsid eea19f9c-f9d2-4aad-92be-2d401cfeb20a devid 1 transid 1303430 /dev/loop2 scanned by mount (14870)
Jun  7 09:03:29 Antares kernel: BTRFS info (device loop2): first mount of filesystem eea19f9c-f9d2-4aad-92be-2d401cfeb20a
Jun  7 09:03:29 Antares kernel: BTRFS info (device loop2): using crc32c (crc32c-intel) checksum algorithm
Jun  7 09:03:29 Antares kernel: BTRFS info (device loop2): using free space tree
Jun  7 09:03:29 Antares root: mount: /var/lib/docker: can't read superblock on /dev/loop2.
Jun  7 09:03:29 Antares root:        dmesg(1) may have more information after failed mount system call.
Jun  7 09:03:29 Antares root: mount error

antares-diagnostics-20240607-0905.zip

Edited June 7 by odrel

[JorgeB]

Run a correcting scrub on the pool and post the results and new diags.

[odrel]

Since the disk was fully readable, I opted to copy it to the array, then stopped the array and reformated the drive before copying my backup back onto the drive.

 

Had to fix permissions a bit after that but everything seems to be working again.