Hello,

 

In the older of the two log files you can see that there were constant error messages in the log like this one:

Tower kernel: BTRFS error (device loop2: state EA): parent transid verify failed on logical 208311271424 mirror 1 wanted 4372964 found 4372437

 

After that I restarted the array/tower to look if the error disappears.

 

In maintenance mode there were no errors in the file system check for cache_appdata ssd.

 

In the log everything looked fine to me at first, then I tried to install an app from the community store and got the following error message:

 

"docker: Error response from daemon: error creating temporary lease: write

/var/lib/docker/containerd/daemon/io.containerd.metadata.v1.bolt/meta.db: read-only file system: unknown."

 

The new error messages can be seen in the newer of the two log files, copied out again here:

Aug 28 11:42:46 Tower kernel: XFS (dm-12): start block 0x88e094d block count 0x8288ffff
Aug 28 11:42:46 Tower kernel: XFS (dm-12): Block Freespace BTree record corruption in AG 0 detected!
Aug 28 11:42:46 Tower kernel: XFS (dm-12): start block 0x88e094d block count 0x8288ffff
Aug 28 11:42:46 Tower kernel: dm-12: writeback error on inode 47039578, offset 109555220480, sector 932647328
Aug 28 11:42:46 Tower kernel: I/O error, dev loop3, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 2
Aug 28 11:42:46 Tower kernel: BTRFS error (device loop3): bdev /dev/loop3 errs: wr 0, rd 0, flush 1, corrupt 1, gen 0
Aug 28 11:42:46 Tower kernel: BTRFS warning (device loop3): chunk 13631488 missing 1 devices, max tolerance is 0 for writable mount
Aug 28 11:42:46 Tower kernel: BTRFS: error (device loop3) in write_all_supers:4369: errno=-5 IO failure (errors while submitting device barriers.)
Aug 28 11:42:46 Tower kernel: BTRFS info (device loop3: state E): forced readonly
Aug 28 11:42:46 Tower kernel: BTRFS: error (device loop3: state EA) in btrfs_sync_log:3198: errno=-5 IO failure

 

What can I do in this case?

tower-diagnostics-20230828-1151.zip tower-diagnostics-20230828-1022.zip

Edited August 28, 20232 yr by aurevo

Check filesystem on the xfs cache, run it without -n.

Since loop3 referers to where the docker image is mounted you may also have a corrupt docker.img file so in that case you should recreate and reinstall your docker apps.

54 minutes ago, JorgeB said:

Check filesystem on the xfs cache, run it without -n.

 

Should this be all?

 

Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... invalid start block 143526221 in record 124 of bno btree block 0/182773 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 clearing reflink flag on inodes when possible Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... done

 

It seems to work now, after checking the filesystem. At least the Dockers can be updated again.

1 minute ago, aurevo said:

 

Should this be all?

 

Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... invalid start block 143526221 in record 124 of bno btree block 0/182773 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 clearing reflink flag on inodes when possible Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... done

 

It seems to work now, after checking the filesystem. At least the Dockers can be updated again.

 

Worth checking to see if you now have a lost+found folder where the repair process would put any files for which it could not locate the directory entry giving the correct name.