I am getting my Cache Drive (not pool) mounting as read only and very few of my docker containers are able to start. I have attached my Full Diag zip file below, and I did run a readonly BTRFS check on the cache drive, but I am at a loss on how to repair it as it isnt a pool so SCRUB isnt working. SCRUB actually is aborted when I click the button immediately. I also have these errors in my log:

 

Quote

May 30 11:50:19 AncientValeMkII kernel: loop: Write error at byte offset 4255567872, length 4096.
May 30 11:50:19 AncientValeMkII kernel: print_req_error: I/O error, dev loop2, sector 8311656
May 30 11:50:19 AncientValeMkII kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
May 30 11:50:19 AncientValeMkII kernel: loop: Write error at byte offset 4255567872, length 4096.
May 30 11:50:19 AncientValeMkII kernel: print_req_error: I/O error, dev loop2, sector 8311656
May 30 11:50:19 AncientValeMkII kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
May 30 11:50:20 AncientValeMkII kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
May 30 11:50:20 AncientValeMkII kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
May 30 11:50:20 AncientValeMkII kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
May 30 11:50:20 AncientValeMkII kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
May 30 11:50:20 AncientValeMkII kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0
May 30 11:50:20 AncientValeMkII kernel: BTRFS: error (device loop2) in btrfs_commit_transaction:2236: errno=-5 IO failure (Error while writing out transaction)

 

ancientvalemkii-diagnostics-20190530-1559.zip

Edited June 25, 20197 yr by AncientVale
Marking Solved

loop2 refers to the docker image file.  It sounds as if it is corrupt?   If so the easiest thing to do is stop the docker service; delete the current image file; restart docker service to create a new clean docker image fine. and then use the Previous Apps option on the Apps tab to restore your container with their settings intact

When i try to recreate the docker image I get the following:

Quote

May 30 12:27:36 AncientValeMkII root: ERROR: unable to resize '/var/lib/docker': Read-only file system

And when I look at the check filesystem status, I get the below:

Quote

[1/7] checking root items [2/7] checking extents ERROR: ignore invalid data extent, length 18446612733170907240 is not aligned to 4096 bad key ordering 186 187 bad block 655830777856 ERROR: errors found in extent allocation tree or chunk allocation [3/7] checking free space cache there is no free space entry for 378999562240-379051835392 cache appears valid but isn't 377978093568 there is no free space entry for 18446613197078861928-463908003840 cache appears valid but isn't 463877439488 [4/7] checking fs roots [5/7] checking only csums items (without verifying data) bad key ordering 186 187 Error looking up extent record -1 csum exists for 378992041984-378992168960 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378992173056-378992287744 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378992291840-378992381952 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378992386048-378992746496 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378992750592-378992889856 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378992893952-378993352704 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378993356800-378993696768 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378993700864-378994229248 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378994233344-378994319360 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378994323456-378994823168 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378994827264-378994872320 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378994876416-378994946048 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378994950144-378995458048 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378995462144-378995490816 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378995494912-378996244480 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378996248576-378996584448 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378996588544-378996953088 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378996957184-378998325248 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378998329344-378999164928 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378999169024-378999427072 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378999431168-378999869440 but there is no extent record bad key ordering 186 187 Error looking up extent record -1 csum exists for 378999873536-378999992320 but there is no extent record ERROR: errors found in csum tree [6/7] checking root refs [7/7] checking quota groups skipped (not enabled on this FS) Opening filesystem to check... Checking filesystem on /dev/sdg1 UUID: 6f423ce3-1272-4cd6-a6bd-285607db665f found 133638533120 bytes used, error(s) found total csum bytes: 0 total tree bytes: 105627648 total fs tree bytes: 0 total extent tree bytes: 105463808 btree space waste bytes: 25854939 file data blocks allocated: 61014016 referenced 61014016

 

Cache filesystem is fully allocated, see here for how to fix, though there might be other problems.

I tried running the command that you give in the thread there, and I get the following:

Quote

ERROR: error during balancing '/mnt/cache': Read-only file system There may be more info in syslog - try dmesg | tail

How can I force the cache to mount not read only?

 

Some additional Errors when I go to mount the system:

Quote

May 30 13:04:57 AncientValeMkII kernel: BTRFS critical (device sdg1): corrupt leaf: root=2 block=655830777856 slot=187, bad key order, prev (15670389631644491776 229 4294936715) current (378999640064 168 208896) May 30 13:04:57 AncientValeMkII kernel: BTRFS: error (device sdg1) in btrfs_run_delayed_refs:2935: errno=-5 IO failure May 30 13:04:57 AncientValeMkII kernel: BTRFS info (device sdg1): forced readonly May 30 13:05:00 AncientValeMkII kernel: BTRFS info (device loop2): disk space caching is enabled May 30 13:05:00 AncientValeMkII kernel: BTRFS info (device loop2): has skinny extents May 30 13:05:00 AncientValeMkII kernel: BTRFS warning (device loop2): log replay required on RO media May 30 13:05:00 AncientValeMkII root: mount: /var/lib/docker: can't read superblock on /dev/loop2. May 30 13:05:00 AncientValeMkII kernel: BTRFS error (device loop2): open_ctree failed May 30 13:05:00 AncientValeMkII root: mount error May 30 13:05:00 AncientValeMkII emhttpd: shcmd (698): exit status: 1 May 30 13:05:00 AncientValeMkII emhttpd: nothing to sync May 30 13:05:00 AncientValeMkII unassigned.devices: Mounting 'Auto Mount' Remote Shares... May 30 13:10:00 AncientValeMkII root: Fix Common Problems Version 2019.05.29 May 30 13:10:06 AncientValeMkII root: Fix Common Problems: Error: Unable to write to cache

 

Edited May 30, 20197 yr by AncientVale
Additional, potentially helpful, information

Disable docker/VM services and reboot, then try again, if it fails post complete diags.

I tried the above and still got a "read only filesystem" error when trying the balance. Please find attached my diag, I also have an earlier copy in the OP of this thread.

ancientvalemkii-diagnostics-20190530-1730.zip

File system has metadata corruption:

 

May 30 13:28:29 AncientValeMkII kernel: BTRFS critical (device sdg1): corrupt leaf: root=2 block=655830777856 slot=187, bad key order, prev (15670389631644491776 229 4294936715) current (378999640064 168 208896)

 

You'll need to backup all important data on cache, format and restore data.

How would I go about that if the system is read only? I tried running the appdata backup plugin (as that is really the only thing on there) and it didnt back anything up.

As long as it stays read only you should be able to use your preferred copy command/util, like cp or midnight commander, you can also see here for some recovery options if it goes unmountable or regular copy doesn't work..

I did what you told me and it was fine for a little over a month. now I am getting the below in the log:

 

Quote

Jul 3 22:22:43 AncientValeMkII kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 85, rd 0, flush 0, corrupt 0, gen 0
Jul 3 22:22:43 AncientValeMkII kernel: loop: Write error at byte offset 12355469312, length 4096.
Jul 3 22:22:43 AncientValeMkII kernel: print_req_error: I/O error, dev loop2, sector 24131776
Jul 3 22:22:43 AncientValeMkII kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 86, rd 0, flush 0, corrupt 0, gen 0
Jul 3 22:22:49 AncientValeMkII kernel: loop: Write error at byte offset 12193988608, length 4096.
Jul 3 22:22:49 AncientValeMkII kernel: print_req_error: I/O error, dev loop2, sector 23816384
Jul 3 22:22:49 AncientValeMkII kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 87, rd 0, flush 0, corrupt 0, gen 0
Jul 3 22:22:49 AncientValeMkII kernel: loop: Write error at byte offset 12278943744, length 4096.
Jul 3 22:22:49 AncientValeMkII kernel: print_req_error: I/O error, dev loop2, sector 23982312
Jul 3 22:22:49 AncientValeMkII kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 88, rd 0, flush 0, corrupt 0, gen 0

 

Should I do that again? And does this mean my Cache drive is dying?

Please post the diagnostics: Tools -> Diagnostics

Here they are

ancientvalemkii-diagnostics-20190704-1125.zip

Cache filesystem is again corrupt, since it only lasted a month likely due to some hardware problem (or incompatibility), try formatting xfs next time and see if it's stable that way.

Thank you!