Hello!

 

I really don't know what actually started this problem.. I wanted to delete my Home Assistant VM, cause I migrated it to different hardware some time ago. I deleted the VM and disk and a bit after that I started to notice my Docker applications acting strange.

 

When I tried restarting my docker applications I got Execution error 403, so I rebooted my machine (from unraid GUI).

 

After the reboot I still had some issues with my docker applications, also with restarting them.

Checked the logs and had this error:

Oct 16 20:25:27 Tower kernel: BTRFS: error (device sdb1: state A) in btrfs_run_delayed_refs:2150: errno=-117 Filesystem corrupted Oct 16 20:25:27 Tower kernel: BTRFS info (device sdb1: state EA): forced readonly

 

Tried to search for this on google and I tried stopping my array to check the disk:

  " [1/7] checking root items
    [2/7] checking extents
    data extent[3231965184, 73728] referencer count mismatch (root 5 owner 561351 offset 29724672) wanted 0 have 1
    data extent[3231965184, 73728] bytenr mimsmatch, extent item bytenr 3231965184 file item bytenr 0
    data extent[3231965184, 73728] referencer count mismatch (root 5 owner 562949953982663 offset 29724672) wanted 1 have 0
    backpointer mismatch on [3231965184 73728]
    ref mismatch on [3232342016 73728] extent item 2199023255553, found 1
    data extent[3251593216, 4096] referencer count mismatch (root 5 owner 29254 offset 0) wanted 0 have 1
    data extent[3251593216, 4096] bytenr mimsmatch, extent item bytenr 3251593216 file item bytenr 0
    data extent[3251593216, 4096] referencer count mismatch (root 5 owner 9007199254770246 offset 0) wanted 1 have 0
    backpointer mismatch on [3251593216 4096]
    ref mismatch on [3290152960 815104] extent item 562949953421313, found 1
    ERROR: errors found in extent allocation tree or chunk allocation
    [3/7] checking free space tree
    [4/7] checking fs roots
    [5/7] checking only csums items (without verifying data)
    [6/7] checking root refs
    [7/7] checking quota groups skipped (not enabled on this FS)
    Opening filesystem to check...
    Checking filesystem on /dev/sdb1
    UUID: b1938f96-4aad-4196-9324-1cf0c29b614d
    found 375251902464 bytes used, error(s) found
    total csum bytes: 359642868
    total tree bytes: 1185349632
    total fs tree bytes: 381976576
    total extent tree bytes: 330072064
    btree space waste bytes: 228839528
    file data blocks allocated: 5968136134656
     referenced 293417140224"

 

When I now start my array the disk cannot mount at all: "Unmountable: Unsupported or no file system".

 

Downloaded a log file as the system is right now.

tower-diagnostics-20241016-2128.zip

I can see more errors in the logs now:

 

Oct 16 21:13:11 Tower kernel: BTRFS error (device sdb1): tree first key mismatch detected, bytenr=595296256 parent_transid=1243609 key expected=(17021235232,168,4096) has=(17021235200,168,4096)
Oct 16 21:13:11 Tower kernel: BTRFS error (device sdb1): tree first key mismatch detected, bytenr=595296256 parent_transid=1243609 key expected=(17021235232,168,4096) has=(17021235200,168,4096)
Oct 16 21:13:11 Tower kernel: BTRFS: error (device sdb1) in btrfs_replay_log:2426: errno=-117 Filesystem corrupted (Failed to recover log tree)
Oct 16 21:13:11 Tower root: mount: /mnt/cache: mount(2) system call failed: Structure needs cleaning.
Oct 16 21:13:11 Tower root:        dmesg(1) may have more information after failed mount system call.
Oct 16 21:13:11 Tower emhttpd: shcmd (444): exit status: 32
Oct 16 21:13:11 Tower emhttpd: /mnt/cache mount error: Unsupported or no file system

 

Should I try running a scrub?

Try

 

btrfs rescue zero-log /dev/sdb1

 

Then restart the array, if it mounts I would recommend backing up and re-formatting the pool, since it looks like the fs has other issues other than just the log tree.

55 minutes ago, JorgeB said:

Try

 

btrfs rescue zero-log /dev/sdb1

 

Then restart the array, if it mounts I would recommend backing up and re-formatting the pool, since it looks like the fs has other issues other than just the log tree.

I'm guessing that I need to stop the array berfore performing this action or should I start it in maintenance mode?

It can be running since the pool didn't mount

1 hour ago, JorgeB said:

It can be running since the pool didn't mount

Thank you! Just tried the command that you sent earlier and it seems like it worked. The cache disk is now mounted. Though I cannot start two of my docker applications.

 

You mentioned that I should back-up and re-format the pool. Can I just move all the data the pool to my array, reformat the cache pool and then move it back?

Edited October 17, 20241 yr by Pedrofranken
Adding info about docker apps

Yes.

Docker image may need to be recreate, but post new diags after the backups/restore if it still doesn't start

Thank you so much!

 

I got this error from Fix Common Problems: "Share Lagring set to not use the cache, but files / folders exist on the cache drive".

 

As of now I changed that share to use cache and added the mover action to: "Cache -> Array" and then started the mover. It's currently running, could that be the reason for my issues with the FS?

That should not cause filesystem corruption.

Alright! It looks like you were right, I'm still having issues.

 

Falling back to moving everything from cache pool to the array and then format the cache pool.

6 hours ago, JorgeB said:

Try

 

btrfs rescue zero-log /dev/sdb1

 

Then restart the array, if it mounts I would recommend backing up and re-formatting the pool, since it looks like the fs has other issues other than just the log tree.

The disk mounted after I used the command. I've set all the shares that was on the cache to mover action: "cache -> array" and started the mover. It seems like the mover action isn't able to complete.

 

When I started the mover I had 425 GB available on my array and 213 GB on my cache. I left the machine for quite a while and it seems like the mover is done (or cancelled) as I now have 420 GB available on my array and 212 GB on the cache.

 

I can also see that Fix Common Problems found this error: "Unable to write to cache - Drive mounted read-only or being completely full".

 

I'm uploading a new diag file.

tower-diagnostics-20241017-1650.zip

Pool is going read only, you can try copying the data, instead of moving it.

1 hour ago, JorgeB said:

Pool is going read only, you can try copying the data, instead of moving it.

Hmm... I disabled Docker and VM Manager in Unraid and then I compared the folders in /mnt/cache/ and /mnt/user0/. I began to copy my appdata-folder, but it started off with some issues right off the bat..

 

root@Tower:/mnt/cache/appdata# cp -r /mnt/cache/appdata /mnt/user0/appdata/ 
cp: cannot stat '/mnt/cache/appdata/mariadb/databases': No such file or directory 
cp: cannot stat '/mnt/cache/appdata/mariadb/log/mysql/mariadb-bin.index': No such file or directory 
cp: cannot stat '/mnt/cache/appdata/mariadb/log/mysql/mariadb-bin.000301': No such file or directory 
cp: cannot stat '/mnt/cache/appdata/mariadb/log/mysql/mariadb-bin.000302': No such file or directory

There could be some corruption, try copying what you can.

13 hours ago, JorgeB said:

There could be some corruption, try copying what you can.

Copied everything that I could, re-formated the cache pool, changed back the settings on the shares that I want on the cachepool and started the mover.

 

After that I enabled docker and VM and it seems like the docker img was broken. Atleast I had my data left so I could reinstall the docker applications and it works!