Array & Cache Read Only

[vb543]

Hello! I've been having some issues where my array and cache drives go read only after a few hours. Parity checks complete without any issues. Everything works fine for a few hours after a reboot -- but eventually the array will go read only. Attached are my diagnostics. Any suggestions? Thanks!

enterprise-diagnostics-20220317-2102.zip

[JorgeB]

Pool filesystem is corrupt, best bet to fix it is to backup and re-format the pool.

[vb543]

That's unfortunate, any other options other than a backup and re-format? I have a cloud backup, but with 45TB of data - that's going to take a while...

 

-- edit --

This seems to be the error in the syslog right before the entire array goes read only.

 

Mar 18 23:06:45 Enterprise kernel: BTRFS critical (device sdd1): corrupt leaf: root=10 block=602423296 slot=522, unexpected item end, have 2290802566 expect 16275
Mar 18 23:06:45 Enterprise kernel: BTRFS error (device sdd1): block=602423296 read time tree block corruption detected
Mar 18 23:06:45 Enterprise kernel: BTRFS: error (device sdd1) in add_to_free_space_tree:1039: errno=-5 IO failure
Mar 18 23:06:45 Enterprise kernel: BTRFS info (device sdd1): forced readonly
Mar 18 23:06:45 Enterprise kernel: BTRFS: error (device sdd1) in __btrfs_free_extent:3221: errno=-5 IO failure
Mar 18 23:06:45 Enterprise kernel: BTRFS: error (device sdd1) in btrfs_run_delayed_refs:2144: errno=-5 IO failure

 

-- edit 2 --

Well, I seem to have been mistaken in stating that the entire array goes read only. It seems that the share I was testing from was set to utilize the cache, so it would throw a read only error. It appears that it is just my cache that is going read only.

Edited March 19, 2022 by vb543

[ChatNoir]

6 hours ago, vb543 said:

it is just my cache that is going read only.

yes, that's what Jorge said :

23 hours ago, JorgeB said:

Pool filesystem is corrupt, best bet to fix it is to backup and re-format the pool.

Not the Array.

[vb543]

Ahh, "Pool" aka the Cache pool. Sorry for the misunderstanding!

 

It seems the mover is going extremely slow / not clearing the cache pool. Is there a recommended method for manually copying its contents over to the array?

 

-- edit --

 

Was able to use the unbalance plugin to copy everything to the array. Then I used 'blkdiscard /dev/sd#' to wipe the cache drives and formatted them after starting the array. So far so good. Thanks everyone!

Edited March 19, 2022 by vb543

[vb543]

I seem to be having an issue with most of my docker containers now. I used the mover and set my appdata to prefer the cache. I can see all the appdata back on the cache pool, however - when I start most of my dockers, they appear to have lost all their settings/configs and come up as if they were just installed. Any ideas?

 

-- edit --

Take this UniFi Controller docker for example. The appdata is there, I start the container and can see it's accessing the same appdata -- but then the docker takes me through the 'new setup' wizard as if it has never been configured before.

 

root@Enterprise:/mnt/user/appdata/unifi-controller# ls -la
total 0
drwxr-xr-x 1 nobody users   22 Dec  3  2019 ./
drwxrwxrwx 1 nobody users 1098 Mar 19 15:21 ../
drwxr-xr-x 1 nobody users   42 Mar  5 15:48 data/
drwxr-xr-x 1 nobody users    0 May 22  2021 logs/
drwxr-xr-x 1 nobody users    8 Mar  3 15:33 run/
root@Enterprise:/mnt/user/appdata/unifi-controller# ls -la
total 0
drwxr-xr-x 1 nobody users   22 Dec  3  2019 ./
drwxrwxrwx 1 nobody users 1098 Mar 19 15:21 ../
drwxr-xr-x 1 nobody users  200 Mar 19 20:35 data/
drwxr-xr-x 1 nobody users   52 Mar 19 20:34 logs/
drwxr-xr-x 1 nobody users   86 Mar 19 20:35 run/
root@Enterprise:/mnt/user/appdata/unifi-controller#

 

Edited March 20, 2022 by vb543