March 18, 20224 yr Hello! I've been having some issues where my array and cache drives go read only after a few hours. Parity checks complete without any issues. Everything works fine for a few hours after a reboot -- but eventually the array will go read only. Attached are my diagnostics. Any suggestions? Thanks! enterprise-diagnostics-20220317-2102.zip
March 18, 20224 yr Community Expert Pool filesystem is corrupt, best bet to fix it is to backup and re-format the pool.
March 19, 20224 yr Author That's unfortunate, any other options other than a backup and re-format? I have a cloud backup, but with 45TB of data - that's going to take a while... -- edit -- This seems to be the error in the syslog right before the entire array goes read only. Mar 18 23:06:45 Enterprise kernel: BTRFS critical (device sdd1): corrupt leaf: root=10 block=602423296 slot=522, unexpected item end, have 2290802566 expect 16275 Mar 18 23:06:45 Enterprise kernel: BTRFS error (device sdd1): block=602423296 read time tree block corruption detected Mar 18 23:06:45 Enterprise kernel: BTRFS: error (device sdd1) in add_to_free_space_tree:1039: errno=-5 IO failure Mar 18 23:06:45 Enterprise kernel: BTRFS info (device sdd1): forced readonly Mar 18 23:06:45 Enterprise kernel: BTRFS: error (device sdd1) in __btrfs_free_extent:3221: errno=-5 IO failure Mar 18 23:06:45 Enterprise kernel: BTRFS: error (device sdd1) in btrfs_run_delayed_refs:2144: errno=-5 IO failure -- edit 2 -- Well, I seem to have been mistaken in stating that the entire array goes read only. It seems that the share I was testing from was set to utilize the cache, so it would throw a read only error. It appears that it is just my cache that is going read only. Edited March 19, 20224 yr by vb543
March 19, 20224 yr 6 hours ago, vb543 said: it is just my cache that is going read only. yes, that's what Jorge said : 23 hours ago, JorgeB said: Pool filesystem is corrupt, best bet to fix it is to backup and re-format the pool. Not the Array.
March 19, 20224 yr Author Ahh, "Pool" aka the Cache pool. Sorry for the misunderstanding! It seems the mover is going extremely slow / not clearing the cache pool. Is there a recommended method for manually copying its contents over to the array? -- edit -- Was able to use the unbalance plugin to copy everything to the array. Then I used 'blkdiscard /dev/sd#' to wipe the cache drives and formatted them after starting the array. So far so good. Thanks everyone! Edited March 19, 20224 yr by vb543
March 19, 20224 yr Author I seem to be having an issue with most of my docker containers now. I used the mover and set my appdata to prefer the cache. I can see all the appdata back on the cache pool, however - when I start most of my dockers, they appear to have lost all their settings/configs and come up as if they were just installed. Any ideas? -- edit -- Take this UniFi Controller docker for example. The appdata is there, I start the container and can see it's accessing the same appdata -- but then the docker takes me through the 'new setup' wizard as if it has never been configured before. root@Enterprise:/mnt/user/appdata/unifi-controller# ls -la total 0 drwxr-xr-x 1 nobody users 22 Dec 3 2019 ./ drwxrwxrwx 1 nobody users 1098 Mar 19 15:21 ../ drwxr-xr-x 1 nobody users 42 Mar 5 15:48 data/ drwxr-xr-x 1 nobody users 0 May 22 2021 logs/ drwxr-xr-x 1 nobody users 8 Mar 3 15:33 run/ root@Enterprise:/mnt/user/appdata/unifi-controller# ls -la total 0 drwxr-xr-x 1 nobody users 22 Dec 3 2019 ./ drwxrwxrwx 1 nobody users 1098 Mar 19 15:21 ../ drwxr-xr-x 1 nobody users 200 Mar 19 20:35 data/ drwxr-xr-x 1 nobody users 52 Mar 19 20:34 logs/ drwxr-xr-x 1 nobody users 86 Mar 19 20:35 run/ root@Enterprise:/mnt/user/appdata/unifi-controller# Edited March 20, 20224 yr by vb543
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.