I believe my 2 drive BTRFS cache pool is falling and is stuck in read-only mode. I cannot get my Docker service to start:

Sep 30 08:35:59 unRAID root: ERROR: unable to resize '/var/lib/docker': Read-only file system
Sep 30 08:35:59 unRAID root: Resize '/var/lib/docker' of 'max'
Sep 30 08:35:59 unRAID emhttpd: shcmd (216): /etc/rc.d/rc.docker start
Sep 30 08:35:59 unRAID root: starting dockerd ...
Sep 30 08:36:14 unRAID emhttpd: shcmd (218): umount /var/lib/docker

I am trying to use the mover to clear out my cache drives so I can replace them, but that will not work either. I figured that would be fine for a read-only file system, but I guess not. Should be moving from cache to disk1.

Sep 30 08:32:32 unRAID root: mover: started
Sep 30 08:32:32 unRAID move: move: file /mnt/cache/Movies/MOVIE1.mp4
Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies/MOVIE1.mp4 error: Read-only file system
Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies error: Read-only file system
Sep 30 08:32:32 unRAID move: move: file /mnt/cache/Movies/MOVIE2.mkv
Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies/MOVIE2.mkv error: Read-only file system
Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies error: Read-only file system
Sep 30 08:32:32 unRAID move: move: file /mnt/cache/Movies/MOVIE3.mkv
Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies/MOVIE3.mkv error: Read-only file system
Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies error: Read-only file system
Sep 30 08:32:32 unRAID move: move: file /mnt/cache/Movies/MOVIE4.mkv
Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies/MOVIE4.mkv error: Read-only file system
Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies error: Read-only file system
Sep 30 08:32:32 unRAID move: move: file /mnt/cache/Movies/MOVIE5.mp4
Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies/MOVIE5.mp4 error: Read-only file system
Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies error: Read-only file system
Sep 30 08:32:32 unRAID move: move_object: /mnt/cache/Movies: Read-only file system
Sep 30 08:32:33 unRAID move: move_object: /mnt/disk1/isos: Read-only file system
Sep 30 08:32:33 unRAID move: move: file /mnt/disk2/isos/ubuntu-20.04.1-desktop-amd64.iso
Sep 30 08:32:33 unRAID move: move: create_parent: /mnt/disk2/isos error: Read-only file system
Sep 30 08:32:33 unRAID move: move_object: /mnt/disk2/isos: Read-only file system

This issue came about because one of the main drives in my array had some read errors recently. So yesterday I stopped the array, pulled the drive and replaced it. I started the array and allowed it to rebuild. This morning I noticed my Docker service failed to start so I did a little digging. Fix Common Problems called out that my cache drive pool was mounted in read-only mode. I am assuming because of the number of errors? One other strange thing is when I start the array I get a notification that one of the cache pool disks is missing, but it doesn't show as missing after the array starts. 

 

Tried starting and stopping the array again with no change. I just rebooted the server as well just to see - no change either. 

 

I'd like to try and move everything off the cache pool into the array so I can replace both cache drives as both have issues. 

 

Looking for some guidance and I am a unRAID newbie and a little lost with my current situation. 

 

 

unraid-diagnostics-20200930-0844.zip

You have a 3 device cache pool with a device missing and another one failing, so it's beyond its redundancy, you can try to manually copy everything you can, but there will likely be some data loss, especially since the metadata was on the missing device and in the failing device.

It shouldn't be a 3 device pool. I set it up with only 2 drives. I replaced one a while back, could that be the 3rd drive? There are no historical drives listed if I stop the array.

Plus the drive it thinks is missing, it is also reporting as present?

 

Edited September 30, 20205 yr by ati
Added info on historical drives

             Data      Metadata  System              
Id Path      RAID1     RAID1     RAID1    Unallocated
-- --------- --------- --------- -------- -----------
 1 missing    62.00GiB   1.00GiB 32.00MiB   -63.03GiB
 2 /dev/sdd1 185.00GiB   1.00GiB 32.00MiB     3.46TiB
 3 /dev/sdf1 123.00GiB         -        -     3.52TiB
-- --------- --------- --------- -------- -----------
   Total     185.00GiB   1.00GiB 32.00MiB     6.91TiB
   Used      178.58GiB 332.02MiB 48.00KiB   

DevId #1 is missing, sdd is failing.

This missing device can be from some time earlier, and the pool never been able to rebalanced due to the errors on sdd, you were also having hardware errors on all devices, after this is solved look here for some more info.

 

Sep 30 08:29:39 unRAID kernel: BTRFS info (device sdd1): bdev (null) errs: wr 2337, rd 206, flush 49, corrupt 0, gen 0
Sep 30 08:29:39 unRAID kernel: BTRFS info (device sdd1): bdev /dev/sdd1 errs: wr 1, rd 5717, flush 0, corrupt 0, gen 0
Sep 30 08:29:39 unRAID kernel: BTRFS info (device sdd1): bdev /dev/sdf1 errs: wr 0, rd 9, flush 0, corrupt 0, gen 0

 

That makes sense. Thank you for the help.

I am assuming the hardware errors were on the cache pool, or were you referring to the data array as well? 

 

What is the best way to recover from this being that my system, appdata and domains folders are on the failing cache pool? 

You can try btrfs restore (option 2 here), but like mentioned at least the metada is only available on the failing disk, so data loss is likely, for the future it's good practice to have a backup of any important data.

I am not worried about data loss. I am just worried about how unRAID will handle the system, appdata and domains folders going missing. Will it automatically recreate them on the array until I replace the cache setup and move them over or something?