Cache is Read Only - Cannot Replace Drives


ati

Recommended Posts

I believe my 2 drive BTRFS cache pool is falling and is stuck in read-only mode. I cannot get my Docker service to start:

Sep 30 08:35:59 unRAID root: ERROR: unable to resize '/var/lib/docker': Read-only file system
Sep 30 08:35:59 unRAID root: Resize '/var/lib/docker' of 'max'
Sep 30 08:35:59 unRAID emhttpd: shcmd (216): /etc/rc.d/rc.docker start
Sep 30 08:35:59 unRAID root: starting dockerd ...
Sep 30 08:36:14 unRAID emhttpd: shcmd (218): umount /var/lib/docker

I am trying to use the mover to clear out my cache drives so I can replace them, but that will not work either. I figured that would be fine for a read-only file system, but I guess not. Should be moving from cache to disk1.

Sep 30 08:32:32 unRAID root: mover: started
Sep 30 08:32:32 unRAID move: move: file /mnt/cache/Movies/MOVIE1.mp4
Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies/MOVIE1.mp4 error: Read-only file system
Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies error: Read-only file system
Sep 30 08:32:32 unRAID move: move: file /mnt/cache/Movies/MOVIE2.mkv
Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies/MOVIE2.mkv error: Read-only file system
Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies error: Read-only file system
Sep 30 08:32:32 unRAID move: move: file /mnt/cache/Movies/MOVIE3.mkv
Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies/MOVIE3.mkv error: Read-only file system
Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies error: Read-only file system
Sep 30 08:32:32 unRAID move: move: file /mnt/cache/Movies/MOVIE4.mkv
Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies/MOVIE4.mkv error: Read-only file system
Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies error: Read-only file system
Sep 30 08:32:32 unRAID move: move: file /mnt/cache/Movies/MOVIE5.mp4
Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies/MOVIE5.mp4 error: Read-only file system
Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies error: Read-only file system
Sep 30 08:32:32 unRAID move: move_object: /mnt/cache/Movies: Read-only file system
Sep 30 08:32:33 unRAID move: move_object: /mnt/disk1/isos: Read-only file system
Sep 30 08:32:33 unRAID move: move: file /mnt/disk2/isos/ubuntu-20.04.1-desktop-amd64.iso
Sep 30 08:32:33 unRAID move: move: create_parent: /mnt/disk2/isos error: Read-only file system
Sep 30 08:32:33 unRAID move: move_object: /mnt/disk2/isos: Read-only file system

This issue came about because one of the main drives in my array had some read errors recently. So yesterday I stopped the array, pulled the drive and replaced it. I started the array and allowed it to rebuild. This morning I noticed my Docker service failed to start so I did a little digging. Fix Common Problems called out that my cache drive pool was mounted in read-only mode. I am assuming because of the number of errors? One other strange thing is when I start the array I get a notification that one of the cache pool disks is missing, but it doesn't show as missing after the array starts. 

 

Tried starting and stopping the array again with no change. I just rebooted the server as well just to see - no change either. 

 

I'd like to try and move everything off the cache pool into the array so I can replace both cache drives as both have issues. 

 

Looking for some guidance and I am a unRAID newbie and a little lost with my current situation. 

 

 

unraid-diagnostics-20200930-0844.zip

Link to comment

It shouldn't be a 3 device pool. I set it up with only 2 drives. I replaced one a while back, could that be the 3rd drive? There are no historical drives listed if I stop the array.

Plus the drive it thinks is missing, it is also reporting as present?

 

Cache.thumb.jpg.8ed1464dc006d6cd62eb37a1b2446f92.jpg

Edited by ati
Added info on historical drives
Link to comment
             Data      Metadata  System              
Id Path      RAID1     RAID1     RAID1    Unallocated
-- --------- --------- --------- -------- -----------
 1 missing    62.00GiB   1.00GiB 32.00MiB   -63.03GiB
 2 /dev/sdd1 185.00GiB   1.00GiB 32.00MiB     3.46TiB
 3 /dev/sdf1 123.00GiB         -        -     3.52TiB
-- --------- --------- --------- -------- -----------
   Total     185.00GiB   1.00GiB 32.00MiB     6.91TiB
   Used      178.58GiB 332.02MiB 48.00KiB   

DevId #1 is missing, sdd is failing.

Link to comment

This missing device can be from some time earlier, and the pool never been able to rebalanced due to the errors on sdd, you were also having hardware errors on all devices, after this is solved look here for some more info.

 

Sep 30 08:29:39 unRAID kernel: BTRFS info (device sdd1): bdev (null) errs: wr 2337, rd 206, flush 49, corrupt 0, gen 0
Sep 30 08:29:39 unRAID kernel: BTRFS info (device sdd1): bdev /dev/sdd1 errs: wr 1, rd 5717, flush 0, corrupt 0, gen 0
Sep 30 08:29:39 unRAID kernel: BTRFS info (device sdd1): bdev /dev/sdf1 errs: wr 0, rd 9, flush 0, corrupt 0, gen 0

 

Link to comment

That makes sense. Thank you for the help.

I am assuming the hardware errors were on the cache pool, or were you referring to the data array as well? 

 

What is the best way to recover from this being that my system, appdata and domains folders are on the failing cache pool? 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.