ati Posted September 30, 2020 Share Posted September 30, 2020 I believe my 2 drive BTRFS cache pool is falling and is stuck in read-only mode. I cannot get my Docker service to start: Sep 30 08:35:59 unRAID root: ERROR: unable to resize '/var/lib/docker': Read-only file system Sep 30 08:35:59 unRAID root: Resize '/var/lib/docker' of 'max' Sep 30 08:35:59 unRAID emhttpd: shcmd (216): /etc/rc.d/rc.docker start Sep 30 08:35:59 unRAID root: starting dockerd ... Sep 30 08:36:14 unRAID emhttpd: shcmd (218): umount /var/lib/docker I am trying to use the mover to clear out my cache drives so I can replace them, but that will not work either. I figured that would be fine for a read-only file system, but I guess not. Should be moving from cache to disk1. Sep 30 08:32:32 unRAID root: mover: started Sep 30 08:32:32 unRAID move: move: file /mnt/cache/Movies/MOVIE1.mp4 Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies/MOVIE1.mp4 error: Read-only file system Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies error: Read-only file system Sep 30 08:32:32 unRAID move: move: file /mnt/cache/Movies/MOVIE2.mkv Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies/MOVIE2.mkv error: Read-only file system Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies error: Read-only file system Sep 30 08:32:32 unRAID move: move: file /mnt/cache/Movies/MOVIE3.mkv Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies/MOVIE3.mkv error: Read-only file system Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies error: Read-only file system Sep 30 08:32:32 unRAID move: move: file /mnt/cache/Movies/MOVIE4.mkv Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies/MOVIE4.mkv error: Read-only file system Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies error: Read-only file system Sep 30 08:32:32 unRAID move: move: file /mnt/cache/Movies/MOVIE5.mp4 Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies/MOVIE5.mp4 error: Read-only file system Sep 30 08:32:32 unRAID move: move: create_parent: /mnt/cache/Movies error: Read-only file system Sep 30 08:32:32 unRAID move: move_object: /mnt/cache/Movies: Read-only file system Sep 30 08:32:33 unRAID move: move_object: /mnt/disk1/isos: Read-only file system Sep 30 08:32:33 unRAID move: move: file /mnt/disk2/isos/ubuntu-20.04.1-desktop-amd64.iso Sep 30 08:32:33 unRAID move: move: create_parent: /mnt/disk2/isos error: Read-only file system Sep 30 08:32:33 unRAID move: move_object: /mnt/disk2/isos: Read-only file system This issue came about because one of the main drives in my array had some read errors recently. So yesterday I stopped the array, pulled the drive and replaced it. I started the array and allowed it to rebuild. This morning I noticed my Docker service failed to start so I did a little digging. Fix Common Problems called out that my cache drive pool was mounted in read-only mode. I am assuming because of the number of errors? One other strange thing is when I start the array I get a notification that one of the cache pool disks is missing, but it doesn't show as missing after the array starts. Tried starting and stopping the array again with no change. I just rebooted the server as well just to see - no change either. I'd like to try and move everything off the cache pool into the array so I can replace both cache drives as both have issues. Looking for some guidance and I am a unRAID newbie and a little lost with my current situation. unraid-diagnostics-20200930-0844.zip Quote Link to comment
JorgeB Posted September 30, 2020 Share Posted September 30, 2020 You have a 3 device cache pool with a device missing and another one failing, so it's beyond its redundancy, you can try to manually copy everything you can, but there will likely be some data loss, especially since the metadata was on the missing device and in the failing device. Quote Link to comment
ati Posted September 30, 2020 Author Share Posted September 30, 2020 (edited) It shouldn't be a 3 device pool. I set it up with only 2 drives. I replaced one a while back, could that be the 3rd drive? There are no historical drives listed if I stop the array. Plus the drive it thinks is missing, it is also reporting as present? Edited September 30, 2020 by ati Added info on historical drives Quote Link to comment
JorgeB Posted September 30, 2020 Share Posted September 30, 2020 Data Metadata System Id Path RAID1 RAID1 RAID1 Unallocated -- --------- --------- --------- -------- ----------- 1 missing 62.00GiB 1.00GiB 32.00MiB -63.03GiB 2 /dev/sdd1 185.00GiB 1.00GiB 32.00MiB 3.46TiB 3 /dev/sdf1 123.00GiB - - 3.52TiB -- --------- --------- --------- -------- ----------- Total 185.00GiB 1.00GiB 32.00MiB 6.91TiB Used 178.58GiB 332.02MiB 48.00KiB DevId #1 is missing, sdd is failing. Quote Link to comment
JorgeB Posted September 30, 2020 Share Posted September 30, 2020 This missing device can be from some time earlier, and the pool never been able to rebalanced due to the errors on sdd, you were also having hardware errors on all devices, after this is solved look here for some more info. Sep 30 08:29:39 unRAID kernel: BTRFS info (device sdd1): bdev (null) errs: wr 2337, rd 206, flush 49, corrupt 0, gen 0 Sep 30 08:29:39 unRAID kernel: BTRFS info (device sdd1): bdev /dev/sdd1 errs: wr 1, rd 5717, flush 0, corrupt 0, gen 0 Sep 30 08:29:39 unRAID kernel: BTRFS info (device sdd1): bdev /dev/sdf1 errs: wr 0, rd 9, flush 0, corrupt 0, gen 0 Quote Link to comment
ati Posted September 30, 2020 Author Share Posted September 30, 2020 That makes sense. Thank you for the help. I am assuming the hardware errors were on the cache pool, or were you referring to the data array as well? What is the best way to recover from this being that my system, appdata and domains folders are on the failing cache pool? Quote Link to comment
JorgeB Posted September 30, 2020 Share Posted September 30, 2020 You can try btrfs restore (option 2 here), but like mentioned at least the metada is only available on the failing disk, so data loss is likely, for the future it's good practice to have a backup of any important data. Quote Link to comment
ati Posted September 30, 2020 Author Share Posted September 30, 2020 I am not worried about data loss. I am just worried about how unRAID will handle the system, appdata and domains folders going missing. Will it automatically recreate them on the array until I replace the cache setup and move them over or something? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.