Cache Drive Error Leading To Docker Issues

[UpperCenter]

Hi,

 

Earlier today I noticed my Docker containers were no longer responding. I checked the Main tab, and see this error from my cache drives:

 

Cache: Unmountable: Unsupported or no file system

 

I "resolved" this error by Googling and searching the forums, and I ended up running `btrfs rescue zero-log /dev/sdg1` and `btrfs rescue zero-log /dev/sdc1` I rebooted the server and now the cache drives seem to be mounted OK, and I can navigate to the share and see the files just fine. However, at this point my docker.img also seemed to be corrupted for some reason. I deleted the vDisk file as I read on the forums that this is a common fix, and I can simply re-create my containers with the existing config.

 

However, the cache drives seem to be mounted in read-only filesystem. This whole situation is quite strange to me, as the server was idle all day, the SSD pool has ~400GB free, and I've not made any changes to cables or drives. The SSDs were both purchased new, and are passing SMART OK, so I'm really confused as to why this has happened.

 

I think the underlying issue is that my cache disks are always being mounted as read only:

root@io:~# touch /mnt/user/system/docker/docker.img
touch: cannot touch '/mnt/user/system/docker/docker.img': Read-only file system

 

How can I recover from this, and ensure no data loss?

Thanks

 

Diagnostics uploaded.

io-diagnostics-20240813-2224.zip

Edited August 14 by UpperCenter
More info

[UpperCenter]

I should add I'm on Unraid 6.12.11. I also noticed this in the diagnostics:

 

Aug 13 22:18:55 io avahi-dnsconfd[20255]: Successfully connected to Avahi daemon.
Aug 13 22:18:55 io emhttpd: shcmd (157): /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 50
Aug 13 22:18:55 io root: truncate: cannot open '/mnt/cache/system/docker/docker.img' for writing: Read-only file system
Aug 13 22:18:55 io kernel: loop3: detected capacity change from 0 to 104857600
Aug 13 22:18:55 io root: mount: /var/lib/docker: wrong fs type, bad option, bad superblock on /dev/loop3, missing codepage or helper program, or other error.
Aug 13 22:18:55 io root:        dmesg(1) may have more information after failed mount system call.
Aug 13 22:18:55 io root: mount error
Aug 13 22:18:55 io kernel: squashfs: Unknown parameter 'space_cache'
Aug 13 22:18:55 io kernel: fuseblk: Unknown parameter 'space_cache'
Aug 13 22:18:55 io kernel: UDF-fs: bad mount option "space_cache=v2" or missing value
Aug 13 22:18:55 io kernel: xfs: Unknown parameter 'space_cache'
Aug 13 22:18:55 io emhttpd: shcmd (157): exit status: 1
Aug 13 22:18:56 io avahi-daemon[20246]: Server startup complete. Host name is io.local. Local service cookie is 3086686641.

 

[JorgeB]

Pool is going read-only, suggest backup and re-format.

[UpperCenter]

7 minutes ago, JorgeB said:

Pool is going read-only, suggest backup and re-format.

Thanks, do you know what might have caused this to happen? Is there no other solution to bring the drives out of read only?

 

Thanks

[JorgeB]

25 minutes ago, UpperCenter said:

Is there no other solution to bring the drives out of read only?

You could try a filesystem repair, but with btrfs I usually recommend re-formatting, and don't try the repair without backups, because sometimes it can makes things worse.

[UpperCenter]

Even running Mover from my Cache to Array is giving read-only filesystem errors. How am I meant to backup the data if it can't be moved?

Aug 14 19:41:11 io move: create_parent: /mnt/cache/data/images/itemIcons error: Read-only file system
Aug 14 19:41:11 io move: file: /mnt/cache/data/images/itemIcons/sugarbell.png
Aug 14 19:41:11 io move: create_parent: /mnt/cache/data/images/itemIcons error: Read-only file system
Aug 14 19:41:11 io move: file: /mnt/cache/data/images/itemIcons/sumac.png

 

Do I just have to use SFTP or something to another machine? I can tell it's not working because the "writes" on my array are not going up.

Edited August 14 by UpperCenter

[JorgeB]

13 hours ago, UpperCenter said:

Even running Mover from my Cache to Array is giving read-only filesystem errors.

That is expected since the mover tries to delete the data and can't, it should still copy it, but you can also use a different tool, like Dynamix File Manager plugin, or Midnight Commander (mc on the CLI)