[SOLVED] Unable to resize '/var/lib/docker': Read-only file system

[liquidate]

I'm getting the following error when I attempt to recreate my docker image (by deleting the current image):

 

Apr 29 21:03:59 Tower root: ERROR: unable to resize '/var/lib/docker': Read-only file system

 

Any suggestions?  I was getting 403 errors trying to start one of my dockers, now I can't start anything.

tower-diagnostics-20210429-2111.zip

[JorgeB]

Btrfs was detecting data corruption before going read-only, you're using ECC RAM so unlikely to be that, but probably there's some hardware issue, try a different NVMe device if available.

[liquidate]

Thank you for the response!  Could it be the instability I saw when I updated to Unraid 6.9 from 6.8.3?  I found that Unraid would just hang on me, so I downgraded back to 6.8.3 and things have been solid since.

 

What steps should I take to retain my data and fix this issue?  I do not want to loose my bitwarden or other docker appdata.

[JorgeB]

Reboot and if cache mounts run a scrub, if uncorrectable errors are found check the syslog for the affected files, those need to be deleted or replaced, if it doesn't mount there are some recovery options here.

[liquidate]

Thank you for your help!  I've run scrub multiple times and deleted the files that were reported in the log.  It appeared all of the corrupted files were in my unifi-controller appdata folder.  Once those were all deleted, and scrub returned no errors, I tried to delete the entire unifi-controller app data folder.  While windows reported the folder was deleted, it still existed.  I then put the array into maintenance mode and the file system check came back with errors.  I ran a repair, did another scrub, deleted the unifi-controller folder (for real this time), and everything seems back to normal.

 

I am concerned about upgrading this system to 6.9.2, given the hangs that occurred with 6.9, but that's another issue and not a pressing matter given 6.8.3 runs rock solid for me.

[2112]

@JorgeB I have another one for you if you're able to help.  I am getting "ERROR: unable to resize '/var/lib/docker': Read-only file system" in my log.  Docker service won't start.

plex-bu-diagnostics-20240208-1646.zip

[trurl]

cache is read-only again, just like 2 years ago.

[2112]

@trurl I have read through the post.  I am not seeing anything in the log file about bad files that should be removed.  I ran a scrub but am getting this error in the log "scrub: not finished on devid 1 with status: -30".  

@JorgeB I took the array offline and did a check filesystem on my cache pool.  This is what it came back with, "attempting to repair backref discrepancy for bytenr 686580432896 backrefs don't agree with each other and extent record doesn't agree with anybody, so we can't fix bytenr 686580432896 bytes 67268608 failed to repair damaged filesystem, aborting."  Earlier in the day I accidentally assigned a new drive to my cache pool then removed it.  I am not sure if that factors into any of this.  I think I am screwed lol!

[JorgeB]

Since the pool is still mounting I would recommend copying what you can/need and reformat.