Help please! - Warning: file_put_contents(/var/lib/docker/unraid/images/plex-icon.png): failed to open stream: Read-only file system

[nitewolfgtr]

Hello, 

 

I've been getting this error often lately and trying to figure out what is happening. 

When I go to Docker tab in Unraid, I get this error message and none of my Docker apps work. I have tried rebuilding the docker image few weeks ago and it worked for couple weeks and started happening again. 

 

Warning: file_put_contents(/var/lib/docker/unraid/images/plex-icon.png): failed to open stream: Read-only file system in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 93

Warning: file_put_contents(/var/lib/docker/unraid/images/radarr-icon.png): failed to open stream: Read-only file system in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 93

 

 

I have attached the Diag zip file. What should I be looking for? Can someone please help? Thank you!

nas-diagnostics-20230303-1705.zip

[Squid]

The root of your problem:

Mar  2 19:23:54 NAS  emhttpd: shcmd (88): mount -t btrfs -o noatime,space_cache=v2,discard=async -U 5e1bc338-27cf-4a34-8163-3495a924be67 /mnt/cache
Mar  2 19:23:54 NAS kernel: BTRFS info (device sdf1): turning on async discard
Mar  2 19:23:54 NAS kernel: BTRFS info (device sdf1): using free space tree
Mar  2 19:23:54 NAS kernel: BTRFS info (device sdf1): has skinny extents
Mar  2 19:23:54 NAS kernel: BTRFS info (device sdf1): bdev /dev/sdf1 errs: wr 0, rd 0, flush 0, corrupt 203, gen 0
Mar  2 19:23:54 NAS kernel: BTRFS info (device sdf1): bdev /dev/sdq1 errs: wr 0, rd 0, flush 0, corrupt 203, gen 0

Corrupt on a BTRFS pool is usually because of memory issues.  Run Memtest for a minimum of a single pass (ideally for a couple of passes)

 

This has trickled down to the docker.img being effectively read-only

 

 

[couzin2000]

I am having this same exact issue after having run a few updates to Docker images. I updated Unassigned devices, Recycle Bin, then I ran Dynamix File Manager.

After Dynamix having updated correctly, I went in to update "Crafty-4" Docker package. 
The update is supposed to delete the current container and replace it with the new one... in this case, it refused to delete the original becaise somehow the "filesystem is now read-only"?  It also says BTRFS, but the file it's pointing at is on an XFS format. Not sure what's going on here. 

I have also enclosed my diag file. 

As of right now, I cannot use any of my Minecraft servers (Crafty is the server manager) and it's stuck in "Removal in progress" status. The new image is currently "oprhan image" and I can't do anything with it. The update does not want to remove the older one. 

I can certainly run memtest, but I'd like to know HOW to change the drive's current state to one I can use. Should I copy files from my cache drives over to the array, and flush the SSD cache drive and start that whole process again? 
Please keep in mind I am running a Plex server and I want to make sure I don't have to start everything from scratch again. Thanks.

sebunraid1-diagnostics-20230610-1459.zip

Edited June 10, 2023 by couzin2000

[couzin2000]

I'm finding a few things in here, I'm wondering if this suggests the failure of my SSD:

Jun 8 08:32:03 sebunraid1 kernel: ACPI Error: Aborting method \_SB.PCI0.SAT0.SPT4._GTF due to previous error (AE_NOT_FOUND) (20220331/psparse-529)
Jun 8 08:32:03 sebunraid1 kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.SAT0.SPT3._GTF.DSSP], AE_NOT_FOUND (20220331/psargs-330)
Jun 8 08:32:03 sebunraid1 kernel: ACPI Error: Aborting method \_SB.PCI0.SAT0.SPT3._GTF due to previous error (AE_NOT_FOUND) (20220331/psparse-529)
Jun 8 08:32:07 sebunraid1 mcelog: failed to prefill DIMM database from DMI data
Jun 8 08:33:06 sebunraid1 kernel: BTRFS error (device loop2): error loading props for ino 11884 (root 314): -5
Jun 8 08:33:47 sebunraid1 kernel: BTRFS error (device loop2): parent transid verify failed on 285835264 wanted 6628 found 6564
Jun 8 08:33:47 sebunraid1 kernel: BTRFS info (device loop2): failed to delete reference to python3.10.xpm, inode 11884 parent 3189
Jun 8 08:33:47 sebunraid1 kernel: BTRFS: error (device loop2: state A) in __btrfs_unlink_inode:4325: errno=-5 IO failure

 

Edited June 10, 2023 by couzin2000

[JorgeB]

15 hours ago, couzin2000 said:

I'm wondering if this suggests the failure of my SSD:

It shows that your docker image is corrupt and needs to be recreated, btrfs is also detecting some data corruption on cache, so good idea to run memtest and a scrub after.

[couzin2000]

40 minutes ago, JorgeB said:

It shows that your docker image is corrupt and needs to be recreated, btrfs is also detecting some data corruption on cache, so good idea to run memtest and a scrub after.

Can you be a little more precise with me? I don't wanna mess up anything here, i' kind of a noob at Unraid.

 

you're talking about the "Crafty-4" Docker image that needs to be recreated? Or THE DOCKER image, i.e. the actual Docker program? 
if we're talking about Crafty, i'm not sure how to remove it because that is a part of the error I've been getting - during updating this app, the older Crafty container refused to remove itself, and the newer one was downloaded, ready to go. The old one still doesnt delete itslef vecause somehow the BTRFS is in read-only mode (which has never been the case, and I'm actually writing this image on an XFS drive) so the new one can't run a new container...

 

 

And as for the scrub you're referring to, what exactly is this? A clean up how?

[JonathanM]

4 hours ago, couzin2000 said:

Can you be a little more precise with me?

Have you read through the link he posted? If so, what parts of the instructions are you having issues with?

[couzin2000]

On 6/11/2023 at 7:04 AM, JorgeB said:

It shows that your docker image is corrupt and needs to be recreated, btrfs is also detecting some data corruption on cache, so good idea to run memtest and a scrub after.

Before I jump in and recreate the Docker.img (I didn't understand we were referring to the actual Docker program's image, sorry), I'd like to ask if there is something underlying that I need to address. I wouldn't want to blast the image and recreate it only to end up with the same problem. 

Does this imply I must put the array offline? Is there an easy guide I can follow? As said prior, I am fairly uneducated when it comes to Unraid, I havent had it long, so I need some step-by-step guidance. 

[couzin2000]

Edit: I tried Scrub in the Docker tools. I get this in the syslog:
Jun 12

Jun 12 16:40:54 sebunraid1 ool www[27058]: /usr/local/emhttp/plugins/dynamix/scripts/btrfs_scrub 'start' '/var/lib/docker' '-r' 
Jun 12 16:40:54 sebunraid1 kernel: BTRFS info (device loop2: state EA): scrub: started on devid 1 
Jun 12 16:40:54 sebunraid1 kernel: BTRFS info (device loop2: state EA): scrub: not finished on devid 1 with status: -30

 

[JorgeB]

I'm not seeing issues with pool, so for now solution is to recreate the image.

[couzin2000]

I hadn't had time to follow the instructions in the link. I went today to do it. As I'm following the first post's steps, I realize I'm reading a 2017 post. The options are not the same anymore.

I went and ran the "Scrub" option in Docker. Didn't find any errors. However, I cam back to the man screen and saw my docker images all good and running, and I was able to delete the orphaned image that was still there.  No corrections needed to the Docker.img file. 
Thanks anyway!

(please pin the new version of these instructions, could be very useful)

[JorgeB]

11 hours ago, couzin2000 said:

(please pin the new version of these instructions, could be very useful)

The instructions are current, note that they were updated earlier this year.