Docker Service Crashing - Read Only (Drive Not Full)

[rguinn]

Hello All,

 

I am in some need of help from my logs i am not seeing any issues but i am having a issue with unraid and docker it will stop working and the fix common problems will tell me it is Read Only but my drives are not Full.

 

I have deleted my Docker Image

I have verify all the shares are set to high water

 

i am adding two diag and a few photo of the disk and shares showing free space

 

Also when this happen i can't reboot/shutdown my sever via GUI or Terminal it does not do anything and only a hard boot solves it for a few hours. 

 

 

thedoctor-diagnostics-20221231-1018.zip thedoctor-diagnostics-20221228-2140.zip

[Squid]

DOCKER_IMAGE_FILE="/mnt/user/system/docker/docker.img"
DOCKER_IMAGE_SIZE="75"

Setting a 75Gig docker image is akin to taking your Ferrari down the block to pick up your mail.  Sure it works, but it's massive overkill.

 

The 75G also implies that you've had serious problems with the image filling up in the past.  By and large, 99.99% of users will never even use the default 20G space.  My suggestion is to either sort out why the image keeps filling up (in the past), or switch to a docker folder so that you don't need to ever worry about it again.

 

(As an aside, 3G for the libvirt image is also overkill, and I can't think of anyone who has ever needed more than the default 1G)

 

But, the real problem here is

Dec 28 19:11:41 TheDoctor kernel: BTRFS critical (device sdo1): corrupt leaf: root=5 block=509462364160 slot=6 ino=496263061 file_offset=339968, invalid compression for file extent, have 136 expect range [0, 3]

 

And because of the corruption the cache drive gets remounted as read-only and is causing your issues. I'm going to leave it to the resident guru @JorgeB to assist further in recovery on this.

[rguinn]

First Thank you for the help and yes I was having issues with it as I was running a lot of dockers (80) and had to expand the size. I can reduce it as I am no longer running them all on UNRaid and have spread across others 

 

I should add this is a new issue that happened in the last update. 

 

as far as the libvirt image I don’t think I ever changed that setting as I don’t utilize VM much I have like 1 on this box I have a different server for VMs. 
 

but as I agree that not standard I don’t this the a root cause as I seem to only be using 10GB 

 

[rguinn]

1 hour ago, Squid said:

DOCKER_IMAGE_FILE="/mnt/user/system/docker/docker.img"
DOCKER_IMAGE_SIZE="75"

Setting a 75Gig docker image is akin to taking your Ferrari down the block to pick up your mail.  Sure it works, but it's massive overkill.

 

The 75G also implies that you've had serious problems with the image filling up in the past.  By and large, 99.99% of users will never even use the default 20G space.  My suggestion is to either sort out why the image keeps filling up (in the past), or switch to a docker folder so that you don't need to ever worry about it again.

 

(As an aside, 3G for the libvirt image is also overkill, and I can't think of anyone who has ever needed more than the default 1G)

 

But, the real problem here is

Dec 28 19:11:41 TheDoctor kernel: BTRFS critical (device sdo1): corrupt leaf: root=5 block=509462364160 slot=6 ino=496263061 file_offset=339968, invalid compression for file extent, have 136 expect range [0, 3]

 

And because of the corruption the cache drive gets remounted as read-only and is causing your issues. I'm going to leave it to the resident guru @JorgeB to assist further in recovery on this.

@Squid and @JorgeB should i just move all the items on the cache to the Array and blow out the cache and move the files back ?

 

Also i went to verify and the image is showing only 1 GB - so ill blow that out too just to be safe

 

 

I think my Cache Drive is got major issues i can't even move the files

Edited December 31, 2022 by rguinn

[rguinn]

Well I have moved all the files via unbalanced plugin to drive 11 and then deleted the Cache and have recreated and formatted the drive I am moving the AppData back and allowed the system to create new Docker and VM image

[trurl]

5 hours ago, Squid said:

 

Dec 28 19:11:41 TheDoctor kernel: BTRFS critical (device sdo1): corrupt leaf: root=5 block=509462364160 slot=6 ino=496263061 file_offset=339968, invalid compression for file extent, have 136 expect range [0, 3]

 

Have you done memtest recently?

 

 

[rguinn]

2 minutes ago, trurl said:

Have you done memtest recently?

 

 

Like 2 Months ago no issues

[trurl]

You should do memtest

[rguinn]

8 minutes ago, trurl said:

You should do memtest

I am copying all my files back as soon as its done ill run a test

[trurl]

19 minutes ago, rguinn said:

I am copying all my files back as soon as its done ill run a test

Bad memory can corrupt file copying.

 

Maybe RAM is OK. But, you shouldn't even attempt to run any computer unless memory is working perfectly. Everything goes through RAM, OS and other executable code, your data, everything. CPU can't do anything with anything until it is loaded in RAM.

[rguinn]

Just now, trurl said:

Bad memory can corrupt file copying.

 

Maybe RAM is OK. But, you shouldn't even attempt to run any computer unless memory is working perfectly. Everything goes through RAM, OS and other executable code, your data, everything. CPU can't do anything with anything until it is loaded in RAM.

Yes and this is why i had run a memory test recently but ill run it again as soon as the file copy is done

 

[rguinn]

4 hours ago, trurl said:

You should do memtest

Memtest done and all passed

[rguinn]

Well this morning I woke up and my server won’t even load GUI

 

the array is still working and mounted but Docker are not working and now web gui is not responding 

 

I am working on getting a diagnostic file before I reboot 

Edited January 1, 2023 by rguinn

[trurl]

What do you get from command line with this?

df -h /

 

[trurl]

You should setup syslog server

[rguinn]

5 minutes ago, trurl said:

What do you get from command line with this?

df -h /

 

 

Here is the new logs @Squid and @JorgeB and @trurl

thedoctor-diagnostics-20230101-0932.zip

Edited January 1, 2023 by rguinn
add tags

[trurl]

Looks like your monthly parity check started at 1am. Is it still running?

[rguinn]

1 minute ago, trurl said:

Looks like your monthly parity check started at 1am. Is it still running?

 

@trurl The Parity Check should not take down the system or cause docker to crash right ?

[trurl]

should not if things are working well. Do any of your disks have SMART warnings on the Dashboard page?

[rguinn]

4 minutes ago, trurl said:

should not if things are working well. Do any of your disks have SMART warnings on the Dashboard page?

no they did not let me hardboot so i can access again and ill share

[rguinn]

11 minutes ago, trurl said:

should not if things are working well. Do any of your disks have SMART warnings on the Dashboard page?

 

@trurl Nope all so Healthy

 

[rguinn]

Well since i out of ideas and it only started with the new update i created a new USB with 6.11.3 and moved only these config files

 

Lets see if a clean config with fix it

[JonathanM]

Just an aside, your signature is pretty much unreadable to a large percentage of forum users. Your font color is too close to the light theme background.

[rguinn]

42 minutes ago, JonathanM said:

Just an aside, your signature is pretty much unreadable to a large percentage of forum users. Your font color is too close to the light theme background.

I have update it! sadly i attempted to just downgrade and same issue woke up to non working server 

[trurl]

1 minute ago, rguinn said:

I have update it!

If you mean your sig, still not working. Did you test with the light theme?