Today i noticed i couldnt start my docker containers once stopped. Began receiving a Error 403 message when trying to start them up.

I have attached a system log - looks like a lot of errors begin Dec18 at 10:30+

Can someone please make sense of it for me.

Am i going to have to nuke the whole docker image file?

unraid-syslog-20241218-1221.zip

btrfs is detecting data corruption on the pool, start by running memtest, then scrub the pool, but yes, you will also need to recreate the docker image.

Thanks for the quick reply.

-What would cause this? Is it common? Its the second time I've been in this predicament.

Along with failing flash drives and bzfirmware firmware checksum errors, I seem to be always trying to keep this system running. Is there something I should be checking to avoid these issues?

45 minutes ago, Gibbo2787 said:

What would cause this?

Bad RAM for example, hence why I recommend running memtest.

Finally got around to doing the memtest today - Completed 5 passes with 0 errors

upon reboot was met again with bzfirmware checksum error - see attached photo

recreated the flash drive again and copied over contents of the config folder.

Booted back up fine then.

Started the array

Performed a scrub on the cache pool - 0 errors found.

Now do i have to delete the docker image and recreate or is there something else I should test first?

9 minutes ago, Gibbo2787 said:

Now do i have to delete the docker image and recreate or is there something else I should test first?

You can try doing that.

4 hours ago, Gibbo2787 said:

upon reboot was met again with bzfirmware checksum error - see attached photo

This error had to either be RAM or problems with that file on the flash drive.    I would suggests downloading the zip file for the Unraid version you are using and then extracting all the bz* type files overwriting the ones on the flash drive.

Well i tried.

 

Deleted the docker image and now I'm reinstalling the containers from previously installed app, watching them fail 1 by 1 . Now nothing works FML

1 hour ago, itimpi said:

This error had to either be RAM or problems with that file on the flash drive.    I would suggests downloading the zip file for the Unraid version you are using and then extracting all the bz* type files overwriting the ones on the flash drive.

This always seems to be the problem - not sure why it kept reoccuring

 

Containers not installing now because of an error saying its a read only file system.

When I stop and restart the docker service, it fails to start, and I have to recreate it again.

How do i nuke this thing and start again

I've got most of my containers reinstalled to some extent - however theres still things going on - Can someone please point me in the right direction. Ive been up for hours trying to get this sorted and its wearing thin.

 

When i go to install my last container (regardless of what it is) I'm met with this error now

docker: Error response from daemon: error creating temporary lease: write /var/lib/docker/containerd/daemon/io.containerd.metadata.v1.bolt/meta.db: read-only file system: unknown.

I have around 20 containers installed. I increased the docker image from 20gb to 30gb. I read somewhere to perform a balance. I found an option to perform a full balance on the cache but it doesnt appear to do anything. It says above the button "No balance found on '/mnt/cache'".

 

Found a command on another thread to run in terminal - btrfs balance start -dusage=75 /mnt/cache

but was met with this message.
"ERROR: error during balancing '/mnt/cache': Input/output error
There may be more info in syslog - try dmesg | tail"

 

I also just checked the log and its littered with error messages, all relating to btrfs i beleive.

 

I have attached the log and diagnostics, hoping someone with more knowledge can help me out.

unraid-syslog-20241223-0226.zip

unraid-diagnostics-20241223-1329.zip

Edited December 23, 20241 yr by Gibbo2787

Dec 23 13:13:13 unRAID kernel: I/O error, dev loop2, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 2
Dec 23 13:13:13 unRAID kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 1, corrupt 0, gen 0
Dec 23 13:13:13 unRAID kernel: BTRFS warning (device loop2): chunk 13631488 missing 1 devices, max tolerance is 0 for writable mount
Dec 23 13:13:13 unRAID kernel: BTRFS: error (device loop2) in write_all_supers:4379: errno=-5 IO failure (errors while submitting device barriers.)
Dec 23 13:13:13 unRAID kernel: BTRFS info (device loop2: state E): forced readonly
Dec 23 13:13:13 unRAID kernel: BTRFS warning (device loop2: state E): Skipping commit of aborted transaction.
Dec 23 13:13:13 unRAID kernel: BTRFS: error (device loop2: state EA) in cleanup_transaction:1992: errno=-5 IO failure
Dec 23 13:13:13 unRAID kernel: BTRFS: error (device loop2: state EA) in btrfs_sync_log:3161: errno=-5 IO failure

 

New docker image is going read only, you may also have problems with the cache filesystem:

 

Dec 23 12:36:35 unRAID kernel: BTRFS error (device sdc1): incorrect extent count for 39758856192; counted 1370, expected 1367

 

There's also a lot of corruption detected, run a correcting scrub on that pool, or probably better, backup and reformat

 

 

I appreciate you trying to help, i really do.
I ended up getting it going again last night, got everything finally installed - created new databases etc but that was shortlived. Plex went down again after about 6 hours reporting database corruption again. I have also started noticed some media file wont playback, even from a different device say the file itself is corrupted making me think the corruption is a bit deeper than just the cache drive.
I've ran checks on all the disks but nothing jumps out - I have been trying to work out what is what by getting chatgpt to dumb it down for me.
I can't work out a way to scan my whole media directory to see what else is corrupted, I only notice if i happen to try and play it. I'm sure there is a way, I checked out Dynamix File Integrity, but it looks like I should have had that setup before all this happened so it could compare checksums.

I have a second SSD installed, I will backup all files to that and reformat the orginal and see what happens.

 

I just backed up everything - removed the cache from the array and reformatted. Now upon restart its saying - Unmountable: Unsupported or no file system. Is it cactus?

Edited December 24, 20241 yr by Gibbo2787
Further steps taken

I would recommend running memtest first to rule out any obvious RAM issues.

Another one? I ran one when this problem first started. I ran it for hours and it showed no problems with RAM.
I mean, I'm happy to do one, but i just feel like I'm going in circles

If you are getting multiple corruption issues, there's likely some hardware problem, also note that memtest is only definitive if it finds errors, but if you have already run it, post new diags to see what the pool issue is now.

unraid-diagnostics-20241225-0155.zip

Pool failed to format because the device was busy:

 

Dec 25 01:39:16 unRAID root: blkdiscard: cannot open /dev/sdc1: Device or resource busy

 

Reboot and try again, it should then format.

24 minutes ago, JorgeB said:

Pool failed to format because the device was busy:

 

Dec 25 01:39:16 unRAID root: blkdiscard: cannot open /dev/sdc1: Device or resource busy

 

Reboot and try again, it should then format.

Thanks that worked a treat
I'll make new databases again - If they keep getting corrupted what do you suggest? New drive?
I have a second one installed as a second cache and I was about to put it in place of this drive before you gave me this solution.

11 minutes ago, Gibbo2787 said:

New drive?

It's worth a try, if you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.