Hello,

 

This morning I woke up to several emails from my NAS saying there were errors trying to start up all docker containers after doing a backup via the Backu/Restore Appdata tool.

 

Upon trying to manually restart a docker container, it throws an Execution Error 403. Attached are my logs. I can see a few places where Python was throwing an error

Jun 12 03:11:37 unraid kernel: CPU: 3 PID: 19483 Comm: python3 Tainted: P    B   W  O       6.1.79-Unraid #1

and also the BTRFS was showingwarnings

Jun 12 05:01:22 unraid kernel: BTRFS error (device loop2: state EA): parent transid verify failed on logical 51642368 mirror 1 wanted 366778 found 366768

 

Then I started getting this error:

Jun 12 05:05:09 unraid kernel: CPU: 3 PID: 10273 Comm: qbittorrent-nox Tainted: P    B   W  O       6.1.79-Unraid #1

 

at that point, I ran Diagnostics to get a snapshot of what's going on. I disabled my VMs and then went into the docker settings. I tried to run the BRTFS scrub command, but it says status:aborted. Then I disabled the docker service and re-enabled it. It was at this point that the docker containers all started up again without any issue, as though nothing had happened earlier. I'm not sure what was the cause of the errors, and not sure why just "turning it off and on again" worked?

 

From what I read across multiple threads on the forum this morning, Error 403 is often caused by a corrupt docker image, corrupt cache, a full cache pool, corrupt files on the cache, etc. I can say for certain that it's not a full cache since it's a 1TB drive and only about 160 GB on the drive. And I don't think it is a failing cache drive since all of the SMART tests come back without any errors; my understanding of SSDs is that they don't fail gracefully or give many warning signs of impending death - they just die on you.

 

Any help to figure out what those the causes for the Execution Error 403 would be greatly appreciated!

unraid-diagnostics-20240612-0900.zip

I've been getting this issue too with my deluge container. I now have the backup/restore appdata tool skip it. After I stopped and started docker and the container worked again, I manually stopped and started the container and it started up just fine. So my current theory is that it's something in the backup/restore appdata plugin. Gonna give it some time and see if the container dies again on its own.

On 7/16/2024 at 11:24 AM, Mickery said:

I've been getting this issue too with my deluge container. I now have the backup/restore appdata tool skip it. After I stopped and started docker and the container worked again, I manually stopped and started the container and it started up just fine. So my current theory is that it's something in the backup/restore appdata plugin. Gonna give it some time and see if the container dies again on its own.

In my case, starting and stopping individual containers did nothing. I needed to stop the docker service, stop the array, restart Unraid, and start up the docker service again (which is automatic on booting up Unraid). That's what solved the issue for me.

 

As well, that problem never came back, so I'm unsure what the root cause is for my original incident.