"filesystem layer verification failed" on all containers

[DforDoom]

Hey All,

 

Posting this because I've completely run out of ideas and could really use some help figuring out what is causing this.

 

About a week ago I stopped being able to update my docker containers, with all of them returning the same "filesystem layer verification failed for digest sha256:XXX" error message. I did a bit of googling and found multiple people recommending deletion of the docker image file and creating a new one to resolve the problem, so I did that. Even with a freshly created docker image file the same error persists when trying to install any new(previously uninstalled containers), or restoring any previously installed containers.

 

So far I've tried:

• Deleting and recreating my docker image file
• Switching Docker to a directory-based setup
• Making sure I'm able to pull images fine on other machines on the same network as my server
• Upgrading UNRAID from stable to next branch

 

Server diagnostics are attached, and any assistance would be greatly appreciated!

amdraid-diagnostics-20230509-1700.zip

[DforDoom]

My system is still unable to install a single docker container. Anyone able to assist?

[JorgeB]

Nothing logged that I can see, to rule out any config issues you can try recreating the flash drive, backup current one, recreate with a stock install then restore only the key, super.dat and pools folder, then try adding a container, if it works you can also restore the user-templates folder to avoid re-configuring them all.

[DforDoom]

40 minutes ago, JorgeB said:

Nothing logged that I can see, to rule out any config issues you can try recreating the flash drive, backup current one, recreate with a stock install then restore only the key, super.dat and pools folder, then try adding a container, if it works you can also restore the user-templates folder to avoid re-configuring them all.

Thank you for the suggestion Jorge. I’ll give this a try in the next day or two and report back the result.

[DforDoom]

On 5/11/2023 at 5:12 AM, JorgeB said:

Nothing logged that I can see, to rule out any config issues you can try recreating the flash drive, backup current one, recreate with a stock install then restore only the key, super.dat and pools folder, then try adding a container, if it works you can also restore the user-templates folder to avoid re-configuring them all.

 

On 5/11/2023 at 5:53 AM, DforDoom said:

Thank you for the suggestion Jorge. I’ll give this a try in the next day or two and report back the result.

 

So, I tried this and the first container I tried to install initially gave me a "filesystem layers don't match image configuration" error, but after deleting the docker image and trying again it worked. The second and third containers installed with no errors, so I thought things were on the up and up.

 

The fourth container I tried to install however generated a "filesystem layers don't match image configuration error", and upon trying to install it a second time it started the dreaded endless stream of "filesystem layer verification failed for digest sha256:XXX" errors no matter how many times I retry.

 

I'm at a total loss here, as seeing the same error across both my previous install and the new clean one would logically point to the problem being hardware related.  Going down the hardware train of thought, the types of errors I'm seeing would seemingly indicate an issue with storage, but I've tried creating my docker image on two different storage devices already (cache and one of my array drives).

 

I'm going to do a memtest as well here shortly, so any other advice or guidance that anyone can provide would be greatly appreciated. I've also attached new diagnostics for good measure.

amdraid-diagnostics-20230514-1207.zip

[JorgeB]

Try repeating with just 2 sticks of RAM, if the same try the other two, that should basically rule out any RAM issues.

 

You are also having macvlan call traces, so recommend changing to ipvlan (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

[DforDoom]

So, the issue ended up being with my unraid flash drive. I replaced it with a new one this morning, moved my entire config folder over, booted, and everything worked as it should.