Dockers randomly become unresponsive after a day or two

[Clayton]

Hello, i recently had an issue when upgrading to the latest version of unraid. 

I ended up fixing the issue and i thought i was good. 3 or 4 times now however i have gone to use one of my dockers and they were not responsive. rebooting unraid or the array seemed to fix it for another day or two and the issue would come back. today i went to use a docker and it was not working but this time the webgui would hang after login. i ended up connecting to the console and running the reboot command. that failed so in the end i pressed the restart button. server came up good and is currently running a check on the array. this will likely happen again in a day or two. could anyone take a look at the support bundle and give me an idea what is broken?

darktower-diagnostics-20231126-1444.zip

[JorgeB]

Docker image is corrupt, delete and recreate.

 

[Clayton]

happened again, different dockers. is it all docker images?

 

also, when doing a btrfs scan, this came up.
 

[1/7] checking root items [2/7] checking extents data extent[82600927232, 36864] referencer count mismatch (root 5 owner 260 offset 5683093504) wanted 0 have 1 data extent[82600927232, 36864] bytenr mimsmatch, extent item bytenr 82600927232 file item bytenr 0 data extent[82600927232, 36864] referencer count mismatch (root 9944610707867697157 owner 4294936705 offset 5683093504) wanted 1 have 0 backpointer mismatch on [82600927232 36864] ERROR: errors found in extent allocation tree or chunk allocation [3/7] checking free space tree [4/7] checking fs roots root 5 inode 3518051 errors 200, dir isize wrong root 5 inode 6903716 errors 1, no inode item unresolved ref dir 3518051 index 751667 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903718 errors 1, no inode item unresolved ref dir 3518051 index 751670 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903719 errors 1, no inode item unresolved ref dir 3518051 index 751672 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903720 errors 1, no inode item unresolved ref dir 3518051 index 751674 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903721 errors 1, no inode item unresolved ref dir 3518051 index 751676 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903723 errors 1, no inode item unresolved ref dir 3518051 index 751679 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903725 errors 1, no inode item unresolved ref dir 3518051 index 751682 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903727 errors 1, no inode item unresolved ref dir 3518051 index 751685 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903729 errors 1, no inode item unresolved ref dir 3518051 index 751688 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903730 errors 1, no inode item unresolved ref dir 3518051 index 751690 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903731 errors 1, no inode item unresolved ref dir 3518051 index 751692 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903732 errors 1, no inode item unresolved ref dir 3518051 index 751694 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903733 errors 1, no inode item unresolved ref dir 3518051 index 751696 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903735 errors 1, no inode item unresolved ref dir 3518051 index 751699 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903737 errors 1, no inode item unresolved ref dir 3518051 index 751702 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903739 errors 1, no inode item unresolved ref dir 3518051 index 751705 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903740 errors 1, no inode item unresolved ref dir 3518051 index 751707 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903741 errors 1, no inode item unresolved ref dir 3518051 index 751709 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903742 errors 1, no inode item unresolved ref dir 3518051 index 751711 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903744 errors 1, no inode item unresolved ref dir 3518051 index 751714 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903746 errors 1, no inode item unresolved ref dir 3518051 index 751717 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903748 errors 1, no inode item unresolved ref dir 3518051 index 751720 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903750 errors 1, no inode item unresolved ref dir 3518051 index 751723 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903751 errors 1, no inode item unresolved ref dir 3518051 index 751725 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903752 errors 1, no inode item unresolved ref dir 3518051 index 751727 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903753 errors 1, no inode item unresolved ref dir 3518051 index 751729 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903754 errors 1, no inode item unresolved ref dir 3518051 index 751731 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903756 errors 1, no inode item unresolved ref dir 3518051 index 751734 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903758 errors 1, no inode item unresolved ref dir 3518051 index 751737 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903759 errors 1, no inode item unresolved ref dir 3518051 index 751739 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903760 errors 1, no inode item unresolved ref dir 3518051 index 751741 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903762 errors 1, no inode item unresolved ref dir 3518051 index 751744 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903764 errors 1, no inode item unresolved ref dir 3518051 index 751747 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903765 errors 1, no inode item unresolved ref dir 3518051 index 751749 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903766 errors 1, no inode item unresolved ref dir 3518051 index 751751 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903768 errors 1, no inode item unresolved ref dir 3518051 index 751754 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903770 errors 1, no inode item unresolved ref dir 3518051 index 751757 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903771 errors 1, no inode item unresolved ref dir 3518051 index 751759 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903772 errors 1, no inode item unresolved ref dir 3518051 index 751761 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903774 errors 1, no inode item unresolved ref dir 3518051 index 751764 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903776 errors 1, no inode item unresolved ref dir 3518051 index 751767 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903777 errors 1, no inode item unresolved ref dir 3518051 index 751769 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903778 errors 1, no inode item unresolved ref dir 3518051 index 751771 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903780 errors 1, no inode item unresolved ref dir 3518051 index 751774 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903782 errors 1, no inode item unresolved ref dir 3518051 index 751777 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903783 errors 1, no inode item unresolved ref dir 3518051 index 751779 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903784 errors 1, no inode item unresolved ref dir 3518051 index 751781 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903786 errors 1, no inode item unresolved ref dir 3518051 index 751784 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903788 errors 1, no inode item unresolved ref dir 3518051 index 751787 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903789 errors 1, no inode item unresolved ref dir 3518051 index 751789 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903790 errors 1, no inode item unresolved ref dir 3518051 index 751791 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903792 errors 1, no inode item unresolved ref dir 3518051 index 751794 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903794 errors 1, no inode item unresolved ref dir 3518051 index 751797 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903795 errors 1, no inode item unresolved ref dir 3518051 index 751799 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903796 errors 1, no inode item unresolved ref dir 3518051 index 751801 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903798 errors 1, no inode item unresolved ref dir 3518051 index 751804 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref root 5 inode 6903800 errors 1, no inode item unresolved ref dir 3518051 index 751807 namelen 17 name WiredTiger.turtle filetype 1 errors 5, no dir item, no inode ref ERROR: errors found in fs roots Opening filesystem to check... Checking filesystem on /dev/sdc1 UUID: 0c76de6d-7b0a-4244-b02e-92e26745ad34 found 65529974784 bytes used, error(s) found total csum bytes: 23398580 total tree bytes: 339673088 total fs tree bytes: 251461632 total extent tree bytes: 50266112 btree space waste bytes: 76053004 file data blocks allocated: 362050203648 referenced 57746354176

 

darktower-diagnostics-20231127-1816.zip

Edited November 27, 2023 by Clayton

[JorgeB]

Cache filesystem is also corrupt, suggest to backup and recreate, also make sure to change the docker network to ipvlan.

[Clayton]

ok, so i have several issues. since this sounds like a bit of an overhaul, do you have any guides or documentation references for all the steps? it would be a huge ordeal if i had to start from scratch so i want to make sure i'm not making any assumptions. i have a backup of my OS flash drive but i take it you are saying i need to backup my cache drive?

 

1. backup something?

2. delete cache pool?

3. recreate cache pool?

4. configure docker network to ipvlan

5. restore config/dockers?

[JorgeB]

First backup what you can from current cache, re-format the pool and restore the data, then see where you stand.

 

52 minutes ago, Clayton said:

configure docker network to ipvlan

This should be a simple change.