Hey all,

Basically, around the same time daily, (4-5am my local time), my docker service will fail to start, but state that it is running. I'm basically clueless on this, so reaching out for help. Diagnostics attached.

 

mainserver-diagnostics-20250110-1926.zip

Your appdata backup plugin is way out-of-date. Possibly that is involved at that time daily.

 

Several other plugins need updating or replacing.

 

You should have taken care of all these plugins before upgrading Unraid.

3 minutes ago, trurl said:

Your appdata backup plugin is way out-of-date. Possibly that is involved at that time daily.

 

Several other plugins need updating or replacing.

 

You should have taken care of all these plugins before upgrading Unraid.

Valid point. I will update the plugins now, and leave this open for 1-2 days to verify if the problem continues.

Hey @trurl, your solution did work for about 1-2 weeks. Sadly, the problem has emerged once again. Diagnostics attached. I updated plugins again two days ago, which didn't work this time. If there are plugins that should be replaced or removed, would you be able to direct me to which ones in my list?

Thanks!

mainserver-diagnostics-20250209-1012.zip

Your appdata and system shares have files on the array. Not ideal but shouldn't break anything, will just impact Docker/VM performance and keep array disks spunup since these files are always open.

 

Why do you have 256G docker.img? Have you had problems filling it? Making it larger won't fix filling it, it will only make it take longer to fill. Usage should not be growing except perhaps a small amount when you add a new container.

 

The usual cause of filling docker.img is an application writing to a path that isn't mapped.

 

Your config/shares folder has a large number of non-existent shares listed. This makes me wonder if you have accidentally created a lot of files and folders at the top level of /mnt/user or some disk or pool, perhaps by writing to a host path of /mnt/user.

 

What do you get from the command line with this?

ls -lah /mnt/user

 

I have appdata set to only be on my Docker pool, but I do see that it claims to have files on 2 disks, although I haven't been able to locate these files as of yet. System is set to go from Array > Cache, which may explain that.

 

I set the docker.img to 256gb as I do have 84 images, some of which are quite large, and this was just a simple way to ensure as I add more containers that I don't need to constantly increase the docker.img size. I agree it's obnoxiously large, but even when checking file mappings, everything appears to be mapped which should be mapped for each container, although I'll need a lot more time to go through every container I have, it's a lot.

 

Some of my shares are set up to remain empty and are ingress points for data to enter, then be moved by the container controlling it (such as immich_import , which only imports data then moves it to the correct storage)

 

Command output:

 

total 73K
drwxrwx---  1 nobody       users 4.0K Feb  9 10:32 ./
drwxr-xr-x 26 root         root   520 Feb  9 10:22 ../
drwx------  1 nobody       users   43 Jun 26  2023 .Trash-99/
drwxr-x---  1 kylehandy123  1000    6 Feb  5  2024 .bzvol/
drwxr-xr-x  1 nobody       users    6 Oct 28 03:58 Whisper-Model/
drwxr-xr-x  1 nobody       users   74 Feb  9 04:59 appdata/
drwxrwxrwx  1 nobody       users   65 Feb  3 17:47 backups/
drwxr-xr-x  1 nobody       users    3 Nov  2 19:11 docker_compose/
drwxr-xr-x  1 nobody       users    5 Nov  7 01:48 domains/
drwxrwxrwx  1 nobody       users    6 Oct 27 06:26 downloads/
drwxrwxrwx  1 nobody       users   10 Jun 27  2024 frigate/
drwxrwxrwx  1 nobody       users   85 Nov  5 23:36 general_mass_storage/
drwxrwxrwx  1          777   777  109 Aug 14 08:37 immich_data/
drwxrwxrwx  1 nobody       users    6 Oct 13 04:01 immich_import/
drwxrwxrwx  1 nobody       users    6 Jan 19 16:04 isos/
drwxr-xr-x  1 nobody       users    8 Dec  4 18:02 kasm/
drwxrwxrwx  1 nobody       users    3 Jan  8 11:33 lxc/
drwxr-xr-x  1 nobody       users   46 Oct 12 20:23 mariadb_data/
drwxrwxrwx  1 nobody       users   33 Jun 30  2024 media/
drwxrwxrwx  1 nobody       users   26 Oct 22 04:46 minio/
drwxr-xr-x  1 nobody       users    6 Sep 25 22:17 netbootxyz/
drwxrwxrwx  1 nobody       users   26 Nov  7 21:02 nextcloud/
drwxrwxrwx  1 nobody       users   62 Oct 31 04:28 paperless-ngx/
drwxrwxrwx  1 nobody       users    6 Sep 23 01:44 pirated_software/
drwx------  1          999 users   85 Feb  8 12:29 postgresql17_data/
drwx------  1          999 users  103 Feb  8 12:29 postgresql_immich/
drwxrwxrwx  1 nobody       users   26 Jun 10  2024 pterodactyl-node/
drwxrwxrwx  1 nobody       users   86 Oct 27 04:40 romm/
drwxrwxrwx  1 nobody       users   26 Nov 10 02:20 sftpgo/
drwx------  1 nobody       users   64 Oct 17 10:00 syncthing_data/
drwxrwxrwx  1 nobody       users   14 Oct 11 07:10 system/
drwxrwxrwx  1 nobody       users    6 Nov 13 12:56 temp/
drwxrwxrwx  1 nobody       users    2 Jan  7 18:46 zfs_lxccontainers/

 

3 hours ago, kylehandy123 said:

I have appdata set to only be on my Docker pool, but I do see that it claims to have files on 2 disks, although I haven't been able to locate these files as of yet. System is set to go from Array > Cache, which may explain that.

Nothing can move open files, you have to Disable Docker and VM Manager in Settings before those can be moved.

28 minutes ago, trurl said:

Nothing can move open files, you have to Disable Docker and VM Manager in Settings before those can be moved.

Noted. I'll disable that and have files on the cache then.

Replying again suddenly as the docker service just crashed and failed to restart less than 5 minutes ago. Diagnostics attached. Interestingly, I changed nothing except set system files to remain on cache about 20 minutes ago.

mainserver-diagnostics-20250209-1858.zip

appdata and system shares have no Mover action set, so those files on the array will stay there.

 

In addition to Disabling Docker and VM Manager in Settings, those shares must be set to

Primary:cache; Secondary:array; Mover action:array->cache

and then run mover so they will get moved to cache.

 

None of that has anything to do with your problem though.

Feb  9 18:53:24 MainServer kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: Failed to read block 0x2ca735c: -5
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: Failed to read block 0x2ca735c: -5
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: Failed to read block 0x2ca735c: -5
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: Failed to read block 0x2ca735c: -5
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: Failed to read block 0x2ca735c: -5
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: Failed to read block 0x2ca735c: -5

This suggests problems with your boot flash. Do you have a current backup of flash?

1 minute ago, trurl said:

appdata and system shares have no Mover action set, so those files on the array will stay there.

 

In addition to Disabling Docker and VM Manager in Settings, those shares must be set to

Primary:cache; Secondary:array; Mover action:array->cache

and then run mover so they will get moved to cache.

 

None of that has anything to do with your problem though.

Feb  9 18:53:24 MainServer kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: Failed to read block 0x2ca735c: -5
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: Failed to read block 0x2ca735c: -5
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: Failed to read block 0x2ca735c: -5
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: Failed to read block 0x2ca735c: -5
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: Failed to read block 0x2ca735c: -5
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Feb  9 18:53:24 MainServer kernel: SQUASHFS error: Failed to read block 0x2ca735c: -5

This suggests problems with your boot flash. Do you have a current backup of flash?

I do have backups of my flash. Should I recreate it or swap to a new USB?

See if you can recreate it on the same flash drive. If you have to use a new one you would have to transfer the license.

21 hours ago, trurl said:

See if you can recreate it on the same flash drive. If you have to use a new one you would have to transfer the license.

Making note here that I have just rebuilt the existing USB and am currently booting the server. I'll reply either if it crashes, or stating that its working normally.

Noting new issues, upon rebuilding the USB:

1. I can't access the WebUI but the unraid box does have a working internet connection, and local connection, as far as I can tell.
2. All of my docker containers are gone? So I'll now have to reinstall them all?

Trying to troubleshoot the whole no webui issue now.

Final update for the night.

I am unable to connect to the webui anymore. I can use the physical server with a monitor to get the GUI up, and it's connected to the network, but I can't connect to the webui from any system. IP and port of the unraid machine is the same as before.

I need ideas on how to correct this.

 

Secondly, all docker containers are gone, so I will need to reinstall them all. This is fine, I have the appdata, and everything should work fine. Albeit very frustrating since this will take quite a few hours to do since my backup wasn't super recent, but recent enough it's usable.

 

Any ideas?

Attach Diagnostics to your NEXT post in this thread.

5 hours ago, trurl said:

Attach Diagnostics to your NEXT post in this thread.

Apologies. Apparently I completely blanked. Attached.

Two log files generated back to back, so I've attached them both. I'm not sure why this occurred.

mainserver-diagnostics-20250211-1721.zip mainserver-diagnostics-20250211-1716.zip

Please don't post about the same thing in multiple threads, already replied in your other thread:

 

https://forums.unraid.net/topic/186824-unable-to-access-webui/?do=findComment&comment=1524820

 

15 hours ago, JorgeB said:

already replied in your other thread:

I am locking this thread.