VMs defaulting to "Paused" state, Dockers started but not accesible

[elpeterson]

I just recently created two new VMs, one Ubuntu and one Windows 10. I have 4 docker containers as well; Plex, Sab, Sonarr and Radarr. Shortly after getting everything setup with the VMs they began reverting to Paused state and my dockers became inaccessible. Plex would load its web UI but couldn't navigate to any media. The other 3 just resulted in "Site cannot be found".

 

I recently upgraded the entire server, adding RAM, a few GPUs and drives I found laying around the house. I noticed the 2 "new" drives must have been older than I thought. After preclear they both ended up with "Error" status. I figured what the hell, I had a new drives on the way if one failed. Well, one now has the "Fail" status but it doesn't appear to be dead. My first drive still doesn't arrive for a few days (Thanks COVID) but I am trying to figure out if this failing drive, with ~1% of utilization is what is bringing down my entire server.

 

I've attached the diagnostics (let me know if I did that wrong) as well as the info on both disks

tower-diagnostics-20200413-1819.zip

[trurl]

You have completely filled cache.

 

The shares you want to stay on cache (appdata, domains, system) you have set to be moved to the array (cache-yes). It's probably just as well for now and we can work on getting them moved back to cache after there is room. Go to Settings - Docker and disable and do the same for Settings - VM Manager.

 

The shares that have files on cache that need to be moved to the array are set to be ignored by mover (cache-no). Set those to cache-yes.

 

Then run mover. After it finishes, post new diagnostics.

[elpeterson]

7 minutes ago, trurl said:

You have completely filled cache.

 

The shares you want to stay on cache (appdata, domains, system) you have set to be moved to the array (cache-yes). It's probably just as well for now and we can work on getting them moved back to cache after there is room. Go to Settings - Docker and disable and do the same for Settings - VM Manager.

 

The shares that have files on cache that need to be moved to the array are set to be ignored by mover (cache-no). Set those to cache-yes.

 

Then run mover. After it finishes, post new diagnostics.

Ok just to be clear, I am:

 

Disabling VM Manager

Disabling Docker

Marking all shares as cache-yes)

Running Move

 

Right? Then I'll re-post the diagnostics

[trurl]

Correct

[elpeterson]

3 hours ago, trurl said:

After it finishes, post new diagnostics.

 

tower-diagnostics-20200413-2142.zip

[trurl]

domains and system shares still have files on cache. Probably these are duplicates of files on the array, since mover won't overwrite existing files.

 

From the command line, what do you get with these?

ls -lah /mnt/cache/system

ls -lah /mnt/user0/system

ls -lah /mnt/cache/domains

ls -lah /mnt/user0/domains

 

 

[elpeterson]

36 minutes ago, trurl said:

domains and system shares still have files on cache. Probably these are duplicates of files on the array, since mover won't overwrite existing files.

 

From the command line, what do you get with these?

 

 

Linux 4.19.107-Unraid.
root@Tower:~# ls -lah /mnt/cache/system
total 16K
drwxrwxrwx 1 nobody users 14 Jun 28  2017 ./
drwxrwxrwx 1 nobody users 26 Aug  3  2018 ../
drwxrwxrwx 1 root   root  22 Jun 28  2017 libvirt/
root@Tower:~# ls -lah /mnt/user0/system
total 0
drwxrwxrwx 1 nobody users 35 Jun 28  2017 ./
drwxrwxrwx 1 nobody users 85 Apr 11 17:08 ../
drwxrwxrwx 1 root   root  32 Feb 25  2019 docker/
drwxrwxrwx 1 root   root  10 Jun 28  2017 libvirt/
root@Tower:~# ls -lah /mnt/cache/domains
total 16K
drwxrwxrwx 1 nobody users 52 Apr 11 15:57 ./
drwxrwxrwx 1 nobody users 26 Aug  3  2018 ../
drwxrwxrwx 1 nobody users 20 Apr 11 15:01 Minecraft\ Server/
drwxrwxrwx 1 nobody users 20 Apr 11 15:57 Windows\ 10/
root@Tower:~# ls -lah /mnt/user0/domains
total 0
drwxrwxrwx 1 nobody users  6 Apr 11 15:57 ./
drwxrwxrwx 1 nobody users 85 Apr 11 17:08 ../
drwxrwxrwx 1 nobody users 10 Apr 11 15:01 Minecraft\ Server/
drwxrwxrwx 1 nobody users 32 Apr 11 15:53 Virtual\ Reality/
drwxrwxrwx 1 nobody users 10 Apr 11 15:57 Windows\ 10/

 

[trurl]

The things in /mnt/cache are on cache obviously. The things in /mnt/user0 are the user shares excluding cache, in other words, on your array.

 

As you can see, there are duplicates. You need to delete one or the other. Probably the cache versions are the ones to keep since I think those would be the ones that were being used.

 

You can just rename the ones on the array for now and that will get them out of the way so the cache versions can be moved.

 

Do you know how to rename those?

[elpeterson]

14 minutes ago, trurl said:

Do you know how to rename those?

I do not. Thank you for being so helpful!

[trurl]

mv /mnt/user0/system /mnt/user0/system1

mv /mnt/user0/domains /mnt/user0/domains1

ls -lah /mnt/user

 

[elpeterson]

2 minutes ago, trurl said:

mv /mnt/user0/system /mnt/user0/system1

mv /mnt/user0/domains /mnt/user0/domains1

ls -lah /mnt/user

 

Linux 4.19.107-Unraid.
Last login: Mon Apr 13 19:35:41 -0700 2020 on /dev/pts/2.
root@Tower:~# mv /mnt/user0/system /mnt/user0/system1
root@Tower:~# mv /mnt/user0/domains /mnt/user0/domains1
root@Tower:~# ls -lah /mnt/user
total 16K
drwxrwxrwx  1 nobody users  26 Apr 14 07:42 ./
drwxr-xr-x 13 root   root  260 Apr 13 15:11 ../
drwxrwxrwx  1 nobody users  42 May 12  2019 Media/
drwxrwxrwx  1 nobody users 124 Apr 11 17:15 WindowsStorage/
drwxrwxrwx  1 nobody users  29 Aug 30  2019 appdata/
drwxrwxrwx  1 nobody users  28 Apr 12 18:21 backup/
drwxrwxrwx  1 nobody users  52 Apr 11 15:57 domains/
drwxrwxrwx  1 nobody users   6 Apr 11 15:57 domains1/
drwxrwxr-x  1 nobody users  40 Aug  3  2018 downloads/
drwxrwxrwx  1 nobody users  51 Apr 11 15:55 isos/
drwxrwxrwx  1 nobody users  14 Jun 28  2017 system/
drwxrwxrwx  1 nobody users  35 Jun 28  2017 system1/
root@Tower:~# 

tower-diagnostics-20200414-1041.zip

Edited April 14, 2020 by elpeterson
Add log just in case

[trurl]

OK, that looks good. Run mover again and when it finishes post new diagnostics.

[elpeterson]

1 hour ago, trurl said:

OK, that looks good. Run mover again and when it finishes post new diagnostics.

 

tower-diagnostics-20200414-1240.zip

[trurl]

Apr 14 07:48:00 Tower move: move: skip /mnt/cache/domains/Minecraft Server/vdisk1.img
Apr 14 07:48:00 Tower move: move: skip /mnt/cache/domains/Windows 10/vdisk1.img
Apr 14 07:48:00 Tower move: move: skip /mnt/cache/system/libvirt/libvirt.img

Mover skipped those last files we were trying to get moved, and it looks like for some reason they are back on the array so you have duplicates again. The diagnostics just before had these all on cache since we had renamed the ones on the array.

 

Did you do anything else besides running mover between these latest diagnostics and the ones just previous?

[elpeterson]

1 hour ago, trurl said:

Did you do anything else besides running mover between these latest diagnostics and the ones just previous?

Nothing. Just ran mover and waited. Right after I initially ran the mover when you first had me move the files, the cache dropped to ~400MB, now its back up to 75GB. I don't mind nuking the VMs. I could do that and just set them back up from scratch. How should I have my shares configured regarding cache? All cache-no?

Edited April 14, 2020 by elpeterson

[trurl]

Go to Settings - Docker and delete docker image. Do the same for libvirt in Settings - VM Manager. Then post new diagnostics.

 

[elpeterson]

 

I deleted the libvirt but it appears the docker image already doesnt exist?

tower-diagnostics-20200414-2257.zip