April 13, 20206 yr I just recently created two new VMs, one Ubuntu and one Windows 10. I have 4 docker containers as well; Plex, Sab, Sonarr and Radarr. Shortly after getting everything setup with the VMs they began reverting to Paused state and my dockers became inaccessible. Plex would load its web UI but couldn't navigate to any media. The other 3 just resulted in "Site cannot be found". I recently upgraded the entire server, adding RAM, a few GPUs and drives I found laying around the house. I noticed the 2 "new" drives must have been older than I thought. After preclear they both ended up with "Error" status. I figured what the hell, I had a new drives on the way if one failed. Well, one now has the "Fail" status but it doesn't appear to be dead. My first drive still doesn't arrive for a few days (Thanks COVID) but I am trying to figure out if this failing drive, with ~1% of utilization is what is bringing down my entire server. I've attached the diagnostics (let me know if I did that wrong) as well as the info on both disks tower-diagnostics-20200413-1819.zip
April 13, 20206 yr Community Expert You have completely filled cache. The shares you want to stay on cache (appdata, domains, system) you have set to be moved to the array (cache-yes). It's probably just as well for now and we can work on getting them moved back to cache after there is room. Go to Settings - Docker and disable and do the same for Settings - VM Manager. The shares that have files on cache that need to be moved to the array are set to be ignored by mover (cache-no). Set those to cache-yes. Then run mover. After it finishes, post new diagnostics.
April 13, 20206 yr Author 7 minutes ago, trurl said: You have completely filled cache. The shares you want to stay on cache (appdata, domains, system) you have set to be moved to the array (cache-yes). It's probably just as well for now and we can work on getting them moved back to cache after there is room. Go to Settings - Docker and disable and do the same for Settings - VM Manager. The shares that have files on cache that need to be moved to the array are set to be ignored by mover (cache-no). Set those to cache-yes. Then run mover. After it finishes, post new diagnostics. Ok just to be clear, I am: Disabling VM Manager Disabling Docker Marking all shares as cache-yes) Running Move Right? Then I'll re-post the diagnostics
April 14, 20206 yr Author 3 hours ago, trurl said: After it finishes, post new diagnostics. tower-diagnostics-20200413-2142.zip
April 14, 20206 yr Community Expert domains and system shares still have files on cache. Probably these are duplicates of files on the array, since mover won't overwrite existing files. From the command line, what do you get with these? ls -lah /mnt/cache/system ls -lah /mnt/user0/system ls -lah /mnt/cache/domains ls -lah /mnt/user0/domains
April 14, 20206 yr Author 36 minutes ago, trurl said: domains and system shares still have files on cache. Probably these are duplicates of files on the array, since mover won't overwrite existing files. From the command line, what do you get with these? Linux 4.19.107-Unraid. root@Tower:~# ls -lah /mnt/cache/system total 16K drwxrwxrwx 1 nobody users 14 Jun 28 2017 ./ drwxrwxrwx 1 nobody users 26 Aug 3 2018 ../ drwxrwxrwx 1 root root 22 Jun 28 2017 libvirt/ root@Tower:~# ls -lah /mnt/user0/system total 0 drwxrwxrwx 1 nobody users 35 Jun 28 2017 ./ drwxrwxrwx 1 nobody users 85 Apr 11 17:08 ../ drwxrwxrwx 1 root root 32 Feb 25 2019 docker/ drwxrwxrwx 1 root root 10 Jun 28 2017 libvirt/ root@Tower:~# ls -lah /mnt/cache/domains total 16K drwxrwxrwx 1 nobody users 52 Apr 11 15:57 ./ drwxrwxrwx 1 nobody users 26 Aug 3 2018 ../ drwxrwxrwx 1 nobody users 20 Apr 11 15:01 Minecraft\ Server/ drwxrwxrwx 1 nobody users 20 Apr 11 15:57 Windows\ 10/ root@Tower:~# ls -lah /mnt/user0/domains total 0 drwxrwxrwx 1 nobody users 6 Apr 11 15:57 ./ drwxrwxrwx 1 nobody users 85 Apr 11 17:08 ../ drwxrwxrwx 1 nobody users 10 Apr 11 15:01 Minecraft\ Server/ drwxrwxrwx 1 nobody users 32 Apr 11 15:53 Virtual\ Reality/ drwxrwxrwx 1 nobody users 10 Apr 11 15:57 Windows\ 10/
April 14, 20206 yr Community Expert The things in /mnt/cache are on cache obviously. The things in /mnt/user0 are the user shares excluding cache, in other words, on your array. As you can see, there are duplicates. You need to delete one or the other. Probably the cache versions are the ones to keep since I think those would be the ones that were being used. You can just rename the ones on the array for now and that will get them out of the way so the cache versions can be moved. Do you know how to rename those?
April 14, 20206 yr Author 14 minutes ago, trurl said: Do you know how to rename those? I do not. Thank you for being so helpful!
April 14, 20206 yr Community Expert mv /mnt/user0/system /mnt/user0/system1 mv /mnt/user0/domains /mnt/user0/domains1 ls -lah /mnt/user
April 14, 20206 yr Author 2 minutes ago, trurl said: mv /mnt/user0/system /mnt/user0/system1 mv /mnt/user0/domains /mnt/user0/domains1 ls -lah /mnt/user Linux 4.19.107-Unraid. Last login: Mon Apr 13 19:35:41 -0700 2020 on /dev/pts/2. root@Tower:~# mv /mnt/user0/system /mnt/user0/system1 root@Tower:~# mv /mnt/user0/domains /mnt/user0/domains1 root@Tower:~# ls -lah /mnt/user total 16K drwxrwxrwx 1 nobody users 26 Apr 14 07:42 ./ drwxr-xr-x 13 root root 260 Apr 13 15:11 ../ drwxrwxrwx 1 nobody users 42 May 12 2019 Media/ drwxrwxrwx 1 nobody users 124 Apr 11 17:15 WindowsStorage/ drwxrwxrwx 1 nobody users 29 Aug 30 2019 appdata/ drwxrwxrwx 1 nobody users 28 Apr 12 18:21 backup/ drwxrwxrwx 1 nobody users 52 Apr 11 15:57 domains/ drwxrwxrwx 1 nobody users 6 Apr 11 15:57 domains1/ drwxrwxr-x 1 nobody users 40 Aug 3 2018 downloads/ drwxrwxrwx 1 nobody users 51 Apr 11 15:55 isos/ drwxrwxrwx 1 nobody users 14 Jun 28 2017 system/ drwxrwxrwx 1 nobody users 35 Jun 28 2017 system1/ root@Tower:~# tower-diagnostics-20200414-1041.zip Edited April 14, 20206 yr by elpeterson Add log just in case
April 14, 20206 yr Community Expert OK, that looks good. Run mover again and when it finishes post new diagnostics.
April 14, 20206 yr Author 1 hour ago, trurl said: OK, that looks good. Run mover again and when it finishes post new diagnostics. tower-diagnostics-20200414-1240.zip
April 14, 20206 yr Community Expert Apr 14 07:48:00 Tower move: move: skip /mnt/cache/domains/Minecraft Server/vdisk1.img Apr 14 07:48:00 Tower move: move: skip /mnt/cache/domains/Windows 10/vdisk1.img Apr 14 07:48:00 Tower move: move: skip /mnt/cache/system/libvirt/libvirt.img Mover skipped those last files we were trying to get moved, and it looks like for some reason they are back on the array so you have duplicates again. The diagnostics just before had these all on cache since we had renamed the ones on the array. Did you do anything else besides running mover between these latest diagnostics and the ones just previous?
April 14, 20206 yr Author 1 hour ago, trurl said: Did you do anything else besides running mover between these latest diagnostics and the ones just previous? Nothing. Just ran mover and waited. Right after I initially ran the mover when you first had me move the files, the cache dropped to ~400MB, now its back up to 75GB. I don't mind nuking the VMs. I could do that and just set them back up from scratch. How should I have my shares configured regarding cache? All cache-no? Edited April 14, 20206 yr by elpeterson
April 15, 20206 yr Community Expert Go to Settings - Docker and delete docker image. Do the same for libvirt in Settings - VM Manager. Then post new diagnostics.
April 15, 20206 yr Author I deleted the libvirt but it appears the docker image already doesnt exist? tower-diagnostics-20200414-2257.zip
Archived
This topic is now archived and is closed to further replies.