Docker Containers and VM missing after reboot

[brawny]

I saw another thread with a very similar title from 5 years ago.  Rather than resurrect the old thread, I'm creating a new one. 

 

My unraid server has been chugging away for over a year with no issues.  Last night I shut it down cleanly, opened it up to add another 10TB drive.  Restarted the server, started preclear on the new drive and went to bed.

 

This morning I realized that all of my docker containers are missing, as is the one VM I have configured.  I checked and the docker image file exists:

root@leviathan:/mnt/user/system/docker# ls -l
total 31457284
-rw-rw-rw- 1 nobody users 32212254720 Feb 24 07:20 docker.img
root@leviathan:/mnt/user/system/docker# sudo docker version
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea838
 Built:             Wed Nov 13 07:22:05 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea838
  Built:            Wed Nov 13 07:28:45 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
root@leviathan:/mnt/user/system/docker# 

 

From the other thread, I understand that the data for my various dockers is fine, but not sure how to get the dockers back.  I'm hoping I don't have to download and reconfigure them all, as I can't remember the configurations for them.  (Radarr, Sonarr, Plex, SABNZBD+, LogitechMediaServer, pi-Hole, Krusader and a few others).  Most were done with help from SpaceInvaderOne's videos.

 

I can send whatever logging information is needed to help understand what happened to help resolve the issue.  Not sure whether installing another drive has caused this, and if so, why/how.

 

Thanks in advance for your help!

Brawny

P.S.  I paused the preclear on the new disk and rebooted.  No change - no dockers showing.

 

Edited February 24, 2021 by brawny
add followup

[itimpi]

You should provide your system’s diagnostics zip file (obtained via Tools -> Diagnostics) attached to your next post in this thread so we can see what is going on.

 

you should be able to recreate your dockers with their settings intact without problems, but the fact you got this issue in the first place suggests there may well be something else going on.

[brawny]

Thanks itimpi!

 

I've attached the diagnostics zip file.

leviathan-diagnostics-20210224-0804.zip

[JorgeB]

Log suggests you are using new docker and libvirt images, please post output of:

 

find /mnt -name libvirt.img

 

[brawny]

No output. 

root@leviathan:~# find /mnt -name libvirt.log
root@leviathan:~# 

 

My bad.  Rerunning the command again with the right parameters...

root@leviathan:~# find /mnt -name libvirt.img
/mnt/user/Documents/backups/domains_backup/libvirt.img
/mnt/user/system/libvirt/libvirt.img
/mnt/disk3/system/libvirt/libvirt.img
/mnt/disk2/Documents/backups/domains_backup/libvirt.img
root@leviathan:~# 

 

Edited February 24, 2021 by brawny
update

[JorgeB]

There's only one on disk3, so older one was deleted/replaced, do you know if the one from the backup is recent? Worth trying.

[brawny]

11 minutes ago, JorgeB said:

There's only one on disk3, so older one was deleted/replaced, do you know if the one from the backup is recent? Worth trying.

I checked each of the img files:

 

root@leviathan:~# ls -l /mnt/user/system/libvirt/libvirt.img 
-rw-rw-rw- 1 nobody users 1073741824 Feb 24 07:11 /mnt/user/system/libvirt/libvirt.img
root@leviathan:~# ls -l /mnt/disk3/system/libvirt/libvirt.img 
-rw-rw-rw- 1 nobody users 1073741824 Feb 24 07:11 /mnt/disk3/system/libvirt/libvirt.img
root@leviathan:~# ls -l /mnt/disk2/Documents/backups/domains_backup/libvirt.img 
-rwxrwxrwx 1 nobody users 1073741824 Feb 22 00:00 /mnt/disk2/Documents/backups/domains_backup/libvirt.img*
root@leviathan:~# 

so the /mnt/user/system, /mnt/disk3/system and /mnt/disk2/Documents/backups/domains_backup are all the same size.  The timestamp on my backup is from 2 days ago ) I backup weekly).

 

What do you suggest I do from here?  Thanks for your help, @JorgeB

[JorgeB]

The one from /mnt/user and /mnt/disk3 is the same file, if the current one is empty worth trying the backup one, just copy over it, but first disable VM service.

[itimpi]

14 minutes ago, brawny said:

so the /mnt/user/system, /mnt/disk3/system and /mnt/disk2/Documents/backups/domains_backup are all the same size.

It is worth emphasising that /mnt/user/system is the aggregate of any /mnt/diskX/system locations so the first 2 are just different views of the same folder.

[brawny]

Sorry guys, I'm not getting it...  Only disk3 has /mnt/diskX/system directory.  Its the same size as my backup and the aggregated file shown in /mnt/user/system

 

Thanks for your patience helping me figure this out.

[JorgeB]

copy /mnt/disk2/Documents/backups/domains_backup/libvirt.img to /mnt/disk3/system/libvirt/libvirt.img

[brawny]

Done.  After copying, I rebooted, and I'm still seeing the same thing.  No dockers shown.

[JorgeB]

Libvirt if for VMs, for dockers you can just restore them:

https://forums.unraid.net/topic/57181-docker-faq/?do=findComment&comment=564309

 

[brawny]

Ok - makes sense.  libvirt.img is for VMs.  I just checked, and I get "Libvirt Service failed to start." on the VMS tab.

 

Looking at the logs, I see:

 

Feb 24 11:55:00 leviathan kernel: BTRFS info (device loop2): new size for /dev/loop2 is 32212254720
Feb 24 11:55:00 leviathan emhttpd: shcmd (64): /etc/rc.d/rc.docker start
Feb 24 11:55:00 leviathan root: starting dockerd ...
Feb 24 11:55:00 leviathan avahi-daemon[10164]: Joining mDNS multicast group on interface docker0.IPv4 with address 172.17.0.1.
Feb 24 11:55:00 leviathan avahi-daemon[10164]: New relevant interface docker0.IPv4 for mDNS.
Feb 24 11:55:00 leviathan avahi-daemon[10164]: Registering new address record for 172.17.0.1 on docker0.IPv4.
Feb 24 11:55:00 leviathan kernel: IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
Feb 24 11:55:03 leviathan rc.docker: a56f380d3d4d55e4fe8b59db86cf04e328d27bebe7d9305442480f97edc0ba62
Feb 24 11:55:03 leviathan emhttpd: shcmd (78): /usr/local/sbin/mount_image '/mnt/user/system/libvirt/libvirt.img' /etc/libvirt 1
Feb 24 11:55:03 leviathan kernel: BTRFS: device fsid 34a49792-1bfd-4d9b-821a-43cdc2cf5449 devid 1 transid 10 /dev/loop3
Feb 24 11:55:03 leviathan kernel: BTRFS info (device loop3): disk space caching is enabled
Feb 24 11:55:03 leviathan kernel: BTRFS info (device loop3): has skinny extents
Feb 24 11:55:03 leviathan kernel: BTRFS error (device loop3): bad fsid on block 22036480
Feb 24 11:55:03 leviathan kernel: BTRFS error (device loop3): bad fsid on block 22036480
Feb 24 11:55:03 leviathan kernel: BTRFS error (device loop3): failed to read chunk root
Feb 24 11:55:03 leviathan root: mount: /etc/libvirt: wrong fs type, bad option, bad superblock on /dev/loop3, missing codepage or helper program, or other error.
Feb 24 11:55:03 leviathan root: mount error
Feb 24 11:55:03 leviathan emhttpd: shcmd (78): exit status: 1
Feb 24 11:55:03 leviathan emhttpd: nothing to sync

 

Not sure why I would have a bad fsid, or what that means?

[JorgeB]

6 minutes ago, brawny said:

Not sure why I would have a bad fsid, or what that means?

Not sure, but it suggests the backup wasn't OK, you'll likely need to re-create the VM, you can use the existing vdisk if you use the same configuration.

[brawny]

1 hour ago, JorgeB said:

Libvirt if for VMs, for dockers you can just restore them:

https://forums.unraid.net/topic/57181-docker-faq/?do=findComment&comment=564309

 

Thanks for the pointer @JorgeB

 

I followed what I found there:

On 5/26/2017 at 10:56 AM, Squid said:

 

How do I move or recreate docker.img?

 

The easy way to move docker.img -
* Go to Settings -> Docker -> Enable Docker, and set to No, then click the Apply button  (this disables Docker support)
* If recreating your docker.img file, then switch to Advanced View, then check off the box and press Delete, then skip the next step

* Using mc or any file manager or the command line, move docker.img to the desired location (/mnt/cache/docker.img is recommended)
* In Settings -> Docker, change the path for Docker image to the exact location you just copied to
* Now set Enable Docker back to Yes, and click the Apply button again  (re-enabling Docker support)

The standard way to move or recreate docker.img is to stop Docker support, delete the current image, re-enable Docker support and recreate the image in the desired location (/mnt/cache/docker.img is recommended), then re-add your current templates.  Your settings should be safe, and nothing else is moved or changed, so once your templates are restored and the Dockers are restarted, they *should* work just the same.  All of that is found in a guide by JonP at -   ***OFFICIAL GUIDE*** Restoring your Docker Applications in a New Image File

 

An easier way to reinstall your applications would be to go to the Apps Tab, Previous Apps Section.  Then check off all of your previous applications and hit "Install"

 

When I went to the Web UI for each, it looks like I have to reconfigure each of them.   Ugh.....  That means I have to set up Plex again, and point it at my media, and redo all my customizations.  Same with LogitechMediaServer for my music.  I haven't tried Sonarr or Radarr yet, but expect to have to re-add all the movies/shows that I'm following.

 

This could take days/weeks to get things back the way they were yesterday - and I'm still not sure what the cause of the issue was.  Would also like to know if there's a better way to restore stuff with the existing configuration...  Hopefully I'm missing something here...

 

Thanks @JorgeB for your help getting me back up and running!

Brawny

 

 

[JorgeB]

12 hours ago, brawny said:

When I went to the Web UI for each, it looks like I have to reconfigure each of them.   Ugh.....  That means I have to set up Plex again,

That would only happen if you also lost the appdata folder.

[brawny]

6 hours ago, JorgeB said:

That would only happen if you also lost the appdata folder.

The appdata folder appeared to be intact for all my dockers.  I spent a bunch of time last night re-configuring all the dockers, and letting them chug through my data (Plex, LogitechMediaServer, Radarr, Sonarr, etc).

This morning your note got me thinking - why not restore the app data folders from backup?  I have CA Backup / Restore AppData V2 installed.

It took a couple hours to restore all appdata, and everything appears to be back to normal now.

 

To summarize how I resolved this:

1. Reinstall previous version of all dockers
2. Restore all appdata from backup via CA Backup / Restore Appdata plugin.

Thanks @JorgeB for your help with this!

[ChatNoir]

I'd say for future reference that you should probably first restore the backup, then reinstall the dockers through Community Applications.

[JonathanM]

51 minutes ago, brawny said:

To summarize how I resolved this:

1. Reinstall previous version of all dockers

Just to be clear, you did this through the apps tab, by clicking on Previous Apps on the menu to the left?

[brawny]

25 minutes ago, jonathanm said:

Just to be clear, you did this through the apps tab, by clicking on Previous Apps on the menu to the left?

Yes.  At first I did a couple of them one at a time, and when that was successful, I selected multiple apps, and installed them in a batch. 

 

[brawny]

1 hour ago, ChatNoir said:

I'd say for future reference that you should probably first restore the backup, then reinstall the dockers through Community Applications.

Good to know.  I wish I knew why this happened in the first place.  It would be good to understand why it happened so I can avoid it happening again in the future. 

 

I also wish I'd been able to figure this out right away, rather than by trial and error, but thankfully my server's back up and fully functional now so no worries. 🙂