Server WEB UI goes offline when trying to start docker service

[extremeaudio]

My tower cache disk became undetectable. So I swapped around a few cache disks till I realised a cable was faulty. The data disks are fine. After starting back up the tower web UI becomes unaccessible when trying to start docker. Unable to access the tower via putty from another machine too. Fortunately terminal on the tower itself works. So I typed diagnostics at the terminal. But after a message that said some /log file exists and saying that it is collecting diagnostics for a minute or so, there is no diagnostics zip on the flash drive. What do I do? I have tried checking the flash for errors. Nothing there.

[trurl]

Edit config/docker.cfg to disable Docker and config/domain.cfg to disable VM Manager. Make a copy of your flash drive while you have it out.

 

Then reboot in SAFE mode and see what happens and if you can get diagnostics to post.

[extremeaudio]

I tried typing diagnostics with a keyboard attached to the tower again but same result. The screen says the file is saved but there is none in the flash. Pic attached.

 

After that I booted using the flash by disabling docker and saved a diagnostics that is attached.

 

Then I enabled docker from the web UI and attempted another diagnostics download which failed midway, presumably as the web UI and whichever other devices went down. The text that showed in the diagnostics pop up window are pasted in the text file attached.

 

 

akserver-diagnostics-20240117-1310.zip failed diagnostics.txt

[trurl]

You probably don't have enough RAM to run many dockers. Does the problem occur if you leave Docker disabled?

 

When you first begin to have the problem, what do you get at command line with this?

df -h

 

[extremeaudio]

 

1 hour ago, trurl said:

You probably don't have enough RAM to run many dockers. Does the problem occur if you leave Docker disabled?

 

I removed one of the 2 RAM sticks suspecting a bad RAM and had forgotten to put it back as the problem still persisted.  With docker disabled the server is working fine.

 

1 hour ago, trurl said:

When you first begin to have the problem, what do you get at command line with this?

df -h

 

When should I issue this? Should I start docker now and immediately do this? Currently the output of that command is as below

 

df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          3.8G  232M  3.5G   7% /
tmpfs            32M  520K   32M   2% /run
/dev/sda1        29G  830M   28G   3% /boot
overlay         3.8G  232M  3.5G   7% /lib
overlay         3.8G  232M  3.5G   7% /usr
devtmpfs        8.0M     0  8.0M   0% /dev
tmpfs           3.8G     0  3.8G   0% /dev/shm
tmpfs           128M  224K  128M   1% /var/log
tmpfs           1.0M     0  1.0M   0% /mnt/disks
tmpfs           1.0M     0  1.0M   0% /mnt/remotes
tmpfs           1.0M     0  1.0M   0% /mnt/addons
tmpfs           1.0M     0  1.0M   0% /mnt/rootshare
/dev/md1p1      7.3T  5.7T  1.7T  78% /mnt/disk1
/dev/md2p1      7.3T  6.8T  504G  94% /mnt/disk2
/dev/md3p1      7.3T  305G  7.0T   5% /mnt/disk3
/dev/sdb1        75G   35G   40G  47% /mnt/cache
shfs             22T   13T  9.2T  59% /mnt/user0
shfs             22T   13T  9.2T  59% /mnt/user
tmpfs           762M     0  762M   0% /run/user/0

 

[trurl]

8 hours ago, extremeaudio said:

When should I issue this?

 

9 hours ago, trurl said:

When you first begin to have the problem

Does it happen immediately when you start Docker?

[extremeaudio]

9 hours ago, trurl said:

 

Does it happen immediately when you start Docker?

Yes, within a few seconds of trying to start docker. Docker does not start. 

[JorgeB]

Enable the syslog server and post that log after starting docker and the issues.

[extremeaudio]

syslog attached.syslog-192.168.1.49.log

[JorgeB]

I don't see anything out of the ordinary logged, suggest recreating the docker image, then see if there are no issues just starting the service, if not start restoring the containers from apps, but do it one by one and retest.

[extremeaudio]

Deleted the docker img, re created it and docker service started fine. Now I get some read only filesystem error when trying to install dockers. Btw will all the dockers be restored back perfectly when reinstalled using user template? I also had a zerotier controller docker installed from outside the unraid app store by using stacks in portainer I believe. So should installing portainer with the user template restore that docker?

 

diag attached.

 

akserver-diagnostics-20240121-1922.zip

[trurl]

6 hours ago, extremeaudio said:

So should installing portainer with the user template restore that docker?

Does the method you used actually create a user template? Your templates are on flash (boot) in config/plugins/dockerMan/templates-user and those are what Previous Apps uses.

 

Connection problems with disk1 and with cache. You should disable Docker in Settings until you get your hardware problems fixed.

[extremeaudio]

12 hours ago, trurl said:

Does the method you used actually create a user template? Your templates are on flash (boot) in config/plugins/dockerMan/templates-user and those are what Previous Apps uses.

 

No it does not create a user template. So should restoring the appdata using the appdata backup/ restore plugin recreate the previous state?

 

Quote

Connection problems with disk1 and with cache. You should disable Docker in Settings until you get your hardware problems fixed.

 

Just reseated everything and restarted. All looks ok now? Diag attached.

 

akserver-diagnostics-20240122-1523.zip

[JorgeB]

Looks OK so far.

[extremeaudio]

Now the docker starts. But when I am installing from user templates the user data is not showing. The containers seem to be installing at default. Actually I'm not sure. Some containers seem ok. Others seem default. Cant be that way, right? 

 

Edited January 23 by extremeaudio

[trurl]

11 minutes ago, extremeaudio said:

when I am installing from user templates the user data is not showing. The containers seem to be installing at default. 

I thought you said you didn't have a template. Your user templates are on flash in the folder I mentioned. Those are what Previous Apps on the Apps page will use. If instead you are trying to install from the Apps page without a user template, the only template it has to work with is the default template.

[extremeaudio]

This is not related dorectly to unraid, but I would be glad if someone helped me out.

 

I installed zerotier controller on unraid using stacks in portainer-ce docker installed from the unraid app store. 

 

Now when I restore from the appdata backup and reinstall portainer from user templates, portainer installs properly and also shows a stack called ztncui which is the correct one. But ideally there should have been networks etc which is the userdata of that container. But I cant see that. Any way to restore that?

 

In additon to the backup created by appdata backup plugin I also have a copy of the appdata folder and docker.img saved away, in case that helps restore the data any better.

[extremeaudio]

Dockers disappeared again.
 

akserver-diagnostics-20240125-1233.zip

[itimpi]

Jan 24 14:14:09 AKserver kernel: usb 2-3: reset SuperSpeed USB device number 2 using xhci_hcd
Jan 24 14:14:11 AKserver kernel: sd 1:0:0:0: [sdb] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s
Jan 24 14:14:11 AKserver kernel: sd 1:0:0:0: [sdb] tag#0 Sense Key : 0xb [current] 
Jan 24 14:14:11 AKserver kernel: sd 1:0:0:0: [sdb] tag#0 ASC=0x47 ASCQ=0x1 
Jan 24 14:14:11 AKserver kernel: sd 1:0:0:0: [sdb] tag#0 CDB: opcode=0x2a 2a 00 00 37 74 00 00 08 00 00
Jan 24 14:14:11 AKserver kernel: I/O error, dev sdb, sector 3634176 op 0x1:(WRITE) flags 0x104000 phys_seg 253 prio class 2

Looks like your cache drive dropped offline - a not uncommon occurrence when using USB connected drives.

[extremeaudio]

I have reconnected using SATA. 

Everything seems to be restoring fine except the userdata of the container added to portainer using stacks, outside of unraid app store. Any idea where this might be stored? In the docker.img itself? When I try to copy back the old docker.img I had backed up, the web UI hangs. I can only create the new docker img but then old data is lost.

[itimpi]

1 hour ago, extremeaudio said:

Any idea where this might be stored? In the docker.img itself?

It will be wherever you configured the container to store it!

[extremeaudio]

Sorry for being really dumb about this but my docker knowledge is very limited.

 

The stack used is as below. Where do you think I could possibly retrieve the data from? And I hope I didnt overwrite the data since I have tried to start the stack since Any suggestions?

 

version: "3"
services:
  ztncui:
    image: keynetworks/ztncui
    container_name: ztncui
    environment:
      - USER_UID=99 #adjust to your system
      - USER_GID=100 #adjust to your system
      - NODE_ENV=production
      - HTTPS_PORT=3443
      - ZTNCUI_PASSWD=********  #change this
    volumes:
      - ztncui:/opt/key-networks/ztncui/etc
      - zt1:/var/lib/zerotier-one
    ports:
      - 3443:3443 #dashboard port
      - 3180:3180

 

Edited January 26 by extremeaudio

[itimpi]

No idea I am afraid.   That sort of information is container specific so only another user of that container would be able to help.