• [6.12.8] Docker service fails to start, related to iptables can't initialize table `nat': Table does not exist

    cYnIx

    By cYnIx

      • Minor

    Hello,

     

    My docker service is not starting and is not returning much when it fails. It tries to start, acts like it's going to start, but then stops without much in the error messages. Looks like it may be an issue with "iptables v1.8.9 (legacy): can't initialize iptables table `nat': Table does not exist" based on the logs I could find but I am unsure how to overcome this. 

     

    Server is a Dell R510 with 2*X5670 CPU, 128G RAM, and a DAS. It is not overclocked. Only a few dockers are publicly accessible on a bridge.

     

    I have tried UnRaid 6.12.2, 6.12.3 and lately 6.12.8. The problem started when I upgraded to 6.12.2  from 6.11.5. On 6.11.x the docker service ran many containers just fine but fails to start the docker service its-self on any 6.12.x upgrade. 

    Symptoms:

    1) On the Docker web GUI tab the message "Docker Service failed to start." is displayed in a yellow box without further errors.

    743282867_Screenshotfrom2023-07-1812-00-

     

    2) appropriate snippet of Syslog says:

    Quote

     

    Jul 17 16:57:51 Asgard kernel: BTRFS info (device loop2): enabling ssd optimizations

    Jul 17 16:57:51 Asgard root: Resize device id 1 (/dev/loop2) from 25.00GiB to max

    Jul 17 16:57:51 Asgard emhttpd: shcmd (175): /etc/rc.d/rc.docker start

    Jul 17 16:57:51 Asgard root: starting dockerd ...

    Jul 17 16:57:51 Asgard avahi-daemon[7186]: Server startup complete. Host name is Asgard.local. Local service cookie is 113882372.

    Jul 17 16:57:52 Asgard avahi-daemon[7186]: Service "Asgard" (/services/ssh.service) successfully established.

    Jul 17 16:57:52 Asgard avahi-daemon[7186]: Service "Asgard" (/services/smb.service) successfully established.

    Jul 17 16:57:52 Asgard avahi-daemon[7186]: Service "Asgard" (/services/sftp-ssh.service) successfully established.

    Jul 17 16:58:24 Asgard emhttpd: shcmd (178): umount /var/lib/docker

     

    3) Attempting a CLI start returns

    '/etc/rc.d/rc.docker start
    no image mounted at /var/lib/docker'

    or

    'docker network ls
    Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?'

    4) /var/log/docker.log shows several identical messages

    Quote

    :~# cat /var/log/docker.log 
    time="2023-07-17T16:57:54-07:00" level=warning msg="containerd config version `1` has been deprecated and will be removed in containerd v2.0, please switch to version `2`, see https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md#version-header"
    failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables --wait -t nat -N DOCKER: iptables v1.8.9 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
    Perhaps iptables or your kernel needs to be upgraded.
     (exit status 3)

     

    5) Running `iptables --wait -t nat -N DOCKER` does indeed fail with the error message above

    "iptables v1.8.9 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
    Perhaps iptables or your kernel needs to be upgraded."

     

     

    Attempted fixes:

    1) Removed docker.img and let the system recreate it.

    2) Tried several GUI docker setting options like macvlan and ipvlan, not preserving user defined networks, a larger vdisk size, btrfs vs directory, and a 5 minutes timeout just so it may get over a hang. 

    3) Cleaned out my go file of things not needed due to various upgrades and plugins.

    4) Rebooted.

    5) Installed the newer update and rebooted.

    6) Manually ran: `iptables --wait -N DOCKER` and attempted to start the docker service with cli /etc/rc.d/rc.docker start

    7) I reset my network to default by moving /boot/custom/network.cfg to network.cfg.bak and rebooted the server.

    8.) Reverted to 6.11.5 where docker works to move dockers off of my custom network and then removed the custom network in CLI, undid preserve custom network in the GUI, then updated unraid to 6.12.8 only to get the same error. 

     

    However no matter what I attempt I am unable to make a difference in the symptom and unable to get docker to start in version 6.12.X. 

     

    Reverting to 6.11.5 and docker starts again.

     

    Please advise,

    asgard-diagnostics-20240303-1321.zip

    Go to reports Stable Releases

    User Feedback

    Recommended Comments

    cYnIx

    It has been 1 month with no response.

     

    Does anyone have any idea on something to try?

    Link to comment
    JorgeB

    Since AFAIK there aren't any other reports it could be something with your /config folder, you can try redoing the flash drive, backup the current one first and then redo it and just restore the bare minimum, like the key, super.dat and the pools folder for the assignments, also copy the docker user templates folder, if all works you can then reconfigure the server or try restoring a few config files at a time from the backup to see if you can find the culprit.

    Link to comment
    cYnIx

    Access to USB was not planned. . .

     

    By your logic, I deduce I may be able to backup, delete, and repopulate /boot/config with your min list, upgrade the server, then put other config files back until it breaks? Any thoughts on the following method A:

     

    1. Backup USB 'dd if=/dev/sda of=/mnt/user/Backups/UNRAID_6_11_5_usb.iso'
    2. Backup config 'cp -R /boot/config /mnt/user/Backups/'
    3. Erase 'rm -r /boot/config/*
    4. Repopulate config from /mnt/user/Backups 'cp -R config/plugins/dockerMan config/pools config/super.dat config/*.key /boot/config/'
    5. Upgrade server to 6.12 via GUI
    6. Reboot
    7. Repopulate remaining cfg files and reboot until it no longer works

     

     

    If not would you have any input on a procedure without removing the USB with lets call method B? 

     

    1. Backup USB 'dd if=/dev/sda of=/mnt/user/Backups/UNRAID_6_11_5_usb.iso'
    2. Backup config 'cp /boot/config /mnt/user/Backups/config'
    3. Erase 'rm -r /boot/*'
    4. Rewrite 'unzip /mnt/user/Backups/unRAIDServer-6.12.10-x86_64.zip -d /boot/'
    5. Repopulate from /mnt/user/Backups 'cp -R config/plugins/dockerMan config/pools config/super.dat config/*.key /boot/config/'
    6. Reboot 'powerdown -r' 

     

    Worst case I see would lead to side load an OS with another USB and then rewrite the ISO, though the downtime would be unappreciated.

    Link to comment
    JorgeB
    10 hours ago, cYnIx said:

    Any thoughts on the following method A:

    That's something I never tried, not sure it will work, I would maybe try deleting everything from /config except the files you want to keep ( key, super.dat, pools folder, user templates folder), then copy the remaining stock /config files from stock Unraid zip (except the files you kept) , then upgrade and reboot, that *should* work, but it may leave you stuck if there's no physical access to the server in case it's really needed.

    Link to comment


    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.

    ×

  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.

    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.