October 14, 20169 yr First off, i'm really happy with Unraid and its dockers, so i was eager to try out the new 6.2 release! After updating all plugins and dockers, yesterday i descided to take the gamble and click the update button besides the Unraid listing on the "plugin" page in the dynaMix web interface. Everything was working well. After clicking that button, it downloaded some files, asked for a reboot, so i stopped the array (couldnt unmount untill i stopped a screen session i apparently left running) and it went down. When it came online i found that while i can ping, and ssh, and the shares work, that the webinterface and docker containers do not work. Its idling and i see no change after 12 hours of waiting. I have no idea where to look for errors and am still/forever learning Linux .. suggestions most welcome! Below all details i think might be relevant. I suspect something with Docker.. i did forget to disable the autostart. How to disable that from commandline? I could not find it online. What i also could not find online in v6 documentation, was how to use the "safe mode" thats apparently a feature added in older versions. Has it been deprecated? btower-diagnostics-20161014-1341.zip
October 14, 20169 yr Author OK.. here is all the output i think might be relevant. Look at the docker errors.. I'm no expert but the only errors i found when digging via SSH are: root@bTower:/mnt/cache# docker ps Cannot connect to the Docker daemon. Is the docker daemon running on this host? root@bTower:/mnt/cache# tail /var/log/docker.log time="2016-10-14T10:03:19.713510985+02:00" level=error msg="migration failed for 4921b0eb883769bdb272078d0ca0889b0b1d1d9e808311ca208d8f6ae341e5d5, err: open /var/lib/docker/graph/df2d4f955de259f76569c58e5a32e836c7ccd788e686fdd1d688158e1e958315/.migration-diffid: no such file or directory" time="2016-10-14T10:03:19.713527499+02:00" level=error msg="migration failed for 54f1994713ff8706ba5d602a0c4bffdae7f923c0f7cb113fbf2c291e8b9da7c6, err: open /var/lib/docker/graph/df2d4f955de259f76569c58e5a32e836c7ccd788e686fdd1d688158e1e958315/.migration-diffid: no such file or directory" time="2016-10-14T10:03:19.713543989+02:00" level=error msg="migration failed for 283c3da6cfa67eb155d3e745e7f7e2d4054170cb5b71dfa089931ce30a07f354, err: open /var/lib/docker/graph/df2d4f955de259f76569c58e5a32e836c7ccd788e686fdd1d688158e1e958315/.migration-diffid: no such file or directory" time="2016-10-14T10:03:19.713560468+02:00" level=error msg="migration failed for 797f5e4995e41ec89f013370a107c8dabb07790ba2c7814125bd770ae3a0ff09, err: open /var/lib/docker/graph/df2d4f955de259f76569c58e5a32e836c7ccd788e686fdd1d688158e1e958315/.migration-diffid: no such file or directory" time="2016-10-14T10:03:19.713576754+02:00" level=error msg="migration failed for a1d1943feb6efbd24717645960729f060a44da792b67f59399afb15ab35a4b9b, err: open /var/lib/docker/graph/df2d4f955de259f76569c58e5a32e836c7ccd788e686fdd1d688158e1e958315/.migration-diffid: no such file or directory" time="2016-10-14T10:03:19.713593781+02:00" level=error msg="migration failed for 9168b1d3ced4cc883ec1f6f82595eb19859236177d2094acb47dc52c8de32bf7, err: open /var/lib/docker/graph/df2d4f955de259f76569c58e5a32e836c7ccd788e686fdd1d688158e1e958315/.migration-diffid: no such file or directory" time="2016-10-14T10:03:19.713610168+02:00" level=error msg="migration failed for fda61ed038fa3277d2bfe121b797869541734c7b88da8e64d02e5a5fd1d0a2ed, err: open /var/lib/docker/graph/df2d4f955de259f76569c58e5a32e836c7ccd788e686fdd1d688158e1e958315/.migration-diffid: no such file or directory" time="2016-10-14T10:03:19.889907013+02:00" level=error msg="Graph migration failed: \"open /var/lib/docker/.migration-v1-images.json: no space left on device\". Your old graph data was found to be too inconsistent for upgrading to content-addressable storage. Some of the old data was probably not upgraded. We recommend starting over with a clean storage directory if possible." time="2016-10-14T10:03:19.889966418+02:00" level=info msg="Graph migration to content-addressability took 36003.30 seconds" time="2016-10-14T10:03:20.103472277+02:00" level=fatal msg="Error starting daemon: Error initializing network controller: error obtaining controller instance: mkdir /var/lib/docker/network: no space left on device" root@bTower:/mnt/cache# root@bTower:/# tail /var/log/syslog Oct 14 12:28:26 bTower emhttp: err: shcmd: shcmd (221): exit status: 1 Oct 14 12:31:29 bTower sshd[29498]: Accepted password for root from 192.168.4.200 port 57006 ssh2 Oct 14 12:43:55 bTower emhttp: shcmd (236): set -o pipefail ; /usr/local/sbin/mount_image '/mnt/disk1/docker.img' /var/lib/docker 10 |& logger Oct 14 12:43:55 bTower root: /mnt/disk1/docker.img is in-use, cannot mount Oct 14 12:43:55 bTower emhttp: err: shcmd: shcmd (236): exit status: 1 Oct 14 12:59:23 bTower emhttp: shcmd (251): set -o pipefail ; /usr/local/sbin/mount_image '/mnt/disk1/docker.img' /var/lib/docker 10 |& logger Oct 14 12:59:23 bTower root: /mnt/disk1/docker.img is in-use, cannot mount Oct 14 12:59:23 bTower emhttp: err: shcmd: shcmd (251): exit status: 1 And for good measure, whats currently running: root@bTower:~# top top - 12:31:41 up 12:29, 1 user, load average: 0.00, 0.00, 0.00 Tasks: 275 total, 1 running, 273 sleeping, 0 stopped, 1 zombie %Cpu(s): 0.0 us, 0.1 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 6172348 total, 314412 free, 106320 used, 5751616 buff/cache KiB Swap: 0 total, 0 free, 0 used. 4933860 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1654 root 20 0 9680 2568 2112 S 0.3 0.0 1:13.57 cpuload 29559 root 20 0 16600 2804 2224 R 0.3 0.0 0:00.04 top 1 root 20 0 4372 1648 1540 S 0.0 0.0 0:09.72 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:03.10 ksoftirqd/0 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:+ 7 root 20 0 0 0 0 S 0.0 0.0 0:26.76 rcu_preempt 8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_sched 9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh 10 root rt 0 0 0 0 S 0.0 0.0 0:00.13 migration/0 11 root rt 0 0 0 0 S 0.0 0.0 0:00.06 migration/1 12 root 20 0 0 0 0 S 0.0 0.0 0:00.09 ksoftirqd/1 14 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:+ 15 root rt 0 0 0 0 S 0.0 0.0 0:00.07 migration/2 16 root 20 0 0 0 0 S 0.0 0.0 0:00.08 ksoftirqd/2 17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0 19 root rt 0 0 0 0 S 0.0 0.0 0:00.09 migration/3 root@bTower:~# ps -f UID PID PPID C STIME TTY TIME CMD root 29525 29498 0 12:31 pts/0 00:00:00 -bash root 29851 29525 0 12:32 pts/0 00:00:00 ps -f root@bTower:~# free total used free shared buff/cache available Mem: 6172348 105744 314824 588364 5751780 4934288 Swap: 0 0 0 root@bTower:/mnt/cache# df Filesystem 1K-blocks Used Available Use% Mounted on rootfs 3017320 585464 2431856 20% / tmpfs 3086172 188 3085984 1% /run devtmpfs 3017336 0 3017336 0% /dev cgroup_root 3086172 0 3086172 0% /sys/fs/cgroup tmpfs 131072 2928 128144 3% /var/log /dev/sda1 15621120 380952 15240168 3% /boot /dev/md1 5859568668 5044055040 815513628 87% /mnt/disk1 /dev/md2 5858435620 2513603448 3344832172 43% /mnt/disk2 /dev/sdb1 488386552 109418000 378409024 23% /mnt/cache shfs 11718004288 7557658488 4160345800 65% /mnt/user0 shfs 12206390840 7667076488 4538754824 63% /mnt/user /dev/loop0 10485760 9604996 655004 94% /var/lib/docker /dev/loop1 1048576 16804 926300 2% /etc/libvirt Again, that last line shows another Docker issue: use = 94%. But that might be expected, i dont know.
October 14, 20169 yr Really should post the diagnostics. But where is the docker.img located? Sent from my LG-D852 using Tapatalk
October 14, 20169 yr Author I only just figured out that i can run "diagnostics" at the console, sorry. I attached the resulting ZIP file to my original top post. And /var/log/syslog says its at /mnt/disk1/docker.img and i checked, its actually there. And 10G big!? ...wow.
October 14, 20169 yr Community Expert Did you read the Release Notes for the update? They contain special manual steps that need carrying out relating to docker containers and VMs. The thread containing them also has a post of notes on common issues encountered and the resolution. It has been pointed out that the Update button does not make it obvious that this should be done, and this is something to be made more obvious in the future.
October 14, 20169 yr Author No i had not read the "release notes" in depth, but i have read the "upgrade notes" just above that which only list reasonable 3 points. I had no reason to believe there would be more "upgrade notes" hidden below the "release notes" just below that. I skimmed over those having already read those in the blog.. Here i had typed a really satisfying rant on Unraid info being chaotic here (call it constructive critisism) but that wont help me solve this problem now. So i removed it. Anyway, what could i have missed? I'm reading it now but besides not disabling auto-start i'm not aware of anything that i did wrong. How can i disable autostart from the console / without the web interface?
October 14, 20169 yr Author And all of a sudden (about 18 hours after the update) the Dynamix web interface started working, and even on the custom port 8888 that i previously set it. Strange .. as the machine was idling. The docker tab is missing.. so not sure what to do now.. and if doing a reboot now is unsafe or recommended instead.. insights welcome.
October 14, 20169 yr Author Perhaps the mover moved the docker.img to disk1? Should i move it to cache?
October 14, 20169 yr Author Here it says However, there is a one-time update procedure that each container will need to go through in order to point it towards that new API going forward, even if the container itself truly isn't in need of an update. but i cant find what that procedure is. Or is that just referring to the image hash that needs to be computed?
October 14, 20169 yr Author Since i cant work on this again for the next few hours i descided to just reboot this server and hope for the best. When clicking Stop the drive array, it now just says "Unmounting disks...Retry unmounting disk share(s)...". And i cant find out whats blocking it. root@bTower:~# ls /mnt disk1/ disks/ root@bTower:~# lsof | grep /mnt/disk1/ root@bTower:~#
October 15, 20169 yr Author Well since i seem to be making progress (aka having luck) i'll keep documenting so that others can learn from this, or tell me how its supposed to be done. Current state: after a reboot and increasing the docker image size dockers work but still no webinterface. I was able to change the docker image size because the webinterface momentarily worked, and had to force a reset via the console. I'm hoping that the webinterface magically starts working after 20 hours of uptime like it did last time, but thats not a solution. edit: webinterface just started working now, after 11,5 hours of uptime. No idea why it took so long.. starting a check right now to rule out that one kind of gremlin.
October 15, 20169 yr Author When updating node-red container, i got this error: Removing orphan image: 0188edb887fd Error: Error code The downloads and update and start went OK.. but Unraid could note remove this orphaned image. Does this indicate a problem that needs fixing?
Archived
This topic is now archived and is closed to further replies.