Server fails to start Docker/Server Crash

[Kai]

Hello

 

My unRaid Server starts to do some strange things lately. 

The Docker Service fails to Start/crashes after a while, and after running for about a Week the whole machine crashes (more or less)

First I usually notice that my Plex Docker has crashed.

After trying to access the Webinterface, the Interface resets to the default theme and....feels laggy.... Also my Dark Theme and Banner are no longer displayed.

 

The Error occured first about 6-8 weeks ago. I restart the Server and everything works fine for 1-2 Weeks.

 

Today i cant start Docker again and many services Fail (SMB, Docker, Webui,...)

Accessing the Server is no longer possible via ssh and trying to restart via the Webinterface doesnt work, so i had to hard reset the Server.

In the log I can see a lot of Errors but i have no Idea what they are pointing to. Is the flash drive bad? Maybe a different hardware Issue? Problems with a Plugin/Docker?

I cant download an Error log since the Interface crashes...

Thanks for a helping Hand.

 

a few hardware specs

Core i5 4670

16GB DDR3

Asus Z87-Pro

1 Seagate 3TB (Paritiy)

2x WD Red 3TB

2x WD Red 2TB

1x WD Red 1TB (Cache)

Sandisk Ultra Fit 16GB 

System Log.html

[Squid]

The snippet shows that your flash drive is either pooched or dropped offline.  Try it in a different port (ideally USB2).  You will have to reboot.  And you will get a parity check happen when the system comes back.  But, you can probably cancel the check as the shutdown should wind up being clean.  This is the bulk of your problems

 

Additionally, it appears that your docker.img file may have some corruption on it.  Hard to tell though from a single line in the snip.  Fix the flash problem first and then we'll approach the docker problems.

[Kai]

The Mainboard only has USB 3 Ports on the Rear. The Case i am using doesnt have any other (working) USB Ports sadly.

 

After changing the Flash Drive and a reboot there was no parity check.

 

But Docker would not start again. I Attached the Diagnostics File. 

 

unraid-diagnostics-20180420-1934.zip

[Kai]

Update

After rebooting the Server yesterday it worked normaly. 

Now the Server is running, but no longer accesible. Webui, SMB, SSH is not working.

[Squid]

From the prior diagnostics:

Apr 20 19:29:56 unraid root: ERROR: unable to resize '/var/lib/docker': Read-only file system

 

Now with the system in an inoperable state, can you log in locally via an attached keyboard /monitor and enter

diagnostics

 

if that fails,

cp /var/log/syslog  /boot/syslog.txt

 

And then upload the resulting file.

 

 

[Kai]

Hello

here is the file you requested.

 

I meanwhile salvaged some external USB 2 Ports from another Case, on which i now connected the Flash Boot Drive. Not pretty but seems to work currently.

unraid-diagnostics-20180421-1756.zip

[Kai]

Update

Since i couldnt find the Cause of the Server not being reachable, i replaced the Mainboard and the Power Supply.

Server boots up fine, but i still cant start the Docker Service.

 

Here is the latest Diagnotics file

 

Help would be appreciated.

unraid-diagnostics-20180426-1122.zip

[Squid]

Apr 26 11:14:52 unraid root: ERROR: unable to resize '/var/lib/docker': Read-only file system

Easiest solution is to stop docker (Settings - docker)

 

Switch to advanced view, delete the image

 

Re-enable docker.

 

Apps Tab - Previous Apps section, check off all of your applications you want installed, then hit install.  Couple of minutes later it'll be like nothing happened

[Kai]

If i try to delete the docker file i get this in the log

 

Apr 26 17:51:41 unraid ool www[30755]: /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/docker_rm
Apr 26 17:52:08 unraid ool www[30091]: /usr/local/emhttp/plugins/dynamix/scripts/emhttpd_update
Apr 26 17:52:08 unraid emhttpd: req (15): cmdStatus=apply&csrf_token=****************
Apr 26 17:52:08 unraid emhttpd: Starting services...
Apr 26 17:52:08 unraid emhttpd: shcmd (269): /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 20
Apr 26 17:52:08 unraid root: truncate: cannot open '/mnt/cache/system/docker/docker.img' for writing: Permission denied
Apr 26 17:52:08 unraid kernel: BTRFS: device fsid 2a29075d-4368-4074-bf92-f8bc3942fc6f devid 1 transid 327310 /dev/loop3
Apr 26 17:52:08 unraid kernel: BTRFS info (device loop3): disk space caching is enabled
Apr 26 17:52:08 unraid kernel: BTRFS info (device loop3): has skinny extents
Apr 26 17:52:08 unraid root: ERROR: unable to resize '/var/lib/docker': Read-only file system
Apr 26 17:52:08 unraid root: Resize '/var/lib/docker' of 'max'
Apr 26 17:52:08 unraid emhttpd: shcmd (271): /etc/rc.d/rc.docker start
Apr 26 17:52:08 unraid root: starting dockerd ...
Apr 26 17:52:18 unraid emhttpd: shcmd (273): umount /var/lib/docker

I earlier noticed that on the Cache Drive there are some Files that havent been Moved to the Aray.

I started the Mover Process and its currently running. Might take some time for the ~200GB

Might there be a Problem with the Cache drive?

[Kai]

Update

I think I found the Error.

Something was wrong with the Cache Drive. After deleting all files, unassigning the Drive, formating and reassigning to the Aray everything works fine (at the Moment)