I was setting up some containers on my system and all of a sudden my docker service stopped responding and would not come back up. I deleted my docker.img and started recreating all my containers. Part way through creating containers became slow and then stopped working all together. When I checked the log from the webGui I noticed a ton of segfaults repeating endlessly. I have included a snippet below.

 

Jan 12 21:02:33 DS5 kernel: Code: 41 c7 45 00 00 00 00 00 4c 89 e2 4c 89 f6 e8 37 fb 76 ff 49 39 c4 49 89 c7 0f 84 ab 02 00 00 e8 36 1f 00 00 41 8b 4d 00 85 c9 <89> 08 0f 85 c8 00 00 00 4d 85 ff c7 00 af 00 00 00 0f 84 89 00 00
Jan 12 21:02:34 DS5 kernel: mysqld[15565]: segfault at 0 ip 000055dd7f846200 sp 0000153f01467490 error 6 in mysqld[55dd7ea5a000+13ec000]

I have also attached my diagnostic report below if it helps.

 

I should also mention I was using the community applications (2020.01.11a) plugin to install most of these containers.

 

UPDATE 1

 

I just rebooted the server and brought up all the containers without a problem. As soon as I tried to add another, the issues all started again. I managed to copy the error from my log and went to store it in a file in **appdata** and i was told by vi that the disk was full. I googled this issue and found this post. I'm going to try and follow this post and see if this resolves the issue.

 

 

I also went back to the logs after this and found the endless loop of segfaults occurring again.

 

Here is the error message I saved

Jan 12 22:08:34 DS5 nginx: 2020/01/12 22:08:34 [error] 2259#2259: *546 upstream timed out (110: Connection timed out) while reading upstream, client: 127.0.0.1, server: , request: "POST /Apps/AddContainer?xmlTemplate=default:/tmp/community.applications/tempFiles/templates-community-apps/linuxserversRepository/dillinger.xml HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "localhost:8080", referrer: "http://localhost:8080/Apps/AddContainer?xmlTemplate=default:/tmp/community.applications/tempFiles/templates-community-apps/linuxserversRepository/dillinger.xml"

ds5-diagnostics-20200112-2111.zip

 

UPDATE 2

 

So I ran the first few commands in the link I posted and it seems one of my SSDs is way smaller than I remember and its full.

# btrfs fi show /mnt/cache
Label: none  uuid: 1b175dd0-5756-4f33-87b6-986909856bce
        Total devices 2 FS bytes used 110.06GiB
        devid    1 size 465.76GiB used 111.79GiB path /dev/sdf1
        devid    2 size 111.79GiB used 111.79GiB path /dev/sdg1

# btrfs fi df /mnt/cache
Data, RAID1: total=109.76GiB, used=109.76GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=2.00GiB, used=312.55MiB
GlobalReserve, single: total=111.08MiB, used=0.00B

Would this be the cause of my segfaults?

Edited January 14, 20206 yr by gmihovics

With 2 different size SSDs in the cache and configured as RAID1 (the default) for redundancy, the usable space is only that of the smaller drive.    You either want to forgo the redundancy and switch to a RAID profile that allows all the space on both SSDs to be used or keep the redundancy but replace the smaller SSD with a larger one.

Thanks itimpi, that's what I thought. New SSD is on the way. Thanks for the help.