segfault killed my docker container and corrupted data

[technorati]

I was just checking in on my unraid box, and found that my docker container had crashed, and the data on a volume that was mapped into the container is completely hosed. The only message in the syslog was

 

[32085.673877] shfs[5258]: segfault at 0 ip 0000000000403bd6 sp 0000149d0d945930 error 4 in shfs[403000+c000]
[32085.673882] Code: 89 7d f8 48 89 75 f0 eb 1d 48 8b 55 f0 48 8d 42 01 48 89 45 f0 48 8b 45 f8 48 8d 48 01 48 89 4d f8 0f b6 12 88 10 48 8b 45 f0 <0f> b6 00 84 c0 74 0b 48 8b 45 f0 0f b6 00 3c 2f 75 cd 48 8b 45 f8

Now, my data is ultimately fine - I just recovered the original data from backup. But I'm anxious now about continuing to use unRAID in this circumstance. Is there any way to discover what happened?

[technorati]

Turns out, the problem is much bigger than I thought at first:

[trurl]

Go to Tools - Diagnostics and attach the complete Diagnostics ZIP file to your NEXT post.

 

Also post the docker run command for the container as explained in this very first link in the Docker FAQ:

 

https://forums.unraid.net/topic/57181-docker-faq/?do=findComment&comment=564345

 

[trurl]

You posted again while I was writing my response. You won't be able to post docker run until you get docker going again. Just get us the Diagnostics.

 

[technorati]

Actually, the webUI is now completely unresponsive, and my SSH sessions have all been terminated.

 

[technorati]

I had to physically reboot the machine - I hope that doesn't mean the diagnostics are worthless, but here they are just in case.

hyperion-diagnostics-20200616-1452.zip

[technorati]

@trurl is the docker config still useful, seeing now that it was a crash of

shfs

that seemingly took everything south with it?

Edited June 16, 2020 by technorati

[trurl]

Looks like your cache is unmountable. Did you fill it up?

 

Now that you have restarted without cache it has created your appdata, domains, and system shares on disk1.

 

It also looks like you have SSDs in the parity array. Lots of reasons to NOT do that.

[technorati]

"Fix Common Problems" warned me about the SSD in the array after booting, so I made a new config with no SSD in the array.

 

I never got any alerts from the system that cache was full, and the only thing going in the cache should be the one docker container's appdata...I guess I'll just watch and see if it happens again at this point. Thanks, @trurl.

[technorati]

Got another shfs segfault. Diagnostics attached.

 

hyperion-diagnostics-20200617-1313.zip

[trurl]

Cache is unmountable, cache is part of user shares, user shares broken.

 

I'm not sure how to fix this:

Jun 16 15:04:53 hyperion emhttpd: shcmd (489): mkdir -p /mnt/cache
Jun 16 15:04:53 hyperion emhttpd: /mnt/cache mount error: No pool uuid
Jun 16 15:04:53 hyperion emhttpd: shcmd (490): umount /mnt/cache
Jun 16 15:04:53 hyperion root: umount: /mnt/cache: not mounted.

so I will see if @johnnie.black has anything.

[JorgeB]

No btrfs filesystem is being detected on the SSD, perhaps it was wiped?

[JorgeB]

Or it wasn't btrfs in the first place.

[technorati]

It was formatted once before, but when shfs crashed it may never have come back. I have created a new config and removed the cache drive from the array to see whether the problem occurs again. Thank you both for looking into this.