I have been having trouble with Unraid for the last week or so where I had two Kernel Panics and today all docker containers were frozen and I couldn't restart the server from the webUI or from ssh poweroff script or shutdown -h now

managed to copy the syslog before I hard rebooted

any idea whats wrong?

syslog

Macvlan call traces are usually caused by having dockers with a custom IP address:

 

 

30 minutes ago, johnnie.black said:

Macvlan call traces are usually caused by having dockers with a custom IP address:

 

 

I don't think this is the culprit

 

I suspect Disk1 which holds docker image?

Any indication that it's failing?

25 minutes ago, DjSamLb said:

Any indication that it's failing?

Not on the syslog, there might be if you post the complete diagnostics instead.

18 minutes ago, johnnie.black said:

Not on the syslog, there might be if you post the complete diagnostics instead.

Here you go

I'm suspecting Disk1 is failing because docker image is on that hdd and it has been loading the docker containers very slowly recently

Thanks a lot for your help!

 

tower-diagnostics-20200623-1822.zip

SMART report is incomplete due to a likely bug with smatmoontools, not sure if the new beta uses a newer version, you could try that.

48 minutes ago, johnnie.black said:

SMART report is incomplete due to a likely bug with smatmoontools, not sure if the new beta uses a newer version, you could try that.

Disk1 is a SAS drive and SMART report never worked on it I guess

any other way to check if Disk1 is failing?

Also if it's not Disk1 anything else in the logs or diagnostics that shows any other problems?

 

Here's the syslog from the syslog server if it offers anything plus?

syslog-192.168.100.133.log

16 minutes ago, DjSamLb said:

Disk1 is a SAS drive and SMART report never worked

It will work with an older Unraid version, and possibly with the new beta, so you can run a long test and check the result.

1 hour ago, johnnie.black said:

It will work with an older Unraid version, and possibly with the new beta, so you can run a long test and check the result.

I don't remember it ever did with the older versions; any version in particular?

I am trying now moving the docker image to the cache SSD and clearing disk1 and check if the problem reoccurs

IIRC it should work with v6.8.2, but the long test still needs to be run manually because it's a SAS device.

After moving docker image to cache and spinning down Disk1 as it was no longer used, I just had a kernel panic

help!

And here is the syslog and diagnostics 

syslog192.168.100.133.log tower-diagnostics-20200624-1212.zip

this is another kernel panic I had last week

Last call trace is related to the Mellanox NIC, see if you don't get them by temporarily running without it.

I will remove the Mellanox NIC and restart

Thank you!

14 minutes ago, johnnie.black said:

Last call trace is related to the Mellanox NIC, see if you don't get them by temporarily running without it.

 

Woke up to this   

This is the syslog

it froze around 4:12AM

syslog-192.168.100.133.log

Now it's the other NIC, possibly you have some custom IP config that is causing issues, did you check the link above about docker with custom IP addresses?

 

13 minutes ago, johnnie.black said:

Now it's the other NIC, possibly you have some custom IP config that is causing issues, did you check the link above about docker with custom IP addresses?

 

I've always had custom IP addresses and never had any problems, I don't think it's the issue here

any other ideas?

No.

3 hours ago, johnnie.black said:

No.

Any possibility it might be a PSU or Hardware failure like MB or CPU?

Always possible.

2 minutes ago, johnnie.black said:

Always possible.

Anyway I can troubleshoot further? any tool or logging level?

I would start by ruling out any config issue first, by running in safe mode with all dockers/VMs disable, if stable start enabling one by one.

 

If it's a hardware problem it's difficult to catch anything in the logs, you'd need to start swapping some hardware and test.