Sporadic Unresponsiveness

[ZeroPM]

Hi All,

 

Firstly, thanks in advance for anyone that may be able to spot the issue.

 

I have been running unraid for years now and never an issue until now.

 

Unexpectedly approx. a week or so ago now the server becomes unresponsive. How do I know this happens? I have the server hooked up to my discord server and it pings me across various containers running on the server. One being scrpyted for my doorbell when the server "drops" I get an Apple HomeKit "Front Door not responding".

 

I then try to dial into the server via web UI and it times out. I try SSH and it times out I then plug a monitor into it and get nothing.

 

Simply rebooting the server brings it back temporarily. Approx. every 24/48 hours the server appears to "die" again.

 

During the state of the server being unresponsive, it goes without saying that server itself cannot be reached but also all its docker containers are no longer accessible nor the VMs.

 

I have run over all the logs, and I cannot see anything that is glaringly obvious. When looking over the syslog file it seems the server is still "running" in some capacity during the times the server appears to have "died". I cannot work out what to make of this.

 

PS. I know I have one array drive with read errors. It will be replaced soon. I am of the belief that this array drive would not be causing the issues I am seeing.

odyssey-diagnostics-20230923-1152.zip

[JorgeB]

If this started out of the blue it could be a hardware issue, but enable the syslog serve and post that after a crash, also make sure this has been taken care of.

[rkotara]

Saw this in your syslog.  I'm just a newbie and do not know your setup, but maybe disk3 having issues could be part of the problem?

 

Also, I use the plugin "CA Auto Update Applications" to auto update many of my plugins/dockers that are not critical (probably dont use it for Adguard in your list).

 

 

Sep 23 11:43:14 Odyssey root: Fix Common Problems Version 2023.07.29
Sep 23 11:43:24 Odyssey root: Fix Common Problems: Error: disk3 (WDC_WD30EFRX-68EUZN0_WD-WCC4N7VNY6K8) has read errors
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Plugin unassigned.devices.plg is not up to date
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application AdGuard-Home has an update available for it
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application bazarr has an update available for it
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application Chromium has an update available for it
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application overseerr has an update available for it
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application plexautoskip has an update available for it
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application prowlarr has an update available for it
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application radarr has an update available for it
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application sabnzbd has an update available for it
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application sonarr has an update available for it
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application tautulli has an update available for it
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application UptimeKuma has an update available for it
Sep 23 11:45:11 Odyssey root: Fix Common Problems: Other Warning: Unassigned Devices Plus not installed
Sep 23 11:45:11 Odyssey root: Fix Common Problems: Other Warning: Background notifications not enabled
Sep 23 11:45:11 Odyssey root: Fix Common Problems: Warning: Jumbo Frames detected on eth0 ** Ignored

[JorgeB]

Disk3 appears to be failing, run an extended SMART test to confirm, that should not cause lockups, but this will:

 

Sep 23 11:46:57 Odyssey kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Sep 23 11:46:57 Odyssey kernel: ? _raw_spin_unlock+0x14/0x29
Sep 23 11:46:57 Odyssey kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

 

Switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

[rkotara]

It seems like he is running 6.12.4. I thought the macvlan issue is fixed in that version?

[JorgeB]

13 minutes ago, rkotara said:

I thought the macvlan issue is fixed in that version?

It is if you disable bridging, or just use ipvlan which is usually the best option for most users.

[Mainfrezzer]

20 minutes ago, rkotara said:

It seems like he is running 6.12.4. I thought the macvlan issue is fixed in that version?

Only if you read the patchnotes release notes, lol, and do what they say to make it work. Bridging is enabled and thus doesnt do much.

Edited September 23, 2023 by Mainfrezzer

[rkotara]

1 hour ago, JorgeB said:

It is if you disable bridging, or just use ipvlan which is usually the best option for most users.

Ah, great to know.  I have not made the move to 6.12.x yet but will soon.

[rkotara]

1 hour ago, Mainfrezzer said:

Only if you read the patchnotes release notes, lol, and do what they say to make it work. Bridging is enabled and thus doesnt do much.

Yes!  Finding this out through other posts, for sure.  Not always a simple click-n-upgrade depending on your customization.

[ZeroPM]

I just did a bit of research on the difference between macvlan and ipvlan and it seems like ipvlan is the way forward. I cannot foresee any issues with moving to this.

I have made the switch now and all my containers are starting up just fine so it seems at first glance. I will continue to monitor my Dozzle.

11 hours ago, JorgeB said:

Disk3 appears to be failing, run an extended SMART test to confirm, that should not cause lockups, but this will:

 

Sep 23 11:46:57 Odyssey kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Sep 23 11:46:57 Odyssey kernel: ? _raw_spin_unlock+0x14/0x29
Sep 23 11:46:57 Odyssey kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

 

Switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

 

 

 

I am aware Disk 3 is failing  It is on my to-do list / buy list. Just haven't gotten around to it yet I am going to have to research the method for unraid to swap out this drive.

11 hours ago, rkotara said:

Saw this in your syslog.  I'm just a newbie and do not know your setup, but maybe disk3 having issues could be part of the problem?

 

Also, I use the plugin "CA Auto Update Applications" to auto update many of my plugins/dockers that are not critical (probably dont use it for Adguard in your list).

 

 

Sep 23 11:43:14 Odyssey root: Fix Common Problems Version 2023.07.29
Sep 23 11:43:24 Odyssey root: Fix Common Problems: Error: disk3 (WDC_WD30EFRX-68EUZN0_WD-WCC4N7VNY6K8) has read errors
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Plugin unassigned.devices.plg is not up to date
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application AdGuard-Home has an update available for it
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application bazarr has an update available for it
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application Chromium has an update available for it
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application overseerr has an update available for it
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application plexautoskip has an update available for it
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application prowlarr has an update available for it
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application radarr has an update available for it
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application sabnzbd has an update available for it
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application sonarr has an update available for it
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application tautulli has an update available for it
Sep 23 11:43:25 Odyssey root: Fix Common Problems: Warning: Docker Application UptimeKuma has an update available for it
Sep 23 11:45:11 Odyssey root: Fix Common Problems: Other Warning: Unassigned Devices Plus not installed
Sep 23 11:45:11 Odyssey root: Fix Common Problems: Other Warning: Background notifications not enabled
Sep 23 11:45:11 Odyssey root: Fix Common Problems: Warning: Jumbo Frames detected on eth0 ** Ignored

 

 

 

I thought I did set this up and route the logs to a syslog folder. But the folder is empty, I haven't been able to work out yet why it will not dump its logs in the syslog folder.

12 hours ago, JorgeB said:

If this started out of the blue it could be a hardware issue, but enable the syslog serve and post that after a crash, also make sure this has been taken care of.

 

[itimpi]

2 minutes ago, ZeroPM said:

I thought I did set this up and route the logs to a syslog folder. But the folder is empty, I haven't been able to work out yet why it will not dump its logs in the syslog folder.

You have to either set the mirror to flash option (easiest) or put the Unraid servers /iP address in the remote server field.  I suspect you have not set either of these?

[ZeroPM]

1 minute ago, itimpi said:

You have to either set the mirror to flash option (easiest) or put the Unraid servers /iP address in the remote server field.  I suspect you have not set either of these?

You would be correct sire! I did not do this due to the field being optional lol

[ZeroPM]

Now since changing over to ipvlan I see this error in FCP now.

 

More research to do I think.

[JorgeB]

You need to reboot to clear the macvlan call traces.

[ZeroPM]

Hi All,

 

Thank you all so much for the assistance.

 

This did indeed resolve my issue.

On 9/23/2023 at 8:43 PM, JorgeB said:

Disk3 appears to be failing, run an extended SMART test to confirm, that should not cause lockups, but this will:

 

Sep 23 11:46:57 Odyssey kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Sep 23 11:46:57 Odyssey kernel: ? _raw_spin_unlock+0x14/0x29
Sep 23 11:46:57 Odyssey kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

 

Switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

 

 

For years I have blindly just updated without ever reading the patchnotes I think to myself there are 10s of 1000s of unraid servers out there surely the devs wouldn't cause any major issues. Lesson learned  

On 9/23/2023 at 9:18 PM, Mainfrezzer said:

Only if you read the patchnotes release notes, lol, and do what they say to make it work. Bridging is enabled and thus doesnt do much.

 

This did indeed resolve the issue.

On 9/24/2023 at 7:01 PM, JorgeB said:

You need to reboot to clear the macvlan call traces.

 

For some time now I have had another random issue where my containers approx. 12 or so of them would take a good 30 mins to an hour to update when I do them. This never use to be this slow.

 

Ever since moving to IPVLAN it is now snappy again. 

 

 

Lastly, funny enough my drive is not reporting it is dying anymore lol coincidence no doubt! It will be replaced soon.

 

Fun