Hi, my system has been running smooth for a year until it complete froze up twice in the last two weeks. I have it hooked up to GUI mode as backup in case of networking issues, and GUI was locked out as well as ssh and http access.

 

I've read other users having stability issues due to macvlan recently, I've been using this since the start and never had an issue until recently so i'm not sure if that's it.

 

I've attached the two diagnostics logs, which i took immediately after hard rebooting.

 

Any help would be greatly appreciated!

 

 

unraid-diagnostics-20240127-1517.zip unraid-diagnostics-20240116-2140.zip

The syslog in the diagnostics is the RAM copy and only shows what happened since the reboot.   It could be worth enabling the syslog server to get a log that survives a reboot so we can see what happened prior to the reboot. The mirror to flash option is the easiest to set up, but if you are worried about excessive wear on the flash drive you can put your server’s address into the Remote Server field.

I'll set that up now and provide more useful logs next time it happens. Thanks for the advice!

Hello, unfortunately I'm back.

 

Everything was running smoothly since my last post, until i got a system lockup last week. It happened around the same time that my download cache got full, and i figured it may have been related. Just a couple days later on an empty cache I got another system lockout. 

 

I enabled syslog server following your advice. My log file is extremely long, with 99% of it being composed of the following lines repeated multiple times per minute:

Jun  7 18:48:11 Unraid kernel: docker0: port 20(veth8a464e8) entered blocking state

Jun  7 18:48:11 Unraid kernel: docker0: port 20(veth8a464e8) entered forwarding state

Jun  7 18:48:11 Unraid kernel: veth248cea2: renamed from eth0

Jun  7 18:48:11 Unraid kernel: docker0: port 20(veth8a464e8) entered disabled state

Jun  7 18:48:11 Unraid kernel: docker0: port 20(veth8a464e8) entered disabled state

Jun  7 18:48:11 Unraid kernel: device veth8a464e8 left promiscuous mode

Jun  7 18:48:11 Unraid kernel: docker0: port 20(veth8a464e8) entered disabled state

Jun  7 18:49:11 Unraid kernel: docker0: port 20(veth3e437da) entered blocking state

Jun  7 18:49:11 Unraid kernel: docker0: port 20(veth3e437da) entered disabled state

Jun  7 18:49:11 Unraid kernel: device veth3e437da entered promiscuous mode

Jun  7 18:49:11 Unraid kernel: eth0: renamed from vethf9f02e6

Jun  7 18:49:11 Unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth3e437da: link becomes ready

 

The only content i see in my syslog file (/boot/logs/syslog-previous) aside from the above being repeated, is two errors which I've attached.

 

I am running macvlan, which I'm aware is known to cause issues for many folks. I see it mentioned in my error logs, but it's not clear to me what they mean. If we can conclude with evidence that that is the culprit of my issues, I'll bite the bullet and switch to ipvlan and work my way through drawback of it. I've got over a dozen docker containers setup and running smoothly on this unraid server for well over a year, which I'd like to avoid messing with unless absolutely necessary. 

 

Happy to add a diagnostics dump if that would be helpful, but not a lot has changed since the last time I posted one. I'm currently on version 6.12.6, will update to 6.12.10 after my parity check finishes (usually takes ~30 hours). 

 

Thanks for all the help!

 

syslog-errors.txt

Jun  7 04:08:41 Unraid kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Jun  7 04:08:41 Unraid kernel: ? _raw_spin_unlock+0x14/0x29
Jun  7 04:08:41 Unraid kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

 

Macvlan call traces will usually end up crashing the server, switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)), then reboot.

It's been consistently crashing about once per month since my initial post. I finally switch to IPVLAN and to my surprise, it's been crashing more frequently. Anecdotally about once per week, and just now it crashed less than an hour after hard rebooting from a prior crash during a parity check. I've attached the complete remote syslog between the two crashes. The only notable thing I see is `dnsmasq-dhcp[24984]: DHCP, IP range 192.168.122.2 -- 192.168.122.254`, i've recently upgraded to ubiquiti gear and changed my network ip range to 10.0.*.*

Any ideas? I know this isn't a lot of information to go off of, how can I provide more data? Thanks!

unraid.logs.txt
 

That syslog only covers a few minutes up time, it does show filesystem corruption in your pool, check filesystem there, if issues persist update to latest stable since the macvlan is fixed there, then enable the syslog server and post that after a crash

Thanks! I do have remote syslog enabled, it was short because that's how long it was up before crashing again. I ran xfs_repair and fixed the corruptions, which were on the system cache drive. Already on the latest Unraid version, and moved back to macvlan.

 

Hopefully that was it! Is there a way to get notifications of corruptions like this in the future, without having to dig through syslogs following a hang? Thanks again!

46 minutes ago, baboolian said:

Is there a way to get notifications of corruptions like this in the future, without having to dig through syslogs following a hang?

Nope

So far so good! Seems that the system drive cache corruption was the culprit (hopefully I'm not jynxing it). Thanks for the help!!

I'm back

 

Got another crash just now. Grep'd the syslog for corruptions, but only saw the old ones from before my last post. Similar logs from docker as before. I've noticed these repetitive logs significantly increase in frequency when I am downloading data via a docker container, and I kicked off a large series of downloads shortly before this crash. The following are repeated many, many, many times:

 

 

Oct 10 13:19:40 Unraid kernel: docker0: port 19(veth03f4177) entered disabled state

Oct 10 13:19:40 Unraid kernel: veth4f08576: renamed from eth0

Oct 10 13:19:40 Unraid kernel: docker0: port 19(veth03f4177) entered disabled state

Oct 10 13:19:40 Unraid kernel: device veth03f4177 left promiscuous mode

Oct 10 13:19:40 Unraid kernel: docker0: port 19(veth03f4177) entered disabled state

Oct 10 13:20:41 Unraid kernel: docker0: port 19(veth443690a) entered blocking state

Oct 10 13:20:41 Unraid kernel: docker0: port 19(veth443690a) entered disabled state

Oct 10 13:20:41 Unraid kernel: device veth443690a entered promiscuous mode

Oct 10 13:20:44 Unraid kernel: eth0: renamed from veth9b5dc70

Oct 10 13:20:44 Unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth443690a: link becomes ready

Oct 10 13:20:44 Unraid kernel: docker0: port 19(veth443690a) entered blocking state

Oct 10 13:20:44 Unraid kernel: docker0: port 19(veth443690a) entered forwarding state

Oct 10 13:20:44 Unraid kernel: docker0: port 19(veth443690a) entered disabled state

Oct 10 13:20:44 Unraid kernel: veth9b5dc70: renamed from eth0

Oct 10 13:20:44 Unraid kernel: docker0: port 19(veth443690a) entered disabled state

Oct 10 13:20:44 Unraid kernel: device veth443690a left promiscuous mode

Oct 10 13:20:44 Unraid kernel: docker0: port 19(veth443690a) entered disabled state

Oct 10 13:21:44 Unraid kernel: docker0: port 19(veth9bb2bb4) entered blocking state

Oct 10 13:21:44 Unraid kernel: docker0: port 19(veth9bb2bb4) entered disabled state

Oct 10 13:21:44 Unraid kernel: device veth9bb2bb4 entered promiscuous mode

Oct 10 13:21:44 Unraid kernel: eth0: renamed from veth934a655

Oct 10 13:21:44 Unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth9bb2bb4: link becomes ready

Oct 10 13:21:44 Unraid kernel: docker0: port 19(veth9bb2bb4) entered blocking state

Oct 10 13:21:44 Unraid kernel: docker0: port 19(veth9bb2bb4) entered forwarding state

Oct 10 13:21:44 Unraid kernel: docker0: port 19(veth9bb2bb4) entered disabled state

Oct 10 13:21:44 Unraid kernel: veth934a655: renamed from eth0

Oct 10 13:21:44 Unraid kernel: docker0: port 19(veth9bb2bb4) entered disabled state

Oct 10 13:21:44 Unraid kernel: device veth9bb2bb4 left promiscuous mode

Oct 10 13:21:44 Unraid kernel: docker0: port 19(veth9bb2bb4) entered disabled state

Oct 10 13:22:44 Unraid kernel: docker0: port 19(veth26ae910) entered blocking state

Oct 10 13:22:44 Unraid kernel: docker0: port 19(veth26ae910) entered disabled state

Oct 10 13:22:44 Unraid kernel: device veth26ae910 entered promiscuous mode

Oct 10 13:22:45 Unraid kernel: eth0: renamed from veth5bb72fd

Oct 10 13:22:45 Unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth26ae910: link becomes ready

Oct 10 13:22:45 Unraid kernel: docker0: port 19(veth26ae910) entered blocking state

Oct 10 13:22:45 Unraid kernel: docker0: port 19(veth26ae910) entered forwarding state

Oct 10 13:22:45 Unraid kernel: docker0: port 19(veth26ae910) entered disabled state

Oct 10 13:22:45 Unraid kernel: veth5bb72fd: renamed from eth0

Oct 10 13:22:45 Unraid kernel: docker0: port 19(veth26ae910) entered disabled state

Oct 10 13:22:45 Unraid kernel: device veth26ae910 left promiscuous mode

Oct 10 13:22:45 Unraid kernel: docker0: port 19(veth26ae910) entered disabled state

Oct 10 13:23:45 Unraid kernel: docker0: port 19(veth630b3c9) entered blocking state

Oct 10 13:23:45 Unraid kernel: docker0: port 19(veth630b3c9) entered disabled state

Oct 10 13:23:45 Unraid kernel: device veth630b3c9 entered promiscuous mode

Oct 10 13:23:45 Unraid kernel: eth0: renamed from veth4f7d494

Oct 10 13:23:45 Unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth630b3c9: link becomes ready

Oct 10 13:23:45 Unraid kernel: docker0: port 19(veth630b3c9) entered blocking state

Oct 10 13:23:45 Unraid kernel: docker0: port 19(veth630b3c9) entered forwarding state

Oct 10 13:23:46 Unraid kernel: docker0: port 19(veth630b3c9) entered disabled state

Oct 10 13:23:46 Unraid kernel: veth4f7d494: renamed from eth0

Oct 10 13:23:46 Unraid kernel: docker0: port 19(veth630b3c9) entered disabled state

Oct 10 13:23:46 Unraid kernel: device veth630b3c9 left promiscuous mode

Oct 10 13:23:46 Unraid kernel: docker0: port 19(veth630b3c9) entered disabled state

Oct 10 13:24:46 Unraid kernel: docker0: port 19(veth87a97e4) entered blocking state

Oct 10 13:24:46 Unraid kernel: docker0: port 19(veth87a97e4) entered disabled state

Oct 10 13:24:46 Unraid kernel: device veth87a97e4 entered promiscuous mode

Oct 10 13:24:46 Unraid kernel: eth0: renamed from veth2c87665

Oct 10 13:24:46 Unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth87a97e4: link becomes ready

Oct 10 13:24:46 Unraid kernel: docker0: port 19(veth87a97e4) entered blocking state

Oct 10 13:24:46 Unraid kernel: docker0: port 19(veth87a97e4) entered forwarding state

Oct 10 13:24:46 Unraid kernel: docker0: port 19(veth87a97e4) entered disabled state

Oct 10 13:24:46 Unraid kernel: veth2c87665: renamed from eth0

Oct 10 13:24:46 Unraid kernel: docker0: port 19(veth87a97e4) entered disabled state

Oct 10 13:24:46 Unraid kernel: device veth87a97e4 left promiscuous mode

Oct 10 13:24:46 Unraid kernel: docker0: port 19(veth87a97e4) entered disabled state

Oct 10 13:25:46 Unraid kernel: docker0: port 19(veth82b9037) entered blocking state

Oct 10 13:25:46 Unraid kernel: docker0: port 19(veth82b9037) entered disabled state

Oct 10 13:25:46 Unraid kernel: device veth82b9037 entered promiscuous mode

Oct 10 13:25:46 Unraid kernel: eth0: renamed from veth4629c81

Oct 10 13:25:46 Unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth82b9037: link becomes ready

Oct 10 13:25:46 Unraid kernel: docker0: port 19(veth82b9037) entered blocking state

Oct 10 13:25:46 Unraid kernel: docker0: port 19(veth82b9037) entered forwarding state

Oct 10 13:25:46 Unraid kernel: docker0: port 19(veth82b9037) entered disabled state

Oct 10 13:25:46 Unraid kernel: veth4629c81: renamed from eth0

Oct 10 13:25:46 Unraid kernel: docker0: port 19(veth82b9037) entered disabled state

Oct 10 13:25:46 Unraid kernel: device veth82b9037 left promiscuous mode

Oct 10 13:25:46 Unraid kernel: docker0: port 19(veth82b9037) entered disabled state

Oct 10 13:26:46 Unraid kernel: docker0: port 19(veth2253b30) entered blocking state

Oct 10 13:26:46 Unraid kernel: docker0: port 19(veth2253b30) entered disabled state

Oct 10 13:26:46 Unraid kernel: device veth2253b30 entered promiscuous mode

Oct 10 13:26:47 Unraid kernel: eth0: renamed from vethf99b22f

Oct 10 13:26:47 Unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth2253b30: link becomes ready

Oct 10 13:26:47 Unraid kernel: docker0: port 19(veth2253b30) entered blocking state

Oct 10 13:26:47 Unraid kernel: docker0: port 19(veth2253b30) entered forwarding state

Oct 10 13:26:47 Unraid kernel: docker0: port 19(veth2253b30) entered disabled state

Oct 10 13:26:47 Unraid kernel: vethf99b22f: renamed from eth0

Oct 10 13:26:47 Unraid kernel: docker0: port 19(veth2253b30) entered disabled state

Oct 10 13:26:47 Unraid kernel: device veth2253b30 left promiscuous mode

Oct 10 13:26:47 Unraid kernel: docker0: port 19(veth2253b30) entered disabled state

Oct 10 13:27:47 Unraid kernel: docker0: port 19(veth1088522) entered blocking state

Oct 10 13:27:47 Unraid kernel: docker0: port 19(veth1088522) entered disabled state

Oct 10 13:27:47 Unraid kernel: device veth1088522 entered promiscuous mode

Oct 10 13:27:47 Unraid kernel: eth0: renamed from veth1252bc2

Oct 10 13:27:47 Unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth1088522: link becomes ready

Oct 10 13:27:47 Unraid kernel: docker0: port 19(veth1088522) entered blocking state

Oct 10 13:27:47 Unraid kernel: docker0: port 19(veth1088522) entered forwarding state

Oct 10 13:27:47 Unraid kernel: docker0: port 19(veth1088522) entered disabled state

Oct 10 13:27:47 Unraid kernel: veth1252bc2: renamed from eth0

Oct 10 13:27:47 Unraid kernel: docker0: port 19(veth1088522) entered disabled state

Oct 10 13:27:47 Unraid kernel: device veth1088522 left promiscuous mode

Oct 10 13:27:47 Unraid kernel: docker0: port 19(veth1088522) entered disabled state
// *---- crash ---*
 

 

 

Edited October 11, 20241 yr by baboolian

6 hours ago, baboolian said:

these repetitive logs significantly increase in frequency when I am downloading data via a docker container

You may have a containers constantly re-starting, check their up-times.