since 6.12 hard freezes cought a call trace

Try switching to ipvlan (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

yes, thats what the plugin suggested and what i already did. (so this trace should be after)

 

before this shutdown i had some freezes which referenced my second network card (atlantic) which i was not using anyway, thats why i removed it, then the plugin said hey yo change that for stability thats why i changed it and it still keeps happening...

 

Changed Priority to Urgent

 

I can confirm now that this happens every day, so we startet at one call trace a week with 6.12 to once a day with 6.12.1, sadly i again couldnt find a log, im trying again tomorrow.

 

I dont know if it helps, heres a diagnostics.

unraid-server-diagnostics-20230626-1949.zip

Edited June 26, 2023 by nuhll

Enable the syslog server and post that after the next crash.

9 minutes ago, JorgeB said:

Enable the syslog server and post that after the next crash.

Yes, i had it enabled in tweaks plugin, but that only seems to work when you shutdown the server yourself, how i got that one call trace is a miracle tho. Ill post my log after next crash. 

 

Also i edited my post.

 

This crash i also lost most of my shares... 

Edited June 26, 2023 by nuhll

20 minutes ago, nuhll said:

i had it enabled in tweaks plugin

It's not there, there's an option for that in the GUID, see the link, you can enable the easier mirror to syslog option.

Like i said i did now.

 

And there is an option.

 

Ive found this now: 

Jun 26 20:14:22 Unraid-Server rc.docker: container webserver-watchtower-1 has an additional network that will be restored: webserver_webserver;

Jun 26 20:14:22 Unraid-Server root: Error response from daemon: error while removing network: network br0 id 427975d4250521d799587e98a81c2bc012564839e02b2b0cf7ee705111c4517a has active endpoints

Jun 26 20:14:22 Unraid-Server root: Error response from daemon: network with name br0 already exists

Jun 26 20:14:22 Unraid-Server root: Error response from daemon: endpoint with name pihole already exists in network br0

Jun 26 20:14:22 Unraid-Server root: RTNETLINK answers: Device or resource busy

Jun 26 20:14:22 Unraid-Server kernel: br0: Device is already in use.

Jun 26 20:14:22 Unraid-Server root: /etc/rc.d/rc.docker: line 380: /proc/sys/net/ipv6/conf/shim-br0/disable_ipv6: No such file or directory Jun 26 20:14:22 Unraid-Server root: Cannot find device "shim-br0"

Jun 26 20:14:22 Unraid-Server rc.docker: created network shim-br0 for host access

Jun 26 20:14:22 Unraid-Server root: Device "shim-br0" does not exist.

Jun 26 20:14:22 Unraid-Server root: Cannot find device "shim-br0"

Jun 26 20:14:22 Unraid-Server root: Cannot find device "shim-br0"

Jun 26 20:14:22 Unraid-Server root: Cannot find device "shim-br0"

Jun 26 20:14:22 Unraid-Server root: Cannot find device "shim-br0"

Jun 26 20:14:22 Unraid-Server root: Cannot find device "shim-br0"

Edited June 26, 2023 by nuhll

Okay another cpu call trace, without freeze.

 

Jun 26 20:25:06 Unraid-Server kernel: ------------[ cut here ]------------
Jun 26 20:25:06 Unraid-Server kernel: WARNING: CPU: 0 PID: 18872 at net/netfilter/nf_conntrack_core.c:1210 __nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
Jun 26 20:25:06 Unraid-Server kernel: Modules linked in: tcp_diag udp_diag inet_diag xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap ipvlan af_packet nvidia_uvm(PO) macvlan veth xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter md_mod xfs dm_crypt dm_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) nct6775 nct6775_core hwmon_vid iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge stp llc x86_pkg_temp_thermal intel_powerclamp coretemp i915 kvm_intel kvm crct10dif_pclmul crc32_pclmul nvidia_drm(PO) iosf_mbi crc32c_intel ghash_clmulni_intel drm_buddy nvidia_modeset(PO) i2c_algo_bit sha512_ssse3 nvidia(PO) mei_pxp
Jun 26 20:25:06 Unraid-Server kernel: mei_hdcp wmi_bmof ttm mxm_wmi aesni_intel drm_display_helper crypto_simd drm_kms_helper cryptd mpt3sas nvme drm rapl intel_gtt intel_cstate agpgart intel_uncore e1000e nvme_core joydev i2c_i801 mei_me i2c_smbus input_leds syscopyarea raid_class i2c_core led_class ahci sysfillrect mei sysimgblt libahci scsi_transport_sas fb_sys_fops thermal fan video wmi backlight intel_pmc_core acpi_pad button unix [last unloaded: md_mod]
Jun 26 20:25:06 Unraid-Server kernel: CPU: 0 PID: 18872 Comm: kworker/u16:26 Tainted: P           O       6.1.34-Unraid #1
Jun 26 20:25:06 Unraid-Server kernel: Hardware name: System manufacturer System Product Name/Z170 PRO GAMING, BIOS 3805 05/16/2018
Jun 26 20:25:06 Unraid-Server kernel: Workqueue: events_unbound macvlan_process_broadcast [macvlan]
Jun 26 20:25:06 Unraid-Server kernel: RIP: 0010:__nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
Jun 26 20:25:06 Unraid-Server kernel: Code: 44 24 10 e8 e2 e1 ff ff 8b 7c 24 04 89 ea 89 c6 89 04 24 e8 7e e6 ff ff 84 c0 75 a2 48 89 df e8 9b e2 ff ff 85 c0 89 c5 74 18 <0f> 0b 8b 34 24 8b 7c 24 04 e8 18 dd ff ff e8 93 e3 ff ff e9 72 01
Jun 26 20:25:06 Unraid-Server kernel: RSP: 0018:ffffc90000003d98 EFLAGS: 00010202
Jun 26 20:25:06 Unraid-Server kernel: RAX: 0000000000000001 RBX: ffff888269168300 RCX: 6fe5e4f394199920
Jun 26 20:25:06 Unraid-Server kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff888269168300
Jun 26 20:25:06 Unraid-Server kernel: RBP: 0000000000000001 R08: 232a105d3a08a54d R09: 4e4a4f8303b296b7
Jun 26 20:25:06 Unraid-Server kernel: R10: eadba36fbf5fd44f R11: ffffc90000003d60 R12: ffffffff82a11440
Jun 26 20:25:06 Unraid-Server kernel: R13: 0000000000013d31 R14: ffff888a788f5200 R15: 0000000000000000
Jun 26 20:25:06 Unraid-Server kernel: FS:  0000000000000000(0000) GS:ffff889006c00000(0000) knlGS:0000000000000000
Jun 26 20:25:06 Unraid-Server kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 26 20:25:06 Unraid-Server kernel: CR2: 000014f5118f2e30 CR3: 0000000161742006 CR4: 00000000003706f0
Jun 26 20:25:06 Unraid-Server kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 26 20:25:06 Unraid-Server kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jun 26 20:25:06 Unraid-Server kernel: Call Trace:
Jun 26 20:25:06 Unraid-Server kernel: <IRQ>
Jun 26 20:25:06 Unraid-Server kernel: ? __warn+0xab/0x122
Jun 26 20:25:06 Unraid-Server kernel: ? report_bug+0x109/0x17e
Jun 26 20:25:06 Unraid-Server kernel: ? __nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
Jun 26 20:25:06 Unraid-Server kernel: ? handle_bug+0x41/0x6f
Jun 26 20:25:06 Unraid-Server kernel: ? exc_invalid_op+0x13/0x60
Jun 26 20:25:06 Unraid-Server kernel: ? asm_exc_invalid_op+0x16/0x20
Jun 26 20:25:06 Unraid-Server kernel: ? __nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
Jun 26 20:25:06 Unraid-Server kernel: ? __nf_conntrack_confirm+0x9e/0x2b0 [nf_conntrack]
Jun 26 20:25:06 Unraid-Server kernel: ? nf_nat_inet_fn+0x60/0x1a8 [nf_nat]
Jun 26 20:25:06 Unraid-Server kernel: nf_conntrack_confirm+0x25/0x54 [nf_conntrack]
Jun 26 20:25:06 Unraid-Server kernel: nf_hook_slow+0x3a/0x96
Jun 26 20:25:06 Unraid-Server kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Jun 26 20:25:06 Unraid-Server kernel: NF_HOOK.constprop.0+0x79/0xd9
Jun 26 20:25:06 Unraid-Server kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Jun 26 20:25:06 Unraid-Server kernel: __netif_receive_skb_one_core+0x77/0x9c
Jun 26 20:25:06 Unraid-Server kernel: process_backlog+0x8c/0x116
Jun 26 20:25:06 Unraid-Server kernel: __napi_poll.constprop.0+0x28/0x124
Jun 26 20:25:06 Unraid-Server kernel: net_rx_action+0x159/0x24f
Jun 26 20:25:06 Unraid-Server kernel: __do_softirq+0x126/0x288
Jun 26 20:25:06 Unraid-Server kernel: do_softirq+0x7f/0xab
Jun 26 20:25:06 Unraid-Server kernel: </IRQ>
Jun 26 20:25:06 Unraid-Server kernel: <TASK>
Jun 26 20:25:06 Unraid-Server kernel: __local_bh_enable_ip+0x4c/0x6b
Jun 26 20:25:06 Unraid-Server kernel: netif_rx+0x52/0x5a
Jun 26 20:25:06 Unraid-Server kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Jun 26 20:25:06 Unraid-Server kernel: ? _raw_spin_unlock+0x14/0x29
Jun 26 20:25:06 Unraid-Server kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]
Jun 26 20:25:06 Unraid-Server kernel: process_one_work+0x1a8/0x295
Jun 26 20:25:06 Unraid-Server kernel: worker_thread+0x18b/0x244
Jun 26 20:25:06 Unraid-Server kernel: ? rescuer_thread+0x281/0x281
Jun 26 20:25:06 Unraid-Server kernel: kthread+0xe4/0xef
Jun 26 20:25:06 Unraid-Server kernel: ? kthread_complete_and_exit+0x1b/0x1b
Jun 26 20:25:06 Unraid-Server kernel: ret_from_fork+0x1f/0x30
Jun 26 20:25:06 Unraid-Server kernel: </TASK>
Jun 26 20:25:06 Unraid-Server kernel: ---[ end trace 0000000000000000 ]---
Jun 26 21:29:26 Unraid-Server emhttpd: Starting services...
Jun 26 21:29:26 Unraid-Server emhttpd: shcmd (2224): chmod 0777 '/mnt/user/google_foto'
Jun 26 21:29:26 Unraid-Server emhttpd: shcmd (2225): chown 'nobody':'users' '/mnt/user/google_foto'
Jun 26 21:29:26 Unraid-Server emhttpd: shcmd (2226): /etc/rc.d/rc.samba restart

 

 

Seems to be known problem:

forum.proxmox.com /threads/warning-cpu-0-pid-openvswitch.109578/ (u guys really banned proxmox? wtf)

https://github.com/openshift/okd/issues/1189

https://www.mail-archive.com/[email protected]/msg65275.html

https://lore.kernel.org/netdev/[email protected]/t/

unraid-server-diagnostics-20230626-2129.zip

Edited June 26, 2023 by nuhll

First time I see macvlan call traces when using ipvlan...

deoes it have something to do with VM? but its not in use.

"Sadly" no freeze. But i think ive got another problem. Docker just stopped working and i think i might have some corruption after those many hard resets.

 

How to progress?

 

If i click scrub it instant finishes, displayin "no errors".

But ive googled this:

root@Unraid-Server:~# btrfs dev stats /mnt/cache/
[/dev/mapper/sdb1].write_io_errs    568
[/dev/mapper/sdb1].read_io_errs     0
[/dev/mapper/sdb1].flush_io_errs    0
[/dev/mapper/sdb1].corruption_errs  3767
[/dev/mapper/sdb1].generation_errs  0
[/dev/mapper/sdc1].write_io_errs    0
[/dev/mapper/sdc1].read_io_errs     0
[/dev/mapper/sdc1].flush_io_errs    0
[/dev/mapper/sdc1].corruption_errs  0
[/dev/mapper/sdc1].generation_errs  0

 

So there is a problem, docker also doesnt like to start anymore. Those ssds are relativly new, so i dont think they have  a hardware problem. btw i know disk8 is in bad shape, didnt came to change it, theres no data on it.

 

edit:
im using unraid a long time now, but i never had such a hard locked and f.ed system. I couldnt stop array, i had to manually stop "ghost" docker containers (?) and even then, it didnt work. It couldnt unmount, but it said it wasnt there. 

Jun 28 03:04:04 Unraid-Server emhttpd: shcmd (271797): umount /mnt/cache
Jun 28 03:04:04 Unraid-Server root: umount: /mnt/cache: target is busy.
Jun 28 03:04:04 Unraid-Server emhttpd: shcmd (271797): exit status: 32
Jun 28 03:04:04 Unraid-Server emhttpd: Retry unmounting disk share(s)...

 

What i also dont understand, where the f. is 192.168.3 coming from? It was from that netword card i removed because of those freezes, but still its there?!

 

after restart all seems to be normal again...

 

UUID: 4282d67a-2c28-4ae1-a03d-d3c3de4006ed

Scrub started: Wed Jun 28 03:14:19 2023

Status: finished Duration: 0:27:15

Total to scrub: 1.20TiB Rate: 768.64MiB/s

Error summary: no errors found

 

 

unraid-server-diagnostics-20230628-0238.zip

Edited June 28, 2023 by nuhll

6 hours ago, nuhll said:

btrfs dev stats /mnt/cache/

Type btrfs dev stats -z /mnt/cache to reset the stats, then keep monitoring.

So what do we do now? ive posted some logs, we all know what the problem is?

 

Do the devs work on this?

 

Do they need more information? Ill post my log when i come back home, server is again not reachable.

 

Is it software problem? Do i have wrong Mainboard? (I could change mainboard and cpu)

Here is finally a log, which shows literally nothing.  On screen was again the same message as always.

 

Quote

Jun 28 03:41:34 Unraid-Server kernel: BTRFS info (device dm-8): scrub: finished on devid 1 with status: 0
Jun 28 03:41:34 Unraid-Server kernel: BTRFS info (device dm-8): scrub: finished on devid 2 with status: 0
Jun 28 19:11:50 Unraid-Server kernel: microcode: microcode updated early to revision 0xf0, date = 2021-11-12
Jun 28 19:11:50 Unraid-Server kernel: Linux version 6.1.34-Unraid (root@Develop) (gcc (GCC) 12.2.0, GNU ld version 2.40-slack151) #1 SMP PREEMPT_DYNAMIC Fri Jun 16 11:48:38 PDT 2023

 

 

syslog (1).txt

Edited June 28, 2023 by nuhll

If only to rule it out, have you run a memtest?  Can you set the memory in the BIOS to run at its stock speed of 2133 instead of an overclocked (xmp profile) of 2800

 

I already did a memtest on this PC. Also it never crashed before 6.12, also this is a "known" problem.

See this links:

https://forums.unraid.net/bug-reports/stable-releases/since-612-hard-freezes-cought-a-call-trace-r2518/?do=findComment&comment=25054

Im not technical enought to understand the problem:

 

1.) i mean why do unraid still have 192.168.3 in his tables, if the network card is removed?

there is only one phys. connection 192.168.0

 

2.) if macvlan is disabled, why does it make problems?

 

3.) can i maybe fix this thru MB and CPU swap? (i have spare parts for upgrade anyway)

Edited June 29, 2023 by nuhll

I am also seeing this issue, I've switched to ipvlan but in the past needed macvlan for some reasons, so will need to audit my many containers as to why.

 

diagnostics attached, kernel panic below

 

Hopefully upstream fix will appear in next release anyway.

 

4 xt_addrtype br_netfilter md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO)
 icp(PO) zcommon(PO) znvpair(PO) spl(O) jc42 regmap_i2c w83795 tcp_diag inet_dia
g ip6table_filter ip6_tables iptable_filter ip_tables x_tables 8021q garp mrp br
idge stp llc bonding tls e1000e x86_pkg_temp_thermal intel_powerclamp coretemp k
vm_intel nvidia_drm(PO) nvidia_modeset(PO) kvm crct10dif_pclmul crc32_pclmul crc
32c_intel ghash_clmulni_intel sha512_ssse3 aesni_intel ast drm_vram_helper i2c_a
lgo_bit crypto_simd drm_ttm_helper cryptd nvidia(PO) ttm
Jun 27 21:01:58 unraid kernel: rapl drm_kms_helper ipmi_ssif intel_cstate drm mp
t3sas i2c_i801 agpgart i2c_smbus ahci raid_class i2c_core libahci intel_uncore a
cpi_ipmi video syscopyarea input_leds sysfillrect joydev led_class sysimgblt wmi
 scsi_transport_sas fb_sys_fops fan thermal ipmi_si backlight button unix [last
unloaded: e1000e]
Jun 27 21:01:58 unraid kernel: CPU: 3 PID: 31345 Comm: kworker/u16:2 Tainted: P
       W  O       6.1.34-Unraid #1
Jun 27 21:01:58 unraid kernel: Hardware name: TYAN S5512/S5512, BIOS V2.05 08/23
/2013
Jun 27 21:01:58 unraid kernel: Workqueue: events_unbound macvlan_process_broadca
st [macvlan]
Jun 27 21:01:58 unraid kernel: RIP: 0010:nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Jun 27 21:01:58 unraid kernel: Code: a8 80 75 26 48 8d 73 58 48 8d 7c 24 20 e8 1
8 db fc ff 48 8d 43 0c 4c 8b bb 88 00 00 00 48 89 44 24 18 eb 54 0f ba e0 08 73
07 <0f> 0b e9 75 06 00 00 48 8d 73 58 48 8d 7c 24 20 e8 eb da fc ff 48
Jun 27 21:01:58 unraid kernel: RSP: 0018:ffffc90000164c78 EFLAGS: 00010282
Jun 27 21:01:58 unraid kernel: RAX: 0000000000000180 RBX: ffff8886f9d07400 RCX: ffff888105f9d9c0
Jun 27 21:01:58 unraid kernel: RDX: 0000000000000000 RSI: ffffc90000164d5c RDI: ffff8886f9d07400
Jun 27 21:01:58 unraid kernel: RBP: ffffc90000164d40 R08: 00000000151aa8c0 R09: 0000000000000000
Jun 27 21:01:58 unraid kernel: R10: 0000000000000098 R11: 0000000000000000 R12: ffffc90000164d5c
Jun 27 21:01:58 unraid kernel: R13: 0000000000000000 R14: ffffc90000164e40 R15: 0000000000000001
Jun 27 21:01:58 unraid kernel: FS:  0000000000000000(0000) GS:ffff88880fcc0000(0000) knlGS:0000000000000000
Jun 27 21:01:58 unraid kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033Jun 27 21:01:58 unraid kernel: CR2: 0000
01d23031f9c0 CR3: 000000000220a001 CR4: 00000000001726e0
Jun 27 21:01:58 unraid kernel: Call Trace:
Jun 27 21:01:58 unraid kernel: <IRQ>
Jun 27 21:01:58 unraid kernel: ? __warn+0xab/0x122
Jun 27 21:01:58 unraid kernel: ? report_bug+0x109/0x17e
Jun 27 21:01:58 unraid kernel: ? nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Jun 27 21:01:58 unraid kernel: ? handle_bug+0x41/0x6f
Jun 27 21:01:58 unraid kernel: ? exc_invalid_op+0x13/0x60
Jun 27 21:01:58 unraid kernel: ? asm_exc_invalid_op+0x16/0x20
Jun 27 21:01:58 unraid kernel: ? nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Jun 27 21:01:58 unraid kernel: ? nf_nat_setup_info+0x44/0x7d1 [nf_nat]
Jun 27 21:01:58 unraid kernel: ? percpu_counter_add_batch+0x85/0xa2
Jun 27 21:01:58 unraid kernel: ? xt_write_recseq_end+0xf/0x1c [ip_tables]
Jun 27 21:01:58 unraid kernel: ? __local_bh_enable_ip+0x56/0x6b
Jun 27 21:01:58 unraid kernel: ? ipt_do_table+0x57a/0x5bf [ip_tables]
Jun 27 21:01:58 unraid kernel: ? rt_dst_alloc+0x2f/0xa0
Jun 27 21:01:58 unraid kernel: ? xt_write_recseq_end+0xf/0x1c [ip_tables]
Jun 27 21:01:58 unraid kernel: __nf_nat_alloc_null_binding+0x66/0x81 [nf_nat]
Jun 27 21:01:58 unraid kernel: nf_nat_inet_fn+0xc0/0x1a8 [nf_nat]
Jun 27 21:01:58 unraid kernel: nf_nat_ipv4_local_in+0x2a/0xaa [nf_nat]
Jun 27 21:01:58 unraid kernel: nf_hook_slow+0x3d/0x96
Jun 27 21:01:58 unraid kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Jun 27 21:01:58 unraid kernel: NF_HOOK.constprop.0+0x79/0xd9
Jun 27 21:01:58 unraid kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Jun 27 21:01:58 unraid kernel: __netif_receive_skb_one_core+0x77/0x9c
Jun 27 21:01:58 unraid kernel: process_backlog+0x8c/0x116
Jun 27 21:01:58 unraid kernel: __napi_poll.constprop.0+0x2b/0x124
Jun 27 21:01:58 unraid kernel: net_rx_action+0x159/0x24f
Jun 27 21:01:58 unraid kernel: __do_softirq+0x129/0x288
Jun 27 21:01:58 unraid kernel: do_softirq+0x7f/0xab
Jun 27 21:01:58 unraid kernel: </IRQ>
Jun 27 21:01:58 unraid kernel: <TASK>
Jun 27 21:01:58 unraid kernel: __local_bh_enable_ip+0x4c/0x6b
Jun 27 21:01:58 unraid kernel: netif_rx+0x52/0x5a
Jun 27 21:01:58 unraid kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Jun 27 21:01:58 unraid kernel: ? _raw_spin_unlock+0x14/0x29
Jun 27 21:01:58 unraid kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]
Jun 27 21:01:58 unraid kernel: process_one_work+0x1ab/0x295
Jun 27 21:01:58 unraid kernel: __local_bh_enable_ip+0x4c/0x6b
Jun 27 21:01:58 unraid kernel: netif_rx+0x52/0x5a
Jun 27 21:01:58 unraid kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Jun 27 21:01:58 unraid kernel: ? _raw_spin_unlock+0x14/0x29
Jun 27 21:01:58 unraid kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]
Jun 27 21:01:58 unraid kernel: process_one_work+0x1ab/0x295
Jun 27 21:01:58 unraid kernel: worker_thread+0x18b/0x244
Jun 27 21:01:58 unraid kernel: ? rescuer_thread+0x281/0x281
Jun 27 21:01:58 unraid kernel: kthread+0xe7/0xef
Jun 27 21:01:58 unraid kernel: ? kthread_complete_and_exit+0x1b/0x1b
Jun 27 21:01:58 unraid kernel: ret_from_fork+0x22/0x30
Jun 27 21:01:58 unraid kernel: </TASK>
Jun 27 21:01:58 unraid kernel: ---[ end trace 0000000000000000 ]---
Jun 27 21:30:10 unraid kernel: w83795 0-002f: Failed to read from register 0x045
, err -6
Jun 27 21:30:10 unraid kernel: w83795 0-002f: Failed to read from register 0x046
, err -6
Jun 27 23:12:25 unraid kernel: w83795 0-002f: Failed to read from register 0x03c
, err -6
Jun 27 23:28:25 unraid kernel: w83795 0-002f: Failed to write to register 0x040,
 err -6
Jun 27 23:31:03 unraid emhttpd: spinning down /dev/sdh
Jun 27 23:31:03 unraid emhttpd: spinning down /dev/sdd
root@unraid:/var/log#
root@unraid:/var/log# logoutl: ? rescuer_thread+0x281/0x281
Jun 27 21:01:58 unraid kernel: kthread+0xe7/0xef
unRAID Server OS version: 6.12.1 kthread_complete_and_exit+0x1b/0x1b
 IPv4 address: 192.168.45.5el: ret_from_fork+0x22/0x30
 IPv6 address: fd5e:1ec7:ed26::5/TASK>
Jun 27 21:01:58 unraid kernel: ---[ end trace 0000000000000000 ]---
unraid login: general protection fault, probably for non-canonical address 0x53ac800000ffb6: 0000 [#1] PREEMPT SMP PTI
CPU: 4 PID: 12374 Comm: syncthing Tainted: P        W  O       6.1.34-Unraid #1
Hardware name: TYAN S5512/S5512, BIOS V2.05 08/23/2013
RIP: 0010:nf_nat_setup_info+0x14a/0x7d1 [nf_nat]
Code: 4c 89 ff e8 1e f8 ff ff 48 8b 15 4a 6a 00 00 89 c0 48 8d 04 c2 4c 8b 30 4d 85 f6 74 2a 49 81 ee 90 00 00 00 eb 21
8a 44 24 46 <41> 38 46 46 74 21 49 8b 96 90 00 00 00 48 85 d2 0f 84 56 ff ff ff
RSP: 0018:ffffc90000190738 EFLAGS: 00010202
RAX: ffff88819efad811 RBX: ffff888474ffaa00 RCX: 7e7e335e583c96a0
RDX: 0053ac8000010000 RSI: 7a6714a18c31b51d RDI: 0a7ed134644f0286
RBP: ffffc90000190800 R08: e5ea1da169d7f210 R09: 45036f9824daec14
R10: 95a518213e8535a6 R11: ffffc90000190710 R12: ffffc90000190814
R13: 0000000000000000 R14: 0053ac800000ff70 R15: ffffffff82a11440
FS:  000000c000100090(0000) GS:ffff88880fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000014c8b567e000 CR3: 00000001674e2002 CR4: 00000000001726e0
Call Trace:
 <IRQ>
 ? __die_body+0x1a/0x5c
 ? die_addr+0x38/0x51
 ? exc_general_protection+0x30f/0x345
 ? asm_exc_general_protection+0x22/0x30
 ? nf_nat_setup_info+0x14a/0x7d1 [nf_nat]
 ? nf_nat_setup_info+0x128/0x7d1 [nf_nat]
 ? krealloc+0x82/0x93
 nf_nat_masquerade_ipv4+0x114/0x13c [nf_nat]
 masquerade_tg+0x48/0x66 [xt_MASQUERADE]
 ipt_do_table+0x51e/0x5bf [ip_tables]
 ? xt_write_recseq_end+0xf/0x1c [ip_tables]
 ? __local_bh_enable_ip+0x56/0x6b
 ? ipt_do_table+0x57a/0x5bf [ip_tables]
 nf_nat_inet_fn+0x126/0x1a8 [nf_nat]
 nf_nat_ipv4_out+0x15/0x91 [nf_nat]
 nf_hook_slow+0x3d/0x96
 ? __ip_finish_output+0x144/0x144
 nf_hook+0xdf/0x110
 ? ethnl_put_bitset32+0xc9/0x328
 ? __ip_finish_output+0x144/0x144
 ip_output+0x78/0x88
 ? __ip_finish_output+0x144/0x144
 ip_sabotage_in+0x52/0x60 [br_netfilter]
 nf_hook_slow+0x3d/0x96
 ? ip_rcv_finish_core.constprop.0+0x3e8/0x3e8
 NF_HOOK.constprop.0+0x79/0xd9
 ? ip_rcv_finish_core.constprop.0+0x3e8/0x3e8
 __netif_receive_skb_one_core+0x77/0x9c
 netif_receive_skb+0xbf/0x127
 br_handle_frame_finish+0x438/0x472 [bridge]
 ? br_pass_frame_up+0xdd/0xdd [bridge]
 br_nf_hook_thresh+0xe5/0x109 [br_netfilter]
 ? br_pass_frame_up+0xdd/0xdd [bridge]
 br_nf_pre_routing_finish+0x2c1/0x2ec [br_netfilter]
 ? br_pass_frame_up+0xdd/0xdd [bridge]
 ? NF_HOOK.isra.0+0xe4/0x140 [br_netfilter]
 ? br_nf_hook_thresh+0x109/0x109 [br_netfilter]
 br_nf_pre_routing+0x236/0x24a [br_netfilter]
 ? br_nf_hook_thresh+0x109/0x109 [br_netfilter]
 br_handle_frame+0x27a/0x2e0 [bridge]
 ? br_pass_frame_up+0xdd/0xdd [bridge]
 __netif_receive_skb_core.constprop.0+0x4fd/0x6e9
 ? rcu_is_cpu_rrupt_from_idle+0x11/0x59
 ? rcu_sched_clock_irq+0x9b2/0xa37
 __netif_receive_skb_one_core+0x40/0x9c
 process_backlog+0x8c/0x116
 __napi_poll.constprop.0+0x2b/0x124
 net_rx_action+0x159/0x24f
 __do_softirq+0x129/0x288
 do_softirq+0x7f/0xab
 </IRQ>
 <TASK>
 __local_bh_enable_ip+0x4c/0x6b
 ip_finish_output2+0x36a/0x39d
 ip_send_skb+0x15/0x3b
 udp_send_skb+0x278/0x2e6
 udp_sendmsg+0x756/0x9bd
 ? ip_neigh_gw4+0x8b/0x8b
 ? sched_clock_cpu+0x12/0xa1
 ? __smp_call_single_queue+0x23/0x35
 ? paravirt_write_msr+0xb/0x11
 ? ttwu_queue_wakelist+0x9a/0xcf
 ? _raw_spin_unlock_irqrestore+0x24/0x3a
 ? try_to_wake_up+0x20e/0x248
 ? sock_sendmsg_nosec+0x2b/0x40
 sock_sendmsg_nosec+0x2b/0x40
 sock_write_iter+0x89/0xb8
 vfs_write+0x10f/0x1b9
 ksys_write+0x76/0xc2
 do_syscall_64+0x6b/0x81
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x40394e
Code: 48 89 6c 24 38 48 8d 6c 24 38 e8 0d 00 00 00 48 8b 6c 24 38 48 83 c4 40 c3 cc cc cc 49 89 f2 48 89 fa 48 89 ce 48
89 df 0f 05 <48> 3d 01 f0 ff ff 76 15 48 f7 d8 48 89 c1 48 c7 c0 ff ff ff ff 48
RSP: 002b:000000c0004c4198 EFLAGS: 00000206 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000000000015 RCX: 000000000040394e
RDX: 0000000000000037 RSI: 000000c000d5e002 RDI: 0000000000000015
RBP: 000000c0004c41d8 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000206 R12: 000000c0004c4318
R13: 0000000000000000 R14: 000000c0003fd1e0 R15: 0000148c54fe5928
 </TASK>
Modules linked in: nvidia_uvm(PO) macvlan xt_mark wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305
chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle
ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conn
track_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br
_netfilter md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) jc42 regmap_i2c
w83795 tcp_diag inet_diag ip6table_filter ip6_tables iptable_filter ip_tables x_tables 8021q garp mrp bridge stp llc bon
ding tls e1000e x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel nvidia_drm(PO) nvidia_modeset(PO) kvm crct10dif
_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 aesni_intel ast drm_vram_helper i2c_algo_bit crypto_s
imd drm_ttm_helper cryptd
 nvidia(PO) ttm rapl drm_kms_helper ipmi_ssif intel_cstate drm mpt3sas i2c_i801 agpgart i2c_smbus ahci raid_class i2c_co
re libahci intel_uncore acpi_ipmi video syscopyarea input_leds sysfillrect joydev led_class sysimgblt wmi scsi_transport
_sas fb_sys_fops fan thermal ipmi_si backlight button unix [last unloaded: e1000e]
---[ end trace 0000000000000000 ]---
RIP: 0010:nf_nat_setup_info+0x14a/0x7d1 [nf_nat]
Code: 4c 89 ff e8 1e f8 ff ff 48 8b 15 4a 6a 00 00 89 c0 48 8d 04 c2 4c 8b 30 4d 85 f6 74 2a 49 81 ee 90 00 00 00 eb 21
8a 44 24 46 <41> 38 46 46 74 21 49 8b 96 90 00 00 00 48 85 d2 0f 84 56 ff ff ff
RSP: 0018:ffffc90000190738 EFLAGS: 00010202
RAX: ffff88819efad811 RBX: ffff888474ffaa00 RCX: 7e7e335e583c96a0
RDX: 0053ac8000010000 RSI: 7a6714a18c31b51d RDI: 0a7ed134644f0286
RBP: ffffc90000190800 R08: e5ea1da169d7f210 R09: 45036f9824daec14
R10: 95a518213e8535a6 R11: ffffc90000190710 R12: ffffc90000190814
R13: 0000000000000000 R14: 0053ac800000ff70 R15: ffffffff82a11440
FS:  000000c000100090(0000) GS:ffff88880fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000014c8b567e000 CR3: 00000001674e2002 CR4: 00000000001726e0
Kernel panic - not syncing: Fatal exception in interrupt
Kernel Offset: disabled
---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

 

unraid-diagnostics-20230629-0950.zip

there seems to be a fix for this. https://github.com/okd-project/okd/issues/1189

 

Edited June 29, 2023 by nuhll

15 hours ago, nuhll said:

1.) i mean why do unraid still have 192.168.3 in his tables, if the network card is removed?

there is only one phys. connection 192.168.0

 

On your flash drive, carefully edit the config/network.cfg file and delete all the entries with [1] in them:

 

IFNAME[1]="br1"
BRNAME[1]="br1"
BRNICS[1]="eth1"
BRSTP[1]="no"
BRFD[1]="0"
DESCRIPTION[1]="LAN2"
PROTOCOL[1]="ipv4"
USE_DHCP[1]="no"
IPADDR[1]="192.168.3.1"
NETMASK[1]="255.255.255.0"
MTU[1]="1500"

 

Then change the SYSNICS line from 2 to 1 and reboot.

 

Also, please upgrade to today's 6.12.2 release. If you continue to have issues we will need to see new diagnostics from 6.12.2

 

all done.

 

Just btw, unraid dhcp doesnt seem to work, my router is set to:

 

but after i removed that file, it just came online with 192.168.0.21...

 

i corrected now the file and updated and restartet, waiting for new crash, or did any of the changes resolve around that issue?

btw i fou nd out my portainer stack isnt auto starting anymore.. and theni found this:

 

but i coudlnt find macvlan here 

could that be the problem?

Edited June 30, 2023 by nuhll

6 hours ago, nuhll said:

all done.

 

Just btw, unraid dhcp doesnt seem to work, my router is set to:

 

but after i removed that file, it just came online with 192.168.0.21...

 

i corrected now the file and updated and restartet, waiting for new crash, or did any of the changes resolve around that issue?

 

Sounds like your router is using the wrong MAC address when assigning IPs.

 

Go into your router and see what MAC address is assigned to 192.168.0.21, that is the one you want to when setting up the DHCP assignments for this server.

 

Once that is straightened out, please provide diagnostics from 6.12.2 and restate what problems remain to be resolved, I was specifically helping with the question about the extra NIC.

 

Note: if you have a Portainer-specific problem, that might be better asked in the thread for that container, the folks handling general support are not likely to know anything about that container.

I dont really understand how that can be a portainer problem when it was working for,  i dont know 5 years? Oo Only after switching to 6.12 problems startet. It is also a bug that unraid doesnt remove the extra nic configs when a nic is removed. anway.

 

Anyway, the router is one of the most used routers in germany, i dont think thats the problem, also  i dont care, it has now a fixed ip again. I just wanted to report it incase unraid want to fix it.

 

Im currently not facing any problems after removing the nic config and updating to 6.12.2. I guess well see in some days.

 

When do you want the diag, before crash, or after?