Helmonder

Thanks ! That is good to read !

Anyone ?

No problem..

After having more then a week of green logfiles I have now received a red one again. Since it looks like the same issue that probably has caused a multitude of system crashes I wanted to post it here hoping that someone can give some guidance. My previous error is described in the following post: The error I had now is: Mar 22 13:09:06 Tower kernel: CPU 0/KVM: page allocation failure: order:4, mode:0x6080c0(GFP_KERNEL|__GFP_ZERO), nodemask=(null) Mar 22 13:09:06 Tower kernel: CPU 0/KVM cpuset=vcpu0 mems_allowed=0 Mar 22 13:09:06 Tower kernel: CPU: 4 PID: 12980 Comm: CPU 0/KVM Tainted: G O 4.19.107-Unraid #1 Mar 22 13:09:06 Tower kernel: Hardware name: Supermicro Super Server/X11SSM-F, BIOS 2.2 05/23/2018 Mar 22 13:09:06 Tower kernel: Call Trace: Mar 22 13:09:06 Tower kernel: dump_stack+0x67/0x83 Mar 22 13:09:06 Tower kernel: warn_alloc+0xd6/0x16c Mar 22 13:09:06 Tower kernel: __alloc_pages_nodemask+0xa81/0xae1 Mar 22 13:09:06 Tower kernel: ? smp_call_function_many+0x1cb/0x200 Mar 22 13:09:06 Tower kernel: ? flush_tlb_kernel_range+0x5e/0x78 Mar 22 13:09:06 Tower kernel: dsalloc_pages+0x38/0x5e Mar 22 13:09:06 Tower kernel: reserve_ds_buffers+0x19e/0x382 Mar 22 13:09:06 Tower kernel: ? reprogram_counter+0x54/0x54 [kvm] Mar 22 13:09:06 Tower kernel: x86_reserve_hardware+0x134/0x14f Mar 22 13:09:06 Tower kernel: x86_pmu_event_init+0x3a/0x1d5 Mar 22 13:09:06 Tower kernel: ? reprogram_counter+0x54/0x54 [kvm] Mar 22 13:09:06 Tower kernel: perf_try_init_event+0x4f/0x7d Mar 22 13:09:06 Tower kernel: perf_event_alloc+0x46e/0x821 Mar 22 13:09:06 Tower kernel: perf_event_create_kernel_counter+0x1a/0xff Mar 22 13:09:06 Tower kernel: pmc_reprogram_counter+0xd9/0x111 [kvm] Mar 22 13:09:06 Tower kernel: reprogram_fixed_counter+0xd8/0xfc [kvm] Mar 22 13:09:06 Tower kernel: intel_pmu_set_msr+0x176/0x2e4 [kvm_intel] Mar 22 13:09:06 Tower kernel: kvm_set_msr_common+0xc6e/0xd24 [kvm] Mar 22 13:09:06 Tower kernel: handle_wrmsr+0x4b/0x85 [kvm_intel] Mar 22 13:09:06 Tower kernel: kvm_arch_vcpu_ioctl_run+0x10d0/0x1367 [kvm] Mar 22 13:09:06 Tower kernel: ? group_sched_in+0x119/0x119 Mar 22 13:09:06 Tower kernel: kvm_vcpu_ioctl+0x17b/0x4b1 [kvm] Mar 22 13:09:06 Tower kernel: ? __seccomp_filter+0x39/0x1ed Mar 22 13:09:06 Tower kernel: ? __perf_event_task_sched_in+0xdf/0x160 Mar 22 13:09:06 Tower kernel: vfs_ioctl+0x19/0x26 Mar 22 13:09:06 Tower kernel: do_vfs_ioctl+0x533/0x55d Mar 22 13:09:06 Tower kernel: ? finish_task_switch+0xbd/0x222 Mar 22 13:09:06 Tower kernel: ksys_ioctl+0x37/0x56 Mar 22 13:09:06 Tower kernel: __x64_sys_ioctl+0x11/0x14 Mar 22 13:09:06 Tower kernel: do_syscall_64+0x57/0xf2 Mar 22 13:09:06 Tower kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Mar 22 13:09:06 Tower kernel: RIP: 0033:0x153983a274b7 Mar 22 13:09:06 Tower kernel: Code: 00 00 90 48 8b 05 d9 29 0d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d a9 29 0d 00 f7 d8 64 89 01 48 Mar 22 13:09:06 Tower kernel: RSP: 002b:00001539811fe678 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 Mar 22 13:09:06 Tower kernel: RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 0000153983a274b7 Mar 22 13:09:06 Tower kernel: RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000015 Mar 22 13:09:06 Tower kernel: RBP: 000015398183c700 R08: 000055e6c17f0770 R09: 000000003b9aca00 Mar 22 13:09:06 Tower kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000 Mar 22 13:09:06 Tower kernel: R13: 00001539854a4004 R14: 0000000000000608 R15: 0000000000000000 Mar 22 13:09:06 Tower kernel: Mem-Info: Mar 22 13:09:06 Tower kernel: active_anon:3176495 inactive_anon:21942 isolated_anon:0 Mar 22 13:09:06 Tower kernel: active_file:1318622 inactive_file:2866427 isolated_file:0 Mar 22 13:09:06 Tower kernel: unevictable:8 dirty:57 writeback:0 unstable:0 Mar 22 13:09:06 Tower kernel: slab_reclaimable:435507 slab_unreclaimable:63250 Mar 22 13:09:06 Tower kernel: mapped:46344 shmem:205573 pagetables:11491 bounce:0 Mar 22 13:09:06 Tower kernel: free:75441 free_pcp:63 free_cma:0 Mar 22 13:09:06 Tower kernel: Node 0 active_anon:12705980kB inactive_anon:87768kB active_file:5274488kB inactive_file:11465708kB unevictable:32kB isolated(anon):0kB isolated(file):0kB mapped:185376kB dirty:228kB writeback:0kB shmem:822292kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 5279744kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no Mar 22 13:09:06 Tower kernel: Node 0 DMA free:15892kB min:32kB low:44kB high:56kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15976kB managed:15892kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Mar 22 13:09:06 Tower kernel: lowmem_reserve[]: 0 1778 31854 31854 Mar 22 13:09:06 Tower kernel: Node 0 DMA32 free:126108kB min:3768kB low:5588kB high:7408kB active_anon:1539660kB inactive_anon:36kB active_file:74384kB inactive_file:191344kB unevictable:0kB writepending:4kB present:2033736kB managed:2016512kB mlocked:0kB kernel_stack:144kB pagetables:312kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Mar 22 13:09:06 Tower kernel: lowmem_reserve[]: 0 0 30076 30076 Mar 22 13:09:06 Tower kernel: Node 0 Normal free:159764kB min:63776kB low:94572kB high:125368kB active_anon:11166320kB inactive_anon:87732kB active_file:5200216kB inactive_file:11274116kB unevictable:32kB writepending:224kB present:31322112kB managed:30798348kB mlocked:32kB kernel_stack:12832kB pagetables:45652kB bounce:0kB free_pcp:252kB local_pcp:0kB free_cma:0kB Mar 22 13:09:06 Tower kernel: lowmem_reserve[]: 0 0 0 0 Mar 22 13:09:06 Tower kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Mar 22 13:09:06 Tower kernel: Node 0 DMA32: 2128*4kB (UME) 567*8kB (UME) 588*16kB (UMEH) 376*32kB (UMH) 234*64kB (UMH) 150*128kB (UMEH) 135*256kB (UMEH) 35*512kB (UMEH) 5*1024kB (ME) 0*2048kB 0*4096kB = 126264kB Mar 22 13:09:06 Tower kernel: Node 0 Normal: 24680*4kB (UME) 7700*8kB (UME) 31*16kB (UME) 2*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 160880kB Mar 22 13:09:06 Tower kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Mar 22 13:09:06 Tower kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Mar 22 13:09:06 Tower kernel: 4283153 total pagecache pages Mar 22 13:09:06 Tower kernel: 0 pages in swap cache Mar 22 13:09:06 Tower kernel: Swap cache stats: add 0, delete 0, find 0/0 Mar 22 13:09:06 Tower kernel: Free swap = 0kB Mar 22 13:09:06 Tower kernel: Total swap = 0kB Mar 22 13:09:06 Tower kernel: 8342956 pages RAM Mar 22 13:09:06 Tower kernel: 0 pages HighMem/MovableOnly Mar 22 13:09:06 Tower kernel: 135268 pages reserved Mar 22 13:09:06 Tower kernel: 0 pages cma reserved Diagnostics are ofcourse attached. tower-diagnostics-20200323-1634.zip

Just run the SYNCTHING docker on both machines.. I do this and it works fine. They have to be able to reach each other but a wireguard connection should lbe able to take care of that ... Or another VPN solution.. I used to work with Crashplan, that works to but backupping to another machine was no longer free..

I have a 10gb in my unraid server and a 10gb in my desktop, both are connected to a mikrotik switch at 10gig. For regular use there is not a real big difference. Large transfers -are- quicker. But to be honoust does it really care if you wait 15 seconds are 10 seconds for a transfer to complete ? Unless it becomes nearly instantaneous it does not really make a difference. It is cool to see the quick speeds though

System has been fully stable for a week. I will keep it in this state.

Bit of a long shot, but are you using docker containers with their own ip addresses? This caused the same issue with me... System has been fully stable for a week since I removed that...

Great idea and thanks for the suggestion. My server is now maxing out on 100% cpu with BOINC on rosetta !

Still no crashes... Updated to 6.8.3. No errors (But for the long list of rsyslogd errors at startup that show up because it gets launched before network is available).

36 hours further and still no crashes. I now installed cache dirs... Hoping I will keep stable.. Would really miss this one..

Still no errors... Installed Dynamix SSD Trim, Wireguard and SystemStats as the first ones to bring back. Since these are not continuously doing something I am not suspecting those of issues... Will report back in a few days.

24 hours no errors without the dynamics plugins... Not what I expected to be honoust... I will keep it like this for a few days just to be sure and will then slowly bring back plugin after plugin..

Restarted without safe mode yesterday and errors are comming back: Mar 5 17:49:07 Tower kernel: CPU: 4 PID: 15956 Comm: CPU 0/KVM Tainted: G O 4.19.98-Unraid #1 Mar 5 17:49:07 Tower kernel: Call Trace: Mar 5 22:58:33 Tower kernel: CPU: 4 PID: 15956 Comm: CPU 0/KVM Tainted: G O 4.19.98-Unraid #1 Mar 5 22:58:33 Tower kernel: Call Trace: Mar 5 23:04:08 Tower kernel: CPU: 4 PID: 15956 Comm: CPU 0/KVM Tainted: G O 4.19.98-Unraid #1 Mar 5 23:04:08 Tower kernel: Call Trace: Mar 5 23:04:08 Tower kernel: CPU: 4 PID: 15956 Comm: CPU 0/KVM Tainted: G W O 4.19.98-Unraid #1 Mar 5 23:04:08 Tower kernel: Call Trace: Mar 5 23:04:08 Tower kernel: CPU: 4 PID: 15956 Comm: CPU 0/KVM Tainted: G W O 4.19.98-Unraid #1 Mar 5 23:04:08 Tower kernel: Call Trace: Mar 5 23:06:04 Tower kernel: CPU: 4 PID: 15956 Comm: CPU 0/KVM Tainted: G W O 4.19.98-Unraid #1 Mar 5 23:06:04 Tower kernel: Call Trace: Mar 5 23:06:04 Tower kernel: CPU: 4 PID: 15956 Comm: CPU 0/KVM Tainted: G W O 4.19.98-Unraid #1 Mar 5 23:06:04 Tower kernel: Call Trace: Mar 5 23:06:18 Tower kernel: CPU: 4 PID: 15956 Comm: CPU 0/KVM Tainted: G W O 4.19.98-Unraid #1 Mar 5 23:06:18 Tower kernel: Call Trace: Mar 5 23:06:18 Tower kernel: CPU: 4 PID: 15956 Comm: CPU 0/KVM Tainted: G W O 4.19.98-Unraid #1 Mar 5 23:06:18 Tower kernel: Call Trace: Mar 6 01:08:35 Tower kernel: CPU: 4 PID: 15956 Comm: CPU 0/KVM Tainted: G W O 4.19.98-Unraid #1 Mar 6 01:08:35 Tower kernel: Call Trace: Mar 6 01:08:35 Tower kernel: CPU: 4 PID: 15956 Comm: CPU 0/KVM Tainted: G W O 4.19.98-Unraid #1 Mar 6 01:08:35 Tower kernel: Call Trace: Mar 6 16:36:04 Tower kernel: CPU: 4 PID: 15956 Comm: CPU 0/KVM Tainted: G W O 4.19.98-Unraid #1 Mar 6 16:36:04 Tower kernel: Call Trace: Diagnostics are attached. I will prune back my plugins to see if I can find the culprit. I will first remove all my dynamics plugins.. Not because I think those are the culprit, but because I want to put them back asap.. tower-diagnostics-20200306-1724.zip

Did someone change ? Cannot get pihole to work anymore... Had it working in bridge mode over the normal unraid ip address... Now refuses to install... ?

Still stable... System is unusually "quiet", a lot more disks spun down.. All dockers still work and do their thing though. I will keep the system in this state untill end of next week and report back. If everything keeps fine I will then restart without safe mode and see if it stays that way.

Was not able to find this thread (plex docker by plex). Issue has not returned and occured while doing a total system reinstall, scanning all my media files again due to a crash. Since this has finished the system appears to remain stable. Possible relation to the following thread:

Issue appears to be related to a bug in the system that causes specific docker ip assignments to be unstable. More explenation in the following thread: https://forums.unraid.net/topic/89038-fatal-system-crash/?tab=comments#comment-826221

Appears to be the consequence of using the ControlR app .. It does not check if you have unassigned devices but just queries it. Error is nothing to be worried about.

Totally no errors in the log since yesterday... So that points in a good direction I think.. The only errors now visible are rsyslogd errors during startup.. The daemon starts doint network traffic before the network is available.. The following appears to be for redhat, but maybe usefull: copy /usr/lib/systemd/system/rsyslog.service to /etc/systemd/system edit /etc/systemd/system/rsyslogd.service and add "After=network-online.target" to the [Unit] section

Since I cannot get the vlan thing to work I first build back everything to how it was (all on br0), that made everything back again.. Maybe someone can assist me on getting this to work? As an alternative I have now moved as much as possible dockers back to the regular bridge interface (so without a dedicated ip address), I kind of liked having all on a seperate address but it is not really necessary.. Maybe this helps. One thing I for some reason cannot get switched back is Plex.. When I switch it to bridge I end up having no IP address at all.. Also in the allocations I see a lot of plex ports dedicated but when clicking "edit" they are not visible... I moved it to HOST mode, that appears to work. This means that I now have no more Dockers with dedicatec IP addresses. Hopefully that solves my crashes. Good to specify though that my setup (with seperate IP addresses) has been working for as long as Unraid has that functionality and only started giving issues somewhere starting january.. I do hope that @limetech solves this in the end... Different IP's make stuff with firewalls and VPN somewhat easier. At the moment I am still using Safe mode, will keep it that way for a couple of weeks to see if the issues come back..

I now changed it to the following: But this has not fixed anything... I cannot reach my dockers anymore .. I actually changed the VLAN number to 6 since 6 is a VLAN number I also use internally.. Does not seem to do anything though..

I think I need to do something with the routing table, never did that before, it now looks as follows:

Think I have found it... Enabling advanced settings shows more possibilities in the docker setup.. I have now enabled BR0.5 in there (with the same settings as BR0) and disabled BR0.. (or should I keep BR0 active ?) Dockers appear to be running but are not reachable..

That is a lot of info.. I think that I can extrapolate that it looks like an advice to move my dockers with own ip address away from the br0 interface towards another interface. I also think I have understood that that has a link with a seperate VLAN ? I have been able to create a VLAN under settings/network but that does not appear to make a different network available for my dockers ? I am missing something.. EDIT: