February 1, 20179 yr Author I'm getting an OOM error and one or both of my VMs get killed when running the mover, IIRC this started happening with the last couple of rcs. Server has 32GB of RAM, not using dockers at the moment, there are 2 VMs always running using 4GB each, total memory in use is usually at around 32%, mover was set to run at 2:55AM: Jan 30 02:55:22 Tower7 kernel: cpuload invoked oom-killer: gfp_mask=0x24200ca(GFP_HIGHUSER_MOVABLE), nodemask=0, order=0, oom_score_adj=0 Jan 30 02:55:22 Tower7 kernel: cpuload cpuset=/ mems_allowed=0 Jan 30 02:55:22 Tower7 kernel: CPU: 0 PID: 1860 Comm: cpuload Not tainted 4.9.6-unRAID #1 Jan 30 02:55:22 Tower7 kernel: Hardware name: Supermicro Super Server/X11SSM-F, BIOS 1.0b 12/29/2015 Jan 30 02:55:22 Tower7 kernel: ffffc900059ff880 ffffffff813a33fa ffffc900059ffa68 ffffffff8193cef3 Jan 30 02:55:22 Tower7 kernel: ffffc900059ff900 ffffffff8111eab3 0000000000000000 0000000000000000 Jan 30 02:55:22 Tower7 kernel: ffffffff810b2067 0000000000000000 ffffc900059ff8d8 ffffffff81053fe5 Jan 30 02:55:22 Tower7 kernel: Call Trace: Jan 30 02:55:22 Tower7 kernel: [<ffffffff813a33fa>] dump_stack+0x61/0x7e Jan 30 02:55:22 Tower7 kernel: [<ffffffff8111eab3>] dump_header+0x76/0x20e Jan 30 02:55:22 Tower7 kernel: [<ffffffff810b2067>] ? delayacct_end+0x51/0x5a Jan 30 02:55:22 Tower7 kernel: [<ffffffff81053fe5>] ? has_ns_capability_noaudit+0x34/0x3e Jan 30 02:55:22 Tower7 kernel: [<ffffffff810c7a2a>] oom_kill_process+0x81/0x377 Jan 30 02:55:22 Tower7 kernel: [<ffffffff810c81f3>] out_of_memory+0x3aa/0x3e5 Jan 30 02:55:22 Tower7 kernel: [<ffffffff810cbe82>] __alloc_pages_nodemask+0xb2f/0xc31 Jan 30 02:55:22 Tower7 kernel: [<ffffffff8110358d>] alloc_pages_vma+0x189/0x1e4 Jan 30 02:55:22 Tower7 kernel: [<ffffffff810dacb1>] shmem_alloc_page+0x5c/0x7d Jan 30 02:55:22 Tower7 kernel: [<ffffffff81122a8a>] ? get_empty_filp+0x4e/0x162 Jan 30 02:55:22 Tower7 kernel: [<ffffffff811380c1>] ? setattr_copy+0x9b/0xe0 Jan 30 02:55:22 Tower7 kernel: [<ffffffff810dafdf>] shmem_alloc_and_acct_page+0xd4/0x152 Jan 30 02:55:22 Tower7 kernel: [<ffffffff810db49a>] shmem_getpage_gfp+0x43d/0x9a5 Jan 30 02:55:22 Tower7 kernel: [<ffffffff810dbbe3>] shmem_getpage+0x16/0x18 Jan 30 02:55:22 Tower7 kernel: [<ffffffff810dbc23>] shmem_write_begin+0x3e/0x40 Jan 30 02:55:22 Tower7 kernel: [<ffffffff810c49d4>] generic_perform_write+0xd0/0x17a Jan 30 02:55:22 Tower7 kernel: [<ffffffff810c5ac6>] __generic_file_write_iter+0xcc/0x172 Jan 30 02:55:22 Tower7 kernel: [<ffffffff810c5c77>] generic_file_write_iter+0x10b/0x160 Jan 30 02:55:22 Tower7 kernel: [<ffffffff81120be7>] __vfs_write+0xc3/0xec Jan 30 02:55:22 Tower7 kernel: [<ffffffff811215ce>] vfs_write+0xcd/0x176 Jan 30 02:55:22 Tower7 kernel: [<ffffffff811222a9>] SyS_write+0x49/0x83 Jan 30 02:55:22 Tower7 kernel: [<ffffffff8167ce37>] entry_SYSCALL_64_fastpath+0x1a/0xa9 Jan 30 02:55:22 Tower7 kernel: Mem-Info: Jan 30 02:55:22 Tower7 kernel: active_anon:2487715 inactive_anon:16989 isolated_anon:0 Jan 30 02:55:22 Tower7 kernel: active_file:5064310 inactive_file:408609 isolated_file:128 Jan 30 02:55:22 Tower7 kernel: unevictable:0 dirty:416267 writeback:0 unstable:0 Jan 30 02:55:22 Tower7 kernel: slab_reclaimable:40726 slab_unreclaimable:30017 Jan 30 02:55:22 Tower7 kernel: mapped:13084 shmem:122944 pagetables:7034 bounce:0 Jan 30 02:55:22 Tower7 kernel: free:67697 free_pcp:186 free_cma:0 Jan 30 02:55:22 Tower7 kernel: Node 0 active_anon:9950860kB inactive_anon:67956kB active_file:20257240kB inactive_file:1634436kB unevictable:0kB isolated(anon):0kB isolated(file):512kB mapped:52336kB dirty:1665068kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 9377792kB anon_thp: 491776kB writeback_tmp:0kB unstable:0kB pages_scanned:33122412 all_unreclaimable? yes Jan 30 02:55:22 Tower7 kernel: Node 0 DMA free:15864kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15960kB managed:15876kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:12kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Jan 30 02:55:22 Tower7 kernel: lowmem_reserve[]: 0 1832 31909 31909 Jan 30 02:55:22 Tower7 kernel: Node 0 DMA32 free:127948kB min:7756kB low:9692kB high:11628kB active_anon:21880kB inactive_anon:4kB active_file:1651528kB inactive_file:177524kB unevictable:0kB writepending:180844kB present:2033580kB managed:2023600kB mlocked:0kB slab_reclaimable:32384kB slab_unreclaimable:5148kB kernel_stack:160kB pagetables:64kB bounce:0kB free_pcp:248kB local_pcp:0kB free_cma:0kB Jan 30 02:55:22 Tower7 kernel: lowmem_reserve[]: 0 0 30077 30077 Jan 30 02:55:22 Tower7 kernel: Node 0 Normal free:126976kB min:127344kB low:159180kB high:191016kB active_anon:9928980kB inactive_anon:67952kB active_file:18605712kB inactive_file:1456768kB unevictable:0kB writepending:1484224kB present:31322112kB managed:30799232kB mlocked:0kB slab_reclaimable:130520kB slab_unreclaimable:114908kB kernel_stack:7840kB pagetables:28072kB bounce:0kB free_pcp:496kB local_pcp:24kB free_cma:0kB Jan 30 02:55:22 Tower7 kernel: lowmem_reserve[]: 0 0 0 0 Jan 30 02:55:22 Tower7 kernel: Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (U) 3*4096kB (M) = 15864kB Jan 30 02:55:22 Tower7 kernel: Node 0 DMA32: 893*4kB (UM) 413*8kB (U) 3743*16kB (UMEH) 542*32kB (UMEH) 125*64kB (UME) 22*128kB (UME) 21*256kB (M) 26*512kB (UM) 8*1024kB (M) 3*2048kB (M) 0*4096kB = 127948kB Jan 30 02:55:22 Tower7 kernel: Node 0 Normal: 3726*4kB (UME) 537*8kB (UME) 304*16kB (UE) 360*32kB (UMEH) 260*64kB (UME) 188*128kB (UME) 118*256kB (UME) 34*512kB (UM) 1*1024kB (U) 1*2048kB (M) 0*4096kB = 126976kB Jan 30 02:55:22 Tower7 kernel: 5596000 total pagecache pages Jan 30 02:55:22 Tower7 kernel: 0 pages in swap cache Jan 30 02:55:22 Tower7 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jan 30 02:55:22 Tower7 kernel: Free swap = 0kB Jan 30 02:55:22 Tower7 kernel: Total swap = 0kB Jan 30 02:55:22 Tower7 kernel: 8342913 pages RAM Jan 30 02:55:22 Tower7 kernel: 0 pages HighMem/MovableOnly Jan 30 02:55:22 Tower7 kernel: 133236 pages reserved Jan 30 02:55:22 Tower7 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Jan 30 02:55:22 Tower7 kernel: [ 1378] 0 1378 5040 719 13 3 0 -1000 udevd Jan 30 02:55:22 Tower7 kernel: [ 1626] 0 1626 59436 578 25 3 0 0 rsyslogd Jan 30 02:55:22 Tower7 kernel: [ 1769] 81 1769 4900 60 14 3 0 0 dbus-daemon Jan 30 02:55:22 Tower7 kernel: [ 1777] 1 1777 3342 519 10 3 0 0 rpcbind Jan 30 02:55:22 Tower7 kernel: [ 1782] 32 1782 5352 1455 15 3 0 0 rpc.statd Jan 30 02:55:22 Tower7 kernel: [ 1792] 0 1792 1619 393 7 4 0 0 inetd Jan 30 02:55:22 Tower7 kernel: [ 1801] 0 1801 6120 670 16 3 0 -1000 sshd Jan 30 02:55:22 Tower7 kernel: [ 1815] 0 1815 24545 1173 21 3 0 0 ntpd Jan 30 02:55:22 Tower7 kernel: [ 1822] 0 1822 1095 29 7 3 0 0 acpid Jan 30 02:55:22 Tower7 kernel: [ 1831] 0 1831 1621 419 8 3 0 0 crond Jan 30 02:55:22 Tower7 kernel: [ 1833] 0 1833 1618 347 8 3 0 0 atd Jan 30 02:55:22 Tower7 kernel: [ 1839] 0 1839 55384 1448 103 4 0 0 nmbd Jan 30 02:55:22 Tower7 kernel: [ 1841] 0 1841 74644 3888 142 4 0 0 smbd Jan 30 02:55:22 Tower7 kernel: [ 1842] 0 1842 73063 1138 136 4 0 0 smbd-notifyd Jan 30 02:55:22 Tower7 kernel: [ 1843] 0 1843 73067 1087 136 4 0 0 cleanupd Jan 30 02:55:22 Tower7 kernel: [ 1848] 0 1848 68146 1815 128 3 0 0 winbindd Jan 30 02:55:22 Tower7 kernel: [ 1850] 0 1850 68147 1723 129 3 0 0 winbindd Jan 30 02:55:22 Tower7 kernel: [ 1860] 0 1860 2383 591 9 3 0 0 cpuload Jan 30 02:55:22 Tower7 kernel: [ 2910] 0 2910 2929 573 10 3 0 0 autofan Jan 30 02:55:22 Tower7 kernel: [ 8519] 0 8519 22474 954 17 3 0 0 emhttp Jan 30 02:55:22 Tower7 kernel: [ 8550] 0 8550 1627 404 8 3 0 0 agetty Jan 30 02:55:22 Tower7 kernel: [ 8551] 0 8551 1627 398 8 3 0 0 agetty Jan 30 02:55:22 Tower7 kernel: [ 8552] 0 8552 1627 396 8 3 0 0 agetty Jan 30 02:55:22 Tower7 kernel: [ 8553] 0 8553 1627 412 8 3 0 0 agetty Jan 30 02:55:22 Tower7 kernel: [ 8554] 0 8554 1627 415 8 3 0 0 agetty Jan 30 02:55:22 Tower7 kernel: [ 8555] 0 8555 1627 390 8 3 0 0 agetty Jan 30 02:55:22 Tower7 kernel: [ 8601] 0 8601 20203 66 13 4 0 0 apcupsd Jan 30 02:55:22 Tower7 kernel: [ 8624] 61 8624 8623 607 21 3 0 0 avahi-daemon Jan 30 02:55:22 Tower7 kernel: [ 8625] 61 8625 8557 63 21 3 0 0 avahi-daemon Jan 30 02:55:22 Tower7 kernel: [ 8633] 0 8633 3185 26 11 3 0 0 avahi-dnsconfd Jan 30 02:55:22 Tower7 kernel: [ 8939] 0 8939 38321 128 17 4 0 0 shfs Jan 30 02:55:22 Tower7 kernel: [ 8949] 0 8949 527341 3134 84 5 0 0 shfs Jan 30 02:55:22 Tower7 kernel: [ 9036] 0 9036 3457 1338 10 3 0 0 cache_dirs Jan 30 02:55:22 Tower7 kernel: [ 9568] 0 9568 18821 841 38 3 0 0 virtlockd Jan 30 02:55:22 Tower7 kernel: [ 9574] 0 9574 35734 958 41 4 0 0 virtlogd Jan 30 02:55:22 Tower7 kernel: [ 9589] 0 9589 203012 3191 88 3 0 0 libvirtd Jan 30 02:55:22 Tower7 kernel: [ 9724] 99 9724 4378 509 13 3 0 0 dnsmasq Jan 30 02:55:22 Tower7 kernel: [ 9725] 0 9725 4345 53 12 3 0 0 dnsmasq Jan 30 02:55:22 Tower7 kernel: [14216] 0 14216 1277617 1196801 2523 8 0 0 qemu-system-x86 Jan 30 02:55:22 Tower7 kernel: [14771] 0 14771 1266824 1178993 2441 8 0 0 qemu-system-x86 Jan 30 02:55:22 Tower7 kernel: [16490] 0 16490 103483 4444 174 4 0 0 smbd Jan 30 02:55:22 Tower7 kernel: [16491] 0 16491 68146 1004 127 3 0 0 winbindd Jan 30 02:55:22 Tower7 kernel: [21510] 99 21510 105342 6016 180 4 0 0 smbd Jan 30 02:55:22 Tower7 kernel: [23403] 0 23403 1090 172 6 3 0 0 sleep Jan 30 02:55:22 Tower7 kernel: [26064] 0 26064 2374 640 9 3 0 0 sh Jan 30 02:55:22 Tower7 kernel: [26067] 0 26067 2397 673 9 3 0 0 mover Jan 30 02:55:22 Tower7 kernel: [26071] 0 26071 1087 183 6 3 0 0 move Jan 30 02:55:22 Tower7 kernel: [26087] 0 26087 3039 560 10 3 0 0 rsync Jan 30 02:55:22 Tower7 kernel: [26088] 0 26088 2944 378 10 3 0 0 rsync Jan 30 02:55:22 Tower7 kernel: [26089] 0 26089 3009 357 10 3 0 0 rsync Jan 30 02:55:22 Tower7 kernel: [26306] 0 26306 1090 185 7 3 0 0 sleep Jan 30 02:55:22 Tower7 kernel: [26336] 0 26336 43375 5053 75 4 0 0 php Jan 30 02:55:22 Tower7 kernel: [26338] 0 26338 42656 2936 70 3 0 0 notify Jan 30 02:55:22 Tower7 kernel: Out of memory: Kill process 14216 (qemu-system-x86) score 146 or sacrifice child Jan 30 02:55:22 Tower7 kernel: Killed process 14216 (qemu-system-x86) total-vm:5110468kB, anon-rss:4770032kB, file-rss:24kB, shmem-rss:17148kB Jan 30 02:55:22 Tower7 kernel: oom_reaper: reaped process 14216 (qemu-system-x86), now anon-rss:0kB, file-rss:20kB, shmem-rss:24kB Jan 30 02:55:22 Tower7 kernel: ------------[ cut here ]------------ Jan 30 02:55:22 Tower7 kernel: WARNING: CPU: 1 PID: 14216 at arch/x86/mm/pat.c:1023 untrack_pfn+0x69/0xaa Jan 30 02:55:22 Tower7 kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables iptable_filter ip_tables vhost_net tun vhost macvtap macvlan md_mod ipmi_devintf mlx4_en mlx4_core igb ptp pps_core fbcon ast bitblit fbcon_rotate fbcon_ccw fbcon_ud fbcon_cw softcursor font ttm x86_pkg_temp_thermal coretemp drm_kms_helper cfbfillrect kvm_intel cfbimgblt cfbcopyarea drm kvm agpgart syscopyarea i2c_i801 sysfillrect i2c_smbus sysimgblt fb_sys_fops i2c_algo_bit fb nvme fbdev i2c_core nvme_core mpt3sas ahci libahci r8169 raid_class scsi_transport_sas mii ipmi_si video backlight [last unloaded: mlx4_core] Jan 30 02:55:22 Tower7 kernel: CPU: 1 PID: 14216 Comm: qemu-system-x86 Not tainted 4.9.6-unRAID #1 Jan 30 02:55:22 Tower7 kernel: Hardware name: Supermicro Super Server/X11SSM-F, BIOS 1.0b 12/29/2015 Jan 30 02:55:22 Tower7 kernel: ffffc900096dfa58 ffffffff813a33fa 0000000000000000 ffffffff81935f52 Jan 30 02:55:22 Tower7 kernel: ffffc900096dfa98 ffffffff8104d04c 000003ff10d02678 ffff880810e74508 Jan 30 02:55:22 Tower7 kernel: 00002ac519f8b000 00002ac519f8a000 ffffc900096dfb60 ffffc900096dfb60 Jan 30 02:55:22 Tower7 kernel: Call Trace: Jan 30 02:55:22 Tower7 kernel: [<ffffffff813a33fa>] dump_stack+0x61/0x7e Jan 30 02:55:22 Tower7 kernel: [<ffffffff8104d04c>] __warn+0xb8/0xd3 Jan 30 02:55:22 Tower7 kernel: [<ffffffff8104d114>] warn_slowpath_null+0x18/0x1a Jan 30 02:55:22 Tower7 kernel: [<ffffffff81046360>] untrack_pfn+0x69/0xaa Jan 30 02:55:22 Tower7 kernel: [<ffffffff810eadc1>] unmap_single_vma+0x4e/0x72 Jan 30 02:55:22 Tower7 kernel: [<ffffffff810eb056>] unmap_vmas+0x51/0x7e Jan 30 02:55:22 Tower7 kernel: [<ffffffff810f1d94>] exit_mmap+0x71/0x10c Jan 30 02:55:22 Tower7 kernel: [<ffffffff8110bd20>] ? kmem_cache_free+0x15f/0x164 Jan 30 02:55:22 Tower7 kernel: [<ffffffff8110bd20>] ? kmem_cache_free+0x15f/0x164 Jan 30 02:55:22 Tower7 kernel: [<ffffffff8104a9fc>] mmput+0x48/0xec Jan 30 02:55:22 Tower7 kernel: [<ffffffffa001ea0b>] vhost_dev_cleanup+0x351/0x361 [vhost] Jan 30 02:55:22 Tower7 kernel: [<ffffffffa00328ce>] vhost_net_release+0x3a/0xf5 [vhost_net] Jan 30 02:55:22 Tower7 kernel: [<ffffffff811228b8>] __fput+0xed/0x19f Jan 30 02:55:22 Tower7 kernel: [<ffffffff81122996>] ____fput+0x9/0xb Jan 30 02:55:22 Tower7 kernel: [<ffffffff81062569>] task_work_run+0x6a/0x7d Jan 30 02:55:22 Tower7 kernel: [<ffffffff81050163>] do_exit+0x3ae/0x86a Jan 30 02:55:22 Tower7 kernel: [<ffffffff81050687>] do_group_exit+0x3c/0x98 Jan 30 02:55:22 Tower7 kernel: [<ffffffff81058024>] get_signal+0x47e/0x4b0 Jan 30 02:55:22 Tower7 kernel: [<ffffffff8101e446>] do_signal+0x23/0x4a5 Jan 30 02:55:22 Tower7 kernel: [<ffffffff81131c00>] ? do_sys_poll+0x388/0x481 Jan 30 02:55:22 Tower7 kernel: [<ffffffff81130b1e>] ? poll_select_copy_remaining+0xe6/0xfe Jan 30 02:55:22 Tower7 kernel: [<ffffffff81002b09>] exit_to_usermode_loop+0x3a/0x81 Jan 30 02:55:22 Tower7 kernel: [<ffffffff81002c61>] syscall_return_slowpath+0x44/0x47 Jan 30 02:55:22 Tower7 kernel: [<ffffffff8167cec4>] entry_SYSCALL_64_fastpath+0xa7/0xa9 Jan 30 02:55:22 Tower7 kernel: ---[ end trace c09dcb38baeb7750 ]--- Jan 30 02:55:22 Tower7 kernel: virbr0: port 2(vnet0) entered disabled state Jan 30 02:55:22 Tower7 kernel: device vnet0 left promiscuous mode Jan 30 02:55:22 Tower7 kernel: virbr0: port 2(vnet0) entered disabled state Jan 30 02:55:24 Tower7 ntpd[1815]: Deleting interface #4 virbr0, 192.168.122.1#123, interface stats: received=0, sent=0, dropped=0, active_time=150378 secs Jan 30 02:55:24 Tower7 kernel: pci-stub 0000:01:00.0: claimed by stub Jan 30 03:04:31 Tower7 move: rmdir: /mnt/cache/./Downloads Directory not empty Jan 30 04:05:40 Tower7 kernel: mdcmd (53): spindown 0 Jan 30 04:05:41 Tower7 kernel: mdcmd (54): spindown 1 Full diags below, some mover lines removed to make the syslog smaller. In the diags I see the mover process and friends: move, rsync, but don't see in the log where the mover was actually started - maybe you have mover logging off? Also see some csrf-token issues with some plugins. How do you know this is being caused by mover? Also, do you happen to have a lot of hard-linked files? You maybe could benefit setting up a swap file.
February 1, 20179 yr In the diags I see the mover process and friends: move, rsync, but don't see in the log where the mover was actually started - maybe you have mover logging off? Also see some csrf-token issues with some plugins. Yes logging is off, the lines I removed were about open files not moved since those are still logged, but I'm sure the mover ran at that time. How do you know this is being caused by mover? Also, do you happen to have a lot of hard-linked files? You maybe could benefit setting up a swap file. I don't know if it's the actual mover but when it happened the mover was always running, it may be related to file caching, normally on this server all writes are to the cache pool. After posting this I changed the vm.dirty_background_ratio and vm.dirty_ratio from default to the lowest possible values to see if it helps. As for hard-linked files, not sure, 99% of the files moved were torrents downloaded by a vm and files copied from my windows desktop.
February 1, 20179 yr Author unRAID has stopped loading in "dashboard" type web apps since RC9. Tried on both the LSIO Muximux docker and Organizr, neither of them load the page where they would previously. Please post diags.
February 1, 20179 yr Perhaps a bit offtopic but I would have preferred a 6.2.5 release to patch all the security vulnerabilities up until rc-6. (no I don't plan on using beta/release canidate software on a "production" system )
February 2, 20179 yr Author Perhaps a bit offtopic but I would have preferred a 6.2.5 release to patch all the security vulnerabilities up until rc-6. (no I don't plan on using beta/release canidate software on a "production" system ) That's essentially what 6.3.0 is.
February 2, 20179 yr A user using the UD plugin is reporting the following errors: Feb 2 10:21:09 Tower root: error: plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 2 10:21:15 Tower root: error: plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token The message is wrong csrf_token. I believe that the csrf_token is not available at the time the UD plugin needs it. When is the csrf_token generated and available for use? Could it be that my plugin is ahead of when it is generated? His log shows the errors at the time the flash is imported: Feb 2 10:17:01 Tower emhttp: import flash device: sda Feb 2 10:17:06 Tower root: error: plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 2 10:17:11 Tower root: error: plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 2 10:17:18 Tower root: error: plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 2 10:17:23 Tower root: error: plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 2 10:17:29 Tower root: error: plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 2 10:17:35 Tower root: error: plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 2 10:17:40 Tower root: error: plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 2 10:17:45 Tower root: error: plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 2 10:17:50 Tower root: error: plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 2 10:17:56 Tower root: error: plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token I have a UD event at disks_mounted.
February 2, 20179 yr Author The message is wrong csrf_token. If it says "wrong" it means the csrf_token is being included but its value is stale. This would happen, for example, if you had a browser window open and there is some javascript doing some kind of periodic ajax call to the server (like what webGui does to update dashboard), and then the server is rebooted. If you don't refresh the browser window then the token held in the browser page will not match the newly generated token which is generated as system boot.
February 2, 20179 yr The message is wrong csrf_token. If it says "wrong" it means the csrf_token is being included but its value is stale. This would happen, for example, if you had a browser window open and there is some javascript doing some kind of periodic ajax call to the server (like what webGui does to update dashboard), and then the server is rebooted. If you don't refresh the browser window then the token held in the browser page will not match the newly generated token which is generated as system boot. That could explain it. Thanks.
February 3, 20179 yr The message is wrong csrf_token. If it says "wrong" it means the csrf_token is being included but its value is stale. This would happen, for example, if you had a browser window open and there is some javascript doing some kind of periodic ajax call to the server (like what webGui does to update dashboard), and then the server is rebooted. If you don't refresh the browser window then the token held in the browser page will not match the newly generated token which is generated as system boot. That could explain it. Thanks. If this is indeed the cause, is there a fix?
February 3, 20179 yr The message is wrong csrf_token. If it says "wrong" it means the csrf_token is being included but its value is stale. This would happen, for example, if you had a browser window open and there is some javascript doing some kind of periodic ajax call to the server (like what webGui does to update dashboard), and then the server is rebooted. If you don't refresh the browser window then the token held in the browser page will not match the newly generated token which is generated as system boot. That could explain it. Thanks. If this is indeed the cause, is there a fix? Refresh the browser?
February 3, 20179 yr Any news on this ? http://lime-technology.com/forum/index.php?topic=55986.msg534790#msg534790 (SASLP issues)
February 3, 20179 yr Refresh the browser? lol.. I wish things were that easy with this unRAID version. But thanks anyway for the tip Seems as though Tom pinpointed the issue in previous post, but the errors just wont go away. Even after booting into safe mode this is the result: Feb 2 18:02:29 Tower kernel: br0: port 1(eth0) entered blocking state Feb 2 18:02:29 Tower kernel: br0: port 1(eth0) entered forwarding state Feb 2 18:02:31 Tower ntpd[1724]: Listen normally on 3 br0 192.168.1.25:123 Feb 2 18:02:31 Tower ntpd[1724]: new interface(s) found: waking up resolver Feb 2 18:02:35 Tower root: error: webGui/include/ProcessStatus.php: wrong csrf_token Feb 2 18:02:36 Tower root: error: webGui/include/ProcessStatus.php: wrong csrf_token Feb 2 18:02:36 Tower emhttp: shcmd (18): rmmod md-mod |& logger Feb 2 18:02:36 Tower kernel: md: unRAID driver removed Feb 2 18:02:36 Tower emhttp: shcmd (19): modprobe md-mod super=/boot/config/super.dat |& logger Feb 2 18:02:36 Tower kernel: md: unRAID driver 2.7.1 installed Feb 2 18:02:36 Tower emhttp: Pro key detected, GUID: 058F-6387-0000-000079010148 FILE: /boot/config/Pro.key Safe Mode boot syslog attached tower-syslog-20170202-1805.zip
February 3, 20179 yr The message is wrong csrf_token. If it says "wrong" it means the csrf_token is being included but its value is stale. This would happen, for example, if you had a browser window open and there is some javascript doing some kind of periodic ajax call to the server (like what webGui does to update dashboard), and then the server is rebooted. If you don't refresh the browser window then the token held in the browser page will not match the newly generated token which is generated as system boot. How about instead of merely just logging the wrong_csrf error you also throw up a pop up or something telling the user to refresh the page
February 3, 20179 yr Author Refresh the browser? lol.. I wish things were that easy with this unRAID version. But thanks anyway for the tip Seems as though Tom pinpointed the issue in previous post, but the errors just wont go away. Even after booting into safe mode this is the result: Feb 2 18:02:29 Tower kernel: br0: port 1(eth0) entered blocking state Feb 2 18:02:29 Tower kernel: br0: port 1(eth0) entered forwarding state Feb 2 18:02:31 Tower ntpd[1724]: Listen normally on 3 br0 192.168.1.25:123 Feb 2 18:02:31 Tower ntpd[1724]: new interface(s) found: waking up resolver Feb 2 18:02:35 Tower root: error: webGui/include/ProcessStatus.php: wrong csrf_token Feb 2 18:02:36 Tower root: error: webGui/include/ProcessStatus.php: wrong csrf_token Feb 2 18:02:36 Tower emhttp: shcmd (18): rmmod md-mod |& logger Feb 2 18:02:36 Tower kernel: md: unRAID driver removed Feb 2 18:02:36 Tower emhttp: shcmd (19): modprobe md-mod super=/boot/config/super.dat |& logger Feb 2 18:02:36 Tower kernel: md: unRAID driver 2.7.1 installed Feb 2 18:02:36 Tower emhttp: Pro key detected, GUID: 058F-6387-0000-000079010148 FILE: /boot/config/Pro.key Safe Mode boot syslog attached Looking in the syslog, here is emhttp startup: Feb 2 18:02:26 Tower emhttp: unRAID System Management Utility version 6.3.0-rc9 The following syslog lines are generated as a result of emhttp starting up, before it starts accepting HTTP connections. Earliest time for that would be right after this message which occurs very near the end of emhttp initialization: Feb 2 18:02:27 Tower emhttp: autostart disabled At this point emhttp will start accept HTTP connections but we see that the networking subsystem has not completed initialization until a few seconds later marked by this line: Feb 2 18:02:31 Tower ntpd[1724]: new interface(s) found: waking up resolver Now notice the immediate next two lines are for "wrong csrf_token": Feb 2 18:02:35 Tower root: error: webGui/include/ProcessStatus.php: wrong csrf_token Feb 2 18:02:36 Tower root: error: webGui/include/ProcessStatus.php: wrong csrf_token This indicates to me that a browser window is sent an ajax.post() using a stale token, but then that same browser window must have initiated a "cmdStatus" emhttp API call because this line marks the beginning of such a call: Feb 2 18:02:36 Tower emhttp: shcmd (18): rmmod md-mod |& logger Also notice after that point in time, no more "wrong csrf_token" lines. The conclusion is: this is harmless and unavoidable if a stale browser window is still open somewhere.
February 3, 20179 yr Author The message is wrong csrf_token. If it says "wrong" it means the csrf_token is being included but its value is stale. This would happen, for example, if you had a browser window open and there is some javascript doing some kind of periodic ajax call to the server (like what webGui does to update dashboard), and then the server is rebooted. If you don't refresh the browser window then the token held in the browser page will not match the newly generated token which is generated as system boot. How about instead of merely just logging the wrong_csrf error you also throw up a pop up or something telling the user to refresh the page What we should probably do is return HTTP 403 error. This way the window can reload itself. Not going to delay 6.3.0 release for this however since an initial burst of those messages is harmless.
February 3, 20179 yr Author Any news on this ? http://lime-technology.com/forum/index.php?topic=55986.msg534790#msg534790 (SASLP issues) I know lots of people want this issue resolved but it's not a 6.3.0 "pre-release issue", though if someone in the linux community finds a driver bug and pushes a fix upstream it would show up first in a pre-release because we keep up to date with the latest kernel. Speaking of kernel, upcoming 6.3.0 is now on 4.9.7. This issue is almost certainly a problem with the card's bios. Probably there are not enough people complaining to Supermicro for them to care about it.
February 3, 20179 yr May I add that this is just the beginning of the issue. The problem began after updating the Uninstall Devices and PreClear .plg's. See here what happens after rebooting system and then installing a clean version of UD from the APPS tab. After install, the version shown is as of 2/1/17. No other plugins or dockers are engaged, and the array not started. The log entry time here is when I perform install, errors begin shortly thereafter. These errors will flood my syslog if I don't uninstall plugin. Feb 2 13:54:39 Tower emhttp: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin install https://github.com/dlandon/unassigned.devices/raw/master/unassigned.devices.plg Feb 2 13:54:40 Tower root: plugin: running: anonymous Diagnostics attached from this session tower-diagnostics-20170202-1358.zip
February 3, 20179 yr Author May I add that this is just the beginning of the issue. The problem began after updating the Uninstall Devices and PreClear .plg's. See here what happens after rebooting system and then installing a clean version of UD from the APPS tab. After install, the version shown is as of 2/1/17. No other plugins or dockers are engaged, and the array not started. The log entry time here is when I perform install, errors begin shortly thereafter. These errors will flood my syslog if I don't stop the array and uninstall plugin. Feb 2 13:54:39 Tower emhttp: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin install https://github.com/dlandon/unassigned.devices/raw/master/unassigned.devices.plg Feb 2 13:54:40 Tower root: plugin: running: anonymous Diagnostics attached from this session Sorry I don't know what messages you are referring to that flood the syslog. Are you talking about these? Feb 3 05:15:35 Tower shfs/user: err: shfs_fallocate: fallocate: (95) Operation not supported That's a different problem.
February 3, 20179 yr May I add that this is just the beginning of the issue. The problem began after updating the Uninstall Devices and PreClear .plg's. See here what happens after rebooting system and then installing a clean version of UD from the APPS tab. After install, the version shown is as of 2/1/17. No other plugins or dockers are engaged, and the array not started. The log entry time here is when I perform install, errors begin shortly thereafter. These errors will flood my syslog if I don't stop the array and uninstall plugin. Feb 2 13:54:39 Tower emhttp: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin install https://github.com/dlandon/unassigned.devices/raw/master/unassigned.devices.plg Feb 2 13:54:40 Tower root: plugin: running: anonymous Diagnostics attached from this session Sorry I don't know what messages you are referring to that flood the syslog. Are you talking about these? Feb 3 05:15:35 Tower shfs/user: err: shfs_fallocate: fallocate: (95) Operation not supported That's a different problem. So sorry, I posted the incorrect diagnostics session. Correct one posted, refer to log time noted in previous post. Thank you tower-diagnostics-20170202-1358.zip
February 3, 20179 yr Any news on this ? http://lime-technology.com/forum/index.php?topic=55986.msg534790#msg534790 (SASLP issues) I know lots of people want this issue resolved but it's not a 6.3.0 "pre-release issue", though if someone in the linux community finds a driver bug and pushes a fix upstream it would show up first in a pre-release because we keep up to date with the latest kernel. Speaking of kernel, upcoming 6.3.0 is now on 4.9.7. This issue is almost certainly a problem with the card's bios. Probably there are not enough people complaining to Supermicro for them to care about it. Why would it be the bios ? These cards have worker without a problem for years.. driver issue seems more likely no ? Verzonden vanaf mijn iPhone met Tapatalk
February 3, 20179 yr May I add that this is just the beginning of the issue. The problem began after updating the Uninstall Devices and PreClear .plg's. See here what happens after rebooting system and then installing a clean version of UD from the APPS tab. After install, the version shown is as of 2/1/17. No other plugins or dockers are engaged, and the array not started. The log entry time here is when I perform install, errors begin shortly thereafter. These errors will flood my syslog if I don't stop the array and uninstall plugin. Feb 2 13:54:39 Tower emhttp: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin install https://github.com/dlandon/unassigned.devices/raw/master/unassigned.devices.plg Feb 2 13:54:40 Tower root: plugin: running: anonymous Diagnostics attached from this session Sorry I don't know what messages you are referring to that flood the syslog. Are you talking about these? Feb 3 05:15:35 Tower shfs/user: err: shfs_fallocate: fallocate: (95) Operation not supported That's a different problem. So sorry, I posted the incorrect diagnostics session. Correct one posted, refer to log time noted in previous post. Thank you Feb 2 13:58:03 Tower root: error: plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Not an RC-9 issue. Purely an Unassigned Devices issue, and should be posted within its support thread.
February 3, 20179 yr Hi squid! Thanks for the response, but yes... I have done my "Due Dilligence" UD developer is pointing me this way: I posted on the rc9 post and this is Tom's response: If it says "wrong" it means the csrf_token is being included but its value is stale. This would happen, for example, if you had a browser window open and there is some javascript doing some kind of periodic ajax call to the server (like what webGui does to update dashboard), and then the server is rebooted. If you don't refresh the browser window then the token held in the browser page will not match the newly generated token which is generated as system boot. Could this be your problem? Beautiful response from TOM! This could very well be my issue and let me tell you why I think so.. Since updating to the new rc9 there have been a few serious unrelated issues occurring on my server so I now keep the logging window open at all times to monitor events. After the updates to UD & PreClear Disks, I rebooted my server, but left my logging window open. I also launched my IMPI Viewer to monitor reboot routine. When unRAID OS returned, I mounted disks and refreshed the logging window, once refreshed, log events began popping in including errors. Since then, I've uninstalled both plugins, rebooted and the like, but errors wont go away. I use firefox primarily if that matters. *Update* Can you boot in the safe mode and install UD and see if the errors occur then? Uninstalled UD plugin, booted into safe mode and got the following csrf errors, see 18:02:35. I also included full syslog from Safe Mode boot Feb 2 18:02:29 Tower kernel: br0: port 1(eth0) entered blocking state Feb 2 18:02:29 Tower kernel: br0: port 1(eth0) entered forwarding state Feb 2 18:02:31 Tower ntpd[1724]: Listen normally on 3 br0 192.168.1.25:123 Feb 2 18:02:31 Tower ntpd[1724]: new interface(s) found: waking up resolver Feb 2 18:02:35 Tower root: error: webGui/include/ProcessStatus.php: wrong csrf_token Feb 2 18:02:36 Tower root: error: webGui/include/ProcessStatus.php: wrong csrf_token Feb 2 18:02:36 Tower emhttp: shcmd (18): rmmod md-mod |& logger Feb 2 18:02:36 Tower kernel: md: unRAID driver removed Feb 2 18:02:36 Tower emhttp: shcmd (19): modprobe md-mod super=/boot/config/super.dat |& logger Feb 2 18:02:36 Tower kernel: md: unRAID driver 2.7.1 installed Feb 2 18:02:36 Tower emhttp: Pro key detected, GUID: 058F-6387-0000-000079010148 FILE: /boot/config/Pro.key ***Bump*** Anymore on this, or am I stuck? If you booted in safe mode, this has to relate to unRAID and not plugins. I would post this on the rc9 release post along with your diagnostics. As Tom has said, the wrong csrf_token is from a stale csrf_token that can happen from a web page needing to be refreshed.
February 3, 20179 yr Hi squid! Thanks for the response, but yes... I have done my "Due Dilligence" UD developer is pointing me this way: I posted on the rc9 post and this is Tom's response: If it says "wrong" it means the csrf_token is being included but its value is stale. This would happen, for example, if you had a browser window open and there is some javascript doing some kind of periodic ajax call to the server (like what webGui does to update dashboard), and then the server is rebooted. If you don't refresh the browser window then the token held in the browser page will not match the newly generated token which is generated as system boot. Could this be your problem? Beautiful response from TOM! This could very well be my issue and let me tell you why I think so.. Since updating to the new rc9 there have been a few serious unrelated issues occurring on my server so I now keep the logging window open at all times to monitor events. After the updates to UD & PreClear Disks, I rebooted my server, but left my logging window open. I also launched my IMPI Viewer to monitor reboot routine. When unRAID OS returned, I mounted disks and refreshed the logging window, once refreshed, log events began popping in including errors. Since then, I've uninstalled both plugins, rebooted and the like, but errors wont go away. I use firefox primarily if that matters. *Update* Can you boot in the safe mode and install UD and see if the errors occur then? Uninstalled UD plugin, booted into safe mode and got the following csrf errors, see 18:02:35. I also included full syslog from Safe Mode boot Feb 2 18:02:29 Tower kernel: br0: port 1(eth0) entered blocking state Feb 2 18:02:29 Tower kernel: br0: port 1(eth0) entered forwarding state Feb 2 18:02:31 Tower ntpd[1724]: Listen normally on 3 br0 192.168.1.25:123 Feb 2 18:02:31 Tower ntpd[1724]: new interface(s) found: waking up resolver Feb 2 18:02:35 Tower root: error: webGui/include/ProcessStatus.php: wrong csrf_token Feb 2 18:02:36 Tower root: error: webGui/include/ProcessStatus.php: wrong csrf_token Feb 2 18:02:36 Tower emhttp: shcmd (18): rmmod md-mod |& logger Feb 2 18:02:36 Tower kernel: md: unRAID driver removed Feb 2 18:02:36 Tower emhttp: shcmd (19): modprobe md-mod super=/boot/config/super.dat |& logger Feb 2 18:02:36 Tower kernel: md: unRAID driver 2.7.1 installed Feb 2 18:02:36 Tower emhttp: Pro key detected, GUID: 058F-6387-0000-000079010148 FILE: /boot/config/Pro.key ***Bump*** Anymore on this, or am I stuck? If you booted in safe mode, this has to relate to unRAID and not plugins. I would post this on the rc9 release post along with your diagnostics. As Tom has said, the wrong csrf_token is from a stale csrf_token that can happen from a web page needing to be refreshed. A wrong csrf (assuming that Unassigned Devices doesn't have any hardcoded anywhere, and I would be shocked if it did -> everyone would be seeing this) is generally from a stale browser session open after a reboot. Note that this browser session can be on any device. If you've got your phone's browser sitting on the UD screen and then reboot the server, then this stuff is going to get generated until you reload / refresh your browser in the phone. Not saying that its impossible that this isn't LT's issue, but since no one else is seeing it, the odds state that you've got a browser open somewhere sitting idle and continually updating the stats for UD.
February 3, 20179 yr Author Any news on this ? http://lime-technology.com/forum/index.php?topic=55986.msg534790#msg534790 (SASLP issues) I know lots of people want this issue resolved but it's not a 6.3.0 "pre-release issue", though if someone in the linux community finds a driver bug and pushes a fix upstream it would show up first in a pre-release because we keep up to date with the latest kernel. Speaking of kernel, upcoming 6.3.0 is now on 4.9.7. This issue is almost certainly a problem with the card's bios. Probably there are not enough people complaining to Supermicro for them to care about it. Why would it be the bios ? These cards have worker without a problem for years.. driver issue seems more likely no ? Verzonden vanaf mijn iPhone met Tapatalk From what I understand the issue presents itself after changing h/w: eg, different motherboard, adding controller, adding device(s) etc. Is that not the case?
February 3, 20179 yr Hi squid! Thanks for the response, but yes... I have done my "Due Dilligence" UD developer is pointing me this way: I posted on the rc9 post and this is Tom's response: If it says "wrong" it means the csrf_token is being included but its value is stale. This would happen, for example, if you had a browser window open and there is some javascript doing some kind of periodic ajax call to the server (like what webGui does to update dashboard), and then the server is rebooted. If you don't refresh the browser window then the token held in the browser page will not match the newly generated token which is generated as system boot. Could this be your problem? Beautiful response from TOM! This could very well be my issue and let me tell you why I think so.. Since updating to the new rc9 there have been a few serious unrelated issues occurring on my server so I now keep the logging window open at all times to monitor events. After the updates to UD & PreClear Disks, I rebooted my server, but left my logging window open. I also launched my IMPI Viewer to monitor reboot routine. When unRAID OS returned, I mounted disks and refreshed the logging window, once refreshed, log events began popping in including errors. Since then, I've uninstalled both plugins, rebooted and the like, but errors wont go away. I use firefox primarily if that matters. *Update* Can you boot in the safe mode and install UD and see if the errors occur then? Uninstalled UD plugin, booted into safe mode and got the following csrf errors, see 18:02:35. I also included full syslog from Safe Mode boot Feb 2 18:02:29 Tower kernel: br0: port 1(eth0) entered blocking state Feb 2 18:02:29 Tower kernel: br0: port 1(eth0) entered forwarding state Feb 2 18:02:31 Tower ntpd[1724]: Listen normally on 3 br0 192.168.1.25:123 Feb 2 18:02:31 Tower ntpd[1724]: new interface(s) found: waking up resolver Feb 2 18:02:35 Tower root: error: webGui/include/ProcessStatus.php: wrong csrf_token Feb 2 18:02:36 Tower root: error: webGui/include/ProcessStatus.php: wrong csrf_token Feb 2 18:02:36 Tower emhttp: shcmd (18): rmmod md-mod |& logger Feb 2 18:02:36 Tower kernel: md: unRAID driver removed Feb 2 18:02:36 Tower emhttp: shcmd (19): modprobe md-mod super=/boot/config/super.dat |& logger Feb 2 18:02:36 Tower kernel: md: unRAID driver 2.7.1 installed Feb 2 18:02:36 Tower emhttp: Pro key detected, GUID: 058F-6387-0000-000079010148 FILE: /boot/config/Pro.key ***Bump*** Anymore on this, or am I stuck? If you booted in safe mode, this has to relate to unRAID and not plugins. I would post this on the rc9 release post along with your diagnostics. As Tom has said, the wrong csrf_token is from a stale csrf_token that can happen from a web page needing to be refreshed. A wrong csrf (assuming that Unassigned Devices doesn't have any hardcoded anywhere, and I would be shocked if it did -> everyone would be seeing this) is generally from a stale browser session open after a reboot. Note that this browser session can be on any device. If you've got your phone's browser sitting on the UD screen and then reboot the server, then this stuff is going to get generated until you reload / refresh your browser in the phone. Not saying that its impossible that this isn't LT's issue, but since no one else is seeing it, the odds state that you've got a browser open somewhere sitting idle and continually updating the stats for UD. He is also reporting it on the preclear plugin.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.