Jump to content
BBoYTuRBo

Frequent Random Lockups

6 posts in this topic Last Reply

Recommended Posts

So I've had ongoing issues with my dad's unRAID server freezing up every day or 2, with no errors or clues of any kind showing up in logs. I even built a new rig with all new hardware (except the hard disks) and the problems continue. The server is able to be pinged, but the web interface and network shares don't work, and when trying to enter a command through SSH or locally, everything freezes completely.

 

Configuration is pretty much identical to my own server, and I have no issues.

 

Just now I was watching dmesg in realtime after replacing the cache disk (trying to see if that would have any beneficial effect), and this time there was actually an error message before it crashed (but the symptoms of this crash are a bit different than usual).

 

Edit: Forgot to mention, running version 6.4.0

 

In any case, here's what showed up:

[Mon Feb  5 14:57:12 2018] php: page allocation stalls for 42430ms, order:1, mode:0x17080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO|__GFP_NOTRACK), nodemask=(null)
[Mon Feb  5 14:57:12 2018] php cpuset=/ mems_allowed=0
[Mon Feb  5 14:57:12 2018] CPU: 0 PID: 18482 Comm: php Not tainted 4.14.13-unRAID #1
[Mon Feb  5 14:57:12 2018] Hardware name: Gigabyte Technology Co., Ltd. B150M-DS3H/B150M-DS3H-CF, BIOS F20 11/04/2016
[Mon Feb  5 14:57:12 2018] Call Trace:
[Mon Feb  5 14:57:12 2018]  dump_stack+0x5d/0x79
[Mon Feb  5 14:57:12 2018]  warn_alloc+0xdf/0x160
[Mon Feb  5 14:57:12 2018]  ? wakeup_kswapd+0x2c/0xb2
[Mon Feb  5 14:57:12 2018]  __alloc_pages_nodemask+0x578/0xb03
[Mon Feb  5 14:57:12 2018]  ? sched_ttwu_pending+0x75/0x75
[Mon Feb  5 14:57:12 2018]  ? get_mem_cgroup_from_mm+0x82/0x88
[Mon Feb  5 14:57:12 2018]  __get_free_pages+0x5/0x32
[Mon Feb  5 14:57:12 2018]  pgd_alloc+0x14/0xf5
[Mon Feb  5 14:57:12 2018]  mm_init+0x168/0x213
[Mon Feb  5 14:57:12 2018]  do_execveat_common.isra.14+0x230/0x5c8
[Mon Feb  5 14:57:12 2018]  do_execve+0x1a/0x1c
[Mon Feb  5 14:57:12 2018]  SyS_execve+0x23/0x2a
[Mon Feb  5 14:57:12 2018]  do_syscall_64+0x5b/0xf8
[Mon Feb  5 14:57:12 2018]  entry_SYSCALL64_slow_path+0x25/0x25
[Mon Feb  5 14:57:12 2018] RIP: 0033:0x15038177a697
[Mon Feb  5 14:57:12 2018] RSP: 002b:00007ffffff09d48 EFLAGS: 00000246 ORIG_RAX: 000000000000003b
[Mon Feb  5 14:57:12 2018] RAX: ffffffffffffffda RBX: 00007ffffff09d80 RCX: 000015038177a697
[Mon Feb  5 14:57:12 2018] RDX: 0000000000e13e70 RSI: 00007ffffff09d50 RDI: 0000150381840df5
[Mon Feb  5 14:57:12 2018] RBP: 00007ffffff09de0 R08: 00007ffffff09d70 R09: 00007ffffff09da0
[Mon Feb  5 14:57:12 2018] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
[Mon Feb  5 14:57:12 2018] R13: 0000150380e83a98 R14: 0000000000000008 R15: 0000000000000004
[Mon Feb  5 14:57:12 2018] Mem-Info:
[Mon Feb  5 14:57:12 2018] active_anon:231874 inactive_anon:251846 isolated_anon:0
                            active_file:286306 inactive_file:1020921 isolated_file:32
                            unevictable:1027 dirty:379 writeback:8 unstable:0
                            slab_reclaimable:90535 slab_unreclaimable:83135
                            mapped:83572 shmem:191245 pagetables:5194 bounce:0
                            free:42821 free_pcp:59 free_cma:0
[Mon Feb  5 14:57:12 2018] Node 0 active_anon:927496kB inactive_anon:1007384kB active_file:1145224kB inactive_file:4083684kB unevictable:4108kB isolated(anon):0kB isolated(file):128kB mapped:334288kB dirty
:1516kB writeback:32kB shmem:764980kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 231424kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[Mon Feb  5 14:57:12 2018] Node 0 DMA free:15896kB min:268kB low:332kB high:396kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15980kB manage
d:15896kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Mon Feb  5 14:57:12 2018] lowmem_reserve[]: 0 2826 7802 7802
[Mon Feb  5 14:57:12 2018] Node 0 DMA32 free:68380kB min:48876kB low:61092kB high:73308kB active_anon:315484kB inactive_anon:258296kB active_file:423392kB inactive_file:1728652kB unevictable:4kB writependi
ng:1148kB present:3057488kB managed:3044120kB mlocked:4kB kernel_stack:1196kB pagetables:4072kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Mon Feb  5 14:57:12 2018] lowmem_reserve[]: 0 0 4975 4975
[Mon Feb  5 14:57:12 2018] Node 0 Normal free:87008kB min:86020kB low:107524kB high:129028kB active_anon:611652kB inactive_anon:748972kB active_file:721564kB inactive_file:2354936kB unevictable:4104kB writ
epending:400kB present:5226496kB managed:5094960kB mlocked:4104kB kernel_stack:11280kB pagetables:16704kB bounce:0kB free_pcp:236kB local_pcp:116kB free_cma:0kB
[Mon Feb  5 14:57:12 2018] lowmem_reserve[]: 0 0 0 0
[Mon Feb  5 14:57:12 2018] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (U) 3*4096kB (M) = 15896kB
[Mon Feb  5 14:57:12 2018] Node 0 DMA32: 14*4kB (UME) 19*8kB (UME) 1163*16kB (UME) 1149*32kB (UE) 208*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 68896kB
[Mon Feb  5 14:57:12 2018] Node 0 Normal: 31*4kB (MEH) 55*8kB (ME) 511*16kB (UMEH) 1566*32kB (UMEH) 438*64kB (UEH) 1*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 87012kB
[Mon Feb  5 14:57:12 2018] 1499386 total pagecache pages
[Mon Feb  5 14:57:12 2018] 866 pages in swap cache
[Mon Feb  5 14:57:12 2018] Swap cache stats: add 17987, delete 17121, find 5891/7065
[Mon Feb  5 14:57:12 2018] Free swap  = 8130300kB
[Mon Feb  5 14:57:12 2018] Total swap = 8160252kB
[Mon Feb  5 14:57:12 2018] 2074991 pages RAM
[Mon Feb  5 14:57:12 2018] 0 pages HighMem/MovableOnly
[Mon Feb  5 14:57:12 2018] 36247 pages reserved
[Mon Feb  5 14:57:12 2018] 0 pages cma reserved
[Mon Feb  5 14:57:12 2018] php: page allocation stalls for 42430ms, order:1, mode:0x17080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO|__GFP_NOTRACK), nodemask=(null)
[Mon Feb  5 14:57:12 2018] php cpuset=/ mems_allowed=0
[Mon Feb  5 14:57:12 2018] CPU: 0 PID: 18482 Comm: php Not tainted 4.14.13-unRAID #1
[Mon Feb  5 14:57:12 2018] Hardware name: Gigabyte Technology Co., Ltd. B150M-DS3H/B150M-DS3H-CF, BIOS F20 11/04/2016
[Mon Feb  5 14:57:12 2018] Call Trace:
[Mon Feb  5 14:57:12 2018]  dump_stack+0x5d/0x79
[Mon Feb  5 14:57:12 2018]  warn_alloc+0xdf/0x160
[Mon Feb  5 14:57:12 2018]  ? wakeup_kswapd+0x2c/0xb2
[Mon Feb  5 14:57:12 2018]  __alloc_pages_nodemask+0x578/0xb03
[Mon Feb  5 14:57:12 2018]  ? sched_ttwu_pending+0x75/0x75
[Mon Feb  5 14:57:12 2018]  ? get_mem_cgroup_from_mm+0x82/0x88
[Mon Feb  5 14:57:12 2018]  __get_free_pages+0x5/0x32
[Mon Feb  5 14:57:12 2018]  pgd_alloc+0x14/0xf5
[Mon Feb  5 14:57:12 2018]  mm_init+0x168/0x213
[Mon Feb  5 14:57:12 2018]  do_execveat_common.isra.14+0x230/0x5c8
[Mon Feb  5 14:57:12 2018]  do_execve+0x1a/0x1c
[Mon Feb  5 14:57:12 2018]  SyS_execve+0x23/0x2a
[Mon Feb  5 14:57:12 2018]  do_syscall_64+0x5b/0xf8
[Mon Feb  5 14:57:12 2018]  entry_SYSCALL64_slow_path+0x25/0x25
[Mon Feb  5 14:57:12 2018] RIP: 0033:0x15038177a697
[Mon Feb  5 14:57:12 2018] RSP: 002b:00007ffffff09d48 EFLAGS: 00000246 ORIG_RAX: 000000000000003b
[Mon Feb  5 14:57:12 2018] RAX: ffffffffffffffda RBX: 00007ffffff09d80 RCX: 000015038177a697
[Mon Feb  5 14:57:12 2018] RDX: 0000000000e13e70 RSI: 00007ffffff09d50 RDI: 0000150381840df5
[Mon Feb  5 14:57:12 2018] RBP: 00007ffffff09de0 R08: 00007ffffff09d70 R09: 00007ffffff09da0
[Mon Feb  5 14:57:12 2018] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
[Mon Feb  5 14:57:12 2018] R13: 0000150380e83a98 R14: 0000000000000008 R15: 0000000000000004
[Mon Feb  5 14:57:12 2018] php: page allocation stalls for 42431ms, order:1, mode:0x17080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO|__GFP_NOTRACK), nodemask=(null)
[Mon Feb  5 14:57:12 2018] php cpuset=/ mems_allowed=0
[Mon Feb  5 14:57:12 2018] CPU: 0 PID: 18482 Comm: php Not tainted 4.14.13-unRAID #1
[Mon Feb  5 14:57:12 2018] Hardware name: Gigabyte Technology Co., Ltd. B150M-DS3H/B150M-DS3H-CF, BIOS F20 11/04/2016
[Mon Feb  5 14:57:12 2018] Call Trace:
[Mon Feb  5 14:57:12 2018]  dump_stack+0x5d/0x79
[Mon Feb  5 14:57:12 2018]  warn_alloc+0xdf/0x160
[Mon Feb  5 14:57:12 2018]  ? wakeup_kswapd+0x2c/0xb2
[Mon Feb  5 14:57:12 2018]  __alloc_pages_nodemask+0x578/0xb03
[Mon Feb  5 14:57:12 2018]  ? sched_ttwu_pending+0x75/0x75
[Mon Feb  5 14:57:12 2018]  ? get_mem_cgroup_from_mm+0x82/0x88
[Mon Feb  5 14:57:12 2018]  __get_free_pages+0x5/0x32
[Mon Feb  5 14:57:12 2018]  pgd_alloc+0x14/0xf5
[Mon Feb  5 14:57:12 2018]  mm_init+0x168/0x213
[Mon Feb  5 14:57:12 2018]  do_execveat_common.isra.14+0x230/0x5c8
[Mon Feb  5 14:57:12 2018]  do_execve+0x1a/0x1c
[Mon Feb  5 14:57:12 2018]  SyS_execve+0x23/0x2a
[Mon Feb  5 14:57:12 2018]  do_syscall_64+0x5b/0xf8
[Mon Feb  5 14:57:12 2018]  entry_SYSCALL64_slow_path+0x25/0x25
[Mon Feb  5 14:57:12 2018] RIP: 0033:0x15038177a697
[Mon Feb  5 14:57:12 2018] RSP: 002b:00007ffffff09d48 EFLAGS: 00000246 ORIG_RAX: 000000000000003b
[Mon Feb  5 14:57:12 2018] RAX: ffffffffffffffda RBX: 00007ffffff09d80 RCX: 000015038177a697
[Mon Feb  5 14:57:12 2018] RDX: 0000000000e13e70 RSI: 00007ffffff09d50 RDI: 0000150381840df5
[Mon Feb  5 14:57:12 2018] RBP: 00007ffffff09de0 R08: 00007ffffff09d70 R09: 00007ffffff09da0
[Mon Feb  5 14:57:12 2018] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
[Mon Feb  5 14:57:12 2018] R13: 0000150380e83a98 R14: 0000000000000008 R15: 0000000000000004
[Mon Feb  5 14:57:12 2018] php: page allocation stalls for 42431ms, order:1, mode:0x17080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO|__GFP_NOTRACK), nodemask=(null)
[Mon Feb  5 14:57:12 2018] php cpuset=/ mems_allowed=0
[Mon Feb  5 14:57:12 2018] CPU: 0 PID: 18482 Comm: php Not tainted 4.14.13-unRAID #1
[Mon Feb  5 14:57:12 2018] Hardware name: Gigabyte Technology Co., Ltd. B150M-DS3H/B150M-DS3H-CF, BIOS F20 11/04/2016
[Mon Feb  5 14:57:12 2018] Call Trace:
[Mon Feb  5 14:57:12 2018]  dump_stack+0x5d/0x79
[Mon Feb  5 14:57:12 2018]  warn_alloc+0xdf/0x160
[Mon Feb  5 14:57:12 2018]  ? wakeup_kswapd+0x2c/0xb2
[Mon Feb  5 14:57:12 2018]  __alloc_pages_nodemask+0x578/0xb03
[Mon Feb  5 14:57:12 2018]  ? sched_ttwu_pending+0x75/0x75
[Mon Feb  5 14:57:12 2018]  ? get_mem_cgroup_from_mm+0x82/0x88
[Mon Feb  5 14:57:12 2018]  __get_free_pages+0x5/0x32
[Mon Feb  5 14:57:12 2018]  pgd_alloc+0x14/0xf5
[Mon Feb  5 14:57:12 2018]  mm_init+0x168/0x213
[Mon Feb  5 14:57:12 2018]  do_execveat_common.isra.14+0x230/0x5c8
[Mon Feb  5 14:57:12 2018]  do_execve+0x1a/0x1c
[Mon Feb  5 14:57:12 2018]  SyS_execve+0x23/0x2a
[Mon Feb  5 14:57:12 2018]  do_syscall_64+0x5b/0xf8
[Mon Feb  5 14:57:12 2018]  entry_SYSCALL64_slow_path+0x25/0x25
[Mon Feb  5 14:57:12 2018] RIP: 0033:0x15038177a697
[Mon Feb  5 14:57:12 2018] RSP: 002b:00007ffffff09d48 EFLAGS: 00000246 ORIG_RAX: 000000000000003b
[Mon Feb  5 14:57:12 2018] RAX: ffffffffffffffda RBX: 00007ffffff09d80 RCX: 000015038177a697
[Mon Feb  5 14:57:12 2018] RDX: 0000000000e13e70 RSI: 00007ffffff09d50 RDI: 0000150381840df5
[Mon Feb  5 14:57:12 2018] RBP: 00007ffffff09de0 R08: 00007ffffff09d70 R09: 00007ffffff09da0
[Mon Feb  5 14:57:12 2018] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
[Mon Feb  5 14:57:12 2018] R13: 0000150380e83a98 R14: 0000000000000008 R15: 0000000000000004

 

Edited by BBoYTuRBo

Share this post


Link to post

I did have the settings as 

 

Disk Cache 'vm.dirty_background_ratio' (%):    1%

Disk Cache 'vm.dirty_ratio' (%):                            2%

 

But I'll try changing them to 2% and 4% and see what happens.

Share this post


Link to post
Quote
 

I tried changing from 1% and 2% to 2% and 4%, and it didn't seem to have any effect. Server still locks up after about a day.

 

 

Quote

Maybe you have the same problem as described here?

I don't believe so. Memory usage for shfs is just showing around 1.1% (of 8GB)

Edited by BBoYTuRBo

Share this post


Link to post

You probably should post your diagnostics if you want more help.  That might give the experts more help diagnosing the problem.  But in the meantime here are some questions that may help them too: What amount of memory do you have installed in your computer?  Do you have VMs running?  If so what do you have assigned to the VM for memory and CPUs?  What dockers do you have installed?

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.