System Hangs -- Logs Attached


Recommended Posts

Background

  • unRAID version: 6.3.5
  • Plugins: Community Applications, CA Backup, CA Cleanup Appdata, Turbo Write, CA Auto Update, Dynamic Cache Dirs, File Integrity, SSD TRIM, System Buttons, System Info, Fix Common Problems, Tips and Tweaks, Unassigned Devices, 
  • Dockers: Plex Media Server, jackett, Sickrage, Transmission 
  • Hardware: i5-3470s, Gigabyte GA-B75m-D3H, 16gb RAM, 1 x 240gb SSD cache drive, 5 x 1TB data drives
  • VMs: None

 

Problem
System has been hanging regularly (every other day or so) for past ~1-2 weeks. By hang, I mean unresponsive—cannot telnet, no dockers, no network shares, etc., but the system is still on. Usually happens in early morning. Finally captured logs and diagnostics (attached). Need help interpreting the logs.
 

FCPsyslog_tail.txt

tower-diagnostics-20170905-0419.zip

Edited by bumblebee21
Edit to add: no VMs
Link to comment

multiple errors and call traces pertaining to page allocation stalls. many attributed to plex.

 

5 03:43:32 Tower kernel: 3760657 total pagecache pages
Sep  5 03:43:32 Tower kernel: 0 pages in swap cache
Sep  5 03:43:32 Tower kernel: Swap cache stats: add 0, delete 0, find 0/0
Sep  5 03:43:32 Tower kernel: Free swap  = 0kB
Sep  5 03:43:32 Tower kernel: Total swap = 0kB
Sep  5 03:43:32 Tower kernel: 4162352 pages RAM
Sep  5 03:43:32 Tower kernel: 0 pages HighMem/MovableOnly
Sep  5 03:43:32 Tower kernel: 68230 pages reserved
Sep  5 03:43:32 Tower kernel: Plex Script Hos: page allocation stalls for 51032ms, order:0, mode:0x2400840(GFP_NOFS|__GFP_NOFAIL)
Sep  5 03:43:32 Tower kernel: CPU: 1 PID: 9774 Comm: Plex Script Hos Not tainted 4.9.30-unRAID #1
Sep  5 03:43:32 Tower kernel: Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./B75M-D3H, BIOS F15 10/23/2013
Sep  5 03:43:32 Tower kernel: ffffc900032cf5d0 ffffffff813a4a1b 0000000000000001 0000000000000000
Sep  5 03:43:32 Tower kernel: ffffc900032cf660 ffffffff810cb5b1 024008401e5eb700 ffffffff8193d4e2
Sep  5 03:43:32 Tower kernel: ffffc900032cf5f8 0000000000000010 ffffc900032cf670 ffffc900032cf610
Sep  5 03:43:32 Tower kernel: Call Trace:
Sep  5 03:43:32 Tower kernel: [<ffffffff813a4a1b>] dump_stack+0x61/0x7e
Sep  5 03:43:32 Tower kernel: [<ffffffff810cb5b1>] warn_alloc+0x102/0x116
Sep  5 03:43:32 Tower kernel: [<ffffffff810d7980>] ? try_to_free_pages+0x9e/0xa5
Sep  5 03:43:32 Tower kernel: [<ffffffff810cbb67>] __alloc_pages_nodemask+0x541/0xc71
Sep  5 03:43:32 Tower kernel: [<ffffffff8106f3a5>] ? __enqueue_entity+0x67/0x69
Sep  5 03:43:32 Tower kernel: [<ffffffff81102d82>] alloc_pages_current+0xbe/0xe8
Sep  5 03:43:32 Tower kernel: [<ffffffff810c4d78>] __page_cache_alloc+0x89/0x9f
Sep  5 03:43:32 Tower kernel: [<ffffffff810c4ecc>] pagecache_get_page+0x13e/0x1e6
Sep  5 03:43:32 Tower kernel: [<ffffffff8130e678>] alloc_extent_buffer+0xf7/0x375
Sep  5 03:43:32 Tower kernel: [<ffffffff812e8e67>] btrfs_find_create_tree_block+0x10/0x12
Sep  5 03:43:32 Tower kernel: [<ffffffff812e8fbc>] read_tree_block+0x14/0x4c
Sep  5 03:43:32 Tower kernel: [<ffffffff812ce3b8>] read_block_for_search.isra.12+0x25b/0x296
Sep  5 03:43:32 Tower kernel: [<ffffffff813232ba>] ? btrfs_clear_lock_blocking_rw+0x79/0xc1
Sep  5 03:43:32 Tower kernel: [<ffffffff812d0320>] btrfs_search_slot+0x710/0x803
Sep  5 03:43:32 Tower kernel: [<ffffffff812e475a>] btrfs_lookup_csum+0x3a/0x108
Sep  5 03:43:32 Tower kernel: [<ffffffff812e4a51>] __btrfs_lookup_bio_sums+0x21a/0x451
Sep  5 03:43:32 Tower kernel: [<ffffffff812e4f74>] btrfs_lookup_bio_sums+0x11/0x13
Sep  5 03:43:32 Tower kernel: [<ffffffff812f26cc>] btrfs_submit_bio_hook+0xcc/0x145
Sep  5 03:43:32 Tower kernel: [<ffffffff8130916a>] submit_one_bio+0x66/0x84
Sep  5 03:43:32 Tower kernel: [<ffffffff8130e0af>] extent_readpages+0x1ce/0x1ec
Sep  5 03:43:32 Tower kernel: [<ffffffff812f2e9a>] ? inode_tree_add+0x140/0x140
Sep  5 03:43:32 Tower kernel: [<ffffffff812f2c19>] btrfs_readpages+0x1a/0x1c
Sep  5 03:43:32 Tower kernel: [<ffffffff810d0cc5>] __do_page_cache_readahead+0x15d/0x21f
Sep  5 03:43:32 Tower kernel: [<ffffffff810c68b8>] filemap_fault+0x184/0x458
Sep  5 03:43:32 Tower kernel: [<ffffffff810c68b8>] ? filemap_fault+0x184/0x458
Sep  5 03:43:32 Tower kernel: [<ffffffff810e8f38>] __do_fault+0x68/0xbb
Sep  5 03:43:32 Tower kernel: [<ffffffff810edf55>] handle_mm_fault+0x6b1/0xf96
Sep  5 03:43:32 Tower kernel: [<ffffffff81042252>] __do_page_fault+0x24a/0x3ed
Sep  5 03:43:32 Tower kernel: [<ffffffff81042438>] do_page_fault+0x22/0x27
Sep  5 03:43:32 Tower kernel: [<ffffffff81680f18>] page_fault+0x28/0x30
Sep  5 03:43:32 Tower kernel: Mem-Info:
Sep  5 03:43:32 Tower kernel: active_anon:275271 inactive_anon:11547 isolated_anon:0
Sep  5 03:43:32 Tower kernel: active_file:3277207 inactive_file:320960 isolated_file:640
Sep  5 03:43:32 Tower kernel: unevictable:0 dirty:318692 writeback:1284 unstable:0
Sep  5 03:43:32 Tower kernel: slab_reclaimable:92119 slab_unreclaimable:23546
Sep  5 03:43:32 Tower kernel: mapped:21506 shmem:161880 pagetables:3599 bounce:0
Sep  5 03:43:32 Tower kernel: free:50153 free_pcp:103 free_cma:0
Sep  5 03:43:32 Tower kernel: Node 0 active_anon:1101084kB inactive_anon:46188kB active_file:13108828kB inactive_file:1283840kB unevictable:0kB isolated(anon):0kB isolated(file):2560kB mapped:86024kB dirty:1274768kB writeback:5136kB shmem:647520kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 145408kB writeback_tmp:0kB unstable:0kB pages_scanned:5994981 all_unreclaimable? no
Sep  5 03:43:32 Tower kernel: Node 0 DMA free:15900kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15984kB managed:15900kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep  5 03:43:32 Tower kernel: lowmem_reserve[]: 0 3252 15769 15769
Sep  5 03:43:32 Tower kernel: Node 0 DMA32 free:77764kB min:27852kB low:34812kB high:41772kB active_anon:88836kB inactive_anon:28kB active_file:3016672kB inactive_file:299756kB unevictable:0kB writepending:296716kB present:3552848kB managed:3542852kB mlocked:0kB slab_reclaimable:34880kB slab_unreclaimable:5888kB kernel_stack:176kB pagetables:428kB bounce:0kB free_pcp:252kB local_pcp:132kB free_cma:0kB
Sep  5 03:43:32 Tower kernel: lowmem_reserve[]: 0 0 12516 12516
Sep  5 03:43:32 Tower kernel: Node 0 Normal free:106948kB min:107180kB low:133972kB high:160764kB active_anon:1012248kB inactive_anon:46160kB active_file:10092156kB inactive_file:983864kB unevictable:0kB writepending:983188kB present:13080576kB managed:12817736kB mlocked:0kB slab_reclaimable:333596kB slab_unreclaimable:88296kB kernel_stack:8816kB pagetables:13968kB bounce:0kB free_pcp:160kB local_pcp:16kB free_cma:0kB
Sep  5 03:43:32 Tower kernel: lowmem_reserve[]: 0 0 0 0
Sep  5 03:43:32 Tower kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (U) 3*4096kB (M) = 15900kB
Sep  5 03:43:32 Tower kernel: Node 0 DMA32: 94*4kB (UME) 630*8kB (UME) 484*16kB (UME) 157*32kB (UME) 100*64kB (UME) 9*128kB (UE) 2*256kB (ME) 1*512kB (U) 2*1024kB (UE) 0*2048kB 12*4096kB (M) = 77960kB
Sep  5 03:43:32 Tower kernel: Node 0 Normal: 5540*4kB (UMEH) 5084*8kB (UMEH) 1947*16kB (UEH) 170*32kB (UMH) 44*64kB (H) 29*128kB (H) 3*256kB (H) 1*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 107232kB

 

and after lots of  that, more page allocation stalls for different services.

 

and then this

Sep  5 04:21:24 Tower kernel: Out of memory: Kill process 5400 (mono) score 2 or sacrifice child

 

 

I'm not an expert on these, but perhaps you're running out of ram. Maybe plex is transcoding to ram and filling it up, causing the page stalls.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.