bumblebee21 Posted September 5, 2017 Share Posted September 5, 2017 (edited) Background unRAID version: 6.3.5 Plugins: Community Applications, CA Backup, CA Cleanup Appdata, Turbo Write, CA Auto Update, Dynamic Cache Dirs, File Integrity, SSD TRIM, System Buttons, System Info, Fix Common Problems, Tips and Tweaks, Unassigned Devices, Dockers: Plex Media Server, jackett, Sickrage, Transmission Hardware: i5-3470s, Gigabyte GA-B75m-D3H, 16gb RAM, 1 x 240gb SSD cache drive, 5 x 1TB data drives VMs: None Problem System has been hanging regularly (every other day or so) for past ~1-2 weeks. By hang, I mean unresponsive—cannot telnet, no dockers, no network shares, etc., but the system is still on. Usually happens in early morning. Finally captured logs and diagnostics (attached). Need help interpreting the logs. FCPsyslog_tail.txt tower-diagnostics-20170905-0419.zip Edited September 5, 2017 by bumblebee21 Edit to add: no VMs Quote Link to comment
bumblebee21 Posted September 6, 2017 Author Share Posted September 6, 2017 Bump. Any ideas? Quote Link to comment
1812 Posted September 6, 2017 Share Posted September 6, 2017 multiple errors and call traces pertaining to page allocation stalls. many attributed to plex. 5 03:43:32 Tower kernel: 3760657 total pagecache pages Sep 5 03:43:32 Tower kernel: 0 pages in swap cache Sep 5 03:43:32 Tower kernel: Swap cache stats: add 0, delete 0, find 0/0 Sep 5 03:43:32 Tower kernel: Free swap = 0kB Sep 5 03:43:32 Tower kernel: Total swap = 0kB Sep 5 03:43:32 Tower kernel: 4162352 pages RAM Sep 5 03:43:32 Tower kernel: 0 pages HighMem/MovableOnly Sep 5 03:43:32 Tower kernel: 68230 pages reserved Sep 5 03:43:32 Tower kernel: Plex Script Hos: page allocation stalls for 51032ms, order:0, mode:0x2400840(GFP_NOFS|__GFP_NOFAIL) Sep 5 03:43:32 Tower kernel: CPU: 1 PID: 9774 Comm: Plex Script Hos Not tainted 4.9.30-unRAID #1 Sep 5 03:43:32 Tower kernel: Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./B75M-D3H, BIOS F15 10/23/2013 Sep 5 03:43:32 Tower kernel: ffffc900032cf5d0 ffffffff813a4a1b 0000000000000001 0000000000000000 Sep 5 03:43:32 Tower kernel: ffffc900032cf660 ffffffff810cb5b1 024008401e5eb700 ffffffff8193d4e2 Sep 5 03:43:32 Tower kernel: ffffc900032cf5f8 0000000000000010 ffffc900032cf670 ffffc900032cf610 Sep 5 03:43:32 Tower kernel: Call Trace: Sep 5 03:43:32 Tower kernel: [<ffffffff813a4a1b>] dump_stack+0x61/0x7e Sep 5 03:43:32 Tower kernel: [<ffffffff810cb5b1>] warn_alloc+0x102/0x116 Sep 5 03:43:32 Tower kernel: [<ffffffff810d7980>] ? try_to_free_pages+0x9e/0xa5 Sep 5 03:43:32 Tower kernel: [<ffffffff810cbb67>] __alloc_pages_nodemask+0x541/0xc71 Sep 5 03:43:32 Tower kernel: [<ffffffff8106f3a5>] ? __enqueue_entity+0x67/0x69 Sep 5 03:43:32 Tower kernel: [<ffffffff81102d82>] alloc_pages_current+0xbe/0xe8 Sep 5 03:43:32 Tower kernel: [<ffffffff810c4d78>] __page_cache_alloc+0x89/0x9f Sep 5 03:43:32 Tower kernel: [<ffffffff810c4ecc>] pagecache_get_page+0x13e/0x1e6 Sep 5 03:43:32 Tower kernel: [<ffffffff8130e678>] alloc_extent_buffer+0xf7/0x375 Sep 5 03:43:32 Tower kernel: [<ffffffff812e8e67>] btrfs_find_create_tree_block+0x10/0x12 Sep 5 03:43:32 Tower kernel: [<ffffffff812e8fbc>] read_tree_block+0x14/0x4c Sep 5 03:43:32 Tower kernel: [<ffffffff812ce3b8>] read_block_for_search.isra.12+0x25b/0x296 Sep 5 03:43:32 Tower kernel: [<ffffffff813232ba>] ? btrfs_clear_lock_blocking_rw+0x79/0xc1 Sep 5 03:43:32 Tower kernel: [<ffffffff812d0320>] btrfs_search_slot+0x710/0x803 Sep 5 03:43:32 Tower kernel: [<ffffffff812e475a>] btrfs_lookup_csum+0x3a/0x108 Sep 5 03:43:32 Tower kernel: [<ffffffff812e4a51>] __btrfs_lookup_bio_sums+0x21a/0x451 Sep 5 03:43:32 Tower kernel: [<ffffffff812e4f74>] btrfs_lookup_bio_sums+0x11/0x13 Sep 5 03:43:32 Tower kernel: [<ffffffff812f26cc>] btrfs_submit_bio_hook+0xcc/0x145 Sep 5 03:43:32 Tower kernel: [<ffffffff8130916a>] submit_one_bio+0x66/0x84 Sep 5 03:43:32 Tower kernel: [<ffffffff8130e0af>] extent_readpages+0x1ce/0x1ec Sep 5 03:43:32 Tower kernel: [<ffffffff812f2e9a>] ? inode_tree_add+0x140/0x140 Sep 5 03:43:32 Tower kernel: [<ffffffff812f2c19>] btrfs_readpages+0x1a/0x1c Sep 5 03:43:32 Tower kernel: [<ffffffff810d0cc5>] __do_page_cache_readahead+0x15d/0x21f Sep 5 03:43:32 Tower kernel: [<ffffffff810c68b8>] filemap_fault+0x184/0x458 Sep 5 03:43:32 Tower kernel: [<ffffffff810c68b8>] ? filemap_fault+0x184/0x458 Sep 5 03:43:32 Tower kernel: [<ffffffff810e8f38>] __do_fault+0x68/0xbb Sep 5 03:43:32 Tower kernel: [<ffffffff810edf55>] handle_mm_fault+0x6b1/0xf96 Sep 5 03:43:32 Tower kernel: [<ffffffff81042252>] __do_page_fault+0x24a/0x3ed Sep 5 03:43:32 Tower kernel: [<ffffffff81042438>] do_page_fault+0x22/0x27 Sep 5 03:43:32 Tower kernel: [<ffffffff81680f18>] page_fault+0x28/0x30 Sep 5 03:43:32 Tower kernel: Mem-Info: Sep 5 03:43:32 Tower kernel: active_anon:275271 inactive_anon:11547 isolated_anon:0 Sep 5 03:43:32 Tower kernel: active_file:3277207 inactive_file:320960 isolated_file:640 Sep 5 03:43:32 Tower kernel: unevictable:0 dirty:318692 writeback:1284 unstable:0 Sep 5 03:43:32 Tower kernel: slab_reclaimable:92119 slab_unreclaimable:23546 Sep 5 03:43:32 Tower kernel: mapped:21506 shmem:161880 pagetables:3599 bounce:0 Sep 5 03:43:32 Tower kernel: free:50153 free_pcp:103 free_cma:0 Sep 5 03:43:32 Tower kernel: Node 0 active_anon:1101084kB inactive_anon:46188kB active_file:13108828kB inactive_file:1283840kB unevictable:0kB isolated(anon):0kB isolated(file):2560kB mapped:86024kB dirty:1274768kB writeback:5136kB shmem:647520kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 145408kB writeback_tmp:0kB unstable:0kB pages_scanned:5994981 all_unreclaimable? no Sep 5 03:43:32 Tower kernel: Node 0 DMA free:15900kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15984kB managed:15900kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Sep 5 03:43:32 Tower kernel: lowmem_reserve[]: 0 3252 15769 15769 Sep 5 03:43:32 Tower kernel: Node 0 DMA32 free:77764kB min:27852kB low:34812kB high:41772kB active_anon:88836kB inactive_anon:28kB active_file:3016672kB inactive_file:299756kB unevictable:0kB writepending:296716kB present:3552848kB managed:3542852kB mlocked:0kB slab_reclaimable:34880kB slab_unreclaimable:5888kB kernel_stack:176kB pagetables:428kB bounce:0kB free_pcp:252kB local_pcp:132kB free_cma:0kB Sep 5 03:43:32 Tower kernel: lowmem_reserve[]: 0 0 12516 12516 Sep 5 03:43:32 Tower kernel: Node 0 Normal free:106948kB min:107180kB low:133972kB high:160764kB active_anon:1012248kB inactive_anon:46160kB active_file:10092156kB inactive_file:983864kB unevictable:0kB writepending:983188kB present:13080576kB managed:12817736kB mlocked:0kB slab_reclaimable:333596kB slab_unreclaimable:88296kB kernel_stack:8816kB pagetables:13968kB bounce:0kB free_pcp:160kB local_pcp:16kB free_cma:0kB Sep 5 03:43:32 Tower kernel: lowmem_reserve[]: 0 0 0 0 Sep 5 03:43:32 Tower kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (U) 3*4096kB (M) = 15900kB Sep 5 03:43:32 Tower kernel: Node 0 DMA32: 94*4kB (UME) 630*8kB (UME) 484*16kB (UME) 157*32kB (UME) 100*64kB (UME) 9*128kB (UE) 2*256kB (ME) 1*512kB (U) 2*1024kB (UE) 0*2048kB 12*4096kB (M) = 77960kB Sep 5 03:43:32 Tower kernel: Node 0 Normal: 5540*4kB (UMEH) 5084*8kB (UMEH) 1947*16kB (UEH) 170*32kB (UMH) 44*64kB (H) 29*128kB (H) 3*256kB (H) 1*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 107232kB and after lots of that, more page allocation stalls for different services. and then this Sep 5 04:21:24 Tower kernel: Out of memory: Kill process 5400 (mono) score 2 or sacrifice child I'm not an expert on these, but perhaps you're running out of ram. Maybe plex is transcoding to ram and filling it up, causing the page stalls. Quote Link to comment
bumblebee21 Posted September 7, 2017 Author Share Posted September 7, 2017 Thanks for your reply. Interesting that it could be Plex. Just got an update for the PMS docker, so I'll give that a shot. Quote Link to comment
bumblebee21 Posted September 7, 2017 Author Share Posted September 7, 2017 Also, found this after searching reading through a few dozen pages of the PMS docker thread. Will give this a shot, as well. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.