acbaldwi Posted September 21, 2022 Share Posted September 21, 2022 Hey Guys, My unraid seems to like to crash for seemingly no reason lately could someone please take a look at the attached logs/diags. i suspect it has to do with my plex docker as it seems to happen when using plex of course. and there seem to be a few errors in that log. basically the plex goes unresponsive, then i try to get to unraid, it is pingable but the web gui wont respond. putty does and a few moments later the web gui does. i suspect it's rebooting at that point as the logs are "cleared" but i have them being sent to a log server so there ya go. Thanks in advance for any help I really appreciate it. argos-diagnostics-20220920-2121.zip Plex Media Server.1.log syslog-192.168.1.12.log Quote Link to comment
JorgeB Posted September 21, 2022 Share Posted September 21, 2022 Do you know the time it last crashed? Not seeing anything relevant in the syslog. Quote Link to comment
acbaldwi Posted September 21, 2022 Author Share Posted September 21, 2022 6 hours ago, JorgeB said: Do you know the time it last crashed? Not seeing anything relevant in the syslog. crash occured about 9:18pm, the plex errors are all over the place sadly, and you should see a entry "Sep 20 21:21:05 Argos rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="14171" x-info="https://www.rsyslog.com"] start" where my syslog started running again, it was down for just a moment or 2. It also kille dthe pre-clears i had running Quote Link to comment
JorgeB Posted September 21, 2022 Share Posted September 21, 2022 I see now, unfortunately there's nothing logged, could be this issue: Quote Link to comment
trurl Posted September 21, 2022 Share Posted September 21, 2022 Probably unrelated, but why do you have 128G docker.img? 20G is often more than enough. Have you had problems filling it? Quote Link to comment
acbaldwi Posted September 21, 2022 Author Share Posted September 21, 2022 29 minutes ago, trurl said: Probably unrelated, but why do you have 128G docker.img? 20G is often more than enough. Have you had problems filling it? I have crashplan pro backing up several tb of data and that in the past has filled it up, plus the plex library is rather large. Ive turned on debug logging for the plex docker in the hopes of catching something there rather disappointing it didnt catch the unraid drop, it's happened a couple times now in the lats couple days. Is there a way to turn on more logging? It's my understanding that a sql error on the plex db could cause the memory to ballon up and possibly cause a crash but i havent seen any evidence that is the case. TIA Quote Link to comment
trurl Posted September 21, 2022 Share Posted September 21, 2022 6 minutes ago, acbaldwi said: plex library is rather large That is in appdata, not in docker.img 7 minutes ago, acbaldwi said: crashplan pro backing up several tb of data and that in the past has filled it up The usual reason for filling docker.img is an application writing to a path that isn't mapped. Quote Link to comment
trurl Posted September 21, 2022 Share Posted September 21, 2022 Another possible way to misconfigure containers is having a host path that isn't actual storage. That could cause rootfs to fill, leaving no room for the OS to work. What do you get from command line with this? df -h / Quote Link to comment
acbaldwi Posted September 21, 2022 Author Share Posted September 21, 2022 1 hour ago, JorgeB said: I see now, unfortunately there's nothing logged, could be this issue: Thanks for that , could be the issue, ill start by pulling c-states in the bios i guess as that fixed one guys issue... not sure how the double post happened i had started writing that one up last night, then decided it belonged here lol never hit go on the other one Quote Link to comment
acbaldwi Posted September 21, 2022 Author Share Posted September 21, 2022 27 minutes ago, trurl said: Another possible way to misconfigure containers is having a host path that isn't actual storage. That could cause rootfs to fill, leaving no room for the OS to work. What do you get from command line with this? df -h / Here is what i get root@Argos:~# df -h Filesystem Size Used Avail Use% Mounted on rootfs 16G 1.8G 14G 12% / tmpfs 32M 968K 32M 3% /run /dev/sda1 15G 719M 14G 5% /boot overlay 16G 1.8G 14G 12% /lib/firmware overlay 16G 1.8G 14G 12% /lib/modules devtmpfs 8.0M 0 8.0M 0% /dev tmpfs 16G 0 16G 0% /dev/shm cgroup_root 8.0M 0 8.0M 0% /sys/fs/cgroup tmpfs 128M 364K 128M 1% /var/log tmpfs 1.0M 0 1.0M 0% /mnt/disks tmpfs 1.0M 0 1.0M 0% /mnt/remotes tmpfs 1.0M 0 1.0M 0% /mnt/rootshare /dev/md1 11T 9.4T 1.6T 87% /mnt/disk1 /dev/md2 11T 9.5T 1.6T 87% /mnt/disk2 /dev/md3 11T 9.6T 1.4T 88% /mnt/disk3 /dev/md4 11T 11T 747G 94% /mnt/disk4 /dev/mapper/md5 9.1T 8.0T 1.2T 88% /mnt/disk5 /dev/mapper/md6 7.3T 6.8T 577G 93% /mnt/disk6 /dev/mapper/md7 7.3T 6.5T 807G 90% /mnt/disk7 /dev/mapper/md8 7.3T 6.5T 833G 89% /mnt/disk8 /dev/nvme0n1p1 932G 408G 523G 44% /mnt/cache shfs 75T 67T 8.5T 89% /mnt/user0 shfs 75T 67T 8.5T 89% /mnt/user /dev/sdb1 1.9T 76G 1.8T 5% /mnt/disks/plexappdata /dev/loop2 128G 13G 110G 11% /var/lib/docker root@Argos:~# ^C root@Argos:~# df -h / Filesystem Size Used Avail Use% Mounted on rootfs 16G 1.8G 14G 12% / root@Argos:~# I'm not opposed to re-creating it smaller if you think it would help. I was also using some things like krusader etc in the past to xfer files and i think that was improperly configured causing ballooning Quote Link to comment
trurl Posted September 21, 2022 Share Posted September 21, 2022 That first line, rootfs, was what I was after and would have been all of the result from df -h / but you left off the / I wouldn't expect smaller docker.img to fix your crashing, just thought it might be an indication that you had misconfigured things. Filling rootfs would be a more likely cause of OS problems than filling docker.img. Neither rootfs usage or docker.img usage should be growing so that is something you can check. Quote Link to comment
acbaldwi Posted September 21, 2022 Author Share Posted September 21, 2022 30 minutes ago, trurl said: That first line, rootfs, was what I was after and would have been all of the result from df -h / but you left off the / I wouldn't expect smaller docker.img to fix your crashing, just thought it might be an indication that you had misconfigured things. Filling rootfs would be a more likely cause of OS problems than filling docker.img. Neither rootfs usage or docker.img usage should be growing so that is something you can check. the / is in the bottom of the snip i took both in case Quote Link to comment
acbaldwi Posted September 23, 2022 Author Share Posted September 23, 2022 Alrighty folks think im getting somewhere, havent been able to shutdown to try the cstates but i turned on tidar and let her tip and sure enough invoked a crash during transcoding. Take a look at the attached logs at about 230am this morning If im reading this right, im running out of memory at some point?? Sep 23 02:30:02 Argos kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=1d1bef63799784e241207b1cd9bd605b573eba57ea7bdba88f647df8ec4e4c63,mems_allowed=0,global_oom,task_memcg=/docker/1d1bef63799784e241207b1cd9bd605b573eba57ea7bdba88f647df8ec4e4c63,task=code42,pid=19715,uid=99 Sep 23 02:30:02 Argos kernel: Out of memory: Killed process 19715 (code42) total-vm:17031436kB, anon-rss:9276kB, file-rss:0kB, shmem-rss:10208kB, UID:99 pgtables:416kB oom_score_adj:200 Sep 23 02:34:23 Argos kernel: tdarr-ffmpeg invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 Sep 23 02:34:23 Argos kernel: CPU: 4 PID: 13431 Comm: tdarr-ffmpeg Tainted: P O 5.15.46-Unraid #1 Sep 23 02:34:23 Argos kernel: Hardware name: ASUS System Product Name/PRIME Z590-P WIFI, BIOS 0820 04/27/2021 Sep 23 02:34:23 Argos kernel: Call Trace: Sep 23 02:34:23 Argos kernel: <TASK> Sep 23 02:34:23 Argos kernel: dump_stack_lvl+0x46/0x5a Sep 23 02:34:23 Argos kernel: dump_header+0x4a/0x1f3 Sep 23 02:34:23 Argos kernel: oom_kill_process+0x80/0xfb Sep 23 02:34:23 Argos kernel: out_of_memory+0x3e3/0x416 Sep 23 02:34:23 Argos kernel: __alloc_pages_slowpath.constprop.0+0x6f5/0x86f Sep 23 02:34:23 Argos kernel: __alloc_pages+0x10b/0x1ac Sep 23 02:34:23 Argos kernel: pagecache_get_page+0x15b/0x1ff Sep 23 02:34:23 Argos kernel: filemap_fault+0x2c6/0x49c Sep 23 02:34:23 Argos kernel: __do_fault+0x4b/0x69 Sep 23 02:34:23 Argos kernel: __handle_mm_fault+0x98a/0xc5c Sep 23 02:34:23 Argos kernel: handle_mm_fault+0x11c/0x1e2 Sep 23 02:34:23 Argos kernel: do_user_addr_fault+0x342/0x50b Sep 23 02:34:23 Argos kernel: exc_page_fault+0xe2/0x101 Sep 23 02:34:23 Argos kernel: ? asm_exc_page_fault+0x8/0x30 Sep 23 02:34:23 Argos kernel: asm_exc_page_fault+0x1e/0x30 Sep 23 02:34:23 Argos kernel: RIP: 0033:0x55f22f66502b Sep 23 02:34:23 Argos kernel: Code: Unable to access opcode bytes at RIP 0x55f22f665001. Sep 23 02:34:23 Argos kernel: RSP: 002b:00007fff397d4328 EFLAGS: 00010283 Sep 23 02:34:23 Argos kernel: RAX: 0000000080000000 RBX: 0000000000000000 RCX: 00000000000001f4 Sep 23 02:34:23 Argos kernel: RDX: 00000000000003e8 RSI: 00000000000f4240 RDI: 000000000028eaa8 Sep 23 02:34:23 Argos kernel: RBP: 7fffffffffffffff R08: 0000000000000001 R09: 00000000000003e8 Sep 23 02:34:23 Argos kernel: R10: 8000000000000001 R11: 7fffffffffffffff R12: 000055f22fc84f18 Sep 23 02:34:23 Argos kernel: R13: 0000000000000001 R14: 000f424000000001 R15: 000055f2306cf880 Sep 23 02:34:23 Argos kernel: </TASK> Sep 23 02:34:23 Argos kernel: Mem-Info: Sep 23 02:34:23 Argos kernel: active_anon:308559 inactive_anon:6997233 isolated_anon:0 Sep 23 02:34:23 Argos kernel: active_file:635 inactive_file:20281 isolated_file:168 Sep 23 02:34:23 Argos kernel: unevictable:67 dirty:332 writeback:0 Sep 23 02:34:23 Argos kernel: slab_reclaimable:132310 slab_unreclaimable:99593 Sep 23 02:34:23 Argos kernel: mapped:249219 shmem:493046 pagetables:187104 bounce:0 Sep 23 02:34:23 Argos kernel: kernel_misc_reclaimable:0 Sep 23 02:34:23 Argos kernel: free:73049 free_pcp:4340 free_cma:0 Sep 23 02:34:23 Argos kernel: Node 0 active_anon:1234236kB inactive_anon:27988932kB active_file:2540kB inactive_file:81124kB unevictable:268kB isolated(anon):0kB isolated(file):672kB mapped:996876kB dirty:1328kB writeback:0kB shmem:1972184kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 534528kB writeback_tmp:0kB kernel_stack:76640kB pagetables:748416kB all_unreclaimable? no Sep 23 02:34:23 Argos kernel: Node 0 DMA free:15360kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Sep 23 02:34:23 Argos kernel: lowmem_reserve[]: 0 2088 31727 31727 Sep 23 02:34:23 Argos kernel: Node 0 DMA32 free:122748kB min:4444kB low:6580kB high:8716kB reserved_highatomic:0KB active_anon:2920kB inactive_anon:1873964kB active_file:416kB inactive_file:452kB unevictable:0kB writepending:44kB present:2369888kB managed:2280484kB mlocked:0kB bounce:0kB free_pcp:7628kB local_pcp:124kB free_cma:0kB Sep 23 02:34:23 Argos kernel: lowmem_reserve[]: 0 0 29638 29638 Sep 23 02:34:23 Argos kernel: Node 0 Normal free:154088kB min:248804kB low:279152kB high:309500kB reserved_highatomic:0KB active_anon:1231316kB inactive_anon:26114936kB active_file:1784kB inactive_file:81724kB unevictable:268kB writepending:1284kB present:30924800kB managed:30349948kB mlocked:160kB bounce:0kB free_pcp:9760kB local_pcp:720kB free_cma:0kB Sep 23 02:34:23 Argos kernel: lowmem_reserve[]: 0 0 0 0 Sep 23 02:34:23 Argos kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15360kB Sep 23 02:34:23 Argos kernel: Node 0 DMA32: 43*4kB (UME) 196*8kB (UME) 330*16kB (UME) 1205*32kB (UME) 590*64kB (UME) 208*128kB (UME) 44*256kB (UME) 3*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 122764kB Sep 23 02:34:23 Argos kernel: Node 0 Normal: 2370*4kB (UME) 1943*8kB (UME) 4472*16kB (UME) 1842*32kB (UE) 1*64kB (E) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 155584kB Sep 23 02:34:23 Argos kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Sep 23 02:34:23 Argos kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Sep 23 02:34:23 Argos kernel: 514556 total pagecache pages Sep 23 02:34:23 Argos kernel: 0 pages in swap cache Sep 23 02:34:23 Argos kernel: Swap cache stats: add 0, delete 0, find 0/0 Sep 23 02:34:23 Argos kernel: Free swap = 0kB Sep 23 02:34:23 Argos kernel: Total swap = 0kB Sep 23 02:34:23 Argos kernel: 8327670 pages RAM Sep 23 02:34:23 Argos kernel: 0 pages HighMem/MovableOnly Sep 23 02:34:23 Argos kernel: 166222 pages reserved Sep 23 02:34:23 Argos kernel: 0 pages cma reserved syslog-192.168.1.12.log Quote Link to comment
trurl Posted September 23, 2022 Share Posted September 23, 2022 3 minutes ago, acbaldwi said: transcoding Where do transcodes go? Quote Link to comment
acbaldwi Posted September 23, 2022 Author Share Posted September 23, 2022 17 minutes ago, trurl said: Where do transcodes go? they are currently going to /temp for tidarr and /transcodes for plex /temp is on my array cache not enabled a and /transcodes got to a 1tb nvme cache Quote Link to comment
JonathanM Posted September 23, 2022 Share Posted September 23, 2022 42 minutes ago, acbaldwi said: /temp is on my array cache not enabled a and /transcodes got to a 1tb nvme cache Are you sure the mappings are correct for those locations? Quote Link to comment
Solution acbaldwi Posted September 26, 2022 Author Solution Share Posted September 26, 2022 Looks like my cpu was causing issues, swapped it with another and nary a crash since 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.