unraid crashing might be smethign to do with plex docker


Go to solution Solved by acbaldwi,

Recommended Posts

Hey Guys,

 

My unraid seems to like to crash for seemingly no reason lately could someone please take a look at the attached logs/diags. i suspect it has to do with my plex docker as it seems to happen when using plex of course. and there seem to be a few errors in that log.

 

basically the plex goes unresponsive, then i try to get to unraid, it is pingable but the web gui wont respond. putty does and a few moments later the web gui does. i suspect it's rebooting at that point as the logs are "cleared"  but i have them being sent to a log server so there ya go.

 

Thanks in advance for any help I really appreciate it.

 

 

argos-diagnostics-20220920-2121.zip Plex Media Server.1.log syslog-192.168.1.12.log

Link to comment
6 hours ago, JorgeB said:

Do you know the time it last crashed? Not seeing anything relevant in the syslog.

crash occured about 9:18pm, the plex errors are all over the place sadly, and you should see a entry "Sep 20 21:21:05 Argos rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="14171" x-info="https://www.rsyslog.com"] start" where my syslog started running again, it was down for just a moment or 2. It also kille dthe pre-clears i had running  

Link to comment
29 minutes ago, trurl said:

Probably unrelated, but why do you have 128G docker.img? 20G is often more than enough. Have you had problems filling it?

 

I have crashplan pro backing up several tb of data and that in the past has filled it up, plus the plex library is rather large. Ive turned on debug logging for the plex docker in the hopes of catching something there rather disappointing it didnt catch the unraid drop, it's happened a couple times now in the lats couple days. Is there a way to turn on more logging? 

 

It's my understanding that a sql error on the plex db could cause the memory to ballon up and possibly  cause a crash but i havent seen any evidence that is the case. 

 

TIA

Link to comment
6 minutes ago, acbaldwi said:

plex library is rather large

That is in appdata, not in docker.img

 

7 minutes ago, acbaldwi said:

crashplan pro backing up several tb of data and that in the past has filled it up

The usual reason for filling docker.img is an application writing to a path that isn't mapped.

Link to comment
1 hour ago, JorgeB said:

I see now, unfortunately there's nothing logged, could be this issue:

 

 

 

 

Thanks for that , could be the issue, ill start by pulling c-states in the bios i guess as that fixed one guys issue...

 

not sure how the double post happened i had started writing that one up last night, then decided it belonged here lol never hit go on the other one :)

Link to comment
27 minutes ago, trurl said:

Another possible way to misconfigure containers is having a host path that isn't actual storage. That could cause rootfs to fill, leaving no room for the OS to work.

 

What do you get from command line with this?

df -h /

 

 

 

Here is what i get

 

root@Argos:~# df -h
Filesystem       Size  Used Avail Use% Mounted on
rootfs            16G  1.8G   14G  12% /
tmpfs             32M  968K   32M   3% /run
/dev/sda1         15G  719M   14G   5% /boot
overlay           16G  1.8G   14G  12% /lib/firmware
overlay           16G  1.8G   14G  12% /lib/modules
devtmpfs         8.0M     0  8.0M   0% /dev
tmpfs             16G     0   16G   0% /dev/shm
cgroup_root      8.0M     0  8.0M   0% /sys/fs/cgroup
tmpfs            128M  364K  128M   1% /var/log
tmpfs            1.0M     0  1.0M   0% /mnt/disks
tmpfs            1.0M     0  1.0M   0% /mnt/remotes
tmpfs            1.0M     0  1.0M   0% /mnt/rootshare
/dev/md1          11T  9.4T  1.6T  87% /mnt/disk1
/dev/md2          11T  9.5T  1.6T  87% /mnt/disk2
/dev/md3          11T  9.6T  1.4T  88% /mnt/disk3
/dev/md4          11T   11T  747G  94% /mnt/disk4
/dev/mapper/md5  9.1T  8.0T  1.2T  88% /mnt/disk5
/dev/mapper/md6  7.3T  6.8T  577G  93% /mnt/disk6
/dev/mapper/md7  7.3T  6.5T  807G  90% /mnt/disk7
/dev/mapper/md8  7.3T  6.5T  833G  89% /mnt/disk8
/dev/nvme0n1p1   932G  408G  523G  44% /mnt/cache
shfs              75T   67T  8.5T  89% /mnt/user0
shfs              75T   67T  8.5T  89% /mnt/user
/dev/sdb1        1.9T   76G  1.8T   5% /mnt/disks/plexappdata
/dev/loop2       128G   13G  110G  11% /var/lib/docker
root@Argos:~# ^C
root@Argos:~# df -h /
Filesystem      Size  Used Avail Use% Mounted on
rootfs           16G  1.8G   14G  12% /
root@Argos:~# 

 

I'm not opposed to re-creating it smaller if you think it would help. I was also using some things like krusader etc in the past to xfer files and i think that was improperly configured causing ballooning

 

Link to comment

That first line, rootfs, was what I was after and would have been all of the result from

df -h /

but you left off the /

 

I wouldn't expect smaller docker.img to fix your crashing, just thought it might be an indication that you had misconfigured things. Filling rootfs would be a more likely cause of OS problems than filling docker.img. Neither rootfs usage or docker.img usage should be growing so that is something you can check.

 

 

Link to comment
30 minutes ago, trurl said:

That first line, rootfs, was what I was after and would have been all of the result from

df -h /

but you left off the /

 

I wouldn't expect smaller docker.img to fix your crashing, just thought it might be an indication that you had misconfigured things. Filling rootfs would be a more likely cause of OS problems than filling docker.img. Neither rootfs usage or docker.img usage should be growing so that is something you can check.

 

 

 the / is in the bottom of the snip :) i took both in case 

Link to comment

Alrighty folks think im getting somewhere, havent been able to shutdown to try the cstates but i turned on tidar and let her tip and sure enough invoked a crash during transcoding. Take a look at the attached logs at about 230am this morning

 

If im reading this right, im running out of memory at some point?? 

 

 

Sep 23 02:30:02 Argos kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=1d1bef63799784e241207b1cd9bd605b573eba57ea7bdba88f647df8ec4e4c63,mems_allowed=0,global_oom,task_memcg=/docker/1d1bef63799784e241207b1cd9bd605b573eba57ea7bdba88f647df8ec4e4c63,task=code42,pid=19715,uid=99
Sep 23 02:30:02 Argos kernel: Out of memory: Killed process 19715 (code42) total-vm:17031436kB, anon-rss:9276kB, file-rss:0kB, shmem-rss:10208kB, UID:99 pgtables:416kB oom_score_adj:200
Sep 23 02:34:23 Argos kernel: tdarr-ffmpeg invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Sep 23 02:34:23 Argos kernel: CPU: 4 PID: 13431 Comm: tdarr-ffmpeg Tainted: P           O      5.15.46-Unraid #1
Sep 23 02:34:23 Argos kernel: Hardware name: ASUS System Product Name/PRIME Z590-P WIFI, BIOS 0820 04/27/2021
Sep 23 02:34:23 Argos kernel: Call Trace:
Sep 23 02:34:23 Argos kernel: <TASK>
Sep 23 02:34:23 Argos kernel: dump_stack_lvl+0x46/0x5a
Sep 23 02:34:23 Argos kernel: dump_header+0x4a/0x1f3
Sep 23 02:34:23 Argos kernel: oom_kill_process+0x80/0xfb
Sep 23 02:34:23 Argos kernel: out_of_memory+0x3e3/0x416
Sep 23 02:34:23 Argos kernel: __alloc_pages_slowpath.constprop.0+0x6f5/0x86f
Sep 23 02:34:23 Argos kernel: __alloc_pages+0x10b/0x1ac
Sep 23 02:34:23 Argos kernel: pagecache_get_page+0x15b/0x1ff
Sep 23 02:34:23 Argos kernel: filemap_fault+0x2c6/0x49c
Sep 23 02:34:23 Argos kernel: __do_fault+0x4b/0x69
Sep 23 02:34:23 Argos kernel: __handle_mm_fault+0x98a/0xc5c
Sep 23 02:34:23 Argos kernel: handle_mm_fault+0x11c/0x1e2
Sep 23 02:34:23 Argos kernel: do_user_addr_fault+0x342/0x50b
Sep 23 02:34:23 Argos kernel: exc_page_fault+0xe2/0x101
Sep 23 02:34:23 Argos kernel: ? asm_exc_page_fault+0x8/0x30
Sep 23 02:34:23 Argos kernel: asm_exc_page_fault+0x1e/0x30
Sep 23 02:34:23 Argos kernel: RIP: 0033:0x55f22f66502b
Sep 23 02:34:23 Argos kernel: Code: Unable to access opcode bytes at RIP 0x55f22f665001.
Sep 23 02:34:23 Argos kernel: RSP: 002b:00007fff397d4328 EFLAGS: 00010283
Sep 23 02:34:23 Argos kernel: RAX: 0000000080000000 RBX: 0000000000000000 RCX: 00000000000001f4
Sep 23 02:34:23 Argos kernel: RDX: 00000000000003e8 RSI: 00000000000f4240 RDI: 000000000028eaa8
Sep 23 02:34:23 Argos kernel: RBP: 7fffffffffffffff R08: 0000000000000001 R09: 00000000000003e8
Sep 23 02:34:23 Argos kernel: R10: 8000000000000001 R11: 7fffffffffffffff R12: 000055f22fc84f18
Sep 23 02:34:23 Argos kernel: R13: 0000000000000001 R14: 000f424000000001 R15: 000055f2306cf880
Sep 23 02:34:23 Argos kernel: </TASK>
Sep 23 02:34:23 Argos kernel: Mem-Info:
Sep 23 02:34:23 Argos kernel: active_anon:308559 inactive_anon:6997233 isolated_anon:0
Sep 23 02:34:23 Argos kernel: active_file:635 inactive_file:20281 isolated_file:168
Sep 23 02:34:23 Argos kernel: unevictable:67 dirty:332 writeback:0
Sep 23 02:34:23 Argos kernel: slab_reclaimable:132310 slab_unreclaimable:99593
Sep 23 02:34:23 Argos kernel: mapped:249219 shmem:493046 pagetables:187104 bounce:0
Sep 23 02:34:23 Argos kernel: kernel_misc_reclaimable:0
Sep 23 02:34:23 Argos kernel: free:73049 free_pcp:4340 free_cma:0
Sep 23 02:34:23 Argos kernel: Node 0 active_anon:1234236kB inactive_anon:27988932kB active_file:2540kB inactive_file:81124kB unevictable:268kB isolated(anon):0kB isolated(file):672kB mapped:996876kB dirty:1328kB writeback:0kB shmem:1972184kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 534528kB writeback_tmp:0kB kernel_stack:76640kB pagetables:748416kB all_unreclaimable? no
Sep 23 02:34:23 Argos kernel: Node 0 DMA free:15360kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep 23 02:34:23 Argos kernel: lowmem_reserve[]: 0 2088 31727 31727
Sep 23 02:34:23 Argos kernel: Node 0 DMA32 free:122748kB min:4444kB low:6580kB high:8716kB reserved_highatomic:0KB active_anon:2920kB inactive_anon:1873964kB active_file:416kB inactive_file:452kB unevictable:0kB writepending:44kB present:2369888kB managed:2280484kB mlocked:0kB bounce:0kB free_pcp:7628kB local_pcp:124kB free_cma:0kB
Sep 23 02:34:23 Argos kernel: lowmem_reserve[]: 0 0 29638 29638
Sep 23 02:34:23 Argos kernel: Node 0 Normal free:154088kB min:248804kB low:279152kB high:309500kB reserved_highatomic:0KB active_anon:1231316kB inactive_anon:26114936kB active_file:1784kB inactive_file:81724kB unevictable:268kB writepending:1284kB present:30924800kB managed:30349948kB mlocked:160kB bounce:0kB free_pcp:9760kB local_pcp:720kB free_cma:0kB
Sep 23 02:34:23 Argos kernel: lowmem_reserve[]: 0 0 0 0
Sep 23 02:34:23 Argos kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15360kB
Sep 23 02:34:23 Argos kernel: Node 0 DMA32: 43*4kB (UME) 196*8kB (UME) 330*16kB (UME) 1205*32kB (UME) 590*64kB (UME) 208*128kB (UME) 44*256kB (UME) 3*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 122764kB
Sep 23 02:34:23 Argos kernel: Node 0 Normal: 2370*4kB (UME) 1943*8kB (UME) 4472*16kB (UME) 1842*32kB (UE) 1*64kB (E) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 155584kB
Sep 23 02:34:23 Argos kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Sep 23 02:34:23 Argos kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 23 02:34:23 Argos kernel: 514556 total pagecache pages
Sep 23 02:34:23 Argos kernel: 0 pages in swap cache
Sep 23 02:34:23 Argos kernel: Swap cache stats: add 0, delete 0, find 0/0
Sep 23 02:34:23 Argos kernel: Free swap  = 0kB
Sep 23 02:34:23 Argos kernel: Total swap = 0kB
Sep 23 02:34:23 Argos kernel: 8327670 pages RAM
Sep 23 02:34:23 Argos kernel: 0 pages HighMem/MovableOnly
Sep 23 02:34:23 Argos kernel: 166222 pages reserved
Sep 23 02:34:23 Argos kernel: 0 pages cma reserved

 

syslog-192.168.1.12.log

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.