Hey guys,

 

So I woke up this morning to another "Call traces found on your server" email in my Gmail inbox. This has been happening for awhile now, and I'm starting to get concerned. Before, I assumed it was safe because my uptime was unusually high (28 days and counting up!) and I'd never reached this uptime before.

 

But the error seems bad so I decided to check around. I was on the rc7a beta, then realized the rc9q or whatever the latest version is was available and downloaded it. Before rebooting, I remembered you guys needed the diagnostics to check anything so I grabbed them before rebooting.

 

Can anybody check if the call traces are potentially harmful to my server or my data? Thanks.

 

Attached diags (before the reboot). This is AFTER I installed the rc9q beta so it may show that in the logs but I haven't rebooted so all the call traces should be on there.

derrickserver-diagnostics-20170916-0017.zip

 

---- EDIT ---

 

Oh crap, I rebooted the server and the WebGUI isn't showing anymore. I can still access the server via SSH, but the WebGUI shoves me out with a CONNECTION_REFUSED.

 

This isn't looking very good... I'm going to try rebooting the server via SSH to see if I can get it back up. Server is in a remote location so I can't easily access it.

 

--- EDIT again ---

 

No dice. The WebGUI will never show up. The SSH service still works though.

 

Any suggestions? Is this due to the beta update?

 

 

Edited September 15, 20178 yr by Guest

You have a billion of these errors

 

Server/Cache/PhotoTranscoder/34/34f1ef74375ef2f6fdf0239d76b7f7ebab17823e.jpg is corrupted
Sep  1 02:33:26 derrickserver bunker: error: BLAKE2 hash key mismatch, /mnt/disk2/appdata/plex/Library/Application Support/Plex Media Server/Cache/PhotoTranscoder/34/34eb920cc7dabc01a64992513496abf947b7fb7a.jpg is corrupted
Sep  1 02:33:26 derrickserver bunker: error: BLAKE2 hash key mismatch, /mnt/disk2/appdata/plex/Library/Application Support/Plex Media Server/Cache/PhotoTranscoder/34/34d2a77f394a85f4fea9ea97b0ef3dadc3d68c62.jpg is corrupted
Sep  1 02:33:26 derrickserver bunker: error: BLAKE2 hash key mismatch, /mnt/disk2/appdata/plex/Library/Application Support/Plex Media Server/Cache/PhotoTranscoder/34/34c6440d6be418c5c3fcc6e23937b478776e4152.jpg is corrupted
Sep  1 02:33:26 derrickserver bunker: error: BLAKE2 hash key mismatch, /mnt/disk2/appdata/plex/Library/Application Support/Plex Media Server/Cache/PhotoTranscoder/34/34474f5a7ce483f3e9d25266dbf5a9ba2e40b16c.jpg is corrupted
Sep  1 02:33:26 derrickserver bunker: error: BLAKE2 hash key mismatch, /mnt/disk2/appdata/plex/Library/Application Support/Plex Media Server/Cache/PhotoTranscoder/34/340224285e0ba0e7023dccf4d9714c44e7984ce1.jpg is corrupted
Sep  1 02:33:27 derrickserver bunker: error: BLAKE2 hash key mismatch, /mnt/disk2/appdata/plex/Library/Application Support/Plex Media Server/Cache/PhotoTranscoder/34/34c95166198cd9ac02aefa7f8a6f63785a57611d.jpg is corrupted
Sep  1 02:33:27 derrickserver bunker: error: BLAKE2 hash key mismatch, /mnt/disk2/appdata/plex/Library/Application Support/Plex Media Server/Cache/PhotoTranscoder/34/34611f3fe9cf9b9e24c37361f1f87ff0bdedd5bc.jpg is corrupted
Sep  1 02:33:27 derrickserver bunker: error: BLAKE2 hash key mismatch, /mnt/disk2/appdata/plex/Library/Application Support/Plex Media Server/Cache/PhotoTranscoder/34/34cfa035b900594d7a632ae41cf1b5a4390607a8.jpg is corrupted
Sep  1 02:33:27 derrickserver bunker: error: BLAKE2 hash key mismatch, /mnt/disk2/appdata/plex/Library/Application Support/Plex Media Server/Cache/PhotoTranscoder/34/344249c98776d2d9d27eb70349d7a141c738e16c.jpg is corrupted
Sep  1 02:33:27 derrickserver bunker: error: BLAKE2 hash key mismatch, /mnt/disk2/appdata/plex/Library/Application Support/Plex Media Server/Cache/PhotoTranscoder/34/34e3066dd68c61e5d7f8854bfe2f31308578472b.jpg is corrupted
Sep  1 02:33:27 derrickserver bunker: error: BLAKE2 hash key mismatch, /mnt/disk2/appdata/plex/Li

 

 

which means those files are probably corrupted, or at least reporting as such.

 

 

and I believe you are running out of memory (someone else will have to verify)

 

Sep  1 02:11:30 derrickserver kernel: lowmem_reserve[]: 0 0 4718 4718
Sep  1 02:11:30 derrickserver kernel: Node 0 Normal free:80812kB min:80868kB low:101084kB high:121300kB active_anon:2541236kB inactive_anon:144296kB active_file:491480kB inactive_file:1032244kB unevictable:0kB writepending:0kB present:4964348kB managed:4832200kB mlocked:0kB slab_reclaimable:340160kB slab_unreclaimable:101132kB kernel_stack:6688kB pagetables:17212kB bounce:0kB free_pcp:256kB local_pcp:0kB free_cma:0kB
Sep  1 02:11:30 derrickserver kernel: lowmem_reserve[]: 0 0 0 0
Sep  1 02:11:30 derrickserver kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (U) 3*4096kB (M) = 15904kB
Sep  1 02:11:30 derrickserver kernel: Node 0 DMA32: 954*4kB (UMEH) 1005*8kB (UMEH) 1132*16kB (UMEH) 644*32kB (UMEH) 220*64kB (UME) 56*128kB (UME) 1*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 72592kB
Sep  1 02:11:30 derrickserver kernel: Node 0 Normal: 127*4kB (UME) 1346*8kB (UME) 196*16kB (UM) 1609*32kB (UM) 236*64kB (MH) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 81004kB
Sep  1 02:11:30 derrickserver kernel: 1054030 total pagecache pages
Sep  1 02:11:30 derrickserver kernel: 0 pages in swap cache
Sep  1 02:11:30 derrickserver kernel: Swap cache stats: add 0, delete 0, find 0/0
Sep  1 02:11:30 derrickserver kernel: Free swap  = 0kB
Sep  1 02:11:30 derrickserver kernel: Total swap = 0kB
Sep  1 02:11:30 derrickserver kernel: 2090006 pages RAM
Sep  1 02:11:30 derrickserver kernel: 0 pages HighMem/MovableOnly
Sep  1 02:11:30 derrickserver kernel: 51693 pages reserved
Sep  1 02:11:30 derrickserver kernel: 0 pages cma reserved
Sep  1 02:13:05 derrickserver kernel: timeout: page allocation stalls for 14908ms, order:0, mode:0x15080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), nodemask=(null)
Sep  1 02:13:05 derrickserver kernel: timeout cpuset=/ mems_allowed=0
Sep  1 02:13:05 derrickserver kernel: CPU: 3 PID: 17388 Comm: timeout Not tainted 4.12.3-unRAID #1
Sep  1 02:13:05 derrickserver kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./A55M-VS, BIOS P1.60 10/02/2013
Sep  1 02:13:05 derrickserver kernel: Call Trace:
Sep  1 02:13:05 derrickserver kernel: dump_stack+0x61/0x7e
Sep  1 02:13:05 derrickserver kernel: warn_alloc+0xdf/0x158
Sep  1 02:13:05 derrickserver kernel: __alloc_pages_nodemask+0x56f/0xb38
Sep  1 02:13:05 derrickserver kernel: alloc_pages_current+0xb3/0xd4
Sep  1 02:13:05 derrickserver kernel: __get_free_pages+0x9/0x37
Sep  1 02:13:05 derrickserver kernel: get_zeroed_page+0x11/0x13
Sep  1 02:13:05 derrickserver kernel: __pud_alloc+0x2c/0xe7
Sep  1 02:13:05 derrickserver kernel: __handle_mm_fault+0xef/0xef3
Sep  1 02:13:05 derrickserver kernel: ? find_lock_entry+0x4a/0x7d
Sep  1 02:13:05 derrickserver kernel: handle_mm_fault+0x130/0x1ae
Sep  1 02:13:05 derrickserver kernel: __get_user_pages+0x373/0x537
Sep  1 02:13:05 derrickserver kernel: get_user_pages_remote+0x8c/0x191
Sep  1 02:13:05 derrickserver kernel: copy_strings.isra.8+0x141/0x294
Sep  1 02:13:05 derrickserver kernel: copy_strings_kernel+0x2f/0x3b
Sep  1 02:13:05 derrickserver kernel: do_execveat_common.isra.15+0x3b7/0x5bc
Sep  1 02:13:05 derrickserver kernel: do_execve+0x1e/0x20
Sep  1 02:13:05 derrickserver kernel: SyS_execve+0x25/0x29
Sep  1 02:13:05 derrickserver kernel: do_syscall_64+0x5e/0xb4
Sep  1 02:13:05 derrickserver kernel: entry_SYSCALL64_slow_path+0x25/0x25
Sep  1 02:13:05 derrickserver kernel: RIP: 0033:0x2b2e59a183f7
Sep  1 02:13:05 derrickserver kernel: RSP: 002b:00007ffeffad5028 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
Sep  1 02:13:05 derrickserver kernel: RAX: ffffffffffffffda RBX: 00007ffeffad5e97 RCX: 00002b2e59a183f7
Sep  1 02:13:05 derrickserver kernel: RDX: 00007ffeffad5370 RSI: 00007ffeffad5348 RDI: 00007ffeffad5e97
Sep  1 02:13:05 derrickserver kernel: RBP: 00007ffeffad50c0 R08: 00007ffeffad5020 R09: 00002b2e59942280
Sep  1 02:13:05 derrickserver kernel: R10: 0000000000000005 R11: 0000000000000202 R12: 00007ffeffad5348
Sep  1 02:13:05 derrickserver kernel: R13: 00007ffeffad5370 R14: 0000000000000009 R15: 0000000000000000
Sep  1 02:13:05 derrickserver kernel: Mem-Info:
Sep  1 02:13:05 derrickserver kernel: active_anon:948230 inactive_anon:36337 isolated_anon:0
Sep  1 02:13:05 derrickserver kernel: active_file:230399 inactive_file:631736 isolated_file:0
Sep  1 02:13:05 derrickserver kernel: unevictable:0 dirty:11 writeback:0 unstable:0
Sep  1 02:13:05 derrickserver kernel: slab_reclaimable:94030 slab_unreclaimable:29574
Sep  1 02:13:05 derrickserver kernel: mapped:27228 shmem:191600 pagetables:4708 bounce:0
Sep  1 02:13:05 derrickserver kernel: free:42349 free_pcp:61 free_cma:0
Sep  1 02:13:05 derrickserver kernel: Node 0 active_anon:3792920kB inactive_anon:145348kB active_file:921596kB inactive_file:2526944kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:108912kB dirty:44kB writeback:0kB shmem:766400kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 2910208kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Sep  1 02:13:05 derrickserver kernel: Node 0 DMA free:15904kB min:264kB low:328kB high:392kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep  1 02:13:05 derrickserver kernel: lowmem_reserve[]: 0 3152 7871 7871
Sep  1 02:13:05 derrickserver kernel: Node 0 DMA32 free:73120kB min:54028kB low:67532kB high:81036kB active_anon:1250436kB inactive_anon:1060kB active_file:429328kB inactive_file:1500344kB unevictable:0kB writepending:12kB present:3379688kB managed:3305148kB mlocked:0kB slab_reclaimable:29840kB slab_unreclaimable:17196kB kernel_stack:400kB pagetables:1348kB bounce:0kB free_pcp:240kB local_pcp:0kB free_cma:0kB

 

 

You also have some errors in you docker log.... and a partridge in a pear tree.

 

 

You probably shouldn't have updated to a non-stable version if this is your main server.

 

 

 

Have you tried booting in safemode or GUI mode?

 

 

 

 

 

 

 

7 hours ago, 1812 said:

You have a billion of these errors

 

(...)

 

which means those files are probably corrupted, or at least reporting as such.

 

and I believe you are running out of memory (someone else will have to verify)

 

(...)

 

You also have some errors in you docker log.... and a partridge in a pear tree.

 

You probably shouldn't have updated to a non-stable version if this is your main server.

 

Have you tried booting in safemode or GUI mode?

 

I’m not very worried about the file hash mismatches. As far as I know, these errors are because while the File Integrity plugin is generating a checksum Plex updates the contents and therefore the file hashes don’t match. I’m not sure if this is a race condition or something but I wouldn’t worry too much about it. 

 

Edit: Unfortunately I can’t access the server because it’s in a remote location but I’ll drive by on my way home and see what I can do. 

 

Specifically what Docker image are you talking about? I only have three dockers installed if I remember correctly.

 

I’m not really sure why it’s running out of memory because it has 8GB of RAM. Can somebody confirm the call traces are due to the memory?

Edited September 16, 20178 yr by Guest
Log too long

Shameful bump. I haven't had access to my data for three days now.

 

Unfortunately my porn fix is encrypted and stored on the server... Just kidding.

 

Any ideas?

30 minutes ago, ideaman924 said:

Shameful bump. I haven't had access to my data for three days now.

 

Unfortunately my porn fix is encrypted and stored on the server... Just kidding.

 

Any ideas?

 

You are going to have to put your hands on it and 

On 9/15/2017 at 8:03 PM, 1812 said:

Have you tried booting in safemode or GUI mode

 

6 hours ago, 1812 said:

 

You are going to have to put your hands on it and 

(...)

 

Okay. One more thing before I try driving there. I got an email this morning saying the array health report had passed. I had kept the server on for three days and it had sent a health report. 

Any ideas before I go and do a physical reboot?

22 hours ago, ideaman924 said:

Okay. One more thing before I try driving there. I got an email this morning saying the array health report had passed. I had kept the server on for three days and it had sent a health report. 

Any ideas before I go and do a physical reboot?

Just means the disks are fine.