nearcatch Posted March 31, 2023 Share Posted March 31, 2023 (edited) Recently have had a lot of instabilty. My jellyfin docker has crashed multiple times a day due to an Out of Memory error, and I'm sporadically getting errors in the terminal that look like the below (I was trying to send a `docker compose down` command). I managed to download diagnostics, although that failed several times. I also got a notification about there being a physical problem with the drive, and then the notification changed to say my drive was blacklisted. After a server reboot where I swapped the drive to another USB slot, the server came up without any errors or those warning notifications. Can someone take a look at the diagnostics and tell me what exactly is happening? EDIT: also have been getting emails like this: ❯ dc down _check_global_aliases:15: fork failed: resource temporarily unavailable zsh: fork failed: resource temporarily unavailable _flush_ysu_buffer:3: fork failed: resource temporarily unavailable ❯ dc down runtime/cgo: pthread_create failed: Resource temporarily unavailable SIGABRT: abort PC=0x175b73a m=10 sigcode=18446744073709551610 goroutine 0 [idle]: runtime: unknown pc 0x175b73a stack: frame={sp:0x15045891f948, fp:0x0} stack=[0x150458900248,0x15045891fe48) 0x000015045891f848: 0x0000000000000037 0x0000000000000000 0x000015045891f858: 0x0000000000000000 0x00000000001062f3 0x000015045891f868: 0x0000000000000001 0x000000000284f160 0x000015045891f878: 0x0000000000000037 0x00000000028ff888 0x000015045891f888: 0x000015045891f8c0 0x000015045891f950 0x000015045891f898: 0x000000000175e58b 0x0000000000040000 0x000015045891f8a8: 0x00000000004b0d6f 0x0000150400000000 0x000015045891f8b8: 0x000015045891f8d8 0x0000003000000010 0x000015045891f8c8: 0x000015045891fb20 0x000015045891fa50 0x000015045891f8d8: 0x0000000000000000 0x0000000000000000 0x000015045891f8e8: 0x0000000000000000 0x0000000000000000 0x000015045891f8f8: 0x0000000000000000 0x5f64616572687470 0x000015045891f908: 0x6620657461657263 0x52203a64656c6961 0x000015045891f918: 0x20656372756f7365 0x7261726f706d6574 0x000015045891f928: 0x76616e7520796c69 0x00656c62616c6961 0x000015045891f938: 0x00000000004c3885 <runtime.persistentalloc+0x0000000000000065> 0x0000000000203000 0x000015045891f948: <0x000000000175b77b 0x0000000000000000 0x000015045891f958: 0x0000000000003fc0 0x0000000000000000 0x000015045891f968: 0x00000000017548ba 0x00000000fffffff5 0x000015045891f978: 0x0000000001756edf 0x0000000000000001 0x000015045891f988: 0x000000000175bd45 0x00000000028ff888 0x000015045891f998: 0x0000000000000000 0x000015045891f9ff 0x000015045891f9a8: 0x0000000000000001 0x000015045891fb28 0x000015045891f9b8: 0x000000000284f160 0x000000000000000a 0x000015045891f9c8: 0x000000000284f1ec 0x0000000000000178 0x000015045891f9d8: 0x000000000284f160 0x000015045891fd08 0x000015045891f9e8: 0x0000000001754e86 0x0000000000000000 0x000015045891f9f8: 0x0a0000000284f160 0x000000000284f160 0x000015045891fa08: 0x000000000175c56a 0x000000000284f160 0x000015045891fa18: 0x000015045891fd08 0x0000000000203000 0x000015045891fa28: 0x00000000017507f9 0x0000003000000008 0x000015045891fa38: 0x000015045891fb20 0x000015045891fa50 runtime: unknown pc 0x175b73a stack: frame={sp:0x15045891f948, fp:0x0} stack=[0x150458900248,0x15045891fe48) 0x000015045891f848: 0x0000000000000037 0x0000000000000000 0x000015045891f858: 0x0000000000000000 0x00000000001062f3 0x000015045891f868: 0x0000000000000001 0x000000000284f160 0x000015045891f878: 0x0000000000000037 0x00000000028ff888 0x000015045891f888: 0x000015045891f8c0 0x000015045891f950 0x000015045891f898: 0x000000000175e58b 0x0000000000040000 0x000015045891f8a8: 0x00000000004b0d6f 0x0000150400000000 0x000015045891f8b8: 0x000015045891f8d8 0x0000003000000010 0x000015045891f8c8: 0x000015045891fb20 0x000015045891fa50 0x000015045891f8d8: 0x0000000000000000 0x0000000000000000 0x000015045891f8e8: 0x0000000000000000 0x0000000000000000 0x000015045891f8f8: 0x0000000000000000 0x5f64616572687470 0x000015045891f908: 0x6620657461657263 0x52203a64656c6961 0x000015045891f918: 0x20656372756f7365 0x7261726f706d6574 0x000015045891f928: 0x76616e7520796c69 0x00656c62616c6961 0x000015045891f938: 0x00000000004c3885 <runtime.persistentalloc+0x0000000000000065> 0x0000000000203000 0x000015045891f948: <0x000000000175b77b 0x0000000000000000 0x000015045891f958: 0x0000000000003fc0 0x0000000000000000 0x000015045891f968: 0x00000000017548ba 0x00000000fffffff5 0x000015045891f978: 0x0000000001756edf 0x0000000000000001 0x000015045891f988: 0x000000000175bd45 0x00000000028ff888 0x000015045891f998: 0x0000000000000000 0x000015045891f9ff 0x000015045891f9a8: 0x0000000000000001 0x000015045891fb28 0x000015045891f9b8: 0x000000000284f160 0x000000000000000a 0x000015045891f9c8: 0x000000000284f1ec 0x0000000000000178 0x000015045891f9d8: 0x000000000284f160 0x000015045891fd08 0x000015045891f9e8: 0x0000000001754e86 0x0000000000000000 0x000015045891f9f8: 0x0a0000000284f160 0x000000000284f160 0x000015045891fa08: 0x000000000175c56a 0x000000000284f160 0x000015045891fa18: 0x000015045891fd08 0x0000000000203000 0x000015045891fa28: 0x00000000017507f9 0x0000003000000008 0x000015045891fa38: 0x000015045891fb20 0x000015045891fa50 goroutine 1 [semacquire, locked to thread]: github.com/docker/cli/vendor/k8s.io/api/core/v1.init() /go/src/github.com/docker/cli/vendor/k8s.io/api/core/v1/types_swagger_doc_generated.go:316 +0x25cc goroutine 34 [chan receive]: github.com/docker/cli/vendor/k8s.io/klog.(*loggingT).flushDaemon(0xc0000c2060?) /go/src/github.com/docker/cli/vendor/k8s.io/klog/klog.go:1010 +0x6a created by github.com/docker/cli/vendor/k8s.io/klog.init.0 /go/src/github.com/docker/cli/vendor/k8s.io/klog/klog.go:411 +0xef rax 0x0 rbx 0x0 rcx 0x175b73a rdx 0x0 rdi 0x2 rsi 0x15045891f950 rbp 0x15045891f950 rsp 0x15045891f948 r8 0xa r9 0x4b0cce r10 0x8 r11 0x246 r12 0x203000 r13 0x178 r14 0x4b0d6f r15 0x15045891fb30 rip 0x175b73a rflags 0x246 cs 0x33 fs 0x0 gs 0x0 ❯ dc down _check_global_aliases:15: fork failed: resource temporarily unavailable zsh: fork failed: resource temporarily unavailable _flush_ysu_buffer:3: fork failed: resource temporarily unavailable ❯ dc down runtime/cgo: pthread_create failed: Resource temporarily unavailable runtime/cgo: pthread_create failed: Resource temporarily unavailable SIGABRT: abort PC=0x175b73a m=4 sigcode=18446744073709551610 goroutine 0 [idle]: runtime: unknown pc 0x175b73a stack: frame={sp:0x151f8605d938, fp:0x0} stack=[0x151f8603e248,0x151f8605de48) 0x0000151f8605d838: 0x0000000000000037 0x0000000000000000 0x0000151f8605d848: 0x0000000000000000 0x0000151f8605d890 0x0000151f8605d858: 0x0000000000000001 0x000000000284f160 0x0000151f8605d868: 0x0000000000000037 0x00000000028ff888 0x0000151f8605d878: 0x0000151f8605d8b0 0x0000151f8605d940 0x0000151f8605d888: 0x000000000175e58b 0x0000151f8605d908 0x0000151f8605d898: 0x00000000004b0d6f 0x0000151f00000000 0x0000151f8605d8a8: 0x0000151f8605d8c8 0x0000003000000010 0x0000151f8605d8b8: 0x0000151f8605db10 0x0000151f8605da40 0x0000151f8605d8c8: 0x0000000000000000 0x0000000000000000 0x0000151f8605d8d8: 0x0000000000000000 0x0000000000000000 0x0000151f8605d8e8: 0x0000000000000000 0x5f64616572687470 0x0000151f8605d8f8: 0x6620657461657263 0x52203a64656c6961 0x0000151f8605d908: 0x20656372756f7365 0x7261726f706d6574 0x0000151f8605d918: 0x76616e7520796c69 0x00656c62616c6961 0x0000151f8605d928: 0x00000000004c3885 <runtime.persistentalloc+0x0000000000000065> 0x0000000000203000 0x0000151f8605d938: <0x000000000175b77b 0x0000000000000000 0x0000151f8605d948: 0x0000151f85f42000 0x0000000002904af8 0x0000151f8605d958: 0x00000000017548ba 0x00000000fffffff5 0x0000151f8605d968: 0x0000000001756edf 0x0000000000000001 0x0000151f8605d978: 0x000000000175bd45 0x00000000028ff888 0x0000151f8605d988: 0x0000000000000000 0x0000151f8605d9ef 0x0000151f8605d998: 0x0000000000000001 0x0000151f8605db18 0x0000151f8605d9a8: 0x000000000284f160 0x000000000000000a 0x0000151f8605d9b8: 0x000000000284f1ec 0x0000000000000178 0x0000151f8605d9c8: 0x000000000284f160 0x0000151f8605dcf8 0x0000151f8605d9d8: 0x0000000001754e86 0x0000000000000000 0x0000151f8605d9e8: 0x0a0000000284f160 0x000000000284f160 0x0000151f8605d9f8: 0x000000000175c56a 0x000000000284f160 0x0000151f8605da08: 0x0000151f8605dcf8 0x0000000000203000 0x0000151f8605da18: 0x00000000017507f9 0x0000003000000008 0x0000151f8605da28: 0x0000151f8605db10 0x0000151f8605da40 runtime: unknown pc 0x175b73a stack: frame={sp:0x151f8605d938, fp:0x0} stack=[0x151f8603e248,0x151f8605de48) 0x0000151f8605d838: 0x0000000000000037 0x0000000000000000 0x0000151f8605d848: 0x0000000000000000 0x0000151f8605d890 0x0000151f8605d858: 0x0000000000000001 0x000000000284f160 0x0000151f8605d868: 0x0000000000000037 0x00000000028ff888 0x0000151f8605d878: 0x0000151f8605d8b0 0x0000151f8605d940 0x0000151f8605d888: 0x000000000175e58b 0x0000151f8605d908 0x0000151f8605d898: 0x00000000004b0d6f 0x0000151f00000000 0x0000151f8605d8a8: 0x0000151f8605d8c8 0x0000003000000010 0x0000151f8605d8b8: 0x0000151f8605db10 0x0000151f8605da40 0x0000151f8605d8c8: 0x0000000000000000 0x0000000000000000 0x0000151f8605d8d8: 0x0000000000000000 0x0000000000000000 0x0000151f8605d8e8: 0x0000000000000000 0x5f64616572687470 0x0000151f8605d8f8: 0x6620657461657263 0x52203a64656c6961 0x0000151f8605d908: 0x20656372756f7365 0x7261726f706d6574 0x0000151f8605d918: 0x76616e7520796c69 0x00656c62616c6961 0x0000151f8605d928: 0x00000000004c3885 <runtime.persistentalloc+0x0000000000000065> 0x0000000000203000 0x0000151f8605d938: <0x000000000175b77b 0x0000000000000000 0x0000151f8605d948: 0x0000151f85f42000 0x0000000002904af8 0x0000151f8605d958: 0x00000000017548ba 0x00000000fffffff5 0x0000151f8605d968: 0x0000000001756edf 0x0000000000000001 0x0000151f8605d978: 0x000000000175bd45 0x00000000028ff888 0x0000151f8605d988: 0x0000000000000000 0x0000151f8605d9ef 0x0000151f8605d998: 0x0000000000000001 0x0000151f8605db18 0x0000151f8605d9a8: 0x000000000284f160 0x000000000000000a 0x0000151f8605d9b8: 0x000000000284f1ec 0x0000000000000178 0x0000151f8605d9c8: 0x000000000284f160 0x0000151f8605dcf8 0x0000151f8605d9d8: 0x0000000001754e86 0x0000000000000000 0x0000151f8605d9e8: 0x0a0000000284f160 0x000000000284f160 0x0000151f8605d9f8: 0x000000000175c56a 0x000000000284f160 0x0000151f8605da08: 0x0000151f8605dcf8 0x0000000000203000 0x0000151f8605da18: 0x00000000017507f9 0x0000003000000008 0x0000151f8605da28: 0x0000151f8605db10 0x0000151f8605da40 goroutine 1 [runnable, locked to thread]: github.com/docker/cli/vendor/k8s.io/apimachinery/pkg/runtime.(*Scheme).AddKnownTypeWithName(0xc000511b20, {{0x0, 0x0}, {0x200529, 0x2}, {0x17865ef, 0xc}}, {0x1b1a938?, 0xc0004219f0?}) /go/src/github.com/docker/cli/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:213 +0x47a github.com/docker/cli/vendor/k8s.io/apimachinery/pkg/runtime.(*Scheme).AddKnownTypes(0xc000511b20, {{0x0?, 0x0?}, {0x200529?, 0xa?}}, {0xc00061f418?, 0x7, 0x1b1a870?}) /go/src/github.com/docker/cli/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:180 +0x237 github.com/docker/cli/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.AddToGroupVersion(0x26ca5e?, {{0x0?, 0x0?}, {0x200529?, 0x0?}}) /go/src/github.com/docker/cli/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/register.go:51 +0x385 github.com/docker/cli/vendor/k8s.io/client-go/plugin/pkg/client/auth/exec.init.0() /go/src/github.com/docker/cli/vendor/k8s.io/client-go/plugin/pkg/client/auth/exec/exec.go:56 +0x34 goroutine 34 [chan receive]: github.com/docker/cli/vendor/k8s.io/klog.(*loggingT).flushDaemon(0xc0000c2060?) /go/src/github.com/docker/cli/vendor/k8s.io/klog/klog.go:1010 +0x6a created by github.com/docker/cli/vendor/k8s.io/klog.init.0 /go/src/github.com/docker/cli/vendor/k8s.io/klog/klog.go:411 +0xef rax 0x0 rbx 0x0 rcx 0x175b73a rdx 0x0 rdi 0x2 rsi 0x151f8605d940 rbp 0x151f8605d940 rsp 0x151f8605d938 r8 0xa r9 0x4b0cce r10 0x8 r11 0x246 r12 0x203000 r13 0x178 r14 0x4b0d6f r15 0x151f8605db20 rip 0x175b73a rflags 0x246 cs 0x33 fs 0x0 gs 0x0 deepervisor-diagnostics-20230331-0133.zip Edited March 31, 2023 by hasown Quote Link to comment
JorgeB Posted March 31, 2023 Share Posted March 31, 2023 Fist thing to try and fix are the flash drive issues, it's dropping offline, make sure it's using a USB 2.0 port, if issues persist try a different flash drive. Quote Link to comment
nearcatch Posted March 31, 2023 Author Share Posted March 31, 2023 (edited) I shutdown the server and moved the drive to a different port, and I haven't seen any warnings about the boot device yet, but I got two more of the cron emails today. EDIT: Also got this just now: Very confused about what could be causing this, because I haven't changed anything major in the server for months, and the changes I have done in the last couple of weeks I rolled back. I thought the boot drive was loaded into memory when the server started, so even if it *is* going bad, I don't know why it would be the source of this issue. Edited April 1, 2023 by hasown Quote Link to comment
JorgeB Posted April 1, 2023 Share Posted April 1, 2023 Try booting in safe mode. Quote Link to comment
nearcatch Posted April 1, 2023 Author Share Posted April 1, 2023 I actually found a docker container change was spawning a bunch of zombie processes, so I've reversed a change there and hopefully that will fix this. The processes list is much shorter afterwards so I'm hopeful. 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.