Hi,

I am using the Unifi Docker from the brettm357/unifi repository and it keeps locking up with [java.lang.OutOfMemoryError: Java heap space] error message. i have been tuning my new unifi ap's and keep logging in from multiple locations to access the unifi controller and apply settings which i think is whats causing the heap memory to become full and crash. But i cannot seem to kill the unifi docker at all. I have tried docker kill Unifi and also docker reboot and docker stop. Unraid  will not shutdown clean without trying to stop the docker and iit cant so I am forced to do a unclean shutdown which i want to try to avoid this time. This has happened 3 times in the past week.

 

Can anyone please give me some suggestions on how to kill this docker process so i can clean reboot the server?

 

here is the unifi docker log file

2018-04-09 17:29:03,133 CRIT Supervisor running as root (no user in config file)
2018-04-09 17:29:03,133 INFO Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
2018-04-09 17:29:03,142 INFO RPC interface 'supervisor' initialized
2018-04-09 17:29:03,142 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2018-04-09 17:29:03,142 INFO supervisord started with pid 1
2018-04-09 17:29:04,145 INFO spawned: 'unifi' with pid 9
2018-04-09 17:29:05,147 INFO success: unifi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-04-09 20:15:05,828 WARN received SIGTERM indicating exit request
2018-04-09 20:15:05,828 INFO waiting for unifi to die
2018-04-09 20:15:08,831 INFO waiting for unifi to die
2018-04-09 20:15:11,834 INFO waiting for unifi to die
2018-04-09 20:15:14,838 INFO waiting for unifi to die
2018-04-09 20:15:16,830 CRIT Supervisor running as root (no user in config file)
2018-04-09 20:15:16,830 INFO Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
Unlinking stale socket /var/run/supervisor.sock
2018-04-09 20:15:17,142 INFO RPC interface 'supervisor' initialized
2018-04-09 20:15:17,142 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2018-04-09 20:15:17,142 INFO supervisord started with pid 1
2018-04-09 20:15:18,144 INFO spawned: 'unifi' with pid 9
2018-04-09 20:15:19,144 INFO success: unifi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Exception in thread "http-bio-8443-Acceptor-0" java.lang.OutOfMemoryError: Java heap space

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "inform-9"
at java.io.ByteArrayOutputStream.<init>(ByteArrayOutputStream.java:77)
at sun.security.ssl.OutputRecord.<init>(OutputRecord.java:94)
at sun.security.ssl.OutputRecord.<init>(OutputRecord.java:105)
at sun.security.ssl.AppOutputStream.<init>(AppOutputStream.java:52)
at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:641)
at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:524)
at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:343)
at org.apache.tomcat.util.net.jsse.JSSESocketFactory.acceptSocket(JSSESocketFactory.java:279)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:222)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "webapi-31" Exception in thread "inform-197" Exception in thread "webapi-147" Exception in thread "pool-4-thread-1" Exception in thread "pool-3-thread-1" Exception in thread "inform_stat-1" Exception in thread "ContainerBackgroundProcessor[StandardEngine[Tomcat]]" Exception in thread "shutdown-listener" Exception in thread "inform-10" Exception in thread "Timer-0" Exception in thread "http-bio-8080-Acceptor-0" java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects

Exception in thread "inform-23" Exception in thread "inform-77" java.lang.OutOfMemoryError: Java heap space
at java.net.ServerSocket.accept(ServerSocket.java:512)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:222)
at java.lang.Thread.run(Thread.java:748)
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "inform-195" Exception in thread "webapi-180" java.lang.OutOfMemoryError: Java heap space
at org.apache.coyote.http11.InternalInputBuffer.<init>(InternalInputBuffer.java:61)
at org.apache.coyote.http11.Http11Processor.<init>(Http11Processor.java:59)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.createProcessor(Http11Protocol.java:165)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.createProcessor(Http11Protocol.java:103)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
java.lang.OutOfMemoryError: Java heap space
Exception in thread "inform-19" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "cluster-ClusterId{value='5acc021846e0fb0009f22b62', description='null'}-127.0.0.1:27117" java.lang.OutOfMemoryError: Java heap space
Exception in thread "stat-processor" java.lang.OutOfMemoryError: Java heap space

i attached the syslog also bc i see a call trace and not sure if this is related to my issue.

Call Trace:
Apr  9 18:50:00 DNASON kernel: tcp_sendmsg_locked+0xa53/0xbac
Apr  9 18:50:00 DNASON kernel: tcp_sendmsg+0x23/0x35
Apr  9 18:50:00 DNASON kernel: sock_sendmsg+0x14/0x1e
Apr  9 18:50:00 DNASON kernel: SyS_sendto+0xc0/0xe7
Apr  9 18:50:00 DNASON kernel: ? vfs_ioctl+0x19/0x26
Apr  9 18:50:00 DNASON kernel: ? do_vfs_ioctl+0x506/0x519
Apr  9 18:50:00 DNASON kernel: ? SyS_newfstat+0x29/0x34
Apr  9 18:50:00 DNASON kernel: ? syscall_trace_enter+0x154/0x17e
Apr  9 18:50:00 DNASON kernel: do_syscall_64+0xfe/0x107
Apr  9 18:50:00 DNASON kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Apr  9 18:50:00 DNASON kernel: RIP: 0033:0x152a81c7c5ea
Apr  9 18:50:00 DNASON kernel: RSP: 002b:0000152a39abd8c0 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
Apr  9 18:50:00 DNASON kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 0000152a81c7c5ea
Apr  9 18:50:00 DNASON kernel: RDX: 0000000000002017 RSI: 0000152a39abd9a0 RDI: 0000000000000070
Apr  9 18:50:00 DNASON kernel: RBP: 0000152a39abd9a0 R08: 0000000000000000 R09: 0000000000000000
Apr  9 18:50:00 DNASON kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000002017
Apr  9 18:50:00 DNASON kernel: R13: 0000152a39abd9a0 R14: 0000152a39ad1680 R15: 0000152a39abd920
Apr  9 18:50:00 DNASON kernel: Code: d0 75 02 31 c0 41 89 f3 41 81 e3 00 80 00 00 74 1a 44 8b 8f 68 05 00 00 41 d1 e9 44 2b 8f 6c 06 00 00 44 03 8f 74 06 00 00 79 10 <80> 48 38 08 8b 8f 6c 06 00 00 89 8f 74 06 00 00 40 80 e6 01 74 
Apr  9 18:50:00 DNASON kernel: RIP: tcp_push+0x4e/0xee RSP: ffffc9002e0a3d60
Apr  9 18:50:00 DNASON kernel: CR2: 0000000000000038
Apr  9 18:50:00 DNASON kernel: ---[ end trace 57d7fe0241ebad02 ]---

 

docker.txt

syslog.txt

What I would try and do, with the ultimate aim of removing it

 

Docker - switch it so it doesn't autostart

Settings - docker.  Stop the service - Probably won't cleanly stop and will leave the image in place which is what's ultimately causing your shutdown issues

 

Try this to unmount the image

umount -f /var/lib/docker

 

Restart should be good to go

1 hour ago, Squid said:

What I would try and do, with the ultimate aim of removing it

 

Docker - switch it so it doesn't autostart

Settings - docker.  Stop the service - Probably won't cleanly stop and will leave the image in place which is what's ultimately causing your shutdown issues

 

Try this to unmount the image

 

umount -f /var/lib/docker

 

 

Restart should be good to go

thanks for the quick replay...

I tried what you have suggested and it says target is busy after running umount

in unraid gui at the bottom left says array started starting services. But i shutdown all dockers and disabled autostart on unifi so i dont know why unraid would be starting services