1000% your container.
I also found the problem, WebSocket.
After a few minutes of inactivity (i.e. not clicking on anything) the WebSocket is closed.
I have no such settings in NGINX for this but I found a workaround.
In the noVNC setting is WebSocket automatic reconnect
The default is no tick and 5000.
Once I set that and watched it time out it counted to 5 seconds and then connected.
Guess you could set it to 1 if you really wanted.