codyrat

Memory Container Issues and Crashes and Mysterious postgreSQL hangsThis container template has been setup without any memory constraint. By default, the GitLab CE Docker container on Unraid has no memory limit — it can consume all available host RAM. On a shared Unraid server running multiple containers, this creates a dangerous condition known as memory overcommit. You could see this using the following docker command: [my docker container is gitlab-ce] docker exec gitlab-ce cat /proc/meminfo | grep Committed_AS Committed_AS: 41168020 kB if Committed_AS exceeds your total physical RAM and you have no swap, the kernel has no safety net. When that committed memory is actually accessed, the **OOM (Out of Memory) killer** fires — silently terminating processes without warning. In my case over the last several weeks i've been chasing the following issues: 1. OOM killer terminates a PostgreSQL background worker 2. PostgreSQL freezes — process stays alive but stops accepting connections 3. Puma (Rails app server) can no longer reach the database and hangs 4. Workhorse socket goes stale 5. nginx returns 502 Bad Gateway on all pages The deceptive part: gitlab-ctl status shows all services as running. The broken state is not obvious without digging into logs. Emergency fix (when already broken) [Your docker may not be named 'gitlab-ce'] docker exec gitlab-ce gitlab-ctl restart postgresql docker exec gitlab-ce gitlab-ctl restart puma Set a Hard Memory Limit### Choosing a limit Check your current container memory usage first: ```bash docker stats --no-stream gitlab-ce ``` For my personal GitLab instance with default Omnibus settings, 6GB seems like a reasonable starting point. Option 1: Unraid Docker UI 1. Go to Docker tab in Unraid web UI 2. Click the GitLab container icon → Edit 3. Scroll to Extra Parameters 4. Add --memory=6g to the existing parameters 5. Click Apply — Unraid will recreate the container with the limit Or On Unraid host vi /boot/config/plugins/dockerMan/templates-user/my-gitlab-ce.xml ``` Find the <ExtraParams> line and add --memory=6g to it: <ExtraParams>... --memory=6g --shm-size 265m ...</ExtraParams> Verify: [use your docker container name] docker stats --no-stream --format 'Memory: {{.MemUsage}}' gitlab-ce Memory: 3.998GiB / 6GiB Running in a Memory-Contrained EnvironmentI will edit this post with enhancements as I discover recommended settings from https://docs.gitlab.com/omnibus/settings/memory_constrained_envs/ In my environment, I have configured a 6GB container memory limit. I am currently experimenting with the following settings: postgresql['shared_buffers'] = '1536MB' postgresql['log_min_duration_statement'] = 1000 # log queries > 1 second puma['worker_processes'] = 0 puma['worker_processes'] = 2 puma['min_threads'] = 1 puma['max_threads'] = 4 sidekiq['concurrency'] = 10 gitlab_rails['gitlab_signup_enabled'] = false gitlab_rails['usage_ping_enabled'] = false gitlab_rails['service_ping_enabled'] = false gitlab_kas['enable'] = false # GitLab Agent Server — not needed without Kubernetes registry['enable'] = false # Container registry alertmanager['enable'] = false gitlab_exporter['enable'] = false node_exporter['enable'] = false postgres_exporter['enable'] = false prometheus['enable'] = false prometheus_monitoring['enable'] = false puma['exporter_enabled'] = false redis_exporter['enable'] = false sidekiq['metrics_enabled'] = false ## jemalloc faster memory release gitlab_rails['env'] = { 'MALLOC_CONF' => 'dirty_decay_ms:1000,muzzy_decay_ms:1000' } ## Gitaly env (combine with existing gitaly['env'] if present) gitaly['env'] = { 'MALLOC_CONF' => 'dirty_decay_ms:1000,muzzy_decay_ms:1000', 'GITALY_COMMAND_SPAWN_MAX_PARALLEL' => '2' }Long Reboot RecoveryAs a separate issue --- the stop timeout on the container is only 10s. If Unraid reboots, the postgreSQL only has 10s to complete its shutdown before docker sends a 'SIGKILL' LOG: database system was not properly shut down; automatic recovery in progress Use the same technique as above Option 1: Unraid Docker UI 1. Go to Docker tab in Unraid web UI 2. Click the GitLab container icon → Edit 3. Scroll to Extra Parameters 4. Add --stop-timeout=120 --restart=unless-stopped to the existing parameters 5. Click Apply — Unraid will recreate the container with the limit vi /boot/config/plugins/dockerMan/templates-user/my-gitlab-ce.xml Add --stop-timeout=120 to Extra Parameters in the Unraid template alongside the memory limit: ```xml <ExtraParams>... --memory=6g --stop-timeout=120 --restart=unless-stopped ...</ExtraParams> ``` After applying changes or recreating the container, verify settings persisted: ```bash # Restart policy docker inspect gitlab-ce | grep -A2 RestartPolicy "RestartPolicy": { "Name": "unless-stopped", "MaximumRetryCount": 0 # Stop timeout docker inspect gitlab-ce | grep StopTimeout "StopTimeout": 120 Extra ParametersThese are the extra parameters that i currently have set. Remove references the in the --env section if you do not need them Note: The environment is all one statement surrounded by double-quotes for GITLAB_ONIBUS_CONFIG --env GITLAB_OMNIBUS_CONFIG="external_url 'external_url https://gitlab.domain.net; postgresql['shared_buffers'] = '256MB'; sidekiq['concurrency'] = 15; prometheus_monitoring['enable'] = false;" My ExtraParams<ExtraParams>--log-opt max-size=10m --log-opt max-file=1 --env GITLAB_OMNIBUS_CONFIG="external_url 'external_url https://gitlab.domain.net; postgresql['shared_buffers'] = '256MB'; sidekiq['concurrency'] = 15; prometheus_monitoring['enable'] = false;" --add-host=gitlab.domain.net:x.x.x.x --memory=6g --shm-size 265m --pids-limit 2048 --stop-timeout=120 --restart=unless-stopped </ExtraParams> Puma CrashesUpdate: Switched Puma to single mode (worker_processes = 0) After investigating a recurring issue where gitlab-ctl status showed puma as "running" despite the socket refusing connections, Trying to switch to single mode (worker_processes = 0) # /etc/gitlab/gitlab.rb puma['worker_processes'] = 0 puma['min_threads'] = 1 puma['max_threads'] = 4

Thanks for replying. I'll take your advice and comment on the issue thread and see if I can convince the team to promote the defect fix earlier. Thanks for the great container!

While you're active on the 1.5.0 uplift with Alpine, would you consider a possible tweak to the guacamole deployment? I have a minor cosmetic issue with Guacamole and the mouse pointer which is corrupted. The issue is reported fixed in 1.6.0 but a minor patch could resolve issues with Ubuntu 22.04 RDP mouse corruption. guacamole releases are reasonably slow so i am hoping a patch might get us through their release cycle. See below: https://issues.apache.org/jira/browse/GUACAMOLE-1717

Please don't forget to add to the Releases page, so the release notes are searchable off the upgrade notice displayed on Unraid.