Was no llama-swap template on CA and needed it so here we go!

llama-swap is a lightweight proxy that sits in front of llama-server and allows for hot swap models. You can serve many models from a single OpenAI-compatible endpoint without manually restarting anything.

Features

Hot model loading and unloading

Aliases and groups for multi-model concurrency

OpenAI-compatible API

GPU Support

The template includes tags for NVIDIA (cuda), AMD (rocm), Vulkan (AMD/Intel iGPU), and CPU-only. Each tag description includes the required Extra Parameters for that GPU type.

Setup

config.yaml Must be manually created example config is linked to below. You can also view llama-swap github for other examples

Use Unraid terminal to make edits so filesystem watchers can see files been changed for Hot Reload I.E nano /mnt/user/appdata/llama-swap/config/config.yaml

"Waiting for Approval" Community Applications Install

Search for llama-swap in the Apps tab

Set your Models and Config directory paths

Create a config.yaml in your Config directory (example config linked in the Overview)

Reference your models in config.yaml as /models/yourmodel.gguf

Select the tag matching your GPU and update Extra Parameters accordingly

Start the container and access the UI at http://your-server-ip:8080/ui

Manual Install

1. Download the template XML from the link below

2. Place the XML file in /boot/config/plugins/dockerMan/templates-user/ on your Unraid server

3. In the Docker tab click Add Container and select llama-swap from the template dropdown

4. Set your Models and Config directory paths

5. Select the tag matching your GPU and update Extra Parameters accordingly

6. Start the container and access the UI at http://your-server-ip:8080/ui

Template download: https://raw.githubusercontent.com/PikkonMG/unraid-docker-templates/main/templates/llama-swap.xml

Links

GitHub: https://github.com/mostlygeek/llama-swap

Example config: https://github.com/PikkonMG/unraid-docker-templates/blob/main/examples/llama-swap/example-llama-swap-config.yaml

Support: https://github.com/mostlygeek/llama-swap/issues

My Unraid Template Repo: https://github.com/PikkonMG/unraid-docker-templates

Edited March 30Mar 30 by PikkonMG

Thanks for setting this up, it's working great for me so far

36 minutes ago, opticon said:

Thanks for setting this up, it's working great for me so far

🫡 Np.

I'm getting - "did not find expected key"

The config file is in place, wondering if the template needs an update?

1 hour ago, i_max said:

I'm getting - "did not find expected key"

The config file is in place, wondering if the template needs an update

That error is a YAML parse error I think, right now I can’t reproduce it from the current template working fine on my end. Please paste your current config.yaml and the full log line showing "did not find expected key" so I can check and see where it might be coming from.

Edited June 14Jun 14 by PikkonMG

On 6/14/2026 at 12:02 PM, i_max said:

I'm getting - "did not find expected key"

The config file is in place, wondering if the template needs an update?

Wanted to check back, did you figure it out if not please share your config.

Hello! I'm able to get this spun up and access the webUI but can't seem to connect to Home Assitant via the Local LLM Integration. I have tried to add the service via the default 8080 port and port 5800, which is listed in the model logs when running. Here is a link to the Home Assistant integration in question: https://github.com/skye-harris/hass_local_openai_llm This works when spinning up a llama.cpp server on a Linux based PC. My config.yaml is attached. Any advice is greatly appreciated!

config.yaml

1 hour ago, obowan said:

Hello! I'm able to get this spun up and access the webUI but can't seem to connect to Home Assitant via the Local LLM Integration. I have tried to add the service via the default 8080 port and port 5800, which is listed in the model logs when running. Here is a link to the Home Assistant integration in question: https://github.com/skye-harris/hass_local_openai_llm This works when spinning up a llama.cpp server on a Linux based PC. My config.yaml is attached. Any advice is greatly appreciated!

config.yaml

Your config.yaml looks basically fine. The likely issue is the URL used in Home Assistant. For llama-swap Home Assistant should connect to the API at http://<your-unraid-ip>:8080/v1, not port 5800. The 5800 port in the logs is typically the internal llama.cpp backend launched by llama-swap. Before retrying in HA test curl http://<your-unraid-ip>:8080/v1/models from another machine. If that fails, HA will fail too.

Edited July 14Jul 14 by PikkonMG

You're my hero! Thank you so much. Probably should have read the documentation more closely. I really appreciate it!

On 6/22/2026 at 3:48 PM, PikkonMG said:

Wanted to check back, did you figure it out if not please share your config.

I started using llama server router mode.