Kubernetes cluster

Recommended Posts

Would like to request better support for running a kubernetes cluster on unraid. I'm currently using KIND (kubernetes in docker) and it's actually working quite well so far but it's not exactly natively integrated.




Incidentally I offered to help out the OP for the Ultimate Unraid Dashboard with his request for help by someone familiar with Docker to upgrade the setup and make it easier to setup and use. I've currently setup Grafana, Loki, Promtail, Telegraf, and InfluxDB (up to date versions) across 5 separate docker containers. 


The community has adopted helm charts for multi app setup/coordination in a cluster,, I think a lot of everyones time could be better spent adopting this paradigm versus putting more effort into Docker setups, docker itself is being deprecated by kubernetes for containerd. 

  • Upvote 2
Link to comment
  • 4 months later...
  • 1 month later...
  • 3 weeks later...

While I look forward to k8s, I suspect the odds are not good.


We may have better chance setting up a new k8s bare-metal server, and let it be managed by unraid k8s if possible. Let UnRaid do what it does best: NAS.


I find that even the VMs are somewhat painful. Say windows VM (can't upgrade 10 -> 11 directly, lack bluetooth support, gpu passthrough, xml vs UI etc)

Link to comment
  • 1 month later...
On 12/21/2022 at 9:13 AM, ctab579 said:

Running KinD is better than nothing. Can you post your cluster config and deployment as an example?


I couldn't confidently deploy k3d and insure that I wouldn't lose the sqlite data. (I guess I could have just tested it).


Happily! It did take sometime to setup and figure out maintenance as I encountered a growing-size issue I'll discuss below, but now I think it's running really well for my simple use cases. I haven't spent too much time with k8s in general so I would love some help/tips on a couple of things. 



I made a folder `/mnt/user/appdata/cluster` where everything lives. Inside I believe I simply used `wget` to download the following binaries, including kind, kubectl, k9s, helm, etc. I also update them to the latest versions by just using `wget` manually to download the new binaries here.


I have the following bash scripts to facilitate the overall setup/maintenance in numbered order of how they would be used in a new cluster creation.


  1. I cd to /mnt/user/appdata/cluster
  2. Run `source 0-setenv.sh`, which changes the HOME of the shell mainly, so that k9s and kubectl work etc.
  3. Cluster creation I only run once at the start, but in case I need to recreate the cluster with additional port mappings etc this order should work.
  4. When changing env-vars for the cluster, I re-run `2-add-app-envs.sh` to delete/re-apply the env-vars.


# Run `source setenv.sh'
export HOME=$(pwd)
export PATH="$HOME:${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
export KUBECONFIG=$(pwd)/.kube/config.yml
export EDITOR=nano


kind create cluster --config=cluster.yaml
helm repo add grafana https://grafana.github.io/helm-charts
helm install promtail grafana/promtail --set config.lokiAddress=http://MY-UNRAID-RUNNING-LOKI-DOCKER-IP:3100/loki/api/v1/push


kubectl delete secret my-app-env
kubectl apply -f my-app-env.yml



kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
- role: control-plane
  # add a mount from /path/to/my/files on the host to /files on the node
  - hostPath: /mnt/user/my-apps-storage
    containerPath: /my-apps-storage
    # optional: if set, the mount is read-only.
    # default false
    readOnly: false
    # optional: if set, the mount needs SELinux relabeling.
    # default false
    selinuxRelabel: false
    # optional: set propagation mode (None, HostToContainer or Bidirectional)
    # see https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation
    # default None
    propagation: HostToContainer
  - containerPort: 30150
    hostPort: 30150
    # optional: set the bind address on the host
    # is the current default
    listenAddress: ""
    # optional: set the protocol to one of TCP, UDP, SCTP.
    # TCP is the default
    protocol: TCP
   - containerPort: 30151
    hostPort: 30151
    # optional: set the bind address on the host
    # is the current default
    listenAddress: ""
    # optional: set the protocol to one of TCP, UDP, SCTP.
    # TCP is the default
    protocol: TCP


  • I didn't find a way to change a cluster config once it's already created, which is very annoying. If someone has tips on this it would be greatly appreciated.
  • Here I mapped unraids share my-apps-storage to the cluster, as well as multiple ports. The idea being I can't change the cluster so I added multiple port mappings in the beginning. This is just 2 port mappings as I've distilled and scrubbed my actual setup so you'll need to go through each file and personalize the setup a bit. You can add more.
  • As well I made one share my-apps-storage, which has sub folder for each app, since the cluster creation lets you map one folder. So each k8s node is given a persistent volume claim to a folder inside of this mapped folder.
  • I then use these port mappings with my traefik container to map this kubernetes node to my domain.


  name: my-app-env
type: Opaque
  MY_APP_ENV_VAR_KEY: https://example.com


This is just env-vars I want in the cluster for my app.

Deployment (Gitlab)

deployment.yaml (in gitlab repo)

apiVersion: v1
kind: PersistentVolume
  name: pv-first-storage
  storageClassName: standard
    - ReadWriteOnce
    storage: 1000Gi
    path: /my-apps-storage/first-app
apiVersion: v1
kind: PersistentVolumeClaim
  name: pvc-first-storage
  volumeName: pv-first-storage
    - ReadWriteOnce
      storage: 1000Gi
apiVersion: apps/v1
kind: Deployment
  name: my-first-app
    app: my-first-app
  replicas: 1
      app: my-first-app
    type: Recreate
        app: my-first-app
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
        - name: storage
            claimName: pvc-first-storage
        - name: my-first-apps-repo-name
          image: registry.example-domain.com/user/repo:main
          imagePullPolicy: Always
              memory: '8Gi'
              cpu: '4'
              memory: '10Gi'
              cpu: '6'
            - containerPort: 8080
            - name: storage
              mountPath: /storage
            - secretRef:
                name: my-app-env
        - name: gitlab-deploy-token
apiVersion: v1
kind: Service
  name: my-first-app
  type: NodePort
    - name: http
      nodePort: 30150
      port: 8080
    app: my-first-app


  • Here you can see I mapped one of the extraPortMappings to my apps http port 8080
  • Using a PersistentVolume and PersistentVolumeClaim, I'm able to map a folder inside of my unraid share my-apps-storage/my-first-app, into the k8s node as /storage inside of the container.
  • I also map my apps env vars from the cluster here, you can see at the container - envFrom.

Project Repo

.gitlab-ci.yml (in gitlab repo)

    - if [ -n "$KUBE_CONTEXT" ]; then kubectl config use-context "$KUBE_CONTEXT"; fi

  - build
  - deploy
  - notify


  stage: build
    - printenv | sort
    - time docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY
    - 'time docker build
      --cache-from $IMAGE_TAG
      --tag $IMAGE_TAG
      --build-arg PUBLIC_HOSTNAME
    - time docker push $IMAGE_TAG

  extends: [.kube-context]
  stage: deploy
  image: 'registry.gitlab.com/gitlab-org/cluster-integration/cluster-applications:latest'
    name: production
    - gl-ensure-namespace gitlab-managed-apps
    # - gl-helmfile --file $CI_PROJECT_DIR/helmfile.yaml apply --suppress-secrets
    - kubectl delete secret gitlab-deploy-token || true
    - kubectl create secret docker-registry gitlab-deploy-token
      [email protected]
    - kubectl apply -f deployment.yaml
    # - gl-helmfile --file $CI_PROJECT_DIR/helmfile.yaml apply --suppress-secrets
    - kubectl rollout restart deploy my-first-app


This is pretty much my exact gitlab-ci.yml, I've even left some commented statements because if anyone is more experienced they might be able to help in integrating helm better. I added helm at cluster-creation for loki so I can ingest the apps logs.


My gitlab setup is pretty simple, just gitlab-ce running in a container, I have it mapped to my domain through traefik/cloudflared, and then gitlab-runner running in a separate docker container all on my unraid. Here's my gitlab-runner config currently because the access to Docker in Docker is needed to build docker images, and I encountered several issues getting this working, including an issue recently so if anyone wants to take a second look they can. I had to map /var/run/docker.sock to get it to work correctly.


config.toml (in gitlab-runner)

concurrent = 1
check_interval = 0

  #listen_address = "" #  listen on all available interfaces on port 8093
  #listen_address = "[::]:8093"
  advertise_address = "gitlab-runner-1:8093"
  session_timeout = 1800

  name = "docker-runner-1"
  url = "https://my-gitlab-instance.com"
  token = "MY_GITLAB_TOKEN"
  executor = "docker"
  builds_dir = "/builds"
  cache_dir = "/cache"
  environment = [
    enabled = true
    tls_verify = false
    tls_cert_path = "/certs"
    image = "docker"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/mnt/user/appdata/gitlab-runner-1/cache:/cache", "/mnt/user/appdata/gitlab-runner-1/builds:/builds", "/mnt/user/appdata/gitlab-runner-1/certs:/certs", "/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0
    name = "docker"
    alias = "docker"


The only issue I encountered was a growing-cache issue, this is a known issue with KinD and I have more information on the github issue I created. 




You can read more about it on github, but it was easily resolved by adding a Unraid User Script that runs daily the following command


# docker delete dangling images
docker rmi $(docker images --quiet --filter "dangling=true")

# docker prune unused images
docker image prune -f
docker image prune -a --filter "until=4320h" -f

# clear KinD control cache manually
docker exec kind-control-plane /bin/bash -c "crictl rmi --prune"


Mainly the clear KinD control cache part, but because gitlab-runner rebuilds docker images on deploy, I believe the Unraid docker fills up with unused layers over time so I clean that up too.



As you can see, this is definitely not a setup for the faint of heart and probably not even a setup I would recommend for anyone else unless you understand at least the plumbing. But, it works really well. I am able to commit changes, gitlab will deploy my container to the cluster, and I'm able to access my custom codebase running on my unraid cluster. Using `k9s` I can shell into my container. Also worth noting I don't have a fancy cluster setup, I just needed 1 pod to run my app. Inside of that pod I have enough cpu allocated for multiple processes running doing different things etc.


I should say after all of this I really am not very experienced with kubernetes at all. I just needed a way to run a docker container on my unraid. I didn't like the other solutions such as a separate docker container that checks if a new image was added to the registry and then updates it. I also didn't want to use a VM because this is exactly what kubernetes is designed for, and with the benefit of no overhead. Although a really involved setup, it does work.


NOTE: this really wasn't meant as a tutorial, if you're not able to understand what to do from this just ask questions and I'm happy to answer them. If there's enough interest I could maybe do a better walkthrough tutorial. Thanks for reading!



I would like some additional things out of this setup. I currently have been looking into how to do a maintenance window (to bring down my app). As well I want some more control over the networking, because KinD uses the `kind` docker network, all of my other docker containers that are in unraid cannot communicate directly using the docker container name, as you do in a docker custom network. I ended up having to use my Unraids LAN IP to get the cluster node to use a pg database I run in an unraid docker. If anyone has any thoughts on these please share!

  • Like 2
Link to comment
  • 2 weeks later...

@gvkhna Thank you for that lengthy writeup! That will probably help many people extensively.

However, I have a question about your config. Am I understanding that you recreate your cluster (and more importantly your database; e.g.: etcd) every time UnRaid boots? That is what I would like to avoid.



Link to comment
@gvkhna Thank you for that lengthy writeup! That will probably help many people extensively.

However, I have a question about your config. Am I understanding that you recreate your cluster (and more importantly your database; e.g.: etcd) every time UnRaid boots? That is what I would like to avoid.

I don’t believe so, in this setup, I don’t do anything custom at boot. I’m pretty sure KinD takes care of all of that, the KinD container starts on its own with Docker and basically starts the cluster.

But I really don’t know the internals of what’s happening at boot and wouldn’t be able to tell you for sure exactly.

Sent from my iPhone using Tapatalk
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.