Hello there...

First, I'm not even sure my hardware will support it, but I'd like to get started in hosting/running my own ChatGPT equivalent on UnRAID if it's possible. I just need advice on how it should be done, and whether or not to incorporate or offload functions to something on DigitalOcean. I'd rather keep it free/local though.

So what's the first steps to determine? And does anyone know of a current and known working guide for setup on UnRAID? Any help would be greatly appreciated.

Thanks!

local AI docker. prepackegaes everything for use... the web UI and modles out of the box...

be sure to have plenty of gcards and access.

review: https://localai.io/

free modles you can use depends on Gcard and memory. a 2 Billion modle needs a min of 3 GB of graphics ram...

https://localai.io/gallery.html

otherwise theres alot of lxc and proxmox help scripts goin around to run olama and open web UI to run a llm on debain proxmox...

https://digitalspaceport.com/how-to-setup-an-ai-server-homelab-beginners-guides-ollama-and-openwebui-on-proxmox-lxc/

... realy depnds on what hardware you have to run xyz model and how you wnat to interact with xyz models...

That's what I thought... It's pretty beefy on the requirements, which I don't have at present. I just have an onboard graphics card since I built it. I never really imagined this use case when I put it together.

Have you heard of any ways to offload the processing part using a third party? Like, spin up a server you subscribe to on Digital Ocean to do the "thinking", and keep the rest local? If it's a reasonable cost for my occasional search, I wouldn't mind offloading the hard stuff, and just host the portal locally, which submits queries to my own AI hosted elsewhere?

My main reason for this is to avoid the censorship (ahem, "fine-tuning") the big boys are already doing to their models. I want to essentially compile my own, hosting whatever I can myself. Storage isn't a problem, but I'd have to spend some money on a graphics card from the sound of it. Any budget recommendation of a graphics card that works well with UnRAID? The cheapest possible that does a good job? It really doesn't have to be lightning fast for my occasional use.

Sorry for the questions, but I literally have no idea where to start. My hope is to slowly walk through this thread, so others in my shoes can follow along if they decide to do the same...

1 hour ago, House Of Cards said:

Have you heard of any ways to offload the processing part using a third party? Like, spin up a server you subscribe to on Digital Ocean to do the "thinking", and keep the rest local?

Technically yeah but you'll be paying through the nose for a hosted VM with the required capabilities.

That's what I thought...

So I guess the first question is, what is a good "sweet spot" graphics card that plays nice with UnRAID, yet provides acceptable capability for basic stuff? I don't need high-end performance, just flexibility for a fairly capable setup. I don't mind waiting a little while it generates an answer, as long as it's local and uncensored...

It gets confusing because of all the options for models. It's one of those things you just have to know to understand fully.

I was looking at the Ollama docker, which mentions CPU only options and Intel processors. Is it even possible to run that with an 11th Gen Intel® Core™ i5-11400 @ 2.60GHz and no graphics card? I know, but again, I built this for fairly basic stuff, but it has about 20 GB of RAM free, and the CPU doesn't bog down with what I have running. The CPU sits at about 2-5% usually. If I can use some of those GB's and CPU to run basic questions through an AI model, I'd be happy.

That's the idea, I'm just trying to figure if that is even feasible or not. I'd shell out for a video card if I really need to. I just need some guidance from those "in the know"...

Thanks!

1 hour ago, House Of Cards said:

I was looking at the Ollama docker, which mentions CPU only options and Intel processors. Is it even possible to run that with an 11th Gen Intel® Core™ i5-11400 @ 2.60GHz and no graphics card?

Yes, and with 20GB RAM should be pretty fine to try stuff on already, just slow.

1 hour ago, House Of Cards said:

It gets confusing because of all the options for models. It's one of those things you just have to know to understand fully.

Yup, the problem is that there are hundreds of models, each of them with many parameters, so you're essentially spoilt for choice and getting what you hope to get is a matter of either trying a few random things and hoping they work how you want or spending months actively evaluating multiple things and trying to tweak them to your liking.

Edited September 9, 2025Sep 9 by Kilrah

Okay,

First step is configuring the basic setup. I only have about 16 GB available on my Docker vDisk. Should I be mapping out an external share location for the LLM images themselves, or are they going to run directly from the vDisk storage? If they copy directly to the vDisk, I'd have to increase the size first. I'm assuming that The docker image runs on the vDisk, it accesses the LLM from a separate storage location, and loads the LLM into memory on startup?

I think I'm going to use LocalAI (https://localai.io/), but the description implies that the LLM would be directly in the vDisk? Any insight on that?