Hello Unraid folks!

 

I have been tinkering with Home Assistant's Voice Assitant feature and after watching the below video from Network Chuck, I decided to run Ollama 3.2 on my Unraid server for a better AI experience. I have everything up and running using the Ollama container found in the Community Store but when my assistant uses Ollama, there is about a 20 second delay until I get an answer. My Unraid server specs are below. Is this about what I can expect or are there ways to speed up the response time? I tried the tinyllama model in lieu of 3.2 but it's not as capable. Any guidance/information is greatly appreciated!

 

Server specs:

 

Intel i5-14600

32GB DDR4 3200mhz RAM

Seagate Firecuda 530 NVME

 

 

Edited January 6, 20251 yr by obowan

olama like open web UI are jsut front end interfaces AI depending on the modle may need more Gcard Vram. a single nvida 2080 can barely run a 2Billion ai modle with decent performace. 

 

local AI docker.

open web Ui Docker

etc are alos resource intensvie when runnign a AI system you need ALOT of resources. the delay is most likely caused by ram loading and unlaoding in the gcrads vram you need antoher gcard to help with the AI tasks.

Thank you @bmartino1 for your response. Whenever I ask my voice assistant a question that requires Ollama, I see the 6 logical cores of my server spike. I tried to pin all logical and virtual cores to the docker container but it didn’t really change anything. I also attempted to pass my iGPU through to the container too but I don’t think it’s able to utilize it at this point. Thanks again! 

i have also been playing around with this, seems that running the same model without 'assist' on (i have no gpu) takes 3-6 seconds for a response, but when i turn on Assist, it can take 10-20 seconds to scan the exposed entities then figure out what to do with it.

for now i am not going to do much with it in HA until i get a gpu in there.

but without Assist on, it can still be used through HA (click the entity for the conversation) and it works faster then the same model in openwebui. I am not sure if thats just due to the way Home Assistant sets the token and initial prompt to have a simpler response.

I'm also looking into this. How about offloading that to an Intel ARC A310... looks like it should work! https://www.reddit.com/r/unRAID/comments/1n2m7h7/comment/nbc5phh/

https://hub.docker.com/r/uberchuckie/ollama-intel-gpu

Looks promising. My friend just got a arc Intel GPU. I may be able to test on it later...