xer01ne

The Local.ai container isn't working properly for processors that don't have AVX2 support. From my understanding, there is supposed to be a feature that chooses AVX or AVX2, but this isn't happening. So, one of the issues is that models won't load. Not sure if this can be fixed from the image, or if this is an upstream issue for Mudler and Local.ai.

60GB!?! Ouch!! That’s insane! … … … I’m gonna do it!! 🤣😂 Thanks for sharing, I’m glad to see I wasn’t going crazy! I’ll let you know how it goes!

Awesome! Thanks for confirming this... I am interested in your results!

I am trying to get the Local.ai container to work, but I can't figure out why none of the models will work. First, when I try to send a query to Local.ai's chat UI page, none of my video cards are tasked. After reviewing the logs, it appears that the models are failing to load. Has anyone had this issue or know how to fix it? For the record, the Ollama server and Open WebUI containers work just fine, so I know the drivers are good. Here is the log when the container starts... @@@@@ If you are experiencing issues with the pre-compiled builds, try setting REBUILD=true If you are still experiencing issues with the build, try setting CMAKE_ARGS and disable the instructions set as needed: CMAKE_ARGS="-DGGML_F16C=OFF -DGGML_AVX512=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF" see the documentation at: https://localai.io/basics/build/index.html Note: See also https://github.com/go-skynet/LocalAI/issues/288 @@@@@ CPU info: model name : Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts md_clear flush_l1d CPU: AVX found OK CPU: no AVX2 found CPU: no AVX512 found @@@@@ Here are is the logs when I send a query to chat... 3:08PM INF Trying to load the model 'Bunny-Llama-3-8B-Q4_K_M.gguf' with the backend '[llama-cpp llama-ggml llama-cpp-fallback piper rwkv stablediffusion whisper huggingface bert-embeddings /build/backend/python/sentencetransformers/run.sh /build/backend/python/parler-tts/run.sh /build/backend/python/coqui/run.sh /build/backend/python/sentencetransformers/run.sh /build/backend/python/diffusers/run.sh /build/backend/python/exllama2/run.sh /build/backend/python/transformers/run.sh /build/backend/python/rerankers/run.sh /build/backend/python/openvoice/run.sh /build/backend/python/bark/run.sh /build/backend/python/mamba/run.sh /build/backend/python/autogptq/run.sh /build/backend/python/vall-e-x/run.sh /build/backend/python/vllm/run.sh /build/backend/python/transformers-musicgen/run.sh]' 3:08PM INF [llama-cpp] Attempting to load 3:08PM INF Loading model 'Bunny-Llama-3-8B-Q4_K_M.gguf' with backend llama-cpp 3:08PM INF [llama-cpp] Fails: backend not found: /tmp/localai/backend_data/backend-assets/grpc/llama-cpp 3:08PM INF [llama-cpp] Autodetection failed, trying the fallback 3:08PM INF Loading model 'Bunny-Llama-3-8B-Q4_K_M.gguf' with backend 3:08PM INF [llama-cpp] Fails: fork/exec grpc: permission denied 3:08PM INF [llama-ggml] Attempting to load 3:08PM INF Loading model 'Bunny-Llama-3-8B-Q4_K_M.gguf' with backend llama-ggml 3:08PM INF [llama-ggml] Fails: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF 3:08PM INF [llama-cpp-fallback] Attempting to load 3:08PM INF Loading model 'Bunny-Llama-3-8B-Q4_K_M.gguf' with backend llama-cpp-fallback 3:08PM INF [llama-cpp-fallback] Fails: could not load model: rpc error: code = Unavailable desc = error reading from server: read tcp 127.0.0.1:59348->127.0.0.1:33383: read: connection reset by peer 3:08PM INF [piper] Attempting to load 3:08PM INF Loading model 'Bunny-Llama-3-8B-Q4_K_M.gguf' with backend piper 3:08PM INF [piper] Fails: could not load model: rpc error: code = Unknown desc = unsupported model type /build/models/Bunny-Llama-3-8B-Q4_K_M.gguf (should end with .onnx) 3:08PM INF [rwkv] Attempting to load 3:08PM INF Loading model 'Bunny-Llama-3-8B-Q4_K_M.gguf' with backend rwkv 3:08PM INF [rwkv] Fails: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF 3:08PM INF [stablediffusion] Attempting to load 3:08PM INF Loading model 'Bunny-Llama-3-8B-Q4_K_M.gguf' with backend stablediffusion 3:08PM INF [stablediffusion] Loads OK And again, here is my nvidia-smi output: +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 560.35.03 Driver Version: 560.35.03 CUDA Version: 12.6 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA RTX A4000 Off | 00000000:05:00.0 Off | Off | | 45% 64C P0 59W / 140W | 4MiB / 16376MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA RTX A2000 Off | 00000000:42:00.0 Off | Off | | 35% 65C P0 41W / 70W | 4MiB / 6138MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ The container I installed is: localai/localai:master-cublas-cuda12 (I also tried the latest cuda that Unraid offers, both have the same results) Bridge, Bash, 8080, and debug=false. Again, any help would be greatly appreciated.

Okay, I’m outgrowing my current setup. I have a Dell R720 with a dual Intel® Xeon® CPU E5-2665 0 @ 2.40GHz and 256GB of DDR3 memory. I Also have a gaming PC in my rack. I want to merge the two, but the processor isn’t powerful enough for the VMs I want, and I don’t think my RTX 3070ti is going to fix in the server chassis. (And I don’t know if the bios will support, etc…) I currently run about 25 dockers and 2 Win10 VMs. The VMs have always been slow, so I know I’m gonna need some more juice in the processors. I’ve never even come close to using 256GB of memory, so I think I can scale that back. I’d like some ideas on what others are doing or what hardware I could get to get this RTX card in a new case for a more centralized setup. I want to VM my gaming machine and have it as a part of my server setup. So, are there any server, rack mount, chassis’s available on the market that will accept an RTX 3070ti? I know it’s not a “workstation” card, but I’m not in the market for dropping $4k on a card that I don’t think will perform as good with gaming (like the A6000). If possible, I’d like to stick with my RTX desktop card and build around it. thanks in advance. I’m interested to hear and see what everyone is using.

Works perfect! I am now using NginxProxyManager and on the Details tab of the domain, I didn't check any of the options (cache assets, block common exploits, etc) and the clone works perfectly! Thank you very much for sharing!

frakman1, Edit: That didn't work.... I added two variables on the configure page; "log-opt" with a value of max-size=10m and "env" with a value of GITLAB_OMNIBUS_CONFIG="external_url 'https://git.MYSITE.COM/'; postgresql['shared_buffers'] = '256MB'; sidekiq['concurrency'] = 15; prometheus_monitoring['enable'] = false;" So, question... should I use the IP address of my unraid server or the actual url I am expecting to work? and do I include the https if I am using the url? Didn't seem to respond to that. I am still getting 500 errors on imports and other weird things. Of note, the gitlab.rb didn't seem to change either. Edit 2: Another thing I noticed is the logs are going wild. I get about 40 lines every 3 to 4 seconds. Mostly puma, sidekiq and gitaly. It's going so fast that I can't even see if my requests are hitting the logs. (but it doesn't appear to be)

I am using this Gitlab with Hotio's Caddy2 (for serving) and ddclient (for ddns). I have many other projects that work fine, but for some reason, Gitlab will not accept the external_url. I change the external_url to 'https://git.SITE.COM' and run gitlab-ctl reconfigure. When I try to go to the external site, I get a blank page. If I comment it out, everything appears to work fine until I start using Gitlab... I will get 500 errors anytime I try to import a project and when I go to clone projects, The address is localhost instead of SITE.COM. I am obviously improperly configuring Gitlab, but I am not sure what I am doing wrong. Any assistance would be greatly appreciated. - Caddy2 Config - git.SITE.COM { reverse_proxy 192.168.1.7:9080 encode gzip log { output file /config/log/git-SITE.log } } I've left all Gitlab configs default.

Now that this feature is gone, how do we install docker images without templates?

Where can I find the plugin? I didn't realize it was in Community Applications. I got it installed. Thanks!

I am trying to attach two USB thumb drives to a VM at the same time but I am receiving an error about multiple USB devices being present and that I have to choose one. I want to choose both. How can I attach two USB thumb drives to a VM at the same time? Is this possible? Here is the error I am getting: `operation failed: Multiple USB devices for 781:5575, use <address> to specify one`

I now have about 15 docker containers and a few VMs running on various different ports and IPs. I am using Heimdall (and might switch to Muximux) to access all of the sites locally which is fantastic. The only problem I have is that I’d like to access these containers from the outside without having to port forward each container. Right now, when I try to access these containers through Heimdall or Muximux from outside of my network, I get a 404 because it’s trying to access a local IP that isn’t there. So, Is there a web tool like Heimdall or Muximux that would serve up the internal site to an external request allowing me to access all containers through a single source (Muximux or Heimdall-like web interface).

Okay, so I am not sure what I did. About 2 months ago, everything was working fine and now everything seems to have gone to hell. I have an Nvidia GTX 980 that I was successfully passing through to a Win10 VM. No issues. I could reboot the VM multiple times and no problems. Now, I reboot the VM once, I get an error 127. Furthermore, the VM doesn't start when the Nvidia card is passed through. If I switch to VNC, the VM starts fine. AND... every time I reboot Unraid, the address of the Nvidia card changes. REALLY annoying. VT-d is enabled in the bios. I have tried using the rom file (The one I dumped and the downloaded one, both with the same results). I have tried turning the card into a multi-function card in XML. Here is the logs when I start with passthrough: -display none \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=31,server,nowait \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=localtime \ -no-hpet \ -no-shutdown \ -boot strict=on \ -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \ -device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \ -device pcie-root-port,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 \ -device pcie-root-port,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x1.0x3 \ -device pcie-root-port,port=0xc,chassis=5,id=pci.5,bus=pcie.0,addr=0x1.0x4 \ -device pcie-pci-bridge,id=pci.6,bus=pci.1,addr=0x0 \ -device pcie-root-port,port=0xd,chassis=7,id=pci.7,bus=pcie.0,addr=0x1.0x5 \ -device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x7.0x7 \ -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x7 \ -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x7.0x1 \ -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x7.0x2 \ -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \ -blockdev '{"driver":"file","filename":"/mnt/cache/domains/Virtual Gaming OS/vdisk1.img","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \ -device virtio-blk-pci,bus=pci.4,addr=0x0,drive=libvirt-1-format,id=virtio-disk2,bootindex=1,write-cache=on \ -netdev tap,fd=33,id=hostnet0 \ -device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:cf:cc:15,bus=pci.3,addr=0x0 \ -chardev pty,id=charserial0 \ -device isa-serial,chardev=charserial0,id=serial0 \ -chardev socket,id=charchannel0,fd=34,server,nowait \ -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \ -device usb-tablet,id=input0,bus=usb.0,port=1 \ -device vfio-pci,host=0000:02:00.0,id=hostdev0,bus=pci.5,addr=0x0,romfile=/mnt/user/storage/storage/files/dump.rom \ -device vfio-pci,host=0000:02:00.1,id=hostdev1,bus=pci.7,addr=0x0 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on 2021-06-19 21:47:51.396+0000: Domain id=2 is tainted: high-privileges 2021-06-19 21:47:51.396+0000: Domain id=2 is tainted: host-cpu char device redirected to /dev/pts/0 (label charserial0) 2021-06-19T21:47:54.476697Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3 2021-06-19T21:47:54.487652Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3 I don't know where to go from here or what to do. I have wiped the Unraid load and rebuilt the VMs, all with crap results that leave me with a broken system where passthrough doesn't work anymore. Any suggestions or help would be greatly appreciated. Thanks.

Does Unraid support using LXC / Linux Containers or is it just Docker containers?