Do you host your own ML / AI / LLM? What do you use, and what do you use it for?
You must log in or # to comment.
Nope.
No I don’t. Unforunetly using Claude (asking myself everyday why tf cuz I don’t do crazy shit) but trying to move on to LumoAI even meaby will buy a premium version to check this out formyself.
I’m running dwarfstar which is a 2 bit deepseek v4 flash. It’s quite capable even at 2 bit.
I have the setup, never found a use for it though.
Yes, I got a Strix Halo machine before the RAM price hike and use it to run all my ML stuff on it.
Currently using llama-swap with llama.cpp/ComfyUI and opencode/Open WebUI as frontend.
I’m running Qwen3.6-27b, Voxtral Mini 4b, Piper and Qwen Image. Also, some embedding and reranking models.
I use them for:
- Tagging and classification of my documents in Paperless
- Home Assistant (voice assistant)
- Translations (both text and image)
- Transcriptions
- Some light coding and debugging
- Avatar/Backdrop generation for DnD sessions
What sort of tok/s are you getting on the strix?
About 200 t/s prompt processing and 10-20 t/s with MTP.
Greatly depends on the task, predictable things like code generates at 18-20 t/s. Creative writing more like 10-17 t/s.
Damn - I thought strix would do a bit better than that, for how much it costs.
Given the 27b is a dense model, I think the numbers are quite ok. Curious about the quant tho.
The cool thing about the strix is its large unified memory, but it lacks memory bandwith for compute intensive workloads. Something like Qwen3.5-122b MoE with only like 12b active parameters might run at twice the speed if it fits the configuration.
Curious about the quant tho.
Q8 from unsloth.
Something like Qwen3.5-122b
My go to model for knowledge. Definitely much faster at Q5 but it lacks the tool calling quality of the Qwen3.6 models. Really hoping we see a Qwen3.6-122b soon…
Yeah. Though I think theres a new strix out soon (Medusa? Gorgon? Something like that).
Its a bit like my P40. On paper, it has 24GB. But that 24gb is capped at 400GB/s and the ai compute is what…Pascal era?
AI = Good, fast, cheap - pick 2
Found vLLM to be the most efficient local runtime service. And “ray” as a good (but complicated) way to distribute the load: https://docs.ray.io/
I’ve tried a few times but with only 8gig of vram it’s simply not worth it.
Have you tried qwen3.5-9b? It’s pretty solid for its size.
Yeah, it’s “good for its size” but it’s just too flaky for me to use for any significant coding.
Yeah, I wouldn’t use it for coding. It’s a bit dumb unfortunately.
How much CPU RAM do you have?
64G. But CPU inference is painfully slow.
Not anymore. Not with hybrid offloading, where the GPU handles dense tensors and the CPU only runs the sparse MoEs. I’m running a 300B model on a single 3090, and its faster than I can read.
You just need to use the right framework, and the right model.
I’d suggest trying ik_llama.cpp and a MoE like this: https://huggingface.co/sokann/Qwen3.6-27B-GGUF-5.076bpw
And speculative decoding like DFlash or MTP (which you can also get specific models for).
I’ll check that out - speed isn’t my biggest issue so much as coding performance… The qwen 3.5 model I was using can write code, but it’s… Meh? Like sometimes it doesn’t even compile.
I did try tweaking llama.cpp to do some cpu offloading and it does seem to allow for much larger contexts at a modest performance loss. I’ll check out larger models.
CPU offloading is too slow unless you use a hybrid MoE model, with the --n-cpu-moe parameter, specifically.
This only offloads “sparse” parts of the model to the CPU, which take up a lot of RAM but are very compute-lite to run. In practice, thats most of the size of modern MoE LLMs.
Since implementation of the
--fitparameter and its relatives, and--fit onbecoming the default, llama.cpp intelligently decides what to offload. For me, it made--n-cpu-moeobsolete.Mostly, yeah.
Sometimes it’s better to “cut it close,” with (for instance) a 27B model that’s nearly OOMing your VRAM fully offloaded, but you know will be fine in regular use without too many programs open.
In my case, with MiMo 2.5, it fills both my CPU and GPU RAM rather completely, so it’s best to set a static value so I don’t swap CPU RAM, and don’t OOM on the GPU either.
Try qwen3.6-35b-a3b with a lightweight harness like pi.dev
Having it be able to run commands and try to compile or run the code and see the output helps especially on the “doesn’t compile” part of things
Yeah - I’ve been playing around more with the Qwen3-Coder-30B-A3B-Instruct MoE model and it’s still quite… Meh. I’ve been using llama.cpp and I’ve tried a bunch of tuning. It works and performs well enough (15t/s) but the output is just garbage. I can do some simple coding but I’m finding I’m fighting with it more than if I just wrote the code myself. Maybe I just have standards that are too high. Claude Opus 3.7 is just in an entirely different league…
When you run it, do you use unsloth’s recommended settings for coding?
https://unsloth.ai/docs/models/qwen3.6
Also have preserve thinking on, it helps it stay consistent in multi turn work.
Which model version you’re using can also affect results, usually unsloth’s ones are good.
With all that said, it’s of course a small model so it’s not a super coder. The 27b is better (I’d guess 25-35% better), but of course still a small model so…
So it’ll maybe not be good enough still, but should give it the chance to let it do the best it can :)
Ollama with gemma 4 for LLM stuff, coding brainstorming, etc.
Comfy ui with z-image or stable diffusion for images.
How is Gemma compared to others?
I’ve got ollama setup with whisper and piper and a HA voice PE, but I honestly haven’t gotten around to configuring much yet. Most notable thing was being able to use the wake word to start a timer, but it was pickier than old Siri about the precise wording.
I run Handy with Parakeet for speech to text, and home assistant with Whiper for the same. Whisper+ on my phone.
I think that counts. But I have more relevant and useful things to do on my hardware and no 2000€+ to get LLM-capable hardware 😂
I do, I use ollama. I mostly just tinker, but I use with with home assistant for a quasi Alexa like experience with the voice assistant, I use it for summarizing some YouTube transcripts in too lazy to read/watch, and I’ve tried to see how capable it is with coding.
Can you elaborate on what you are using exactly with home assistant ? And is English your primary language in that context ?
Trying to do something similar, English not primary and its a bit… Harder than it seems. Can’t figure out if it is because I’m not using English or something else. (3060 12GB BTW)
English is my primary, so that does make it easier. I use it for general conversion things, like asking it questions about the Titanic or making up a new story or something. It doesn’t work as well as I’d like yet, but like I said, it’s just an other thing for me to mess around with and change.
Yes. Openwebui/ollama for LLM, comfyui for stable diffusion. I just dick around with it as a toy.
I was put off by ComfyUI, seems awfully complex. How is your experience?
Any suggestions to start? I have Fooocus installed now
Same. Its somewhat useful on some very small scripting or tasks…but its mostly just to try out a new model or two. Its not really useful for anything big.
I will have to say…even my tiny models are about as good as Chatgpt/Claude/etc… which makes me think about how much people are spending on tokens regularly. I was able to get the same kind of python script started with my local tiny model that was comparable to the newest Claude code offerings.
What local models have you been using? And what hardware are you running them on? I’ve been playing with local LLMs a bit for exactly your use case.
I have zero interest in vibe coding or full agentic workflows. But having a local LLM generate a Bash script to help me automate parts of my home lab infrastructure would be nice.
What are your hardware specs?
Ryzen 7 5800 X3D Radeon RX 9070XT 32GB of DDR4 system memory.
How hard does it push this setup? How far can you scale up your own models on this hardware?
I was hosting LLM with LMStudio occasionally but can’t access it anymore due to some fuckery with CORS and http vs https in browsers.
I googled it, and it seems like you can just enable cors.
Yes you can enable cors in LMStudio. But since few months it’s blocked by all major web browsers if you aren’t using HTTPS.
Which I don’t. I had LMStudio server open to local network so I can use it on my phone or laptop via third party website.
Why would it be blocked? I can use http sites just fine, and even then you could setup a self-signed certificate.
I don’t know why. It worked half a year ago, now it doesn’t. All I know I can access server via local host on same machine, but if I try to access it over LAN it just doesn’t work no matter what I do. I might look into self signed cert but it feels like to much work for now.
Seems this has been rectified. At least in Brave - it asks for permission to access local network at the first try, so this is now usable again
I recently gave it a try with qwen3.5 and deepseek coder v2. I have a RTX3090 and these are the largest models that can run comfortably on it.
Conclusion, they are both fucking useless. Free tier claude runs circles.
If you just pulled the default version of qwen3.5 from ollama’s repo you downloaded a mediocre one that only uses ~6GB.
Check
ollama show qwen3.5and see if you get something like this in the result:Model architecture qwen35 parameters 9.7B context length 262144 embedding length 4096 quantization Q4_K_MThis is the default version I got when I first tried using ollama without any experience. It worked, but it’s a heavily quantized, lower parameter version of the model – i.e. it’s pretty dumb – compared to what you can actually run on your hardware.
I will check it later. I loaded whichever one cluade suggested lol
Did you serve them with ollama?
It’s basically broken, if you did. Try the same models over API, and you’ll see what I mean.
Is there an alternative to ollama? The point was to run something locally.
Oh, and I just saw you have a 3090.
To get more specific, you can actually run way better models than Qwen 3.5 and Deepseek coder (both of which are very obsolete now). The best that’s practical depends on how much CPU RAM you have, but at the minimum you can do Qwen 3.6 27B, with a more optimal quant like ones here: https://huggingface.co/ubergarm/Qwen3.6-27B-GGUF/tree/main
Or Gemma 31B QAT: https://huggingface.co/unsloth/gemma-4-31B-it-qat-GGUF
If you have 128GB CPU RAM, I can upload my custom MiMo 2.5 quant. That should “beat” the cheapest Claude, give or take.
If you have 64GB, I’d suggest a quantization of Step 3.7.
If you have 32GB or 48, I’m not sure. I’d need to look if any “small” MoE is actually better than Qwen 27B now.
https://sleepingrobots.com/dreams/stop-using-ollama/
And that’s not even all of it. Basically they break models in many ways, and they’re slimey Tech Bros.
LM Studio is better, and easy.
If you’re on Nvidia, and want to run optimally, I would use the ik_llama.cpp fork. On AMD, regular llama.cpp. On a Mac, use an MLX runner (Like LM Studio) with an MLX quant (ideally an MLX-DWQ quant).
It’s all pretty technical, and… thats kinda the point. LLMs are just too performance sensitive and too finicky to not have a grasp of how they work. There is no “easy button” to run them without bad results, there can’t be.
But if you don’t have time for that and just want to see if it’s worth it, I’d suggest self hosing your own UI, and trying the dirt cheap APIs of models you can theoretically run on your setup. This will give you a “best case” taste of what they’re capable of.
Yeah :(
Were not there yet on consumer rigs.
Yes. My Actual Intelligence lives in my head, and runs mostly on coffee.
deleted by creator
With sufficient coffee, mine shows considerable artifice.
Plastic flowers.
I’ll make sure to send you flowers, Algernon lol
@SuspiciousCarrot78@aussie.zone this comment is not (directly) for you, I just want it in context.
Before you report someone for breaking rule 1, please look a the context. Specifically, the username someone may be replying to.
LOL.
https://en.wikipedia.org/wiki/Flowers_for_Algernon
Looks like someone got big mad over a harmless joke. You love to see it. Sorry they wasted your time.
Eh, its fine. Certainly better than the “I don’t like this so I’m going to report it” approach.
critical security bug: if coffee is taken away my head hurts :(
Do you get many hallucinations?
Only when I’m deprived of coffee.
Would flowers work instead?
As we know AI stands for “An Indian”, so if you’re not from India, its actually impossible to self host.
Well, unless you manage to trap one in your basement, but that would violate human rights and hopefully also break the laws of your country.
You may be confusing Indians with gremlins. Which might explain ChatGPTs obsession with gremlins
Just coffee?!? That’s cool.
Mine runs on:
- coffee
- spite
- tortilla chips
- & shame
If that’s not already on a shirt it should be
Mostly on coffee, not exclusively. Noticable amounts of spite & tortilla chips are also present, yes, but… no shame.
Nice!
