You must log in or # to comment.
How about a nice round of global thermonuclear war?
Which part is selfhosted?
all of it. I’m not using anything outside of my own local area network.
Please share the full stack with us
I use the Kaico DSpico Flashcart and on the RPi Zero 2W I use a bash script (because I don’t want to host it permanently) with hostapd, iw, dnsmasq, socat, and iproute2. For HTTP and JSON I use my own implementation, no library. The rest is documented in the post.
Did some experiments with my DS Lite and ended up with this. Using DevkitPro libs (libnds, dswifi, calico). Apfel on macOS and a wifi (WEP) bridge running on a RPi Zero 2W. 🖤
Which LLM are you running and what are you running it on?
This is the model:
{ "data" : [ { "context_window" : 4096, "created" : 1719792000, "id" : "apple-foundationmodel", "notes" : "Apple on-device model via FoundationModels framework. Unsupported parameters are rejected with 400 when present (except n=1 and logprobs=false). Supported languages: it, zh, fr, en, nb, fr, da, es, es, es, vi, zh, de, pt, tr, en, zh, sv, nl, ja, en, ko, pt", "object" : "model", "owned_by" : "apple", "supported_parameters" : [ "temperature", "max_tokens", "seed", "stream", "tools", "tool_choice", "response_format", "x_context_strategy", "x_context_max_turns", "x_context_output_reserve" ], "unsupported_parameters" : [ "logprobs", "n", "stop", "presence_penalty", "frequency_penalty" ] } ], "object" : "list" }and I’m running it on a MacBook Air M4 16 GB. “Apfel” (the CLI tool) is kinda a wrapper around it and exposes an OpenAI compatible API.
Do I understand correctly that what you’ve made is a DSLite client to chat with the apple included model on your laptop?
Exactly, yes. And I use the Raspberry Pi Zero 2W as a WiFi AP (RPi is connected to my home network and to my DS Lite) because the Nintendo DS Lite only supports WEP encryption. So that the DS Lite is able able to send requests to my inference on the MacBook Air
Doesn’t the DS… just have a browser? Meaning you’re just pointing at a web page here?
The DSi had a built-in browser, the DS and DS Lite require a game card plus a memory expansion pack to use a browser. Even at the time, website support was spotty at best so I dread to think how a browser based on Opera 8.5 from 2006 would work on the modern web
Yes but this isn’t a full browser, just an API client.
So, even more extra steps for even less functionality, got it.
Out of topic, but I’m unreasonably happy that you use the more exact word “LLM” instead of AI. Cool project!
Thank you, you know these days small things make me happy
Client on DS Lite. LLM on Mac.
Yeah, that’s not impressive at all compared to what I thought. It’s not much different than playing Crysis on a DS by just using remote desktop.
No actually, that would be impressive. The DS is barely capable of decoding video. Getting that to work in real time, over the network, and low latency is a huge technical achievement.
This is just a text input and output connected to a remote machine.
Maybe not the perfect example, but then again, getting Crysis to run natively on a DS would be a significantly larger achievement than running a video streamer.
Uhm, I figured out that people don’t like it. However, for me, it was a nice little project and it brightened my day.
Yeah, it was definitely the title. It’s a neat project in of itself, but the title made it seem like you accomplished a frankly massive feat.
Videos like this one where they make absolutely tiny llm models for low power hardware are what I had in mind. Sadly I can’t find the other one that I watched several months back.
That’s really cool!
Pretty sure people don’t like the misleading title.
I changed the title
amazing
