I’ve played with it for Home Assistant integration, but I just dont have much interest in it, the whole thing is too inefficient at the moment, and the tiny models that can run in a few gigs of system ram on an ipgu or npu arent good enough in quality or speed to rely on.
Hopefully some future generation micro-models will be more useful for the way I want to use it (aka , ultra light, no dedicated hardware etc.), but for now it’s a lot of compute resources, plus heat and energy for a gimmick.
Agreed. It will be ironic if 1.58B models (Microsoft) turns out to be the great white hope.
I looked at the recent Steam stats (which is a GPU sample of convenience); the most common GPU size was 6GB. Meanwhile you probably need what…64GB unified memory or a 5090 to drive a decent model at a decent speed/context?
There’s a real gap between the haves and the have nots and it’s widening.
I’ve played with it for Home Assistant integration, but I just dont have much interest in it, the whole thing is too inefficient at the moment, and the tiny models that can run in a few gigs of system ram on an ipgu or npu arent good enough in quality or speed to rely on.
Hopefully some future generation micro-models will be more useful for the way I want to use it (aka , ultra light, no dedicated hardware etc.), but for now it’s a lot of compute resources, plus heat and energy for a gimmick.
Agreed. It will be ironic if 1.58B models (Microsoft) turns out to be the great white hope.
I looked at the recent Steam stats (which is a GPU sample of convenience); the most common GPU size was 6GB. Meanwhile you probably need what…64GB unified memory or a 5090 to drive a decent model at a decent speed/context?
There’s a real gap between the haves and the have nots and it’s widening.