SuspiciousCarrot78@aussie.zone to Selfhosted@lemmy.worldEnglish · 11 days agoDo you host your own AI?message-squaremessage-square185linkfedilinkarrow-up10file-text
arrow-up10message-squareDo you host your own AI?SuspiciousCarrot78@aussie.zone to Selfhosted@lemmy.worldEnglish · 11 days agomessage-square185linkfedilinkfile-text
minus-squarehexagonwin@lemmy.todaylinkfedilinkEnglisharrow-up0·10 days agoa haswell xeon e5-1650 machine, i remember running llama 7b in llama.cpp in like 2023 and it was quite sluggish. guess i should try whisper at some point…
minus-squareSuspiciousCarrot78@aussie.zoneOPlinkfedilinkEnglisharrow-up0·10 days agoHa. You were doing inference on CPU on a haswell era. Been there, done that. OTOH…whisper.cpp is heavily optimised for it. Plus, you’re doing batch transcription, not real-time, so slow doesn’t actually matter. Fire Whisper small or medium overnight and wake up to searchable text. PS: if you want a good fast little llm, something like Qwen 3.5 2B will work well on the Xeon.
a haswell xeon e5-1650 machine, i remember running llama 7b in llama.cpp in like 2023 and it was quite sluggish. guess i should try whisper at some point…
Ha. You were doing inference on CPU on a haswell era. Been there, done that.
OTOH…whisper.cpp is heavily optimised for it.
Plus, you’re doing batch transcription, not real-time, so slow doesn’t actually matter.
Fire Whisper small or medium overnight and wake up to searchable text.
PS: if you want a good fast little llm, something like Qwen 3.5 2B will work well on the Xeon.