Slotstream
A Swift-based inference engine that enables running the 125-billion parameter Qwen 3.8-Flash-Next model on Apple Silicon Macs with limited memory by streaming expert weights from storage. Achieves approximately 12 tokens/second on 48GB M-series Macs.
🔗 Visit Slotstream
Description
Big AI language models are usually too large to fit in a normal computer's memory, which is why most people use them through a cloud service instead of running them at home. Slotstream is a small piece of software that gets around this on Apple laptops: instead of trying to cram the whole model into memory at once, it streams the pieces it needs straight from the hard drive, in and out, as the conversation goes.
Slotstream is a Swift-based inference engine, distributed as a single dependency-free binary, that runs the 125-billion-parameter Qwen3.8-Flash-Next mixture-of-experts model on Apple Silicon Macs by streaming individual experts from SSD storage into a managed memory cache instead of requiring the full weights resident in RAM. On a 48GB M-series Mac it reaches roughly 12 tokens/second, and it exposes both Ollama- and OpenAI-compatible chat APIs so existing tooling can point at it without changes. Other features include adaptive memory sizing based on available RAM, SHA256 weight verification, and speculative decoding for extra throughput.
💬 Our review
The short version: Slotstream doesn't make the model smaller (that's what quantization tools do) — it makes a model that wouldn't fit at all fit, by never loading more of it into memory than the current token needs.
Tools like llama.cpp or MLX quantization shrink a model's precision to make it fit in RAM, which trades away some accuracy. Slotstream's expert-streaming approach is a different tradeoff: it keeps the full-precision expert weights and instead pays a speed cost, reading from SSD as it goes, which is why 12 tokens/second on a 48GB Mac is respectable but not fast next to a quantized model that fits fully in memory. It's a young, single-maintainer project with no commercial backing, so treat it as a serious engineering experiment rather than a polished product — but if your goal is running a genuinely large open-weight model locally for privacy or cost reasons rather than speed, it's one of the few ways to do that on consumer Apple hardware without a rack of GPUs.
💰 Pricing
📊 Global score
🤖 AI-enriched data
Gratuit, licence MIT, binaire Swift installable via script shell, aucune offre commerciale.
Pros
Fait tourner un modèle 125B paramètres sur un Mac 48GB, impossible avec la quantization seule
Binaire Swift unique, zéro dépendance Python
Compatible API Ollama et OpenAI, s'intègre aux outils existants
Vérification SHA256 des poids, gestion mémoire adaptative
Cons
Débit modeste (~12 tokens/s), pas adapté à un usage temps réel exigeant
Projet jeune, mono-mainteneur, pas de support commercial
Dépend fortement de la vitesse du SSD et du modèle Mac utilisé