MoE-Direct
An open-source tool that lets you run huge AI language models on an ordinary gaming PC by streaming the parts it needs straight from your SSD instead of cramming everything into RAM.
🔗 Visit MoE-Direct
Description
Big AI language models are usually too large to fit in a normal computer's memory, which is why running a top-tier model at home has meant either buying very expensive hardware or settling for a smaller, weaker model. MoE-Direct works around that for a specific family of models — the kind built from many specialized "expert" sub-models where only a few experts actually answer any given question. Instead of loading every expert into RAM just in case, it fetches only the ones needed from your SSD in real time, which means a model that would normally need hundreds of gigabytes of memory can run on a regular desktop with a decent gaming graphics card.
Technically, MoE-Direct is a Windows-only, open-source layer on top of llama.cpp that streams Mixture-of-Experts weights directly from NVMe storage rather than fully materializing them in RAM, exploiting the sparse per-token activation pattern inherent to MoE architectures. Models must first go through a one-time offline repacking step (5-18 minutes) that reorganizes weights for streaming access without any re-quantization, and SHA-256 verification confirms byte-exact integrity against the reference model. The project reports working performance on models such as Qwen3.5-122B (5.59-5.69 tokens/sec) and Kimi K2.6 (1.03 tokens/sec on a 32GB-RAM machine), requiring an NVIDIA GPU with CUDA and NVMe free space of roughly 1-2x the model size.
💬 Our review
The short version: MoE-Direct is a clever, narrow-purpose open-source tool for one specific problem — running a huge Mixture-of-Experts model on a desktop that doesn't have server-grade RAM — and it's free, but it's brand new, Windows-only, and the speeds it reports (roughly 1-6 tokens/sec depending on the model) are usable for experimentation, not production workloads.
Compared to the mainstream local-inference stack — llama.cpp itself, Ollama, or LM Studio — MoE-Direct doesn't replace them so much as extend what's possible on modest hardware for a specific model class: it's the only one of the bunch built specifically around NVMe-streaming of MoE experts to avoid RAM ceilings, rather than relying on quantization or partial GPU offload alone. That specialization is also its limitation: it needs an NVIDIA/CUDA GPU and a fast NVMe drive with real free space, it's Windows-only today, and as a very fresh (Show HN, August 2026) single-maintainer project, it lacks the polish, community testing, and cross-platform support of llama.cpp or Ollama. If your actual goal is just running a mainstream 7B-70B model reasonably fast, Ollama or LM Studio remain simpler and better supported. But if you specifically want to try a 100B+-parameter MoE model on a 32-64GB RAM desktop and are comfortable with GitHub-README-level documentation, MoE-Direct is a genuinely novel, zero-cost way to do it that nothing else in this space currently offers.
💰 Pricing
📊 Global score
🤖 AI-enriched data
Free, MIT License. No paid tiers. Built on llama.cpp (also MIT). Only cost is your own hardware: an NVIDIA CUDA GPU and NVMe SSD space equal to roughly 1-2x the model size.
Pros
Runs 100B+ parameter MoE models on desktops with as little as 32GB RAM
Free and open-source (MIT), built on the well-established llama.cpp
Byte-exact weight integrity — no re-quantization, SHA-256 verified
Validated against real large models: Qwen3.5-122B, Kimi K2.6, DeepSeek-V4-Flash
Cons
Windows-only — no macOS or Linux support
Requires an NVIDIA CUDA GPU and a fast NVMe drive with real free space
Very new project (launched August 2026), minimal community track record
Reported speeds (roughly 1-6 tokens/sec) are workable for testing, not production use