Kimi K3 (2.78T params) runs on 29GB RAM via NVMe streaming
Original: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
Why This Matters
Enables trillion-parameter model inference on consumer laptops, dramatically lowering the hardware barrier for large-scale AI.
Open-source project WASTE enables the 2.78-trillion-parameter Kimi K3 model to run on a consumer laptop with just 29GB RAM by streaming activated weights directly from NVMe storage, achieving approximately 0.50 tokens per second with no third-party runtime dependencies.
GitHub user sqliteai has released WASTE (Weight-Aware Streaming Tensor Engine), a dependency-free C inference engine that allows the 2.78-trillion-parameter Kimi K3 model to run on consumer hardware with as little as 29GB of RAM. The engine works by streaming activated weights directly from NVMe storage rather than loading the entire model into memory. In a demonstration, the tool ran K3 on a 64GB laptop using 46.24GB of available memory (including a 17.56GB expert cache), answering a simple query in 31.09 seconds at 0.51 tokens per second. The expert cache hit rate was 14% (3,357 hits vs. 20,195 misses), illustrating the trade-off between memory usage and inference speed. Written entirely in C with no third-party runtime dependencies, WASTE is designed to be embeddable and portable. The project is licensed under Apache-2.0 and has already attracted over 370 GitHub stars since its release. The approach represents a practical method for running frontier-scale models on commodity hardware without requiring expensive multi-GPU server setups.