Mesh LLM: Distributed AI Inference via iroh Network
Original: Mesh LLM: distributed AI computing on iroh
Why This Matters
Mesh LLM enables private, cost-controlled LLM inference across distributed commodity hardware without cloud dependency.
Iroh has launched Mesh LLM, a distributed AI inference system that pools GPUs across multiple machines and exposes them as a single OpenAI-compatible API at localhost:9337/v1. It supports 40+ models up to 235B parameters using a pipeline-split architecture called 'Skippy.'
Iroh has introduced Mesh LLM, a distributed AI computing framework built on its iroh peer-to-peer networking stack. The system allows teams to pool existing GPUs — whether in offices, closets, or under desks — and treat them as a unified inference cluster, exposed via an OpenAI-compatible API endpoint at http://localhost:9337/v1.
Requests can be handled three ways: locally on the requesting machine's GPU, routed to a peer that already has the target model loaded, or split across multiple machines using a pipeline mode internally called 'Skippy.' In Skippy mode, a model is partitioned by layer ranges across nodes — for example, layers 0–15 on one machine and 16–31 on the next — allowing large models to run on hardware that individually could not support them.
The model catalog includes 40+ options ranging from sub-billion-parameter models suited for laptops to 235B mixture-of-experts models. The architecture is plugin-based: plugins declare capabilities via a manifest, and the runtime routes calls across MCP, HTTP, inference, and mesh event channels.
Each node boots an iroh endpoint, which serves as its cryptographic identity (public key) and network surface. There is no central server. iroh handles NAT traversal, hole-punching, and relay fallback to establish direct, authenticated QUIC connections between peers. Mesh LLM operates two iroh relay servers in different regions for fallback connectivity. Three ALPN protocols handle mesh gossip and routing, control-plane configuration sync, and Skippy pipeline data transfer respectively.