Ollaya: Run open-source decision models locally
Original: Ollaya – Ollama for open-source, Jev-style decision models
Why This Matters
Local, sub-10ms decision inference could replace hosted classification APIs in latency-sensitive pipelines.
Ollaya is an open-source local runtime for 'decision models' — small, specialized models that answer typed yes/no, choice, and score questions in a single forward pass. Compatible with TypeSafe's API, it runs on your own hardware in as little as 8 ms per five-question request on an RTX 4090.
Ollaya positions itself as the Ollama equivalent for decision models: a local server you point TypeSafe's official Python SDK at by swapping one environment variable. It serves the /v1/systemone and /v1/models endpoints with TypeSafe-compatible request and response shapes, so existing code works unchanged.
The key performance claim is latency. Decision models skip token-by-token generation entirely — answers come from a single forward pass, reading probabilities directly from logits. On an RTX 4090, the flagship 'laya' model answers a five-question HTTP request in 8–10 ms end-to-end. That compares favorably to the TypeSafe hosted API's measured median of 236–276 ms, though Ollaya notes those figures include network time, so it frames the gap as 'order-of-magnitude' rather than apples-to-apples.
The model catalog ships out of the box with several open-weight options: Laya (Convai Innovations, 322m–421m params, 100+ languages), Decider (Mapika on Qwen3.5, 0.75b–1.9b), NLI zero-shot classifiers (Moritz Laurer, ~400m), GLiClass (Knowledgator, 439m), Qwen3Guard (Qwen team, 0.6b, safety classification in 119 languages), and Kev (Jared Palmer, LoRA on Qwen3.5). Data never leaves your hardware.