Cognitora
Cognitora · Coding
Cognitora is an open-source inference orchestration platform that coordinates multiple large language model engines into one distributed cluster. Instead of replacing your engine, it sits above vLLM, SGLang, llama.cpp, MLX and any OpenAI-compatible server, then routes requests to the nodes where the right KV cache already lives. It targets teams that run their own GPUs, from a couple of machines on a desk to a mixed datacenter fleet, and it ships as six static binaries with a single curl install.
About Cognitora
What Is Cognitora
Cognitora is a self-hosted control plane for LLM inference orchestration. You keep whatever engine you already trust and let Cognitora handle the hard parts: routing, cache placement, health, power and observability across many machines. The project is written in Rust and positions itself as a drop-in alternative to orchestrators like NVIDIA Dynamo, with wider engine coverage and a bare-metal-first deployment story. Six binaries. One control plane.
The main problem it solves is fragmentation. A fleet usually ends up running different engines on different hardware, and a plain load balancer sends requests to random nodes even when one machine already holds the relevant cache. Cognitora adds KV-aware routing so requests land where the memory lives. That cuts redundant prefill work and keeps tail latency in check. It also tracks energy per token, a detail most orchestrators ignore.
Think of it as a vLLM orchestrator that never locks you into vLLM. The biggest limitation to plan around is scale. Cognitora is honest that it leads on heterogeneous, energy-aware, ops-light inference rather than enormous NVL72-class deployments, and some advanced tiered-cache and topology features are still on the roadmap. If you need in-flight request migration or GPU-to-GPU weight streaming today, a heavier stack may fit better. For everyone else, the lightweight footprint is the point.
Getting Started
- Install the binaries with a single curl command on Linux, or build from source with cargo if you're on macOS.
- Bootstrap local PKI files with
cgn-ctl pki bootstrap, then decide whether to require mTLS. - Issue an API key with scoped permissions using
cgn-ctl key create. - Write a short
cognitora.tomlthat names the cluster, points at your models and sets the router's HTTP and gRPC ports. - Boot
cgn-router, point your engine driver at a real backend, and query it with the OpenAI SDK to confirm tokens stream end to end.
Product Information
A quick look at Cognitora's pricing, supported platforms, and performance.
Best for
The users, tasks, and scenarios where this tool fits best.
Users
- ML platform engineers
- Startup infrastructure teams
- Research labs with mixed hardware
Tasks
- Running one model across several engines
- Cutting prefill costs
- Tracking power per token
- Standing up a quick demo cluster
Scenarios
- A 16-node mixed fleet under load
- Desk-scale experiments
- Production Kubernetes deployments
- Prefill/decode separation
Key features
Engine-Agnostic Orchestration
Cognitora doesn't replace your inference engine, it coordinates several of them into one cluster. Any backend exposing the OpenAI HTTP surface can join, and the project ships drivers for vLLM, SGLang, llama.cpp, MLX and TensorRT-LLM. Mixed hardware is normal. That means you can run an NVIDIA node on vLLM next to a Mac Studio on MLX and still treat them as one fleet.
KV-Aware Routing
The router tracks where KV caches live and places each request accordingly. If a node already holds the context for a repeated prompt, the request goes there instead of triggering a fresh prefill elsewhere. That's the whole trick. Why does it matter? For chat and agent workloads with long shared system prompts, this is the feature that most directly reduces wasted compute. It adds up fast.
Prefill/Decode Disaggregation
Cognitora is built around separating prefill and decode onto different pools. You can dedicate some machines to processing prompts and others to generating tokens, then let the router and KV cache move work between them. Split the phases. The project also supports multi-tier KV storage across RAM and SSD for larger caches.
Bare-Metal-First Deployment
The whole system installs as six static binaries with a single curl line and runs under systemd, with no Python control plane in the loop. That's the bare-metal LLM cluster story in one sentence. You only add the Kubernetes operator when you actually want orchestration at the pod level, so a couple of desktop machines are a legitimate target.
Zero-Dependency Fleet Dashboard
A standalone dashboard ships in the repo as one static HTML file that reads Prometheus /metrics endpoints directly. It shows live requests per second, tokens per second, latency and TTFT percentiles, queue depth, per-node status, KV-cache utilization, fleet power and energy per token. No Grafana. No backend.
Energy-Aware Scheduling
Cognitora factors power into placement decisions and reports energy per token for the whole fleet. On a mixed set of GPUs, that lets an operator see which nodes deliver the most tokens per watt and steer work toward them. Power is money. It matters when you pay the electricity bill yourself.
Open-Source and Inspectable
The project lives on GitHub under an active repository, so every routing decision, driver and configuration option is open to review. It works as a fully open-source inference server. Teams that can't send inference traffic through a closed third-party service can read the code, fork it and run the entire control plane as a self-hosted AI fleet.
Pros and cons
Pros
- Engine-agnostic design lets you keep existing vLLM, SGLang or llama.cpp setups instead of rewriting them.
- KV-aware routing reduces redundant prefill work, which shows up as lower cost and steadier tail latency.
- Bare-metal-first install means a two-machine setup is realistic, not a degraded demo of a Kubernetes product.
- Built-in dashboard covers latency, tokens per second, KV utilization and energy per token without extra tooling.
- Written in Rust with no Python control plane, so the runtime footprint stays small.
Cons
- Some advanced features like native tiered block managers and topology-aware gang scheduling aren't there yet, so very large NVL72-class fleets need a different stack.
- Setup assumes comfort with config files, PKI bootstrapping and systemd, which rules out non-technical users.
- macOS works only through the from-source build, so most Mac users can't use the one-line installer.
- TensorRT-LLM support is still a spawn driver without a default recipe, so that path needs more manual work.
Frequently asked questions
Cognitora coordinates multiple LLM inference engines into one distributed cluster. You use it to route requests across nodes, place work where KV caches already live, and watch the health and power draw of a whole GPU fleet from one control plane.
Related content
Explore related tools, skills, and articles for Cognitora.
Cognitora Alternatives
Base Code
Base44 (Wix) · Coding · ProductivityBase Code is a shared cloud development environment from Base44. Teammates describe changes to an existing GitHub codebase in chat, preview them live and send them back as pull requests.
Google EmbeddingGemma 2
Google DeepMind · Writing · Image · CodingGoogle EmbeddingGemma 2 is a 740M-parameter open model that turns text, code, images, audio, and video frames into vectors in one 768-dimension space, running on phones and laptops.
GitGlow
GitGlow · Coding · ProductivityGitGlow is a local-first desktop Git client that lets independent developers review their own code and their AI agents' changes in one place before shipping.