RunAnywhere

A single C++ core sits under every binding, so behavior stays the same whether you build for Swift, Kotlin, React Native, Flutter, Web or Electron. Each language gets a native wrapper around that core, and every SDK hangs the same namespaces off a RunAnywhere entry point. That means a call reads almost the same everywhere, and you don't relearn the API per platform. The shared shape is the point: fewer surprises when the same feature has to ship on four devices.

The API is organized into namespaces that cover far more than text. There's llm for generation and streaming, stt and tts for speech, voice for a complete spoken conversation, vlm for image plus prompt, rag for retrieval over your own documents, and embeddings, rerank, lora, images, diarization and segmentation alongside them. A single package exposes all of it, so you add the pieces you need. You don't juggle five libraries for one assistant.

RunAnywhere doesn't run on the CPU alone. Backends include the llama.cpp SDK for language and vision, ONNX and Sherpa for speech, Apple MLX, Apple Neural Engine through MetalRT, and Qualcomm Hexagon NPUs through QHexRT. On an M4 Max the company measures LLM decode at 658 tokens per second and speech-to-text latency at 101 milliseconds, its own on-device figures. The catch is coverage: MLX never runs on a simulator, and QHexRT only targets arm64 Snapdragon chips.

Inference runs on the device, and nothing leaves it unless you ask. Prompts, audio and images stay in local memory, which is the whole reason a privacy-sensitive app would pick this over a cloud API. The hosted path exists for models too large for a laptop, but it's a separate choice you make on purpose. The default keeps data home, and the opt-in is explicit.

The wally command line does double duty. Locally it pulls a model, like the roughly 640 MB qwen3-0.6b, and runs it offline with no account. Pointed at the cloud, it reaches frontier-scale open models on RunAnywhere's hardware through an OpenAI-compatible API, billed against prepaid credits. You can also serve a pulled model on a local OpenAI-compatible URL. The same tool covers both ends of the size range.

The SDK tells you what a given device can actually do. Call RunAnywhere.capabilities() and you get the backends that registered and an unavailable list, so you can hide a feature before a user hits it. The models namespace handles the registry, download, load and unload of weights. This is unglamorous plumbing, but it's what stops a build from offering speech on hardware that was never going to run it.