Base Compute published BaseRT on July 1 with a paper, Hugging Face article, and public GitHub repository for an Apple Silicon LLM inference runtime built directly on Metal.
The paper’s claim is specific: existing Apple Silicon runtimes leave performance on the table because their abstractions were not designed around Metal’s execution model or Apple Silicon’s unified memory topology. BaseRT instead uses native Metal, chip-specific kernel fusion, unified-memory-aware layout, and custom GPU dispatch.
The reported result is up to 1.56x higher decode throughput than llama.cpp and up to 1.35x higher decode throughput than MLX on the paper’s tested workloads, with larger reported prefill margins for mixture-of-experts models.
The claim is about the runtime, not the model
BaseRT’s paper evaluates Qwen3, Llama 3.2, and Gemma 4 families at Q4 and Q8 quantization on M3 and M4 Pro devices. It also says BaseRT supports model families across eight quantization formats, from Q2 to FP16, on Apple M-series devices.
That makes this a runtime story. The model family matters, the quantization matters, and the hardware matters, but the central argument is that a native runtime can recover performance that framework-based stacks miss.
The repository reinforces that framing. It exposes the basert CLI, a .base model format, conversion tools, an OpenAI-compatible server, language bindings for Python, Node, Rust, and Swift, and requirements of Apple Silicon with macOS 14 or later.
Local inference is becoming an engineering stack
Local LLM work used to be framed mostly as a privacy or hobbyist question: can this model run on a laptop at all? The newer question is more operational: can it run fast enough, predictably enough, and with enough tooling to serve real workflows?
BaseRT is aiming at that second question. The GitHub README says the server supports chat, completions, embeddings, transcription, tool calls, continuous batching, paged KV, and prefix caching. It also describes a coding-agent backend path through a local extension.
Those features matter because local inference has to compete with hosted APIs on workflow, not only ideology. Developers need model management, serving, adapters, batching, bindings, and repeatable benchmarks.
The caveat is benchmark transfer
The paper’s numbers should be read as evidence for its tested setup, not a universal promise. Throughput depends on model architecture, quantization, prompt length, decode length, hardware, memory pressure, thermal state, and runtime configuration.
There is also a licensing distinction. The GitHub repository is Apache-2.0, but the README says the prebuilt engine binary distributed through releases is proprietary and ships under its own license.
