A July 13 arXiv paper proposes HCRMap, a runtime system for managing hot expert replicas in mixture-of-experts inference on 3.5D chiplet systems.
The paper’s claim is narrow but useful: as MoE models route tokens to a small subset of experts, the bottleneck is not only whether the right expert exists. It is where that expert’s weights live, which memory tier serves them, and how token traffic, memory banks, compute queues, and package links interact during serving.
The authors report that HCRMap cuts end-to-end latency by 43.6% and 43.0% over Hydra in prefill and decode, by 34.5% and 33.1% over MoEntwine, and by 46.7% and 46.0% over PIMoE. Those are paper-reported results, not an independent commercial benchmark.
Hot experts are a memory problem
MoE models activate only part of the network for each token. That saves compute, but it creates a routing pattern where some experts become hot while others are lightly used.
In a simple view, the system should just route tokens to the right expert. HCRMap argues that this is not enough on a 3.5D package with stacked SRAM, local HBM, and group-shared DRAM. The same hot expert can stress memory banks, shared I/O, interconnect paths, and execution queues.
The paper frames the problem as residency. Hot experts may deserve fast nearby storage. Warm experts may not fit in the closest tier but should not always be pushed all the way to far shared memory. A useful serving runtime has to promote, retain, demote, or evict expert replicas based on pressure, not only popularity.
The design is a serving signal, not a product launch
HCRMap is not a claim that a new chip is shipping. It is a research design that points toward where MoE inference optimization is going.
The system uses a two-timescale control loop. A slower residency loop decides where expert replicas should live across tiers. A faster mapping loop routes token groups to suitable resident replicas while accounting for current pressure.
That matters because frontier and open models are leaning hard on MoE designs. If serving costs increasingly come from expert movement, queueing, and memory-tier pressure, the next efficiency gains may come from runtime placement policy as much as from model architecture or raw accelerator speed.
