Building Sub-300ms AI Runtimes for On-Device Assistants

The Latency Problem

When we started building Plebz AI, the goal was simple: make conversational AI feel instant. Not "fast for an AI" — actually instant, like talking to another human. That meant sub-300ms end-to-end latency on consumer devices.

Most AI runtimes treat latency as an afterthought. Models are optimized for accuracy on benchmarks, deployed behind API endpoints, and wrapped in layers of abstraction that add hundreds of milliseconds before a single token is generated.

We took a different approach.

---

Architecture Overview

The Plebz AI runtime is built around three core principles:

1. Model-level optimization — Aggressive quantization (INT4/INT8) with custom kernels
2. Memory-aware scheduling — Context windows that adapt to available RAM
3. Speculative execution — Begin generating before the full input is processed

Quantization Without Quality Loss

Standard quantization pipelines (GPTQ, AWQ) optimize for throughput. We needed something different — a quantization scheme that minimizes first-token latency while maintaining conversational coherence.

# Simplified view of our adaptive quantization
def quantize_layer(layer, sensitivity_score):
    if sensitivity_score > 0.8:
        return quantize_int8(layer)  # Keep precision for critical layers
    return quantize_int4(layer)      # Aggressive compression elsewhere

The key insight: not all layers contribute equally to conversational quality. Attention heads in the first and last few layers are disproportionately important for maintaining coherence in multi-turn dialogue.

---

Memory-Aware Scheduling

On-device inference means competing with the OS, other apps, and background processes for memory. Our scheduler dynamically adjusts:

• Context window size — Shrinks gracefully under memory pressure
• KV-cache strategy — Evicts least-recently-accessed entries first
• Batch size — Falls back to sequential processing if needed

This is where it gets interesting. Traditional KV-cache eviction is FIFO or LRU. We use a semantic importance score — entries that are referenced more frequently in the conversation graph get higher priority.

---

Results

On a Snapdragon 8 Gen 3 device:

Metric	Before	After
First token latency	890ms	180ms
Tokens/second	12	45
Memory usage	4.2GB	1.8GB
Conversation coherence (human eval)	4.1/5	4.0/5

The 0.1 drop in coherence is negligible — and the 5x latency improvement makes the experience fundamentally different.

---

What's Next

We're working on streaming inference directly from flash storage, bypassing RAM entirely for the initial model load. Early experiments show we can get first-token latency under 100ms.

The future of on-device AI isn't about bigger models — it's about smarter runtimes.