Plebz AI Runtime: Toward Sub-300ms Memory-Rich On-Device AI Companions

Abstract

The proliferation of large language models has enabled conversational AI experiences that rival human fluency — but almost exclusively through cloud-hosted inference. On-device deployment remains constrained by memory footprint, thermal budgets, and latency requirements that cloud architectures can ignore.

This paper presents the Plebz AI Runtime, a system designed from the ground up for sub-300ms end-to-end latency on consumer mobile devices. We address three fundamental challenges: model compression without conversational quality degradation, memory-aware inference scheduling, and speculative execution for first-token optimization.

---

System Architecture

Model Compression Pipeline

Our compression pipeline operates in three stages:

1. Sensitivity analysis — Profiling each layer's contribution to conversational coherence
2. Adaptive quantization — INT4 for insensitive layers, INT8 for critical attention heads
3. Knowledge distillation — Fine-tuning the compressed model against the full-precision teacher

Full Model (7B params, FP16)
  → Sensitivity profiling (2 hours)
  → Adaptive quantization (INT4/INT8 hybrid)
  → Knowledge distillation (4 hours)
  → Compressed Model (1.8GB, <5% quality loss)

Memory-Aware Scheduling

The scheduler dynamically manages three resources:

Resource	Strategy	Fallback
RAM	Adaptive context window	Reduce to 1K tokens
KV-Cache	Semantic importance eviction	LRU fallback
Compute	Speculative batching	Sequential decode

Voice-to-Voice Pipeline

The full voice pipeline runs in a single forward pass:

• ASR → Whisper-tiny (quantized) — 50ms
• LLM → Compressed 7B — 150ms
• TTS → VITS-based vocoder — 80ms
• Total — ~280ms end-to-end

---

Evaluation

We benchmark against three baselines on Snapdragon 8 Gen 3:

System	Latency (p95)	Memory	Quality (MOS)
Cloud API	1200ms	N/A	4.3
ONNX Runtime	650ms	3.8GB	4.0
llama.cpp	420ms	2.4GB	3.8
Plebz Runtime	280ms	1.8GB	4.0

---

Conclusion

The Plebz AI Runtime demonstrates that sub-300ms conversational AI is achievable on consumer hardware without significant quality trade-offs. Our key insight is that latency optimization must be treated as a first-class design constraint — not an afterthought applied to cloud-optimized architectures.

---

Citation

@article{gupta2025plebz,
  title={Plebz AI Runtime: Toward Sub-300ms Memory-Rich On-Device AI Companions},
  author={Gupta, Sauhard},
  year={2025}
}