M. Sanjai — AI Systems Engineer | MoE Quantization Research

Engineering AI Systems

I’m Sanjai — I research how far you can compress a Mixture-of-Experts model before it stops behaving like itself, and I build the products that run on the other side of that question. Currently on an H100 at VIT Chennai, and shipping AI software that people use.

LoRA Fine-Tuning

RAG Systems

Local Inference

Agentic Workflows

Multi-Agent Arch.

sancodes@local-ai: ~

❯ cat about_me.md

— AI systems engineer from India

— Building production-grade local LLM systems

— Researching efficient inference & agent architectures

❯ ls projects/

— multi-agent-coding-system

— rag-portfolio-assistant

— vision-lowlight-detection

❯ echo "Ready to build"

█ deployment ready_

Featured Study

Independent research · VIT Chennai · NVIDIA H100 80GB (DST-FIST research server)

🐙 Code & pipelines

Selective Int8 Quantization for Mixture-of-Experts LLMs

Mixture-of-Experts Post-Training Quantization Fidelity Evaluation Expert Routing

MoE models activate few parameters per token but must keep every expert in memory, and standard post-training quantization treats all layers as equally important. I built a pipeline that scores each layer’s quantization tolerance with five signals — KL divergence, top-1 disagreement, choice-flip rate, log-likelihood MAD and perplexity delta — then quantizes only the tolerant safe band. On Qwen3-30B-A3B that band is 21 of 48 layers; on Qwen3.5-35B-A3B it is 10 of 40.

Multiple-choice benchmarks could not tell the two apart. Selective and full int8 both score 91.5% on GSM8K, and perplexity differs in the fourth decimal. So I measured something else: fidelity to the unquantized model as ground truth — KL divergence over 19,200 teacher-forced token positions, expert-routing overlap, and tokens-until-divergence in free-running generation.

Divergence from the bf16 model, Qwen3-30B-A3B. Lower is better; the axis starts at zero. GSM8K is not plotted — both configs score 91.5%, so the bars would be zero.

Top-1 disagreement 2.29 2.89

Expert-slot mismatch 2.22 5.66

Top-k set mismatch 5.37 13.29

Mean KL (×10⁻³ nats) 1.75 3.56

Safe band — 21 of 48 layers All layers — 48 of 48

The benchmark sees nothing. Everything that looks at what the model actually does sees a factor of two to two and a half. Safe band wins all twelve fidelity metrics on both architectures.

To rule out the obvious objection — that quantizing fewer layers trivially drifts less — I ran a control where every arm quantizes exactly 21 of 48 layers and only the choice of layers changes: the selected band, three random draws, and the 21 most-sensitive layers. The selected band beat count-matched random on 12 of 12 metrics. Paired over the same 150 prompts with 4,000 bootstrap resamples, exact top-k set match improves by +6.52 points, 95% CI [+6.31, +6.73], winning on 150 of 150 prompts.

12 / 12 Fidelity metrics won, both models

+6.52 Top-k set match vs count-matched random

150 / 150 Prompts won in the control

19,200 Token positions scored per config

What this does not yet show

All-layer int8 is still the better memory trade. It saves 15.6 GB more than the safe band at equal perplexity and equal multiple-choice accuracy. The safe band’s measured advantages are fidelity and throughput, not footprint.

On the Qwen models the ranking may be restating depth. Spearman correlation between layer index and sensitivity is −0.852 (30B) and −0.947 (35B), and a plain “deepest N layers” band recovers most of the advantage over random. A last-N control arm would separate the two and has not been run. On the smaller Phase 1 models the safe tier sat mid-network, where a depth heuristic would have picked the wrong layers — so depth is not a universal answer, but the Qwen result is not yet clean.

Long-form output quality has never been measured. Every metric here is accuracy, perplexity, or distance from the bf16 model.

Phase 1 · 2025–26

Apple M4 Pro / M3 Max, PyTorch MPS · four small MoE models

Where a MoE Model Can Afford to Lose Precision

OLMoE-1B-7B JetMoE-8B Layer Ablation Routing Analysis

The work that led to the H100 study. Quantize one layer at a time, measure the perplexity delta, and treat everything under 0.3% as the safe band. On OLMoE that was 4 of 16 layers and on JetMoE 6 of 24 — and across six benchmarks the largest degradation anywhere in the phase was −0.66 pp, with WikiText-2 perplexity unchanged at 6.67.

The more interesting result came from a different intervention: replacing layers with identity instead of quantizing them. Damage compounds super-linearly, and which layers you touch matters more than how many.

OLMoE-1B-7B, layers replaced with identity. Baseline perplexity 6.67.
Layers removed	Perplexity	Change	Mean logit KL
L7 — one layer	13.82	+7.42%	0.040
L6–7 — two consecutive	16.11	+25.15%	0.170
L4–7 — four consecutive	33.06	+156.90%	0.632
L3, L7, L11 — three scattered	42.58	+230.91%	0.646

Four consecutive layers cost roughly seven times the sum of their individual impacts, and three scattered layers cost more than four adjacent ones. That is the observation the whole safe-band idea rests on: position dominates count.

Routing turned out to be strongly structured too. Maximum expert coefficient of variation reached 0.9913, and Jensen–Shannon divergence between per-domain routing distributions spanned an order of magnitude — history against code at 0.0401 [0.0373, 0.0452], biology against chemistry at 0.0038 [0.0035, 0.0073].

Caveats on this phase

The quantization was simulated. Weights were quantized then dequantized back to fp16 and stored that way, so it measures the precision impact only — no memory was actually saved and the 1.73 GB figure is theoretical. Phase 2 on the H100 is where real int8 was run.

The router was quantized too, which contaminates any routing conclusion drawn from this phase.

The domain-classifier result is thin. Router logits classify domain at 95% using all 16 layers, but a keyword baseline reaches 90% on the same 20 items — a margin of one item. It needs a larger evaluation set before it means anything.

Let's Build Something Ambitious

I'm always interested in connecting with people who share my passion for building intelligent systems — whether it's AI engineering internships, ambitious AI projects, research collaborations, or experimental infrastructure work.