Feature request: MoE experts in host RAM with on-demand GPU compute — run 16B/35B MoE on 8GB GPUs without OOM

Important limitations

• This may be one public report, not widespread demand.
• Current independent report count: 1.
• This is currently a single independent report. It has not passed clustering or human review as a verified signal.
• No market size, willingness to pay, competition, or trend claims are inferred here.

What was reported

Who is affected

Application developer using the reported software

Job to be done

Build an application reliably

Workflow

I've built and measured a working approach: keep expert weights in host RAM (pinned), let cuBLAS DMA-read them over PCIe on demand, and let VRAM hold only non-expert layers + KV cache.

Desired outcome

That's why a 16B MoE model (6GB file) needs 23GB VRAM, and 8GB/12GB GPUs OOM on models that *should* fit.

Severity

Not enough evidence yet.

Urgency

Not enough evidence yet.

Evidence indicators

What would upgrade this

• Additional independent reports from distinct communities or authors.
• Human clustering into a review signal, then explicit publication approval.
• Counter-evidence review and durable original-source verification.

Original source

Open the original public report