Running a 125B MoE Model on a 10GB RTX 3080: Speedy 94.6 tok/s & PCIe 4.0 x16 Benchmark Guide

Executive Summary: By every conventional rule of local LLM computing, attempting to run an 80-gigabyte, 125-billion-parameter model on a 10GB graphics card is a recipe for an out-of-memory crash—or a painful 1 to 2 tokens/s slideshow under standard CPU offloading. In this bare-metal engineering benchmark, we paired a single NVIDIA GeForce RTX 3080 (10GB GDDR6X) with 96GB of DDR4 system RAM on Ubuntu 24.04 to run Qwen3.8-Flash-Next (IQ3_S) via the Speedy inference engine. By combining extreme MoE sparsity (~6B active parameters per token out of 512 routed experts) with Speedy’s on-VRAM expert caching (~1,448 hot slots), asynchronous DMA prefetching, and 4B Multi-Token Prediction (MTP), our test rig achieved a conversational peak of 94.6 tokens/s and maintained a median decode speed of 87.2 tokens/s at 200,000 context tokens. Furthermore, diagnosing and fixing a motherboard PCIe lane-splitting bottleneck (PCIe 4.0 x8 to full PCIe 4.0 x16) boosted 120K cold prompt prefill throughput by +26.0% (from 1,009.8 to 1,272.7 tokens/s, shaving 24.6 seconds off initial wait time).

1. Breaking the “10GB VRAM Wall”: Why Standard Offloading Fails and System Synergy Wins

In local AI deployment, hardware sizing usually follows a simple formula: if the quantized model weights exceed your GPU’s physical VRAM, performance falls off a cliff. When you offload dense transformer layers to system RAM in standard llama.cpp, the runtime is forced to shuttle tens of gigabytes of weights across the PCIe bus for every single generated token—dropping generation speed to 1–2 tokens/s.

Yet when we deployed Qwen3.8-Flash-Next (~80GB in IQ3_S split-GGUF format) on a four-year-old 10GB RTX 3080, the live terminal stream delivered instantaneous, production-grade responsiveness:

Key Verified Benchmark Highlights

• Single-Run Conversational Decode Peak: 94.6 tokens/s (~11 ms inter-token latency)
• Automated Stress Sweep (5K to 60K Context via EvalScope): 91.8 to 104.3 tokens/s sustained decode throughput (Batch Mean: 96.25 tok/s, P50 Median: 95.03 tok/s)
• Extreme Long-Context Stability (4K to 200K Context): 87.2 tokens/s median decode rate at 200,000 tokens with zero memory thrashing
• PCIe 4.0 x8 to x16 Hardware Surgery: +26.0% cold prefill throughput gain at 120K context (1,009.8 tok/s → 1,272.7 tok/s), cutting cold prompt ingestion time from 118.9s down to 94.3s (-24.6s saved)

2. Bare-Metal Test Bench Specifications

To ensure our results are reproducible on affordable consumer hardware rather than cherry-picked workstation servers, all tests were executed on a standard AM4 desktop platform:

3. Architecture Deep Dive: How an 80GB Model Runs on a 10GB GPU

So how does an 80-gigabyte model fit into 10 gigabytes of VRAM? The honest engineering answer is: it doesn’t.

The full ~80GB model weights reside permanently inside the 96GB of DDR4 system memory, while the RTX 3080’s 10GB VRAM holds only the active computation working set. Turning that asymmetric memory hierarchy from a 1–2 tok/s bottleneck into a 94.6 tok/s engine requires extreme model sparsity paired with Speedy’s three runtime optimizations:

3.1 Qwen3.8-Flash-Next Radical Sparsity (~6B Active per Token)

Qwen3.8-Flash-Next combines 125 billion base parameters, a 51-billion n-gram lookup table, and a 4-billion speculative draft head. Crucially, out of 512 routed experts, each token calculation activates only the top 10 routed experts plus 1 shared expert. Despite its massive 80GB disk footprint, only ~6 billion parameters are active per token—opening the physical window for a 10GB GPU to compete.

3.2 Speedy’s Three Engineering Scalpels

1. Scalpel 1: Cross-Layer Expert VRAM Caching (~1,448 Hot Slots)
   Even across 512 routed experts, real-world prompts repeatedly trigger the same power-law expert clusters. Speedy dynamically pins approximately 1,448 hot expert slots directly inside the RTX 3080’s 10GB VRAM. With an on-VRAM cache hit rate exceeding 80%, the vast majority of token computations execute entirely on-chip without touching the PCIe bus.
2. Scalpel 2: Asynchronous DMA Pipelining
   When an expert cache miss occurs, Speedy never stalls the GPU. While the Ampere tensor cores execute the current layer, the Direct Memory Access (DMA) controller asynchronously prefetches the next layer’s uncached expert weights across PCIe—overlapping memory transfer with matrix compute.
3. Scalpel 3: Multi-Token Prediction (4B MTP Speculative Decoding)
   Instead of decoding one token per forward pass, the lightweight 4B MTP draft head proposes multiple candidate tokens ahead, and the 125B main model verifies them in a single parallel forward pass—amortizing weight access overhead and nearly doubling effective decode throughput.

4. Long-Context Scaling (4K to 200K) & Raw Log Audit

Speed on short conversational prompts is only half the story. To test whether the 10GB VRAM cache collapses under heavy KV context pressure, we scaled input prompt lengths from 4,096 tokens all the way up to 200,000 tokens (approaching the model’s native 256K window):

4.1 Discrete Measured Checkpoints (4K to 200K Context)

4.2 Auditing the Early “74 tok/s” Benchmark Glitch (Issue #123)

During our initial automated scripting pass, an early parser misidentified a short tail generation chunk as a full cold prefill pass and reported an anomalous 74 tokens/s figure. Instead of glossing over the number, we audited the raw Speedy server logs line by line, corrected the cold-start measurement window, and verified the true 1,009.8 tokens/s baseline at 120K cold prefill (and 91–104 tokens/s sustained decode across our supplementary 5K–60K EvalScope stress sweep).

5. The PCIe Surprise: How Unseating a Secondary Card Boosted Cold Prefill by +26.0%

While decode speed stayed rock solid across all context lengths, our initial 120K cold prompt prefill took a staggering 118.9 seconds.

Inspecting live nvtop bus telemetry revealed the culprit: our RTX 3080’s PCIe receive (RX) bandwidth was hard-capped at 13.38 GiB/s. Because a secondary card was installed in the second slot of our ASUS ROG STRIX X570-F GAMING motherboard, the platform automatically bifurcated the 16 CPU PCIe 4.0 lanes into x8 / x8 mode.

We performed a physical hardware teardown—powering off the rig, unseating the secondary card, and restoring the primary GPU slot to full PCIe 4.0 x16:

Controlled A/B Benchmark: PCIe 4.0 x8 vs. PCIe 4.0 x16 (Same GPU, Same Model, Same Weights)

Critical Distinction — Prefill vs. Decode: Restoring full PCIe 4.0 x16 bandwidth delivered a +26.0% throughput gain specifically on Cold Prefill (when the GPU must ingest massive uncached prompt context across the PCIe bus), cutting wait time by 24.6 seconds (~20.7% shorter wait). Once prompt ingestion finishes, Decode generation barely moved (86.5 vs. 87.2 tok/s) because Speedy’s 1,448-slot VRAM cache and async DMA overlap already hide bus latency during token-by-token generation.

6. 4-Step Bare-Metal Replication Guide (Ubuntu 24.04 + Ampere sm_86)

If you own an RTX 3080 (or a similar 10GB/12GB Ampere GPU) paired with 96GB+ of system RAM, follow these four steps to replicate our setup:

Step 1: Verify Ubuntu 24.04, CUDA 12.4 & Full PCIe x16 Link Width

Before compiling, ensure your NVIDIA driver (550+) and CUDA 12.4 toolkit are installed, and verify that no secondary PCIe device is bifurcating your primary GPU slot:

# 1. Check GPU driver and CUDA version
nvidia-smi
nvcc --version

# 2. Verify your RTX 3080 is negotiating a full x16 link (LnkSta: Speed 16GT/s, Width x16)
sudo lspci -vv -s $(lspci | grep -i "VGA.*NVIDIA" | awk '{print $1}') | grep -E "LnkCap|LnkSta"

# 3. Confirm at least 96GB of available host RAM for full ~80GB weight residency
free -h

Step 2: Clone Speedy & Compile for Ampere (sm_86)

Clone the official Speedy repository and target NVIDIA Ampere (CMAKE_CUDA_ARCHITECTURES=86). If you are compiling under early CUDA 12.0–12.4 toolchains, apply the kBatchRoom constant-memory compatibility guard before building:

# 1. Clone Speedy inference engine
git clone https://github.com/DevDock-AI/Speedy
cd Speedy

# 2. Configure CMake specifically for NVIDIA Ampere (sm_86)
cmake -B build \
  -DSM=86 \
  -DCMAKE_CUDA_ARCHITECTURES=86 \
  -DCMAKE_BUILD_TYPE=Release

# 3. Compile binary targets in parallel
cmake --build build -j$(nproc)

Step 3: Download the ~80GB IQ3_S Split-GGUF & 4B MTP Weights

Download the official Qwen3.8-Flash-Next IQ3_S split-GGUF weights (~80GB across two shards) along with the 4B Multi-Token Prediction (MTP) speculative draft weights into your local models directory:

# Download the IQ3_S split-GGUF shards and MTP draft weights
hf download ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF \
  --include "IQ3_S/*" \
  --local-dir "$HOME/models/Qwen3.8-Flash-Next-GSQ-RCO-GGUF"

Step 4: Launch speedy-server with MTP Enabled & Verify Metrics

Start the OpenAI-compatible HTTP server with both --model and --mtp enabled, then run a streaming completion probe to verify your raw prefill and decode telemetry:

# 1. Launch speedy-server with Qwen3.8-Flash-Next IQ3_S and 4B MTP enabled
./build/bin/speedy-server \
  --model Qwen3.8-Flash-Next-IQ3_S \
  --mtp \
  --port 8080

# 2. Verify local streaming inference over HTTP
curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.8-flash-next-iq3_s",
    "messages": [{"role": "user", "content": "Explain how MoE expert caching and async DMA overlap hide PCIe latency on a 10GB GPU."}],
    "stream": true
  }'

7. Frequently Asked Questions (FAQ)

Q1: Can I run this setup with 64GB of system RAM instead of 96GB?

Answer: Yes, with context-window tuning. While 96GB of DDR4 allows the entire ~80GB split-GGUF footprint and 200K KV cache to reside comfortably without swap pressure, 64GB systems can run the model by capping --max-context to 32768 and reducing the conversation cache allocation to prevent Linux OOM interventions during long prefill passes.

Q2: Why does PCIe 4.0 x8 vs. x16 impact Cold Prefill (+26.0%) so much more than Decode?

Answer: During Cold Prefill (reading a massive 32K–120K token prompt for the first time), the GPU must stream a broad sweep of uncached expert weights across the PCIe bus all at once—making physical bus bandwidth the primary bottleneck (13.38 GiB/s on x8). During Decode (generating one token batch at a time), over 80% of expert activations hit Speedy’s 1,448 resident VRAM slots and the remaining misses are prefetched asynchronously via DMA, keeping decode speed virtually identical (86.5 vs. 87.2 tok/s).

Q3: Where can I find the open-source repositories referenced in this test?

Answer:
– Speedy Inference Engine: https://github.com/DevDock-AI/Speedy
– Qwen3.8-Flash-Next Architecture: https://github.com/QwenLM/Qwen3.8-Flash-Next
– Raw Benchmark Audit Logs: Documented in our engineering tracker (Issue #123 cold-prefill log audit & Issue #124 PCIe x8 vs. x16 controlled comparison).

About The Author

玩客

See author's posts