Running 125B MoE at 200 Tokens/s on a Single 24GB GPU: AMD RX 7900 XTX + Strata Benchmark Guide

Executive Summary: Can a single 24GB consumer graphics card—specifically an AMD Radeon RX 7900 XTX paired with a budget 7-year-old Zen 2 CPU—actually run a massive 125-billion-parameter Mixture-of-Experts (MoE) model at interactive speeds? In the local AI community, conventional wisdom dictates two rigid rules: 24GB cards top out at 30B models, and AMD ROCm is too finicky for serious production workloads. This benchmark report thoroughly dismantles both assumptions. Using the newly optimized Strata-rdna3 runtime on an entry-level AMD Ryzen 7 3700X with standard dual-channel DDR4 RAM, we pushed Qwen3.8-Flash-Next 125B MoE to a single-burst peak decode throughput of 199.67 tokens/s, a sustained prompt prefill speed of 3,455 tokens/s, and flawless end-to-end stability across a massive 256K context window. Here is the complete engineering breakdown, benchmark scorecard, PR #1 CPU instruction fallback fix, and production deployment guide.

1. Dismantling the Myth: 125B MoE on a Single $550 GPU

Historically, hosting a 100B+ foundation model locally required eye-watering hardware investments: either a custom workstation with multi-GPU RTX 4090 configurations running into thousands of dollars, or expensive cloud H100 instances that rack up steep hourly charges.

Compounding this barrier is the prevailing dogma in consumer AI: “CUDA is the only game in town; AMD Radeon cards are strictly for gaming.”

Yet a closer look at physical hardware specifications reveals a compelling alternative:
– VRAM & Memory Bandwidth: The AMD Radeon RX 7900 XTX features 24GB GDDR6 with 960 GB/s of memory bandwidth—fully matching enterprise 24GB accelerators in bus width and raw capacity.
– Compute Architecture: Built on RDNA3 (gfx1100), its compute units incorporate Dual-Issue ALUs and dedicated WMMA (Wave Matrix Multiply-Accumulate) instructions for high-throughput tensor operations.
– Price-to-Performance: On the secondary hardware market, an RX 7900 XTX trades hands at roughly $550 to $600 (~4,000 RMB), representing an unbeatable compute-per-dollar ratio.

Why couldn’t developers run 100B+ models on this card previously? The limitation was never the silicon—it was the inference runtime architecture. Traditional naive weight loading attempts to dump the entire 70GB–90GB model file directly into GPU memory, triggering an instantaneous Out-of-Memory (OOM) crash.

This is where Strata-rdna3 changes the game: an open-source inference engine re-architected from the ground up to exploit RDNA3 matrix hardware and tiered host-device memory scheduling.

2. The Testbed: An Aging Budget Rig (No Lab Cherry-Picking)

To evaluate real-world feasibility for everyday developers, we deliberately rejected flagship workstation configurations (such as an AMD Ryzen 9 9950X3D with high-frequency DDR5). Instead, we conducted all benchmarks on a 7-year-old budget desktop platform:

2.1 Physical Hardware Specification

[!NOTE]
Many developers worry that an older CPU will bottleneck LLM inference. Testing on a 2019 Zen 2 processor demonstrates that when memory tiering and kernel scheduling are architected correctly, legacy platforms handle frontier models with exceptional throughput.

3. Benchmark Scorecard: Sustained 85 tok/s and 200 tok/s Burst

Over 48 hours of continuous stress testing, we logged prompt prefill throughput, generation decode rates, Time to First Token (TTFT), and long-context stability across various input depths.

3.1 Peak Stress Run Telemetry

During a dedicated single-concurrency stress test with a massive 24,064-token input prompt, the single RX 7900 XTX achieved the following verified metrics:

• Input Context Depth: 24,064 tokens (single concurrency)
• Target Output Length: 512 tokens
• Prompt Prefill Throughput: 3,141.43 tokens/s
• Decode Generation Speed: 199.67 tokens/s!
• Time to First Token (TTFT): 7.71 seconds (7,707.76 ms)
• Inter-Token Latency (ITL): 4.99 ms
• Execution Stability: 36 / 36 consecutive automated benchmark requests succeeded with 100% completion.

3.2 Standard 32K–256K Context Scaling Ladder

To evaluate performance across varying context windows, we subjected the recommended Swift 1.5 IQ3_S configuration to a stepped benchmark from 32K up to the theoretical 256K context limit (1 warmup run + 3 measured runs, median recorded, fixed 256 output tokens):

[!TIP]
Understanding the 199.67 tok/s Peak vs 82~85 tok/s Steady State:
In specialized single-concurrency runs with high speculative acceptance, decode throughput peaked at 199.67 tokens/s. Under continuous, full-length generation ranging up to 256K tokens, Swift 1.5 maintained a rock-solid steady-state rate of 82.2 to 85.8 tokens/s (outperforming the official reference baseline of 73.7 tokens/s). Both metrics represent unprecedented performance on a single consumer graphics card.

4. Under the Hood: How 24GB VRAM Tames a 125B Model

Running an 80GB+ model on a 24GB GPU without falling off a performance cliff requires deep synergy across three architectural pillars:

flowchart TD
    subgraph Host["Host Node: 96GB DDR4 System RAM (~94GB Usable Pinned RAM)"]
        RAM_Weights["46.8GB Cold Expert Pool (Swift IQ3_S experts.bin)"]
        CPU_Workers["8-Core CPU Worker Threads (AVX2 Vector Operators)"]
    end

    subgraph GPU["GPU Silicon: AMD Radeon RX 7900 XTX 24GB GDDR6"]
        VRAM_Core["Resident Attention Layers (KV Cache + Core Model)"]
        VRAM_Cache["Resident Expert Arena (~17.61 GiB VRAM)"]
        VRAM_MTP["MTP 4.9GB Multi-Token Speculative Layer (~833 MiB VRAM)"]
        WMMA_Units["WMMA Hardware Tensor Units (hipBLASLt Fused Kernels)"]
    end

    RAM_Weights -- "Dynamic Expert Streaming<br>(PCIe 4.0 Pinned DMA)" --> VRAM_Cache
    CPU_Workers -- "Collaborative Pre-computation" --> VRAM_Cache
    VRAM_Core --> WMMA_Units
    VRAM_Cache --> WMMA_Units
    VRAM_MTP -- "Speculative Draft Verification (62.4% Acceptance Rate)" --> WMMA_Units

4.1 MoE Dynamic Expert Streaming

Qwen 125B is a Mixture-of-Experts (MoE) network: although the model spans 125 billion parameters, each token activation selectively routes through only a subset of expert blocks.

Traditional runtimes either force all weights into VRAM (causing OOM) or perform naive page swapping across the PCIe bus, causing severe bandwidth starvation.

Strata resolves this with Tiered Hotspot Caching and Asynchronous DMA Streaming:
– Attention layers and routing gates remain permanently pinned in GPU VRAM.
– A dedicated 17.61 GiB Expert Arena is reserved directly in GPU memory to host the 9,283 most frequently activated expert slots.
– The remaining cold experts (totaling 46.84 GiB in Swift IQ3_S) reside in host pinned system RAM.
– Measured telemetry shows a resident expert cache hit rate consistently between 94% and 99%, virtually eliminating PCIe transfer bottlenecks during token generation.

4.2 WMMA Matrix Cores & hipBLASLt Kernel Fusion

Reaching a prefill speed of 3,455 tokens/s requires fully saturating RDNA3 compute units.

Standard ROCm ports frequently rely on generic GEMM operators that incur heavy memory round-trips and kernel scheduling bubbles. Strata integrates directly with AMD’s rocWMMA headers, enabling STRATA_HIP_WMMA to dispatch prompt attention directly to the GPU’s hardware matrix units. Coupled with a 26-row hipBLASLt configuration tuned specifically for gfx1100, prefill computation runs entirely GPU-bound at near-peak theoretical throughput.

4.3 MTP (Multi-Token Prediction) Speculative Decoding

Standard autoregressive generation decodes tokens strictly sequentially. Strata leverages Qwen’s official 4.9GB MTP speculative draft weights (--spec 4 --spec-min-p 0.5), which occupy only 833 MiB of VRAM.

In this pipeline, the lightweight draft model speculates 4 candidate tokens ahead; the main 125B model validates the candidate batch in a single forward pass:
– Measured Draft Acceptance Rate: 62.4% on Swift 1.5 IQ3_S.
– This multi-token validation significantly amortizes forward-pass latency, allowing decode speeds to surge toward 200 tokens/s.

5. Critical Gotcha: Resolving the AVX-512 Crash on Legacy CPUs (PR #1)

During initial deployment on our Ryzen 7 3700X, we encountered a severe blocker common to millions of older developer PCs.

5.1 The Root Cause: SIGILL Crash on Zen 2 / Zen 3

Upstream Strata compiled its host expert worker threads with mandatory AVX-512 vector instructions. However, AMD Ryzen 3000 (Zen 2), Ryzen 5000 (Zen 3), and standard consumer Intel chips prior to 11th Gen lack AVX-512 extensions.

Upon initializing the CPU thread pool, the runtime immediately threw:

Illegal instruction (core dumped)

The process crashed instantly before processing a single token.

5.2 The Solution: PR #1 (commit 309f0c8f)

We verified and merged community PR #1, which introduces runtime CPUID capability checking and an automatic AVX2 fallback path. Startup telemetry confirms the fix:

this CPU has no AVX-512: the expert kernels run on AVX-2
8 pool workers on logical processors 1-7, host thread on 0

With PR #1 in place, legacy processors run the host worker pool smoothly without instruction traps.

6. Official Reproduction Guide: 3 Clean Production Steps

Follow this verified sequence to reproduce these benchmark results on any Ubuntu ROCm installation:

Step 1: Clone, Checkout PR #1, and Build

# 1. Clone the rdna3 branch repository
git clone https://github.com/nicejunjie/Strata-rdna3.git
cd Strata-rdna3

# 2. Check out the verified PR #1 commit (mandatory for Zen 2/3 CPUs)
git fetch origin pull/1/head:pr-1
git checkout 309f0c8f

# 3. Execute the automated clean build script
export D="$HOME/AI/strata-rdna3-test"
./setup.sh --setup --build \
  --backend hip \
  --family qwen \
  --model IQ3_S \
  --vision cpu \
  --data-dir "$D" \
  --no-start --yes

Step 2: Weight Quantization & Expert Packing (experts.bin)

Download the model splits, then execute the packing utility to build the memory-mapped expert binary pool:

# Extract and pack the expert pool
.venv/bin/python tools/iq_pack.py \
  --gguf "$D/models/Qwen3.8-125B-Swift-1.5-IQ3_S-00001-of-00004.gguf" \
  --out "$D/packs/swift-iq3" \
  --experts-bin

Step 3: Launch Tailored Single-GPU Configuration

Launch the optimized swift-iq3-1gpu profile targeting GPU 0:

# Start the single-GPU Strata runtime
.venv/bin/python rdna3/run.py swift-iq3-1gpu \
  --data "$D" --gpus 0

Step 4: Verify via OpenAI-Compatible API

Once active, the daemon listens on http://127.0.0.1:18081 with a drop-in OpenAI-compatible endpoint. You can integrate it immediately with Open-WebUI, Cherry Studio, or developer tools like Cursor and Claude Code:

curl http://127.0.0.1:18081/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.8-125b",
    "messages": [{"role": "user", "content": "Summarize the key advantages of running 125B MoE on a 7900 XTX in three bullet points."}],
    "temperature": 0.7
  }'

7. Model Selection & Practical Recommendations

We conducted side-by-side evaluations between IQ3_S and the larger UD-IQ4_XS quantization:

1. Top Recommendation: Swift 1.5 IQ3_S
2. 45% to 55% Higher Decode Throughput: Swift IQ3_S delivers 85.8 tokens/s vs 53.8 tokens/s for UD-IQ4_XS at 32K context, and 83.4 tokens/s vs 54.1 tokens/s at 64K context.
3. Zero Semantic Degradation: Across real-world Python code execution, multi-key needle-in-a-haystack retrieval, and complex reasoning benchmarks, IQ3_S matched IQ4_XS output accuracy.

4. System Memory Footprint: Both models occupy 24GB VRAM fully, but IQ3_S requires only 46.84 GiB of host pinned RAM (compared to 60GB+ for IQ4), making it accessible to standard 64GB–96GB desktops.

5. Hardware Balance Analysis

6. The Strength (GPU Compute): WMMA and hipBLASLt optimizations push prefill speeds past 3,400 tokens/s, handling massive context injections almost instantaneously.
7. The Throughput Ceiling (Host Memory Bandwidth): Because cold experts stream through host memory, older DDR4-3200 memory limits maximum decode throughput. Upgrading host platforms to modern DDR5 configurations unlocks even higher performance reserves from the graphics card.

8. Conclusion

Running a 125B MoE foundation model on a single AMD Radeon RX 7900 XTX—hitting up to 199.67 tokens/s in burst decode, over 3,400 tokens/s in prompt prefill, and sustaining rock-solid 256K context handling on a 2019 Zen 2 processor—proves that open-source runtime optimization has changed the economics of local AI.

Developers no longer need to spend tens of thousands of dollars on enterprise compute clusters to run frontier models. With smart memory tiering, hardware matrix kernel fusion, and speculative decoding, consumer AMD silicon delivers enterprise-grade private intelligence on everyday desktops.

[!TIP]

🎬 Companion Production Assets & Community Links

• Companion Video: Watch on YouTube (4K Ultra HD · GeekLab AI)
• Open-Source Code & PR #1: nicejunjie/Strata-rdna3 PR #1 on GitHub
• Community Discussions & Configs: What throughput and context lengths are you seeing on your AMD rigs? Drop your hardware specs in the comments or join our discussions!

About The Author

玩客

See author's posts