Deploying Qwen3.8-27B on Dual RX 7900 XTX: Dual Independent 256K Context at 114 Tokens/s (Ubuntu 24.04 + ROCm + llama.cpp)

Single Source of Truth (SSOT) Statement:
All core benchmark metrics, memory footprints, and deep long-context stress tests in this guide were conducted on September 14, 2026, on physical hardware (AMD Ryzen 7 3700X, 96GB DDR4, AMD Radeon RX 7900 XTX 24GB ×2, Ubuntu 24.04.4 LTS, ROCm 7.2.4, and upstream llama.cpp b10951-093a2f86c). All performance data represents real, un-cherry-picked physical tests under this specific hardware and software combination.

When using modern AI coding agents like Cursor, Claude Code, Continue, or autonomous coding pipelines on complex codebases, developers consistently run into three critical bottlenecks: prohibitive cloud API expenses, disruptive rate limits (HTTP 429), and latency spikes when feeding massive context windows.

When equipped with two 24GB consumer GPUs, the conventional instinct is to combine them into a single service using Tensor Parallelism (--split-mode tensor). However, real-world multi-agent testing reveals severe flaws in this approach: under concurrent multi-slot execution (-np 2), the context window is halved to roughly 128K per agent, and inter-GPU synchronization introduces drastic compute locking—causing an active decode stream to crater to 1.1 Tokens/s whenever another agent triggers a 100K-token prefill.

This guide adopts an uncoupled service architecture: dedicating each RX 7900 XTX to its own standalone Qwen3.8-27B instance. Paired with UD-IQ4_XS quantization, Q4 KV cache compression, and Multi-Token Prediction (MTP) speculative decoding, this setup unlocks two fully independent 256K context windows, achieves an aggregate throughput of 114.45 Tokens/s on short prompts, sustains 80.59 Tokens/s across two simultaneous 114K code streams, and limits cross-workload interference to a negligible ~5%!

Core Technology Stack:

flowchart TD
  subgraph Clients["Developer Workstations & Client Layer"]
    AgentA["Primary Coding Agent (Cursor / Cline)"]
    AgentB["Review & Audit Agent (Continue / Scripts)"]
  end

  subgraph Host["Ubuntu 24.04 (llama.cpp ROCm Runtime)"]
    subgraph Srv0["Independent Instance A (:8080)"]
      API0["OpenAI API (:8080)"] --> Eng0["llama-server (FA On + MTP n=3)"]
      Eng0 --> M0["Qwen3.8-27B (UD-IQ4_XS)"]
      Eng0 --> KV0["Q4 KV Cache (Dedicated 256K Window)"]
    end

    subgraph Srv1["Independent Instance B (:8081)"]
      API1["OpenAI API (:8081)"] --> Eng1["llama-server (FA On + MTP n=3)"]
      Eng1 --> M1["Qwen3.8-27B (UD-IQ4_XS)"]
      Eng1 --> KV1["Q4 KV Cache (Dedicated 256K Window)"]
    end
  end

  subgraph Hardware["Physical Hardware Layer (Zero Cross-GPU Sync)"]
    VRAM0["GPU 0: RX 7900 XTX 24GB<br/>(Physical VRAM: ~22.4 GiB)"]
    VRAM1["GPU 1: RX 7900 XTX 24GB<br/>(Physical VRAM: ~22.4 GiB)"]
  end

  AgentA -->|Dedicated HTTP| API0
  AgentB -->|Dedicated HTTP| API1
  Eng0 -.->|100% Resident| VRAM0
  Eng1 -.->|100% Resident| VRAM1

  classDef highlight fill:#1e293b,stroke:#38bdf8,stroke-width:2px,color:#fff;
  class AgentA,AgentB,API0,API1,Eng0,Eng1,M0,M1,KV0,KV1,VRAM0,VRAM1 highlight;

⚡ 1. Real Hardware Benchmarks & Performance Matrix

1. Benchmark Matrix: Architectural Comparison

(All metrics measured on physical hardware on September 14, 2026)

2. Memory Footprint: VRAM & Host RAM Breakdown

GPU VRAM Profile (per RX 7900 XTX):

• Model Weights (UD-IQ4_XS): ~13.27 GiB (14.3 GB decimal)
• Attention KV Cache (256K + Q4_0): ~6.50 GiB
• Compute Buffer & ROCm Graph: ~1.98 GiB
• Active VRAM Usage: ~22.4 GiB (Leaves ~1.6 GiB unallocated headroom in 24.0 GiB physical VRAM, preventing runtime OOM errors).

Host System RAM Profile (nvtop Real-World Measurement):

• Instance A (PID 29810 on GPU0): 19,364 MiB Host RSS
• Instance B (PID 29809 on GPU1): 19,746 MiB Host RSS
• Combined Host RAM Active: 39,110 MiB (~38.19 GiB, strictly under 40GB).
• Analysis: Host memory is consumed by ROCm DMA pinned memory, memory-mapped (mmap) page caches, and 256K scheduling indices. While a 32GB system will immediately OOM-crash, a standard 48GB system RAM configuration accommodates the full workload with 8~10GB of buffer for Ubuntu OS operations.

3. Quantization & Speculative Tuning Deep Dive

① Quantization Comparison: UD-IQ4_XS vs Q4_K_M

By reducing memory bandwidth pressure at 114K context, UD-IQ4_XS actually outpaces Q4_K_M in deep decode throughput while liberating VRAM.

② MTP Draft Step Matrix (--spec-draft-n-max)

Conducted on 114K tokens of real-world codebase context:
* n=2: 45.5 Tokens/s
* n=3: 48.0 Tokens/s (Optimal sweet spot between speculation accuracy and rollback cost)
* n=4: 40.8 Tokens/s (Diminishing draft acceptance rate increases rollback latency penalty)

③ Per-Task Capability & Quality Retention

• Code Refactoring & Multi-File Generation (Python / TS / Go / Shell): Negligible difference compared to Q4/Q8. Syntax integrity, docstring compliance, and architectural refactoring remain rock-solid.
• Agent Tool-Calling & Structured JSON Outputs: Reliable schema adherence and bracket closure under multi-tier function nesting.
• 114K Needle-In-A-Haystack & Deep Call-Graph Tracing: Tested across 114,130 tokens of nested repositories; both instances achieved 100% recall of hidden cross-file identifiers without degradation.

💡 Core Takeaway: In a dual-card workstation, Dual Decoupled 256K Instances provide 2x the dedicated context, deliver 114.45 t/s aggregate throughput, and eliminate cross-agent stalls—making it the definitive architecture for local multi-agent programming.

🛠️ 2. Minimal Prerequisites & 10-Second Sanity Check

1. Hardware Requirements

• GPUs: AMD Radeon RX 7900 XTX 24GB × 2 (Navi 31, gfx1100).
• CPU & RAM: 8+ core modern CPU (tested on Ryzen 7 3700X), 48GB+ Host System RAM (measured under 40GB total consumption).
• Storage: 60GB+ free space on NVMe SSD.
• Topology Note: The decoupled architecture binds each process to a discrete GPU (--split-mode none) and does not require PCIe P2P or RCCL. Standard PCIe 4.0 x8/x8 slots directly linked to the CPU are recommended if you also wish to maintain the Tensor Parallel profile.

2. Base Dependencies & ROCm 7.2.4 Installation

sudo apt update && sudo apt upgrade -y
sudo apt install -y git curl wget cmake ninja-build build-essential pciutils psmisc jq python3 python3-pip python3-venv

Install ROCm 7.2.4 (per AMD official Ubuntu 24.04 documentation):

cd ~
wget https://repo.radeon.com/amdgpu-install/7.2.4/ubuntu/noble/amdgpu-install_7.2.4.70204-1_all.deb
sudo apt install ./amdgpu-install_7.2.4.70204-1_all.deb
sudo apt update
sudo apt install -y rocm rccl rccl-dev
sudo usermod -a -G render,video $USER
sudo reboot

Post-reboot verification:

rocm-smi
rocminfo | grep -E "Name:|gfx"

Both cards should report gfx1100.

3. [Optional] PCIe P2P Topology Check (For Tensor Parallel Mode Only)

sudo apt install -y rocm-bandwidth-test
rocm-smi --showtopoaccess
rocm-bandwidth-test plugin --run tb p2p

• Physical benchmark result: GPU0 → GPU1: True, bidirectional aggregate bandwidth ~17.5 GB/s. (Not required for the dual independent setup).

🚀 3. Step-by-Step Installation Guide

Path A: [Recommended] Automated One-Click Deployment Script

Fully compliant with Ubuntu 24.04 PEP 668 policies, this script utilizes an isolated Python virtual environment, compiles llama.cpp with ROCm support, downloads quantized models, and establishes a multi-instance launch manager:

cat << 'EOF' > ~/setup_qwen38_dual_256k.sh
#!/usr/bin/env bash
set -eo pipefail

echo "=== 1. Building llama.cpp with ROCm Support ==="
cd ~
if [ ! -d "llama.cpp" ]; then
  git clone https://github.com/ggml-org/llama.cpp
fi
cd llama.cpp
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
cmake -S . -B build-rocm -G Ninja \
  -DGGML_HIP=ON \
  -DGGML_HIP_RCCL=ON \
  -DGPU_TARGETS=gfx1100 \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build-rocm -j"$(nproc)"

echo "=== 2. Setting Up Virtual Environment & Downloading Model Weights ==="
python3 -m venv ~/hf-cli
source ~/hf-cli/bin/activate
pip install -U huggingface_hub

mkdir -p "$HOME/AI/models/Qwen3.8-27B-GGUF"
cd "$HOME/AI/models/Qwen3.8-27B-GGUF"

# Download primary model
hf download unsloth/Qwen3.8-27B-GGUF Qwen3.8-27B-UD-IQ4_XS.gguf --local-dir .
# Download speculative MTP sidecar (skipped automatically if already embedded in GGUF)
hf download unsloth/Qwen3.8-27B-GGUF MTP/mtp-Qwen3.8-27B-Q4_0.gguf --local-dir . || true
deactivate

echo "=== 3. Generating Production Dual-Instance Startup Script ==="
cat << 'LAUNCH_EOF' > ~/start_qwen_dual_256k.sh
#!/usr/bin/env bash
set -eo pipefail
pkill -9 -f llama-server || true
sleep 2

MODEL="$HOME/AI/models/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-IQ4_XS.gguf"
MTP_DIR="$HOME/AI/models/Qwen3.8-27B-GGUF"
LLAMA="$HOME/llama.cpp/build-rocm/bin/llama-server"

# Auto-detect external MTP sidecar if present
MTP_ARGS=()
if [ -f "$MTP_DIR/MTP/mtp-Qwen3.8-27B-Q4_0.gguf" ]; then
  MTP_ARGS=(-md "$MTP_DIR/MTP/mtp-Qwen3.8-27B-Q4_0.gguf")
elif [ -f "$MTP_DIR/mtp-Qwen3.8-27B-Q4_0.gguf" ]; then
  MTP_ARGS=(-md "$MTP_DIR/mtp-Qwen3.8-27B-Q4_0.gguf")
fi

COMMON_ARGS=(
  -m "$MODEL"
  "${MTP_ARGS[@]}"
  --split-mode none
  -ngl all
  -c 262144
  -np 1
  -ctk q4_0
  -ctv q4_0
  -fa on
  --spec-type draft-mtp
  --spec-draft-n-max 3
  --spec-draft-ngl all
  --cache-ram 16384
  -t 8
  --host 0.0.0.0
)

echo "Starting GPU0 Instance (:8080, Dedicated 256K)..."
"$LLAMA" "${COMMON_ARGS[@]}" --device ROCm0 --spec-draft-device ROCm0 --port 8080 > /tmp/qwen_gpu0.log 2>&1 &

echo "Starting GPU1 Instance (:8081, Dedicated 256K)..."
"$LLAMA" "${COMMON_ARGS[@]}" --device ROCm1 --spec-draft-device ROCm1 --port 8081 > /tmp/qwen_gpu1.log 2>&1 &

echo "=== Deployment Complete! ==="
echo "GPU0 Instance: http://127.0.0.1:8080"
echo "GPU1 Instance: http://127.0.0.1:8081"
LAUNCH_EOF

chmod +x ~/start_qwen_dual_256k.sh
~/start_qwen_dual_256k.sh
EOF

chmod +x ~/setup_qwen38_dual_256k.sh && ~/setup_qwen38_dual_256k.sh

Path B: Manual 4-Step Deployment (Transparent & Granular)

Step 1: Compile llama.cpp

cd ~
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

HIPCXX="$(hipconfig -l)/clang" \
HIP_PATH="$(hipconfig -R)" \
cmake -S . -B build-rocm -G Ninja \
  -DGGML_HIP=ON \
  -DGGML_HIP_RCCL=ON \
  -DGPU_TARGETS=gfx1100 \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build-rocm -j"$(nproc)"

Verify devices with ./build-rocm/bin/llama-cli --list-devices. Output must show ROCm0 and ROCm1.

Step 2: Download Model Weights via Isolated venv

python3 -m venv ~/hf-cli
source ~/hf-cli/bin/activate
pip install -U huggingface_hub

mkdir -p ~/AI/models/Qwen3.8-27B-GGUF
cd ~/AI/models/Qwen3.8-27B-GGUF

hf download unsloth/Qwen3.8-27B-GGUF Qwen3.8-27B-UD-IQ4_XS.gguf --local-dir .
hf download unsloth/Qwen3.8-27B-GGUF MTP/mtp-Qwen3.8-27B-Q4_0.gguf --local-dir .
deactivate

Step 3: Configure Dual-Instance Startup Script

cat << 'EOF' > ~/start_qwen_dual_256k.sh
#!/usr/bin/env bash
set -eo pipefail
pkill -9 -f llama-server || true
sleep 2

MODEL="$HOME/AI/models/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-IQ4_XS.gguf"
MTP="$HOME/AI/models/Qwen3.8-27B-GGUF/MTP/mtp-Qwen3.8-27B-Q4_0.gguf"
LLAMA="$HOME/llama.cpp/build-rocm/bin/llama-server"

COMMON_ARGS=(
  -m "$MODEL"
  -md "$MTP"
  --split-mode none
  -ngl all
  -c 262144
  -np 1
  -ctk q4_0
  -ctv q4_0
  -fa on
  --spec-type draft-mtp
  --spec-draft-n-max 3
  --spec-draft-ngl all
  --cache-ram 16384
  -t 8
  --host 0.0.0.0
)

# Launch GPU0 (:8080)
"$LLAMA" "${COMMON_ARGS[@]}" --device ROCm0 --spec-draft-device ROCm0 --port 8080 > /tmp/qwen_gpu0.log 2>&1 &

# Launch GPU1 (:8081)
"$LLAMA" "${COMMON_ARGS[@]}" --device ROCm1 --spec-draft-device ROCm1 --port 8081 > /tmp/qwen_gpu1.log 2>&1 &
EOF

chmod +x ~/start_qwen_dual_256k.sh

Parameter Breakdown:
* --device ROCm0 / --spec-draft-device ROCm0: Pin instance execution strictly to the target GPU.
* --split-mode none: Explicitly avoid tensor slicing across GPUs.
* -c 262144 -np 1: Allocate a full, unpartitioned 262,144-token slot to this instance.
* -ctk q4_0 -ctv q4_0: 4-bit KV cache quantization, maintaining ~22.4 GiB VRAM footprint at 256K context.
* --spec-type draft-mtp --spec-draft-n-max 3: Hardware-tuned speculative decoding with 3-token prediction horizon.

Step 4: Lifecycle Management Strategy

💡 Design Choice: We recommend manual, on-demand execution rather than an auto-starting systemd service. Running dual 256K instances reserves ~45GB of physical VRAM and ~38GB of system RAM. On-demand execution allows you to instantly terminate models to run SDXL, LoRA fine-tuning, or gaming without memory conflicts.

🧪 4. Testing & Verification (6 Hardcore Benchmarks)

Test 1: Service Health & Memory Status

curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8081/health

# Inspect GPU VRAM utilization
rocm-smi --showmeminfo vram --showuse

Expectation: Both endpoints return {"status":"ok"}. VRAM stays rock-solid at ~22.4 GiB per card.

Test 2: OpenAI Compatible /v1 Endpoint Check

curl http://127.0.0.1:8080/v1/models
curl http://127.0.0.1:8081/v1/models

Both ports return standard JSON representations containing the loaded Qwen3.8-27B model identifier.

Test 3: Concurrent Short-Prompt Benchmark

Sending simultaneous 512-token generation queries to both ports:
* GPU0: 58.19 Tokens/s
* GPU1: 56.26 Tokens/s
* Aggregate Generation Throughput: 114.45 Tokens/s (100% hardware capability achieved without cross-GPU communication penalties).

Test 4: Dual 114K Real-World Code Benchmark

Stress-tested with a 114,130-token prompt composed of the full llama.cpp source codebase and documentation:
* Prompt Prefill: GPU0 reaches 603.93 t/s; GPU1 reaches 602.72 t/s (Combined aggregate ~1206 t/s).
* Deep Decode: GPU0 outputs at 41.89 t/s; GPU1 outputs at 38.70 t/s (Combined aggregate 80.59 t/s).
* Context Integrity: Zero OOM, zero truncations, and >140K context headroom remaining on each stream.

Test 5: Cross-Load Stress Test (Isolation Verification)

Simulating an asymmetric real-world workflow:
1. GPU1 active: Continuously decoding 2,048 tokens on an existing 114K context (baseline ~41 t/s);
2. GPU0 cold burst: An abrupt, massive 114K-token prefill is blasted to GPU0 while GPU1 is actively outputting code.

Results:
* GPU0 achieves 604.10 t/s on its cold prefill without hesitation;
* GPU1 maintains 38.48 Tokens/s during the shock (only a tiny ~5% drop);
* Result: Completely resolves the catastrophic lockup seen in Tensor Parallelism, where an incoming prefill throttles other decode sessions to ~1.1 t/s!

Test 6: Local Network Integration (Cursor / Continue / Cline) & Dual-Agent Setup

Open firewall ports:

sudo ufw allow 8080/tcp
sudo ufw allow 8081/tcp

① Verification via Native API Call

Query GPU0 (:8080):

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen",
    "messages": [{"role": "user", "content": "Explain your key advantage as a local coding assistant in one concise sentence."}],
    "max_tokens": 256
  }'

Query GPU1 (:8081): Simply switch the port to 8081.

② Dual-Agent Architecture in Practice

• Agent A (Primary Interactive Coder, e.g., Cursor):
• Base URL: http://<HOST-IP>:8080/v1
• API Key: local
• Model: qwen
• Role: Real-time inline code completions, function implementations, and interactive file editing.
• Agent B (Architectural Review & Long-Horizon RAG, e.g., Continue / Cline):
• Base URL: http://<HOST-IP>:8081/v1
• API Key: local
• Model: qwen
• Role: Whole-project audits, deep bug hunting, and multi-file documentation indexing. Even when Agent B ingests a 100K-token diff, Agent A continues writing code smoothly with zero stutter.

❓ 5. Frequently Asked Questions (FAQ)

Q1: Why choose Dual Independent Instances over Tensor Parallel?

1. Context Slicing: Tensor mode splits context evenly between slots (-c 262144 -np 2 results in only ~128K per agent).
2. Compute Locking: Tensor parallelism synchronizes every single layer across GPUs. A 114K prefill in one slot drops the other slot’s decode rate from 45 t/s down to ~1.1 t/s, completely stalling interactive typing.

Q2: Why UD-IQ4_XS instead of standard Q4_K_M?

In our physical benchmarks, Q4_K_M weighs ~15.9 GiB compared to UD-IQ4_XS at ~13.27 GiB (16.5% smaller). At 114K context, the reduced memory bandwidth load allows UD-IQ4_XS to reach 48.0 t/s vs. 44.0 t/s (+9.1%) in decode throughput while preserving critical VRAM.

Q3: Why is Q4 KV Cache a physical necessity?

Qwen3.8-27B features 64 layers with 16 full-attention layers (4 KV heads, 256 head dimension). At 256K context, raw F16 KV cache alone consumes ~16 GiB. Combined with ~13.3 GiB model weights and compute buffers, an F16 configuration requires >30 GiB VRAM—instantly exceeding a 24GB card. q4_0 KV compression provides a massive reduction with no noticeable degradation in coding and retrieval tasks.

Q4: Do I need to download an external MTP sidecar file?

Not necessarily. In our September 14, 2026 tests, we used Unsloth’s standalone MTP/mtp-Qwen3.8-27B-Q4_0.gguf via -md. However, newer releases of llama.cpp and Unsloth GGUFs support embedded NextN/MTP tensors directly within the base model. If using embedded weights, simply specify --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-ngl all without -md.

Q5: Why lock MTP draft tokens at n=3?

Speculative decoding yields diminishing returns as draft horizon increases. Empirically, n=2 achieved 45.5 t/s, n=3 peaked at 48.0 t/s, while n=4 fell back to 40.8 t/s due to rollback verification latency.

Q6: Does multi-turn dialogue recompute the entire 114K codebase every time?

No. llama-server natively leverages Longest Common Prefix (LCP) caching. On turn 2 of our 114K codebase test, the server matched 99.9% of the cached prefix, computing only 93 new tokens—collapsing initial prompt evaluation from 124 seconds down to 1.8 seconds.

Q7: Why is --cache-reuse 256 omitted?

The context backend for this architecture explicitly logs cache_reuse is not supported by this context. Supplying it is a no-op that risks errors, so it was removed.

Q8: If models live in VRAM, why does host RAM need 48GB (measured <40GB)?

Real-time nvtop monitoring reveals that each 256K llama-server process maintains ~18.9~19.3 GiB of Host RSS. This overhead originates from ROCm DMA pinned memory (ensuring ultra-fast host-to-device transfers), Linux kernel mmap page allocations, and dynamic graph dispatch structures for 256K slots. Dual concurrent instances hold 39,110 MiB (~38.2 GiB) in system memory. While 32GB RAM will trigger immediate OOM kills, 48GB of host RAM runs the setup smoothly with comfortable operating system headroom.

🔧 6. Advanced Profiles: Dual Scenario Startup Templates

Profile A: Dual Independent 256K Production Setup (Default)

• Target: Multi-agent concurrent coding (e.g., Cursor + Continue / Cline).
• Benefit: Complete GPU isolation, zero inter-agent stalls, independent 256K windows.
• Launch: Run ~/start_qwen_dual_256k.sh.

Profile B: Dual-GPU Tensor Parallel 256K Monolithic Setup

• Target: Single massive task requiring full double-card compute on a single 256K context stream with F16 KV support.

cd ~/llama.cpp

./build-rocm/bin/llama-server \
  -m ~/AI/models/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-IQ4_XS.gguf \
  --device ROCm0,ROCm1 \
  --split-mode tensor \
  -ngl all \
  -c 262144 \
  -np 1 \
  -ctk f16 \
  -ctv f16 \
  -fa on \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-draft-ngl all \
  -t 8 \
  --host 0.0.0.0 \
  --port 8080

Production Cheat Sheet

🏁 Conclusion: Sovereign Local Compute

Deploying Qwen3.8-27B across dual RX 7900 XTX cards proves that true engineering productivity comes from matching memory topology to real developer workflows.

By decoupling two 24GB GPUs into independent 256K instances, you eliminate the inter-GPU bottlenecks and compute stalls inherent in Tensor Parallelism. You achieve 114.45 Tokens/s aggregate short-text generation, sustain 80.59 Tokens/s across deep 114K contexts, and empower two independent coding agents to work autonomously in real time.

Paired with Prefix Caching, your dual-card workstation transforms into a permanent, private, and zero-latency local intelligence center—completely free from external API quotas and rate limits.

About The Author

玩客

See author's posts