The body of this article was written by Claude Code based on human work and instructions, and adjusted by a human. It is a translation of the Japanese original.

Introduction

I split the 125B-class MoE model Qwen3.8-Flash-Next between two GPUs: the Radeon 8060S built into a GMKtec EVO-X3 (Strix Halo, 128GB) and a Radeon AI PRO R9700 32GB attached over OCuLink. It now generates at 80+ tok/s with about 80K tokens of context, and around 70 tok/s at 130K tokens, close to the limit.

On paper, pairing a GPU behind a slow link with an integrated GPU is awkward. This article records where it got stuck and how I got around it. The biggest win was that a small patch to the existing Strata engine was enough.

The idea came from xct’s Local AI with Strix Halo & R9700, Part 2: Writing Our Own Engine. On a similar setup (Strix Halo plus an R9700 over OCuLink), xct runs the same Qwen3.8-Flash-Next on a custom engine and gets 82 tok/s with about 100K tokens of context. Rather than writing an engine, I took a different route and modified the existing Strata. Still, that article set my expectations for how far long contexts can go on this hardware. I also followed it for the quieter settings, such as lowering the R9700’s power limit to 210 W.

The appendix at the end compares actual runs of the same logic puzzles on three setups: mine, lucebox and llama.cpp. It replays 12 sessions at once, so you can see side by side how tokens pile up and how fast each setup goes.

Setup

The two GPUs are the main players. Experts are split between the R9700, which is fast but small, and the 8060S, which is slow but large.

The model’s experts total about 72 GiB. With about 23 GiB on the R9700 and about 49 GiB on the 8060S, every expert fits on a GPU. The entry point is a llama-server router; naming a model such as q38fn-strata starts it as a child process.

The wall: a narrow OCuLink link

The problem was prompt processing. Every turn, even when only 1,000 tokens were added, about 31 GiB of experts were sent to the R9700.

Strata has a helper feature that lets another GPU hold experts. Loading as many as fit onto the 8060S raised generation to 100 tok/s. But the helper only assists generation; prompts are processed on the R9700 alone. So every turn streams all the experts again.

OCuLink delivers about 7.2 GB/s in practice, so the response to the next turn took about 6 seconds. For agent-style use with dozens of turns, that time adds up. Oddly, giving the helper only a few experts backfired: with 2,000 of them, it was slower than no sharing at all.

The patch in brief

In the original prompt path, what crossed between the GPUs was the expert weights themselves. Every turn, all experts, including those placed on the 8060S, were sent to the R9700 again. The plan was simple: keep each expert’s weights on the GPU that owns it, and send only the rows of each layer’s input (activations) and the rows of results between the GPUs. Even for prompts, each GPU computes the experts it holds.

Strata already has --peer-device, which shares prompt processing too. But it cannot be combined with the helper, so --remote-expert-opt, which speeds up generation, stops working. So I kept the helper and --remote-expert-opt for generation, and made the prompt path use the helper’s cache as its peer.

• Added PrefillPeer: the prompt path needs only a few things from a peer, such as has() and slot_ptr(). I pulled those out into an interface and made the helper’s cache present itself the same way.
• STRATA_HELPER_PREFILL=1: makes the helper the prompt peer.
• STRATA_LEND_WITH_HELPERS=1: keeps the contents of the slots the R9700 lends for prompts in RAM.

Direct GPU-to-GPU transfers did not work. HIP reports them as possible, but test copies come out corrupted. Strata’s route via the host worked as is, so in the end this was not a problem. The row limit --peer-prefill-rows also ended up unnecessary. When the helper is the peer, the default is now all rows, so no rows spill over the limit into direct transfers.

Key parts of the code

The core of the change was pulling what the prompt path needs from the other GPU into a small interface. This is the only new file.

// include/strata/core/prefill_peer.hpp
class PrefillPeer {
public:
    virtual ~PrefillPeer() = default;
    virtual bool valid() const = 0;
    virtual int device() const = 0;
    /// Whether the primary and this GPU can read each other's memory directly; false takes the mapped host route.
    virtual bool p2p() const = 0;
    virtual bool has(int64_t layer, int64_t expert) const = 0;
    /// The device address of (layer, expert)'s blob on this GPU, or null when it is not held.
    virtual const uint8_t* slot_ptr(int64_t layer, int64_t expert) = 0;
};

Besides PeerExperts, which was already a peer, I gave this interface to RemoteExperts, the helper’s cache that computes experts on the 8060S during generation. All it does is report whether the cache holds an expert and where it lives. P2P does not work, so it always takes the route via the host.

-class RemoteExperts {
+class RemoteExperts : public PrefillPeer {
 ...
+    /// STRATA_HELPER_PREFILL=1: the prompt path computes this cache's experts here (always the mapped host route).
+    bool valid() const override { return device_ >= 0; }
+    int device() const override { return device_; }
+    bool p2p() const override { return false; }
+    bool has(int64_t layer, int64_t expert) const override { return holds(layer, (int32_t) expert); }
+    const uint8_t* slot_ptr(int64_t layer, int64_t expert) override {
+        const int32_t slot = cache_.slot_of(layer, expert);
+        return slot >= 0 ? cache_.device_slot(slot) : nullptr;
+    }

On the prompt side (Prefill::set_peer and the PeerPrefill inside it), the accepted type changed from PeerExperts* to PrefillPeer*. The computation reuses upstream’s peer code unchanged. The rest is passing the helper as the peer at startup, only when there is no other peer.

+            if (helper_prefill && !peer.valid() && o.peer_prefill_rows != 0 && remote_experts[0].valid()) {
+                const int64_t rows = o.peer_prefill_rows > 0 ? o.peer_prefill_rows : o.prefill_chunk * K;
+                if (!sp.set_peer(&remote_experts[0], rows, err)) {
+                    std::fprintf(stderr, "strata serve: %s - the helper's experts stream to the primary GPU\n", err.c_str());
+                    err.clear();
+                    (void) cudaGetLastError();
+                } else {
+                    std::fprintf(stderr, "strata serve: the prompt path computes the experts of the helper cache on GPU %d "
+                                         "(STRATA_HELPER_PREFILL, up to %lld rows per layer)\n",
+                                 remote_experts[0].device(), (long long) rows);
+                }
+            }

STRATA_LEND_WITH_HELPERS is a one-line change to a condition. Even with a helper present, the R9700 now uses the region from which it lends slots for prompts. The experts in lent slots stay in RAM, so they are no longer reread from the file for every prompt chunk. Upstream skips this region whenever a helper is present.

-    const bool lend = lend_from_slot >= 0 && lend_from_slot < n_slots && additional_gpu_pairs.empty();
+    const bool lend = lend_from_slot >= 0 && lend_from_slot < n_slots &&
+                      (additional_gpu_pairs.empty() || (lend_with_helpers && stage_lend_regions_.empty()));

There are also changes to support an MXFP4 derivative model and small changes on the server side (serve/server.py). The server changes let an environment variable replace the sentence used to cut off thinking, and fix the conversation reuse mentioned in the logic puzzle appendix. The excerpts are changes against 146df3c4 of Strata (MIT License, Copyright (c) 2026 Niko1221 and the Strata contributors).

Results

The next turn dropped from about 6 seconds to about 2, and generation stayed at about 1.5 times the no-sharing speed. Only the first load of 130K is about 10% slower than without sharing.

With this setup, prompt processing (prefill) runs at 1,670 tok/s at 31K, 1,710 tok/s at 74K and 1,810 tok/s at 130K. It does not drop as the context grows; if anything, it rises a little.

Each condition was measured once or twice, so treat these as rough figures. The helper-only column uses the read setting STRATA_PREFILL_HELPER_SOURCE=1.

Even by a conservative measure, generation with long context does not fall below the 70 tok/s level. The table shows conservative results. I have measured more, but preparing a benchmark on natural, messy data that I could also publish took some effort, so those numbers are not included here. In actual use it sometimes runs at 100 tok/s, and even with 130K tokens of data loaded, close to the limit, it still gets around 80 tok/s and sometimes exceeds 100 tok/s.

In total task time, a long answer finishes in about 60% of llama.cpp’s time. The code-fix task varies a lot in turn count, and some runs finish before the speed difference shows, so the gap is hard to see there. Still, after a few turns the gap quickly grows to about this size. Because prompt processing (prefill) time and the like accumulate, a long task can be expected to finish in about half the time.

The llama.cpp figures run IQ4_XS on the 8060S alone, without the R9700. For R1, only passing runs are compared. llama.cpp ran 4 times, lucebox 7 times and this setup 3 times, and the conditions are not fully matched.

Running derivative models

A derivative model with the same structure sometimes only needs a pack built. The engine needs changes only when the format is new.

• Uncensored (IQ4_XS): no extra implementation was needed. Its experts are smaller, so it runs faster than the original; loading 130K took 62.7 seconds.
• CYBER-FROST (MXFP4): the engine did not support the format. I added the compute path in the same shape as IQ4_NL, and added support to the MMQ and PLE tables.

Finally, I ran the three models through regression tests. Extraction from about 44K tokens, follow-up turns, tool round trips, stream interruption and two concurrent requests all passed.

Gotcha: the missing 70 GiB

After unloading the model, about 70 GiB stayed in use without belonging to any process. It turned out to be the TTM page pool on the 8060S side: not a leak, but memory the driver keeps for reuse.

This pool does not shrink over time and does not show up in MemAvailable. Models loaded onto the 8060S reuse the pool, so it does no harm there. But allocating just 26 GiB of ordinary memory dropped MemAvailable to 11.9 GiB. 84 GiB stayed in the pool and was not reclaimed fast enough. That was low enough for my own memory watchdog to stop things by mistake.

Lowering ttm.page_pool_size to 16 GiB (4,194,304 pages) brought MemAvailable back to about 105 GiB after unloading the model. The cause was the 116 GiB value I had put in the boot parameters, so other Strix Halo machines with the same setting are worth checking.

What’s left

Next I want to try the split of experts between the R9700 and the 8060S. Right now the top experts simply go to the R9700. Balancing the processing time of both GPUs, as STRATA_PEER_HOT does, might shorten the 130K load further. I also want more runs to judge the quality of CYBER-FROST’s answers.

The patch is for my own experiments and has not been sent upstream.

Appendix: the order rules task

All three setups passed the 10 tests, and lucebox was the fastest on this task. The task is to fix order state transitions and shipping fee calculation in Rust with enum and match, following the spec. Initially, every test fails.

One run each, so the numbers include variation in how the model approached the task. The totals include model loading.

Here is the apply written by this setup. Every state and event pair reads in a single match.

pub fn apply(state: OrderState, event: Event) -> Result<OrderState, Rejected> {
    use {Event as E, OrderState as S, Rejected as R};
    match (state, event) {
        (S::Cart, E::Place { total: 0 }) => Err(R::EmptyOrder),
        (S::Cart, E::Place { total }) => Ok(S::Placed { total }),
        (S::Placed { total }, E::Pay { amount, .. }) if amount != total => {
            Err(R::AmountMismatch { expected: total, got: amount })
        }
        (S::Placed { total }, E::Pay { payment, .. }) => Ok(S::Paid { total, payment }),
        (S::Placed { .. }, E::Cancel) => Ok(S::Cancelled { refund: 0 }),
        (S::Paid { .. }, E::Ship { tracking }) if tracking.trim().is_empty() => {
            Err(R::MissingTracking)
        }
        (S::Paid { total, payment }, E::Ship { tracking }) => {
            Ok(S::Shipped { total, payment, tracking })
        }
        (S::Paid { total, payment }, E::Cancel) => {
            Ok(S::Cancelled { refund: cancel_refund(total, payment) })
        }
        (S::Shipped { total, payment, .. }, E::Deliver) => Ok(S::Delivered { total, payment }),
        (S::Delivered { total, payment }, E::Return { days, opened }) => {
            return_refund(total, payment, days, opened).map(|refund| S::Returned { refund })
        }
        _ => Err(R::WrongState),
    }
}

Appendix: solving two logic puzzles with tools

I had each setup solve two original logic puzzles made for benchmarking, with omp’s tools available, and lined up four runs per setup. The point is not the answers but how tokens pile up and how fast. The replay below shows all 12 sessions at once, so you can watch context length and speed change side by side (open in a new tab). The model’s thinking and output are shown as recorded.

All runs use Qwen3.8-Flash-Next: UD-Q4_K_XL for Strata(patched) and lucebox, and UD-IQ4_XS for llama.cpp. For llama.cpp, two runs split layers between the R9700 and the 8060S (3:7) and two use the 8060S alone; adding the R9700 did not change the generation speed.

The Strata(patched) runs come after fixing a problem where the token boundaries disagreed on the turn after writing a file, breaking conversation reuse. Now, when the history’s bytes match the previous generation, the previous tokens are reused as is. Each run was capped at 900 seconds.