How to Run MoE Models on OpenVINO with Disk Streaming

Large Mixture-of-Experts models create an unusual memory problem for integrated GPUs. A model can have a relatively small number of active parameters per token while still having a very large total parameter count. On an Intel integrated GPU, the limiting factor is often not compute but how much of the model can remain resident in GPU-accessible memory.

Recent OpenVINO releases add a practical answer for supported MoE models: weight offloading. Instead of requiring every eligible expert weight to remain resident in GPU memory, OpenVINO can stream a configurable portion of the weights from host storage through an LRU cache.

This article shows the OpenVINO side of that workflow. It does not use llama.cpp.

The feature: offload_ratio

OpenVINO’s Intel GPU plugin exposes ov::intel_gpu::offload_ratio. The value is a percentage from 0 to 100 describing how much of the eligible model weights should be offloaded rather than kept resident.

Conceptually:

                 OpenVINO model
                       |
             +---------+---------+
             |                   |
       resident weights      offloaded weights
             |                   |
             v                   v
        GPU-accessible       host storage
           memory                 |
                                 v
                         LRU weight cache
                                 |
                                 v
                              GPU

The current implementation is intended for MoE expert weights. Increasing the ratio reduces the amount of model weight that needs to stay resident, but it can increase inference latency because weights have to be brought in as they are needed.

The setting is therefore a memory/performance tradeoff, not a way to make a model free.

When disk streaming is useful

Disk-backed offloading is most useful when the model is just beyond what the GPU can comfortably keep resident.

For example, suppose an INT4 MoE model is large enough that loading all of its eligible expert weights into the GPU causes memory pressure. An offload ratio gives OpenVINO permission to keep only part of those weights resident and fetch the rest on demand.

The important word is eligible. offload_ratio=50 does not mean that exactly half of every tensor in the model is written to disk. The property applies to the weights that the GPU plugin can offload, currently centered on MoE expert weights.

Check whether the GPU can use the feature

The disk-offloading path is tied to GPU hardware capabilities. In particular, the current OpenVINO ecosystem uses the GPU’s hardware matrix-multiply capability for this class of optimization.

On a supported OpenVINO installation, inspect the GPU’s optimization capabilities before assuming offloading is active:

import openvino as ov

core = ov.Core()
print(core.get_property("GPU", "OPTIMIZATION_CAPABILITIES"))

Look for the GPU hardware matrix-multiply capability in the returned set.

This distinction matters for Intel integrated graphics because “Core Ultra” is not a single GPU architecture. A feature that works on a newer XMX-equipped integrated GPU should not automatically be assumed to work on every earlier Core Ultra iGPU.

Install the OpenVINO stack

For a stable installation, use the current OpenVINO packages:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -U openvino openvino-genai

For features that are still landing in nightly builds, use the nightly package index instead:

python -m pip install --pre -U openvino openvino-genai \
  --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly

If the model needs an OpenVINO export, install Optimum Intel as well:

python -m pip install -U "optimum-intel[openvino]"

For a model whose current exporter requires development versions, follow the model’s documented version requirements rather than mixing an arbitrary stable Optimum release with a nightly OpenVINO runtime.

Run a model with OFFLOAD_RATIO

The OpenVINO GenAI Python API exposes the GPU offload ratio directly. The basic form is:

import openvino_genai as ov_genai

model_dir = "./model"

pipe = ov_genai.LLMPipeline(
    model_dir,
    "GPU",
    OFFLOAD_RATIO=20,
)

print(pipe.generate("Explain how a CPU cache works.", max_new_tokens=128))

The important part is:

OFFLOAD_RATIO=20

That is a starting point, not a universal optimum.

Try progressively larger values if the model cannot fit comfortably:

OFFLOAD_RATIO=10
OFFLOAD_RATIO=20
OFFLOAD_RATIO=30
OFFLOAD_RATIO=40

Measure actual generation speed and memory use at each step. A higher ratio can make a model fit when it otherwise would not, but there is no reason to offload aggressively if the model already fits and the extra data movement only reduces performance.

Choosing a ratio

A sensible tuning procedure is:

1. Run the model with no offloading if it fits.
2. If GPU memory pressure is excessive, try a small ratio such as 10.
3. Increase to 20, 30, and higher only as necessary.
4. Measure tokens per second and system memory consumption.
5. Keep the lowest ratio that gives the desired stability and memory headroom.

The goal is not “maximum offloading.” The goal is the smallest amount of offloading required to make the model practical.

Qwen3.6-35B-A3B is a good example

Qwen3.6-35B-A3B is a current 35B-total / 3B-active MoE model with 256 experts. OpenVINO officially supports an INT4-MIXED version, and OpenVINO Model Server has a current deployment example for it. citeturn0search0turn0search1turn0search4

The OVMS deployment is straightforward:

mkdir -p models

docker run -d \
  --user $(id -u):$(id -g) \
  --rm \
  -p 8000:8000 \
  -v $(pwd)/models:/models/:rw \
  --device /dev/dri \
  --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) \
  openvino/model_server:weekly \
  --rest_port 8000 \
  --source_model OpenVINO/Qwen3.6-35B-A3B-int4-ov \
  --model_repository_path /models \
  --task text_generation \
  --target_device GPU

OpenVINO’s current OVMS documentation specifically describes Qwen3.6-35B-A3B as a 35B-total / 3B-active vision-language MoE model and documents GPU deployment. Its nightly OVMS documentation also records a 32 GB Panther Lake iGPU test configuration. citeturn0search4turn0search5

The important separation is that OVMS is the serving layer, while offload_ratio is an OpenVINO GPU property. Do not assume that an OVMS command-line option exists for every low-level GPU property. If you need to tune OFFLOAD_RATIO explicitly, use an OpenVINO GenAI pipeline where the property is directly exposed, or configure it through an OVMS mechanism documented for the exact OVMS release you are using.

OpenVINO does not make every MoE model streamable

An MoE architecture by itself does not guarantee disk streaming.

There are three separate questions:

1. Is the model architecture supported by OpenVINO?
2. Is there an OpenVINO export in the required precision?
3. Does the GPU plugin support offloading for the relevant weights on the target hardware?

A model can satisfy the first two and still not benefit from disk offloading on a particular GPU.

This is why current model support tables are more useful than old “VRAM required” lists. OpenVINO currently verifies several Qwen3.5 and Qwen3.6 models and quantization formats, including Qwen3.6-35B-A3B INT4-MIXED. citeturn0search1

Disk streaming versus quantization

Offloading and quantization solve different problems.

Quantization reduces the amount of memory required by the model weights themselves.

Offloading changes where eligible weights live while the model is running.

For a large MoE model, the useful combination is often:

Large MoE checkpoint
        |
        v
      INT4
        |
        v
Smaller weight footprint
        |
        v
GPU memory still insufficient?
        |
        v
   offload_ratio
        |
        v
Some expert weights streamed on demand

Starting with a reasonable quantization is therefore usually preferable to immediately pushing the offload ratio high.

What about OVMS?

OVMS is useful when the goal is to turn an OpenVINO model into a local service rather than manually driving generation from a Python process.

A current OVMS deployment can expose an OpenAI-compatible API for supported generative models. OpenVINO’s documentation demonstrates this approach for large Qwen models and continuous batching. citeturn0search4

For example, a model can be started from the OpenVINO Hugging Face organization with:

docker run -d \
  --user $(id -u):$(id -g) \
  --rm \
  -p 8000:8000 \
  -v "$HOME/models:/models:rw" \
  --device /dev/dri \
  --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) \
  openvino/model_server:weekly \
  --rest_port 8000 \
  --source_model OpenVINO/Qwen3.6-35B-A3B-int4-ov \
  --model_repository_path /models \
  --allowed_media_domains raw.githubusercontent.com

The exact OVMS flags should be checked against the installed OVMS release. In particular, do not invent an OFFLOAD_RATIO server flag just because the underlying OpenVINO GPU plugin has an offload_ratio property.

Nightly status matters

The most interesting OpenVINO GPU features can appear in nightly builds before they are available in a normal stable release. The same applies to newly added model architectures and exporter support.

If a command works in an OpenVINO nightly example but fails on the stable package, first compare the OpenVINO, OpenVINO GenAI, Optimum Intel, and Transformers versions. Do not immediately assume the model itself is unsupported.

For Qwen3.5 and Qwen3.6, OpenVINO’s current documentation already lists official model support, while newer model integrations can still have stricter version requirements. citeturn0search1turn0search6

The practical rule

For a large MoE model on an Intel iGPU, use this order:

Choose a supported OpenVINO model
             |
             v
Use a sensible quantization
             |
             v
Try GPU inference normally
             |
        Does it fit?
          /      \
        yes       no
         |         |
         v         v
       Keep      Enable/tune
       it        offloading
                   |
                   v
             Measure again

The important part is that disk streaming is a memory-management tool, not a replacement for having enough system memory. It can turn an otherwise impractical MoE deployment into a usable one on supported hardware, but every additional layer of memory movement has a performance cost.

For Intel integrated graphics, that tradeoff is often worthwhile because shared system memory gives OpenVINO considerably more room to work with than a small fixed-VRAM GPU. The combination of MoE sparsity, INT4 quantization, and selective weight offloading is what makes models such as Qwen3.6-35B-A3B realistic targets for high-memory Core Ultra systems.