How to Use Qwen3.5 MTP Speculative Decoding with OpenVINO

Qwen3.5 and Qwen3.5-MoE can use a Multi-Token Prediction (MTP) head for speculative decoding with OpenVINO. Recent Optimum Intel work exports the MTP head as a separate stateful OpenVINO model component and integrates it into the target model directory, so OpenVINO GenAI can use the target model and its MTP head without requiring a separate draft checkpoint.

How MTP differs from a separate draft model

Traditional speculative decoding uses two models:

Target model ────────────────→ verified output
      ↑
Draft model ────────────────→ candidate tokens

MTP instead puts a prediction head alongside the target model:

                    ┌───────────────┐
hidden states ─────→│ MTP head      │──→ candidate token
                    └───────────────┘
                           │
                           └────────→ draft KV cache

                    ┌───────────────┐
inputs_embeds ─────→│ Target model  │──→ verified output
                    └───────────────┘

The MTP head consumes hidden states and shared input embeddings from the target-model workflow. This avoids maintaining a completely separate draft model.

Requirements

Use an Optimum Intel version containing the MTP exporter and an OpenVINO GenAI version with the corresponding speculative-decoding support. For development builds, keep the Optimum Intel checkout and OpenVINO/OpenVINO GenAI versions aligned rather than mixing unrelated releases.

A basic environment can be created with:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip

Install Optimum Intel from the development source when the required MTP support has not yet reached the package version you are using:

python -m pip install "optimum-intel[openvino] @ git+https://github.com/huggingface/optimum-intel.git"

For a development OpenVINO stack, install the required nightly packages according to the OpenVINO GenAI release you are targeting.

Check the environment before exporting:

python -c "import openvino; print('OpenVINO:', openvino.__version__)"
python -m pip show optimum-intel

Export the Qwen3.5 model

MTP support is implemented as part of the OpenVINO export configuration for the supported Qwen3.5 architectures. Start with the model checkpoint you intend to run and use the OpenVINO exporter available in your Optimum Intel version.

For a standard text-generation export, the general form is:

optimum-cli export openvino \
  --model <Qwen3.5-model-id> \
  --task text-generation-with-past \
  ./qwen3.5-openvino

For a local checkpoint:

optimum-cli export openvino \
  --model ./qwen3.5-model \
  --task text-generation-with-past \
  ./qwen3.5-openvino

Multimodal Qwen3.5 checkpoints require the VLM export path supported by the Optimum Intel version in use. Do not assume that an older exporter can export every Qwen3.5 vision architecture as an MTP model.

After a successful MTP export, inspect the output directory. The MTP implementation adds a model component named mtp, with the main MTP graph stored as:

qwen3.5-openvino/
├── openvino_model.xml
├── openvino_model.bin
├── openvino_mtp_model.xml
└── openvino_mtp_model.bin

The exact set of auxiliary files can vary by model and exporter, but openvino_mtp_model.xml is the important indication that the MTP head was exported.

Verify that the MTP model exists

Check the export directory:

ls -lh ./qwen3.5-openvino/*mtp*

You should see the MTP OpenVINO IR files. If the MTP graph is missing, the checkpoint or exporter configuration did not enable MTP for that architecture.

The MTP graph is not simply another copy of the entire language model. The implementation exports the MTP component as a single decoder layer with its own one-layer KV cache.

Run MTP speculative decoding

OpenVINO GenAI is responsible for the speculative-decoding runtime. The MTP model is part of the same exported model directory, so the runtime can use it as the assistant component instead of loading a second draft-model directory.

The important conceptual difference is:

MTP workflow

one model directory
        │
        ├── target OpenVINO model
        │
        └── MTP OpenVINO model

The current Optimum Intel tests configure MTP speculative decoding with:

num_assistant_tokens = 1

Use the equivalent setting supported by the OpenVINO GenAI API version you have installed. Do not assume that a larger assistant-token setting has been validated for every supported Qwen3.5 configuration.

VLM MTP models

For Qwen3.5 VLM variants, the MTP component is integrated into OVModelForVisualCausalLM. The MTP model is loaded, saved, and compressed as part of the visual causal-language-model representation.

The speculative-decoding path therefore remains tied to the exported target model rather than requiring a separately exported draft VLM.

For current Qwen3.5 VLM testing, prefix caching is disabled for the DFlash and MTP configurations because the relevant linear-attention architecture has different caching requirements.

Quantized MTP models

The MTP head matters for memory usage because it is a full MoE decoder layer and can be comparable in size to a main-model layer.

Optimum Intel therefore has explicit MTP quantization handling. The MTP model’s default configuration uses 4-bit asymmetric quantization with a group size of 64 and an INT8 symmetric backup configuration.

The important point is that the MTP head is not automatically equivalent to a tiny auxiliary model. If you are memory constrained, account for its weights and KV cache when estimating the total footprint.

A typical OpenVINO export/compression workflow can request INT4 weight compression, but use the exact compression options supported by the Optimum Intel revision you installed. After export, verify that the MTP graph was compressed as intended rather than assuming that compression of the main model automatically covers every auxiliary component.

Validate correctness before measuring speed

Speculative decoding must preserve the target model’s output semantics. The Optimum Intel test suite compares MTP speculative output against baseline generation.

A useful local validation procedure is:

1. Generate a fixed prompt with normal target-model inference.
2. Generate the same prompt with MTP speculative decoding.
3. Keep generation parameters identical.
4. Compare the generated token sequence.
5. Only after correctness matches should you measure throughput.

For reproducible tests, record the model revision, Optimum Intel revision, OpenVINO version, OpenVINO GenAI version, device, precision, prompt, generation length, and assistant-token configuration.

Troubleshooting

openvino_mtp_model.xml is missing

The exporter did not enable MTP for the selected architecture or the installed Optimum Intel revision does not contain the required exporter behavior. Verify the model family and Optimum Intel revision first.

The runtime cannot load the MTP model

Check that OpenVINO GenAI and OpenVINO are from a compatible release family. MTP support requires runtime functionality in addition to the exporter.

Memory usage is unexpectedly high

The MTP head is a real decoder layer, not merely a small metadata object. Quantization can reduce its footprint, but the MTP weights and KV cache still consume memory.

Generation is correct but not faster

Speculative decoding is not guaranteed to improve every workload. Measure acceptance behavior, target-model latency, device utilization, prompt length, generated-token count, and assistant configuration. A correctness test only establishes that the output is preserved.

MTP versus DFlash

DFlash uses a separate draft model:

Target model + separate DFlash draft model

MTP uses a prediction head attached to the target model:

Target model + bundled MTP head

The two approaches are complementary rather than interchangeable. DFlash can be useful when a dedicated draft model is available, while MTP can reduce the need to maintain a separate draft checkpoint and export.

For a DFlash-specific OpenVINO workflow, see How to Convert and Run DFlash Models with OpenVINO.

Summary

Qwen3.5 MTP speculative decoding on OpenVINO consists of a normal target-model export plus an exported MTP head. The MTP head is represented by openvino_mtp_model.xml and its weights, integrated into the Optimum Intel model structure, and consumed by the OpenVINO GenAI speculative-decoding runtime.

The most important checks are simple: use an Optimum Intel revision with Qwen3.5 MTP support, verify that the MTP IR was actually produced, keep OpenVINO and OpenVINO GenAI compatible with that exporter, validate output against normal generation, and only then benchmark the speedup.