How to Export Qwen3.5 with an MTP Head to OpenVINO

Recent Optimum Intel development work adds first-class Multi-Token Prediction (MTP) export support for Qwen3.5 and related architectures. Instead of hiding the MTP head inside the main OpenVINO graph, the exporter writes it as a separate stateful OpenVINO model and stores it alongside the target model.

What the exporter produces

A conventional OpenVINO causal-language-model export produces a target model graph and its weights. An MTP export adds another OpenVINO IR model for the MTP head:

qwen3.5-openvino/
├── openvino_model.xml
├── openvino_model.bin
├── openvino_mtp_model.xml
└── openvino_mtp_model.bin

The exact directory can contain additional tokenizer, configuration, and multimodal files. The important addition is:

openvino_mtp_model.xml
openvino_mtp_model.bin

These files represent the MTP prediction component used by speculative decoding.

Why the MTP head is a separate model

The MTP head does not need another copy of the entire Qwen model. It operates on intermediate information produced by the target-model execution.

Conceptually:

flowchart LR
    A[Input tokens] --> B[Shared input embeddings]
    A --> C[Target Qwen model]
    C --> D[Hidden states]
    B --> E[MTP head]
    D --> E
    E --> F[Candidate prediction]
    E --> G[MTP KV cache]
    C --> H[Target KV cache]

This design lets the MTP component consume shared inputs_embeds rather than duplicating the text embedding layer. For large models, avoiding a second embedding matrix is significant because the text embedding layer itself can be around a gigabyte.

Requirements

Create an isolated environment before experimenting with development exporter code:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip

Install an Optimum Intel revision containing the MTP exporter:

python -m pip install "optimum-intel[openvino] @ git+https://github.com/huggingface/optimum-intel.git"

Then install the compatible OpenVINO and OpenVINO GenAI versions for the MTP runtime you intend to use. Development features can move between releases, so verify the versions rather than copying an old environment blindly.

Check the installation:

python -c "import openvino; print(openvino.__version__)"
python -m pip show optimum-intel
optimum-cli export openvino --help

Export Qwen3.5

The general Optimum Intel export form is:

optimum-cli export openvino \
  --model <Qwen3.5-model-id> \
  --task text-generation-with-past \
  ./qwen3.5-openvino

For a local checkpoint:

optimum-cli export openvino \
  --model ./qwen3.5-model \
  --task text-generation-with-past \
  ./qwen3.5-openvino

Use the model’s supported task and export path for the exact Qwen3.5 variant. Vision-language checkpoints can have additional multimodal export requirements, and the MTP VLM path is separate from a plain text-only export.

Verify the MTP export

After conversion, inspect the output:

find ./qwen3.5-openvino -maxdepth 1 -type f -printf '%f\n' | sort

Look specifically for:

openvino_mtp_model.bin
openvino_mtp_model.xml

If those files are absent, you did not produce an MTP export. Do not treat a normal openvino_model.xml as proof that MTP is available.

The Optimum Intel test infrastructure explicitly checks that openvino_mtp_model.xml is produced for the MTP export tests.

What is inside the MTP graph

The exported MTP model is a stateful OpenVINO model representing a single decoder layer. The current implementation requires mtp_num_hidden_layers to be exactly 1.

Its inputs include:

• hidden_states
• inputs_embeds
• attention_mask
• position_ids
• the MTP layer’s KV-cache state

Its outputs include the new last_hidden_state and the updated KV-cache state.

This is intentionally different from exporting another full Qwen model. The MTP graph is a small prediction component attached to the target-model execution path.

Why inputs_embeds is passed into MTP

The MTP head needs access to the same embedded token representation used by the target model. Reusing inputs_embeds means the exporter does not have to duplicate the large text embedding layer inside the MTP graph.

The architecture is approximately:

                  ┌─────────────────┐
inputs_embeds ───→│                 │
                  │   MTP decoder   │──→ last_hidden_state
hidden_states ───→│     layer       │
                  │                 │
KV cache ────────→│                 │──→ updated KV cache
                  └─────────────────┘

The natural sequence dimension is preserved for the MTP inputs, while the MTP component maintains a one-layer cache for its own recurrent state.

MTP is not just a tiny output layer

A useful mistake to avoid is assuming that an MTP head is equivalent to a cheap linear projection.

The implementation treats the MTP component as a full MoE decoder layer. Depending on the architecture, that can make the MTP head comparable in size to a layer of the main model.

That is why the exporter and quantization pipeline explicitly recognize mtp_model as a separate model component.

Quantizing the MTP graph

Optimum Intel adds explicit quantization handling for the MTP model. The current configuration uses a 4-bit asymmetric representation with group size 64 and an INT8 symmetric backup configuration.

The key implementation detail is that requested precision is applied to the MTP model when it is present rather than silently leaving the MTP graph at a different default precision.

When evaluating a quantized export, inspect both the target model and MTP component. Do not assume that quantizing the main graph automatically tells you how the MTP graph was compressed.

MTP and model saving

The MTP component is registered as an OVMTPModel part of the Optimum Intel model structure. It is loaded, saved, and compressed together with the rest of the model representation.

This means the MTP graph is not intended to be manually loaded as an unrelated standalone model during normal speculative decoding. The model directory is the unit that the higher-level runtime consumes.

VLM exports

The same architecture is extended to OVModelForVisualCausalLM. For Qwen3.5 VLM variants, the MTP component is integrated into the visual causal-language-model representation.

The hidden-state discovery code was also generalized to handle decoder stacks nested inside multimodal text models. Runtime annotations identify the relevant hidden-state outputs so speculative decoding can connect the MTP component to the target graph.

This hidden-state metadata is important because an MTP export needs a reliable way to locate the intermediate representation consumed by the prediction head.

Troubleshooting

The exporter creates only the main OpenVINO model

Check the Optimum Intel revision and the exact model architecture. MTP support is architecture-specific. A normal Qwen checkpoint does not automatically gain MTP just because it is exported to OpenVINO.

The MTP model is present but the runtime cannot use it

Check the OpenVINO GenAI version and the Optimum Intel/OpenVINO versions together. The exporter and runtime need compatible MTP support.

The MTP model is unexpectedly large

That is expected to some degree. The MTP head is a decoder layer rather than a tiny classifier. Quantizing it can substantially reduce the footprint.

A VLM export cannot find hidden states

Make sure the exporter contains the hidden-state runtime annotation support for the multimodal decoder architecture. Older exporter revisions may not know how to locate nested text-model decoder stacks.

Inspecting the exported files

A quick sanity check is:

ls -lh ./qwen3.5-openvino/openvino_model.* \
       ./qwen3.5-openvino/openvino_mtp_model.*

The XML files contain the OpenVINO graph structure and the BIN files contain the associated tensor data.

For a more complete conversion workflow, including development Optimum Intel installation and memory planning, see How to Convert and Run DFlash Models with OpenVINO and How to Convert Models to OpenVINO IR.

Summary

Qwen3.5 MTP export is a two-model-component OpenVINO representation: the normal target graph plus a stateful openvino_mtp_model.xml MTP graph. The MTP component consumes hidden states, shared input embeddings, attention metadata, position IDs, and its KV cache, then returns its updated hidden state and cache.

The separate graph keeps MTP modular while avoiding duplication of the large input-embedding layer. Optimum Intel also treats the MTP graph as a first-class component during saving and quantization, which makes the exported directory ready for higher-level OpenVINO speculative-decoding workflows.