How OpenVINO MTP Speculative Decoding Works Internally

Multi-Token Prediction (MTP) speculative decoding is different from the more familiar architecture where a separate small language model drafts tokens for a larger target model. In the OpenVINO implementation for Qwen3.5 and related architectures, the target model export contains an additional MTP component that operates on the target model’s intermediate representations.

The basic architecture

A separate-draft speculative-decoding system looks like this:

flowchart LR
    P[Prompt] --> D[Separate draft model]
    D --> C[Candidate tokens]
    C --> T[Target model]
    T --> V[Verification]
    V --> O[Accepted output]

MTP changes the draft side:

flowchart LR
    P[Prompt] --> T[Target Qwen model]
    T --> H[Target hidden states]
    T --> E[Shared inputs_embeds]
    H --> M[MTP decoder layer]
    E --> M
    M --> C[Candidate prediction]
    M --> K[MTP KV cache]
    C --> V[Target verification]
    V --> O[Accepted output]

The important property is that the MTP head is derived from the same target-model execution rather than being a completely independent draft network.

Why OpenVINO represents MTP as a separate model

The Optimum Intel exporter writes the MTP component to:

openvino_mtp_model.xml
openvino_mtp_model.bin

This gives the runtime a distinct graph for the MTP decoder layer while keeping it in the same exported model directory as the target model.

The separation is useful because the MTP component has its own inputs, outputs, state, and quantization configuration. It is a real stateful OpenVINO model, not merely an annotation on the main XML graph.

The MTP inputs

The exported MTP graph consumes several pieces of state from the target-model workflow:

hidden_states
inputs_embeds
attention_mask
position_ids
MTP KV cache

These inputs serve different purposes.

hidden_states

The hidden-state tensor carries the target model’s intermediate representation into the MTP decoder layer. This is the key connection between the target model and the MTP head.

inputs_embeds

The MTP component receives the already-created input embeddings instead of owning another copy of the large text embedding matrix.

For a large model, this matters. A second embedding table can consume roughly a gigabyte or more, so sharing the embeddings avoids unnecessary duplication.

attention_mask and position_ids

The MTP layer still needs the normal transformer metadata that determines which positions can attend to which tokens and where the current tokens sit in the sequence.

KV cache

The MTP layer maintains its own cache state. Because the exported MTP component is currently a single decoder layer, it needs only the cache associated with that layer rather than a full stack of target-model KV layers.

Why the MTP graph is a single decoder layer

The current implementation requires:

mtp_num_hidden_layers = 1

The exporter constructs the MTP graph as a single decoder layer and gives that layer a single-layer KV cache.

That keeps the auxiliary component much smaller than exporting another full copy of the target model while still giving the MTP head enough transformer computation to predict candidate tokens.

Conceptually:

Target model
┌──────────────────────────┐
│ Decoder layer 1           │
│ Decoder layer 2           │
│ ...                       │
│ Decoder layer N           │
└────────────┬─────────────┘
             │ hidden states
             ▼
MTP model
┌──────────────────────────┐
│ Decoder layer 1           │
└──────────────────────────┘

The MTP layer is therefore not a shallow vocabulary projection. It has transformer-layer computation and state.

Hidden-state runtime annotations

One of the less visible parts of the implementation is hidden-state runtime metadata.

Speculative decoding needs to know which output from the target graph corresponds to the hidden representation consumed by MTP. The exporter therefore discovers the relevant decoder stack and annotates the OpenVINO graph with runtime information describing the hidden-state location.

The implementation generalizes this discovery so it can handle decoder stacks nested inside multimodal text models. It also supports the embedding-oriented model configuration used by visual causal-language models.

Without this metadata, a generic runtime would have a graph containing many intermediate tensors but no reliable semantic connection between the target decoder output and the MTP input.

Shared embeddings reduce duplication

The embedding path is one of the important memory optimizations:

flowchart TD
    Tokens[Input token IDs] --> Embed[Shared text embedding]
    Embed --> Target[Target model]
    Embed --> MTP[MTP model]
    Target --> Hidden[Hidden states]
    Hidden --> MTP

The target and MTP paths can therefore consume the same embedded representation.

This is particularly useful for large Qwen variants where the vocabulary and hidden dimension make the embedding matrix substantial. MTP does not need to own a second copy.

Stateful execution

OpenVINO’s state mechanism is important because both the target model and MTP component operate incrementally during autoregressive generation.

The target model maintains its normal KV cache. The MTP component maintains a separate one-layer cache:

Target state
┌───────────────────────────┐
│ KV layer 1                │
│ KV layer 2                │
│ ...                       │
│ KV layer N                │
└───────────────────────────┘

MTP state
┌───────────────────────────┐
│ KV layer 1                │
└───────────────────────────┘

This state is updated on each MTP invocation rather than rebuilding the entire MTP sequence from scratch.

Where speculative decoding fits

The MTP head generates a candidate continuation. The target model remains responsible for the final verified generation.

A simplified sequence is:

sequenceDiagram
    participant U as Generation loop
    participant T as Target OpenVINO model
    participant M as MTP OpenVINO model

    U->>T: Run target model
    T-->>U: hidden states + target state
    U->>M: hidden states + inputs_embeds + metadata + MTP state
    M-->>U: candidate prediction + MTP state
    U->>T: Verify candidate
    T-->>U: accepted/rejected tokens

The exact scheduling and verification behavior belongs to the OpenVINO GenAI speculative-decoding runtime. The exporter provides the graphs, state, and metadata required to connect those operations.

MTP versus DFlash

DFlash and MTP both target speculative decoding, but their architecture is different.

DFlash

Target model
      ↑
Separate DFlash draft model

The draft model is an independently exported model and usually has its own model directory.

MTP

Target model
      │
      └── MTP decoder layer

The MTP component lives alongside the target export and consumes target-model intermediate state.

This means MTP avoids the overhead of maintaining a separate full draft-model architecture, but it does not mean the MTP component is free. It is a decoder layer with weights and KV-cache state.

For the separate-draft workflow, see How to Convert and Run DFlash Models with OpenVINO.

Why MTP gets its own quantization configuration

Because the MTP component contains substantial model computation, leaving it at an unnecessarily high precision can undermine the memory savings expected from a quantized target model.

Optimum Intel therefore identifies the component as mtp_model during compression. The current configuration uses 4-bit asymmetric quantization with group size 64 and an INT8 symmetric backup.

This matters operationally because a user might otherwise quantize the target model to INT4 while accidentally leaving the MTP decoder at a different precision.

The resulting memory footprint should be considered as:

Total model footprint
≈ target weights
+ MTP weights
+ target KV cache
+ MTP KV cache
+ runtime overhead

The exact numbers depend on the architecture, precision, sequence length, and runtime.

VLMs make hidden-state discovery harder

Multimodal models add another level of nesting:

Visual causal LM
└── multimodal wrapper
    └── text model
        └── decoder stack

The hidden-state locator therefore cannot simply assume that the decoder is the top-level model object. The newer exporter logic discovers the decoder stack and annotates the relevant graph nodes so MTP can consume the correct hidden representation.

This is also why hidden-state annotation work is important to DFlash VLM support. Both speculative-decoding mechanisms need a reliable connection to the text decoder’s internal state.

Why the current tests use one assistant token

The MTP integration tests explicitly configure:

num_assistant_tokens = 1

That is useful when interpreting current support. It proves the MTP path is integrated into the tested speculative-decoding workflow, but it does not establish that every possible multi-token assistant configuration has been validated for every architecture.

When benchmarking locally, treat the assistant-token count as an experimental parameter rather than assuming a larger value will always improve throughput.

Correctness before performance

The most important validation is output equivalence.

A speculative-decoding implementation should produce the same result as normal target-model generation under equivalent deterministic settings. The Optimum Intel test suite compares MTP speculative output against baseline output for this reason.

Only after correctness is established should you measure:

• tokens per second
• time to first token
• target-model evaluations
• candidate acceptance behavior
• device utilization
• memory usage
• KV-cache growth

A speculative decoder can be functionally correct without being faster for every prompt or device.

What this means on Intel Arc

On an Intel Arc device, MTP can be interesting because the auxiliary computation stays inside the OpenVINO execution stack instead of requiring a separate framework and device path for a draft model.

However, the actual speedup depends on the target model, precision, device, memory bandwidth, generation length, and acceptance behavior. The existence of MTP support does not establish a specific tokens-per-second improvement on Arc 140V or another GPU.

For local testing, record the complete environment and compare against the same target model without speculative decoding.

Summary

OpenVINO MTP speculative decoding is built around a small but real stateful transformer component attached to the target model. Optimum Intel exports that component as openvino_mtp_model.xml, connects it to target hidden states and shared input embeddings, gives it its own one-layer KV cache, and annotates the target graph so the runtime can locate the correct intermediate representation.

The result is a different architecture from DFlash: MTP does not need a separate draft model, but its prediction head still has meaningful compute, weights, and state. OpenVINO GenAI handles the speculative-decoding loop, while Optimum Intel provides the exported graphs and metadata that make the integration possible.