How to Convert and Run DFlash Models with OpenVINO

DFlash is a speculative-decoding method in which a small draft model proposes several tokens and a larger target model verifies them in a single pass. This guide shows how to export both models to OpenVINO IR and run them together with OpenVINO GenAI.

Browse my published OpenVINO IR models, including DFlash drafts, on blaj’s Hugging Face profile.

DFlash support requires two development components:

• Optimum Intel from source, which contains the DFlash export path and the target-model annotations required by the runtime.
• The OpenVINO and OpenVINO GenAI nightly wheels, which contain the DFlash runtime implementation. Stable OpenVINO 2026.3.x does not include this support.

The target model and the DFlash draft are separate exports. The target is the model that produces the final output; the DFlash model is the smaller draft used to propose candidate tokens.

Before you start

Exporting large models can require substantially more RAM than running them. As rough planning figures, a 9B model may need around 80 GB during export, while a 35B model may need around 200 GB. These are estimates, not requirements guaranteed for every architecture.

You also need a target and draft from the same model family. This guide uses two complete pairs from the DFlash model authors:

• Text-only: Qwen/Qwen3-8B + z-lab/Qwen3-8B-DFlash-b16
• VLM: google/gemma-4-31B-it + z-lab/gemma-4-31B-it-DFlash

Do not mix a draft from one pair with a target from another. The DFlash draft is not a standalone model; it is trained for the target named by its model card.

Create the environment

Create and activate a clean virtual environment:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip

Install the development Optimum Intel exporter:

python -m pip install \
  "optimum-intel[openvino] @ git+https://github.com/huggingface/optimum-intel.git"

Install matching OpenVINO and OpenVINO GenAI nightly wheels in the same environment:

python -m pip install --pre --upgrade \
  openvino openvino-genai \
  --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly

Verify that the shell is using this environment and that both packages import:

python -c "import openvino, openvino_genai; print('OpenVINO:', openvino.__version__); print('OpenVINO GenAI:', openvino_genai.__version__)"
optimum-cli export openvino --help

Do not combine a source Optimum Intel exporter with an older stable OpenVINO runtime. The exporter and runtime must be new enough to agree on the DFlash model format.

Example 1: export the text-only Qwen3-8B pair

The target is exported as an ordinary text-generation model. The current Optimum Intel exporter adds the hidden-state metadata that OpenVINO GenAI needs when it attaches the DFlash draft.

optimum-cli export openvino \
  --model Qwen/Qwen3-8B \
  --task text-generation-with-past \
  --weight-format int4 \
  qwen3-8b-openvino

Export the matching DFlash checkpoint separately:

optimum-cli export openvino \
  --model z-lab/Qwen3-8B-DFlash-b16 \
  --task text-generation-with-past \
  --trust-remote-code \
  --weight-format int4 \
  --all-layers \
  --disable-convert-tokenizer \
  qwen3-8b-dflash-openvino

--disable-convert-tokenizer is intentional: the DFlash repository contains the draft weights and custom architecture, but the target directory supplies the tokenizer used by LLMPipeline. --all-layers compresses the draft’s embedding and output layers along with its internal weights.

Example 2: export the Gemma 4 VLM pair

VLM DFlash support uses the VLM target directly. Export the target with image-text-to-text; do not extract its language model and do not use text-generation-with-past for this target.

optimum-cli export openvino \
  --model google/gemma-4-31B-it \
  --task image-text-to-text \
  --weight-format int4 \
  gemma-4-31b-it-openvino

The matching DFlash draft is a separate, text-only draft export. The text-generation-with-past task belongs on this draft command because the draft predicts text tokens; it does not change the VLM target’s export task:

optimum-cli export openvino \
  --model z-lab/gemma-4-31B-it-DFlash \
  --task text-generation-with-past \
  --trust-remote-code \
  --weight-format int4 \
  --all-layers \
  --disable-convert-tokenizer \
  gemma-4-31b-it-dflash-openvino

The VLM target’s hidden-state annotations are written during the direct image-text-to-text export. The draft export contains DFlash metadata in its OpenVINO rt_info, including dflash_mode, mask_token_id, and target_layer_ids. OpenVINO GenAI connects these annotations when it constructs VLMPipeline.

The commands below use the exact target/draft pairs documented by their model cards. For another model family, substitute both IDs with the target and DFlash checkpoint named together by that model’s card; do not reuse either directory with an unrelated model.

Run a text model with DFlash

Do not load the draft with OVModelForCausalLM. That Optimum Intel class is suitable for ordinary causal-language-model inference; DFlash execution is provided by OpenVINO GenAI.

The following script loads a target and its matching draft on the GPU and generates text:

import openvino_genai as ovg

TARGET = "qwen3-8b-openvino"
DRAFT = "qwen3-8b-dflash-openvino"
DEVICE = "GPU"

# Read the DFlash metadata from the exported draft.
draft = ovg.draft_model(DRAFT, DEVICE)

# DFlash with the PA attention backend does not support prefix caching yet.
scheduler = ovg.SchedulerConfig()
scheduler.enable_prefix_caching = False

pipeline = ovg.LLMPipeline(
    TARGET,
    DEVICE,
    ATTENTION_BACKEND="PA",
    draft_model=draft,
    scheduler_config=scheduler,
)

generation_config = ovg.GenerationConfig()
generation_config.max_new_tokens = 256
generation_config.num_assistant_tokens = 7

prompt = "Explain speculative decoding in three paragraphs."
text = pipeline.generate(prompt, generation_config)
print(text)

num_assistant_tokens controls how many candidate tokens the draft proposes in each speculative block. Increase it only after confirming that the pair is stable; the best value depends on the model and hardware.

The enable_prefix_caching = False setting is required by current PA-based DFlash builds. Leaving prefix caching enabled causes pipeline construction to fail with an error stating that DFlash CB/PA does not support prefix caching.

Run the Gemma 4 VLM pair with DFlash

Use VLMPipeline for the VLM target and the matching Gemma DFlash draft. Install Pillow for the image-loading part of this example, then place an RGB image at image.jpg:

python -m pip install pillow

import numpy as np
import openvino_genai as ovg
from openvino import Tensor
from PIL import Image

TARGET = "gemma-4-31b-it-openvino"
DRAFT = "gemma-4-31b-it-dflash-openvino"
DEVICE = "GPU"

image = Tensor(np.asarray(Image.open("image.jpg").convert("RGB")))

draft = ovg.draft_model(DRAFT, DEVICE)
scheduler = ovg.SchedulerConfig()
scheduler.enable_prefix_caching = False

pipeline = ovg.VLMPipeline(
    TARGET,
    DEVICE,
    draft_model=draft,
    scheduler_config=scheduler,
)

config = ovg.GenerationConfig()
config.max_new_tokens = 128
config.num_assistant_tokens = 7

result = pipeline.generate(
    "Describe this image.",
    images=[image],
    generation_config=config,
)
print(result.texts[0])

The OpenVINO GenAI VLM sample accepts images as openvino.Tensor objects containing RGB pixel data. If a different VLM requires another layout or preprocessing convention, follow that model’s processor and pass the resulting tensor instead.

Compatibility warning

Pre-converted model directories are not automatically compatible with DFlash. A text or VLM target may load normally and still fail when a draft is attached because it lacks the hidden-state annotations written by the current exporter. Older hybrid SSM/attention exports can also fail before pipeline construction with errors such as:

Model references undeclared parameters: beam_idx, attention_mask

or:

DFlash target model has no hidden-state RT info and Eagle3 fallback found 2 outputs

These errors describe an export-contract mismatch, not a model-size limit. The same issue can affect both small and large checkpoints. The reliable fix is to export the target with the current Optimum Intel source checkout and run it with the matching OpenVINO GenAI nightly. A runtime flag cannot add metadata that is missing from an existing IR.

The DFlash draft and target must also belong to the same model family. A draft that loads by itself is not proof that it can be attached to an arbitrary target.

Three ways a draft export silently fails

A DFlash draft can export without error, load without error, attach to a target without error, and still produce no speedup at all. The failure is silent: OVMS logs Draft model strategy: DFlash and the model reaches state: AVAILABLE, but every proposal is rejected and decoding runs slower than with no draft, because the draft is pure overhead.

Measured on the same target, to show what a broken draft costs:

Draft	tok/s
none (baseline)	9.0
correct draft	21.2
silently-broken draft	8.5

Three distinct defects produce that outcome. All were hit while converting a third-party draft whose repository shipped custom modeling code.

1. Exporting through the wrong patcher class

Optimum maps a DFlash architecture to two different classes. _CUSTOM_DRAFT_MODEL_MAP in optimum/exporters/openvino/__main__.py reads:

_CUSTOM_DRAFT_MODEL_MAP = {
    "DFlashDraftModel": ("Qwen3DFlashDraftModel", "Qwen3DFlashForCausalLM"),
}

The second entry is what optimum-cli uses for a causal-LM export, and it does something the first does not:

# Drop the seed position so emitted hidden states align 1:1 with the draft
# candidate tokens that the grafted lm_head will score.
last_hidden_state = outputs.last_hidden_state[:, 1:, :]

Subclassing Qwen3DFlashDraftModel (the AutoModel entry) instead of Qwen3DFlashForCausalLM produces a graph that omits that slice. Every emitted hidden state is off by one position relative to the candidate tokens the runtime scores, so every proposal is rejected.

This one is easy to miss because the export succeeds. The tell is the measurement, not an error message.

2. A stateless draft

patch_stateful_decoder in optimum/exporters/openvino/stateful.py converts a graph’s KV-cache inputs and outputs into internal state, and it bails out silently when it cannot find both:

key_value_input_names = [
    key_name for key in ov_model.inputs for key_name in key.get_names() if "key_values" in key_name
]
key_value_output_names = [
    key_name for key in ov_model.outputs for key_name in key.get_names() if "present" in key_name
]
if not key_value_input_names or not key_value_output_names:
    return

use_past_in_inputs defaults to False, which yields a graph with present.* outputs but no matching past_key_values inputs. The guard above then returns early, and the model stays stateless — it recomputes its full context on every step instead of reusing cache.

A correctly stateful draft exposes a beam_idx input and a single last_hidden_state output. Check it directly:

import openvino as ov

model = ov.Core().read_model("openvino_model.xml")
names = [i.get_any_name() for i in model.inputs]
assert "beam_idx" in names, "draft is stateless — proposals will be rejected"
print(names, [o.get_any_name() for o in model.outputs])

A stateless export looks like this — note the absent beam_idx:

inputs : ['inputs_embeds', 'hidden_states', 'position_ids', 'attention_mask']
outputs: ['last_hidden_state', 'present.0.key', 'present.0.value', ...]

Set use_past_in_inputs=True alongside use_past=True to get the cache pair the guard requires.

3. Target-layer geometry mismatch

The draft reads hidden states from a fixed set of target layers, declared as dflash_config.target_layer_ids. Those layers must be a uniform set.

Qwen3.5-9B is a hybrid model: its 32 layers follow a four-layer cycle of three linear_attention layers then one full_attention layer. A draft trained against it samples one consistent layer type:

z-lab/Qwen3.5-9B-DFlash  target_layer_ids [1,5,9,13,17,21,25,29]
                         -> all linear_attention            (uniform)

a broken third-party draft  target_layer_ids [1,8,15,22,29]
                         -> linear, linear, FULL, linear, linear   (mixed)

The mixed set also implies a different fc projection width — 5 × 4096 = 20480 instead of 8 × 4096 = 32768. Since fc is trained for a fixed number of target layers, remapping target_layer_ids cannot repair it: the weight matrix shape would not match. Such a draft cannot work on any Qwen3.5-9B target, at any quantization.

This is worth checking before spending time on an export. Read the draft’s config.json and compare its target_layer_ids against the target’s layer_types:

import json

draft = json.load(open("draft/config.json"))
target = json.load(open("target/config.json"))

ids = draft["dflash_config"]["target_layer_ids"]
types = target["layer_types"]
picked = [types[i] for i in ids]

print(ids, picked)
assert len(set(picked)) == 1, f"non-uniform layer selection: {picked}"

Distinguishing a broken draft from a quantization problem

A natural first suspect is quantization error — that a low-precision draft proposes bad tokens. Test it by exporting the draft unquantized and comparing. If the draft is broken by geometry or statefulness, the BF16 build is the worst result, because the slowdown scales with the draft’s own inference cost:

Draft build	tok/s
int4	8.5
int8	8.0
BF16 (unquantized)	6.6

Quantization error cannot produce that ordering. Unquantized is the most accurate and the slowest, which means accuracy was never the problem — the proposals were never accepted in any build.

A working export for a custom-code draft

Some draft repositories ship their own modeling code and set auto_map to point at it. optimum-cli cannot export those, because the repository’s forward takes a target_hidden argument that Optimum’s dummy-input generators cannot produce. The check fails with:

ValueError: Config dummy inputs are not a subset of the model inputs:
{'attention_mask', 'inputs_embeds', 'past_key_values', 'hidden_states', 'position_ids'}
vs {'labels', 'logits_to_keep', 'use_cache', 'position_ids', 'kwargs',
    'attention_mask', 'inputs_embeds', 'past_key_values', 'input_ids'}

Note that the message does not name the missing argument. The real cause is that the repository’s own forward signature is not the one Optimum expects.

Export through the patched class directly instead:

from transformers import AutoConfig
from safetensors.torch import load_file
from optimum.exporters.openvino import model_patcher
from optimum.exporters.openvino.model_configs import Qwen3OpenVINOConfig
from optimum.exporters.openvino.convert import export as ov_export

cfg = AutoConfig.from_pretrained(SRC, trust_remote_code=False)
cfg.tie_word_embeddings = False
cfg.dflash = True

model = model_patcher.Qwen3DFlashForCausalLM(cfg)   # not the base DraftModel
model.load_state_dict(load_file(f"{SRC}/model.safetensors"), strict=False)
model.eval()

ov_export(
    model=model,
    config=Qwen3OpenVINOConfig(
        cfg,
        task="text-generation",      # not the CLI's "text-generation-with-past"
        use_past=True,
        use_past_in_inputs=True,     # required for a stateful export
    ),
    output="openvino_model.xml",
    stateful=True,
    library_name="transformers",     # skip library inference, which expects .model
)

Three details in that call are not optional:

• task="text-generation". The internal config class rejects the CLI’s text-generation-with-past spelling with KeyError: 'text-generation-with-past', raised from _TASK_TO_COMMON_OUTPUTS in optimum/exporters/base.py.

• library_name="transformers". Without it, export fails while inferring the library from the model class:

  AttributeError: 'Qwen3DFlashForCausalLM' object has no attribute 'model'
  

• strict=False on load_state_dict. The draft checkpoint contains only the draft weights; anything the patched class initializes on its own is expected to be absent.

After exporting, run the beam_idx assertion from the previous section before benchmarking. It takes one second and catches defects 1 and 2 immediately.

Troubleshooting

Check the versions and executable paths first:

which python
which optimum-cli
python -c "import openvino, openvino_genai; print(openvino.__version__); print(openvino_genai.__version__)"
python -m pip show optimum-intel openvino openvino-genai

If the export is killed without a Python traceback, check system logs for an out-of-memory termination and compare available RAM with the model’s uncompressed size. Large exports also require enough free disk space for the output IR and temporary files.

For reproducibility, record:

• the Optimum Intel Git revision;
• the OpenVINO and OpenVINO GenAI versions;
• the model revisions;
• the export tasks and command lines; and
• the target and draft directories used together.

DFlash export is not the same as ordinary OpenVINO weight compression. --weight-format int4 controls weight compression; DFlash support comes from the model architecture, exporter metadata, and OpenVINO GenAI pipeline. For general OpenVINO conversion guidance, see How to Convert Models to OpenVINO IR.

Converted drafts

Working DFlash drafts converted to OpenVINO IR, useful as reference points when checking your own export. Each exposes beam_idx and reaches roughly 2.4x over a no-draft baseline on an Arc iGPU:

• blaj/Qwen3.5-9B-abliterated-DFlash-int8-ov — abliterated Qwen3.5-9B draft, int8 and int4
• circulus/Qwen3.5-9B-DFlash-ov-int4 — the upstream z-lab draft, int4
• blaj/Ornith-1.5-9B-DFlash-int4-ov — Ornith-tuned draft, with blaj/Ornith-1.5-9B-int4-ov (int4) and blaj/Ornith-1.5-9B-int8-ov (int8) as its targets

Abliteration changes model weights, not layer geometry, so a draft trained on a base model also pairs correctly with its abliterated counterpart. Both report the same 32-layer hybrid layout, and the draft reads the same layers in each.

Verifying that a draft is actually running

A DFlash draft can attach without error and still never contribute a token. The failure is silent: the target answers correctly, at baseline speed, because verification simply never accepts anything. Before benchmarking, confirm two things.

1. The target carries hidden-state locators. The draft consumes intermediate hidden states, so the runtime needs to know which graph values correspond to the layers named by target_layer_ids. Optimum Intel writes them as RT info at export time:

import json, openvino as ov
m = ov.Core().read_model("openvino_language_model.xml")
info = json.loads(m.get_rt_info()["hidden_states_decoder_layers"].value)
print(len(info["layers"]))          # -> 32
print(info["layers"]["1"])
# {'producer': '...layers.1/aten::add/Add_1', 'output_index': 0}

This annotation does not change the model’s public I/O — no new inputs or outputs appear. Checking the input/output list therefore proves nothing; you must read the RT info. A target exported without it loads fine and silently disables speculation.

2. The server reports the strategy. OVMS logs the selected mode when it accepts a pairing:

Draft model strategy: DFlash

If that line is missing, the draft was not attached. On OpenVINO GenAI directly, LLMPipeline and VLMPipeline both reject a draft_model keyword argument — DFlash pairing is wired through OVMS’s draft_models_path in graph.pbtxt, not through the Python pipeline constructors.

Confirming the draft is correct

Speculative decoding is exact under greedy decoding: the target verifies every proposed token, so accepted tokens are always the target’s own. That makes the output an equivalence check, not just a quality check. Generate the same prompt twice — once with the draft attached and once without — and compare bytes:

assert with_draft == without_draft   # must hold for greedy decoding

If the strings differ, the draft is proposing something the verifier is accepting incorrectly. If they match but throughput equals the baseline, the draft is attached but never yielding — check step 1. Note that reasoning models emit reasoning_content before content, so give the comparison enough max_tokens for both, or the two runs may both be truncated mid-reasoning and compare equal for the wrong reason.

Measured example

Ornith 1.5 9B target on an Arc 130V iGPU, greedy, 128-token counting workload:

Configuration	Throughput	Speedup
int4 target only	17.4 tok/s	1.00x
+ Ornith DFlash draft (int4, group 16)	51.0 tok/s	2.93x
int8 target only	8.5 tok/s	1.00x
+ same draft	37.0 tok/s	4.35x

Greedy output was byte-identical with and without the draft in both cases.

The int8 target is slower in absolute terms but gets a larger speedup. That is the expected shape when decode is memory-bandwidth bound: a heavier target is a slower baseline, while the draft’s proposals cost roughly the same regardless of target size. Speculation therefore pays off more on the larger model. The practical consequence is that quantization choice is not a simple speed knob — the smaller build wins on absolute throughput, but the draft recovers proportionally more of what quantization costs.

Note the asymmetry when checking a target: hidden-state locators live in the model’s RT info, not the graph, so weight compression does not disturb them. Verify this rather than assume it — read the RT info back after compression and assert the layer count still matches, since a silently dropped annotation yields a target that loads and runs correctly at baseline speed with the draft ignored.

Converting a draft that only ships as GGUF

Some drafts are published only as GGUF, which optimum-cli cannot read. Dequantizing to BF16 and re-exporting works, but the precision story matters:

• Q4_K_M stores 4-bit codes in 16-element sub-blocks: 16 distinct levels per sub-block.
• Dequantizing writes those 16 levels into BF16 — a wider container holding the same information.
• Quantizing that back at group size 16 rounds values that already sit on a 16-level grid, so the second rounding is exact with respect to what was stored. Measured relative error against the held values: 0.000%.

Quantizing at a larger group is not equivalent, because the group scale then spans values that came from different sub-blocks. Measured drift against the fp32 export of the same draft:

Variant	Size	Relative error	Cosine
fp32	5168 MB	0.00%	1.000000
int8 per-channel	1293 MB	2.35%	0.999723
int4, group 16	866 MB	14.69%	0.989211
int4, group 32	769 MB	16.01%	0.987184
int4, group 128	696 MB	18.80%	0.982327

Group size 16 is both the most accurate int4 option and the one that preserves the source’s own quantization grid. The remaining error is NNCF choosing its own scales rather than reproducing Q4_K’s, which is unavoidable without the original BF16 weights — but at group 16 it is the smallest available, and the resulting draft still reaches 2.93x with byte-identical greedy output.

When mapping GGUF tensors to the PyTorch module, two details bite:

• Shape is reported reversed. GGUF lists ne[0] (the contiguous axis) first, so a stored [32768, 4096] is a PyTorch [4096, 32768] linear. The flat buffer is already in PyTorch order — reshape it to the reversed shape rather than transposing it. Getting this wrong yields a model that loads without complaint and correlates 0.0 with the reference weights.
• Norm names differ. llama.cpp writes the draft’s final norm as output_norm and the target-hidden norm as enc.output_norm; the Optimum module calls them norm and hidden_norm. Map them explicitly.

Validate the mapping by correlating dequantized tensors against the reference checkpoint before exporting. A correct map gives correlation ≈ 0.998 against the BF16 originals (the residual is exactly the quantization error); a wrong orientation gives 0.000, which is easy to spot and easy to miss if you only check that the export succeeded.