The Current State of OpenVINO in llama.cpp

The OpenVINO backend in llama.cpp is in an interesting place right now. It works, it is actively being developed, and the backend is targeting Intel CPUs, GPUs, and NPUs. But I would not call it finished yet.

One of the more interesting recent additions is support for the Qwen3.5 MoE architecture. llama.cpp now has a dedicated qwen35moe model implementation, including the hybrid Transformer/SSM architecture used by the Qwen3.5/3.6 family. citeturn0search11

That sounds particularly interesting for Intel Arc because Qwen3.5’s MoE models can offer a lot of parameters without requiring all of them to be active for every token.

Unfortunately, when I actually tried the new Qwen3.5 support with the OpenVINO backend, I couldn’t get it working.

There is already a llama.cpp issue documenting Qwen3.5 models failing to load through OpenVINO, including an error involving a tensor operation that the OpenVINO buffer could not run. citeturn0search0 So this isn’t just a case of me missing one command somewhere. There are still backend-specific problems to work through.

OpenVINO is still marked as in progress

This is probably the most important thing to keep in mind when looking at OpenVINO support in llama.cpp. The project’s backend list currently labels OpenVINO as [In Progress], while listing Intel CPUs, GPUs, and NPUs as its target devices. citeturn0search2

That makes the current situation make a lot more sense. The backend is usable, but compatibility is not necessarily going to match the more mature llama.cpp backends.

There have also been reports of other problems. For example, users have reported very poor OpenVINO performance compared with native OpenVINO inference, as well as model-loading and server issues. citeturn0search1turn0search6

So is OpenVINO worth using?

If you’re running Intel hardware, absolutely worth experimenting with. I just wouldn’t treat the llama.cpp OpenVINO backend as the definitive way to run every model on an Arc GPU yet.

For models that work, it can be a convenient way to stay inside the llama.cpp ecosystem while using Intel’s inference stack. For newer architectures such as Qwen3.5/3.6, though, support can still be rough around the edges.

The backend is moving quickly, and the fact that new model architectures are being added is a good sign. But there is a difference between llama.cpp supporting an architecture and that architecture working correctly through every llama.cpp backend.

That’s basically where OpenVINO is right now.

Follow the backend development

If you’re interested in trying it yourself, the best place to keep track of the actual implementation is the OpenVINO backend in the llama.cpp repository:

llama.cpp OpenVINO backend on GitHub

That’s also where I’d look before following an older tutorial, because the backend is still changing.

I’ll probably revisit this once the Qwen3.5 MoE issues are worked out and see whether it actually becomes a practical option on Intel Arc.