How to Run Local LLMs with OpenVINO on Intel Arc
Use Intel OpenVINO for local LLM inference on Arc GPUs and understand where OpenVINO, OVMS, and related runtimes fit.
What OpenVINO is
OpenVINO is Intel’s toolkit for optimizing and deploying AI inference on Intel hardware. It is different from something like Vulkan or SYCL because it is focused specifically on inference rather than being a general GPU API.
For Intel Arc users, that makes OpenVINO another route for running local AI workloads on the GPU.
The basic stack looks more like this:
Intel Arc
↓
Intel runtime
↓
OpenVINO
↓
LLM application or server
↓
Model
Why use OpenVINO?
The main reason to use OpenVINO is that it is designed around Intel hardware. The toolkit can target Intel CPUs, integrated GPUs, and discrete GPUs through the same general inference ecosystem.
That can make it particularly interesting if you have a system where the CPU and Arc GPU are both important parts of the workload.
OpenVINO Model Server
OpenVINO Model Server (OVMS) takes the idea further by providing a model-serving system rather than just a local inference library.
That makes it useful when you want to expose an inference endpoint to another application instead of sitting at a terminal and generating text manually.
For example, you could have:
Application
↓
API
↓
OVMS
↓
OpenVINO
↓
Intel Arc
This is a different use case from a desktop application such as LM Studio. OVMS is much more interesting when you are building your own AI service or integrating inference into another application.
OpenArc and other projects
You may also run into projects built around OpenVINO that make local LLM inference easier to use on Intel GPUs. OpenArc is one example of an Intel-focused local inference project.
The exact capabilities and supported models of these projects change quickly, so treat the project documentation as the source for current installation instructions.
Model formats and recommended pre-converted IR builds
OpenVINO does not load PyTorch, GGUF, or raw safetensors weights directly. Everything you run on it is an OpenVINO Intermediate Representation (openvino_model.xml + .bin), so a model on Hugging Face reaches an Intel Arc GPU one of three ways:
- A pre-converted IR repo with DFlash hidden states exposed. The
blajOpenVINO IR model family collection publishes ready-to-run INT4 and INT8 OpenVINO IR targets and matching DFlash speculative-decoding drafts with verifiedhidden_states_decoder_layersruntime metadata and statefulbeam_idxKV caches—includingblaj/Ornith-1.5-9B-int4-ov+blaj/Ornith-1.5-9B-DFlash-int4-ov(51.0 tok/s, 2.93x speedup on Arc 130V iGPU; 4.35x onint8),blaj/Qwen3.5-9B-abliterated-int4-ov+blaj/Qwen3.5-9B-abliterated-DFlash-int4-ov(43.7 tok/s, 2.26x), and 64-layerblaj/Qwen3.8-27B-heretic-ara-int4-ov+blaj/Qwen3.8-27B-heretic-ara-DFlash-int4-ov. See Best OpenVINO Models with DFlash Speculative Decoding for Intel Arc and Core Ultra for the full hardware-tier table. - Your own export.
optimum-cli export openvinoconverts a Hugging Face checkpoint to IR, and--weight-format int8orint4compresses the weights in the same pass. How to Convert Models to OpenVINO IR covers the supported source formats. - A two-stage export for large or MoE models. Abliterated and other fine-tunes often ship only full BF16 safetensors. Exporting to fp16 IR first (with
OMP_NUM_THREADS=4) and compressing the IR with NNCF second avoids out-of-memory kills on 30 GB machines. How to Convert an Abliterated (Uncensored) Model to OpenVINO int8 IR shows the exact sequence used forblaj/LFM2.5-2.6B-heretic-int8-ovandblaj/LFM2.5-8B-A1B-heretic-int4-ov.
Once the IR exists, OpenVINO GenAI runs it inside a local application and OVMS serves it as an OpenAI-compatible API endpoint. For custom DFlash exports and verification checks, see How to Convert and Run DFlash Models with OpenVINO.
If you already have a large collection of GGUF models, llama.cpp may be the simpler starting point. If you are building an Intel-oriented inference system, OpenVINO becomes much more interesting.
Arc GPU support
OpenVINO can use Intel GPU devices for supported inference workloads. The exact device and model support depends on the OpenVINO version and the model/runtime combination you are using.
When troubleshooting, first make sure OpenVINO can see the Intel GPU before worrying about the LLM application on top of it.
OpenVINO provides device-selection mechanisms so an application can target CPU or GPU. A typical OpenVINO application will expose a device choice such as:
CPU
GPU
AUTO
The exact syntax depends on the API or application being used.
When OpenVINO makes sense
OpenVINO is worth looking at if you:
- Want an Intel-focused inference stack.
- Want to use Intel CPU and GPU hardware together.
- Are building an application around an inference API.
- Want to use OVMS for model serving.
- Are interested in Intel’s optimization and deployment tooling.
If you mainly want to download a GGUF and chat with it, llama.cpp, Ollama, or LM Studio may be a simpler starting point.
For the other GPU backends, see How to Run llama.cpp on Intel Arc with SYCL and How to Run Local LLMs on Intel Arc with Vulkan. If you’re deciding between all three, the overview at How to Run Local LLMs on Intel Arc compares them side by side.
Comments
One comment per thread every 30 minutes · edits are unlimited.