How to Run AI Models on the Intel Arc B580 (12 GB VRAM Guide)

Running AI models on the Intel Arc B580 is one of the highest-value setups for local LLM inference on a budget: its 12 GB of GDDR6 VRAM, 456 GB/s of memory bandwidth, and 160 Xe2 XMX matrix engines fit 8B and 9B parameter models at INT4 quantization with room for an 8K–16K context window and a DFlash speculative-decoding draft model. Whether you are setting up an Intel Arc B580 for local AI with OpenVINO Model Server (OVMS), benchmarking Intel Arc B580 LLM tokens per second, or choosing the best local LLMs for 12 GB VRAM on Battlemage, the card handles dense 9B vision-language models at 199 tok/s with DFlash and sparse Mixture-of-Experts (MoE) builds at up to 600 tok/s. This guide covers Intel Arc B580 AI specs, the 12 GB VRAM and KV-cache budget, pre-converted OpenVINO IR models to pull, Docker and Windows OVMS setup commands, DFlash speculative decoding, and the hardware limits to avoid.

Intel Arc B580 specs for local AI and LLM inference

Spec	Intel Arc B580
Architecture	Xe2 (Battlemage), BMG-G21
Xe-cores	20
XMX engines	160
Memory	12 GB GDDR6, 192-bit
Memory bandwidth	456 GB/s
Total board power	190 W
PCIe interface	PCIe 4.0 x8
Launch	13 December 2024, $249

Two hardware specifications govern local LLM performance on the Arc B580, and neither is raw TOPS. Memory bandwidth (456 GB/s) sets the ceiling on single-stream token generation because every generated token requires reading the active model weights from GDDR6. Memory capacity (12 GB VRAM) determines which model parameter classes and context lengths fit on the card without spilling over PCIe into system RAM. Meanwhile, the 160 XMX engines accelerate the dense matrix multiplications during prompt processing (prefill), making time-to-first-token on Battlemage fast relative to autoregressive decode.

Intel Arc B580 12 GB VRAM budget and KV cache sizing

GPU memory during local LLM inference on the Arc B580 is consumed by four distinct allocations, and only the first is obvious from a model’s file size:

1. Model weights — parameters multiplied by bytes per parameter. INT4 weight compression uses roughly 0.5 bytes per parameter plus group-scale and embedding overhead, placing a 9B model near 5.0–5.5 GB in VRAM.
2. KV cache — stores attention keys and values across every layer and grows linearly with context length and batch size.
3. Speculative draft model — attaching a DFlash draft or Multi-Token Prediction (MTP) head adds roughly 0.6–1.5 GB.
4. Runtime and driver buffers — scratch activations, Level Zero command queues, OpenVINO graph buffers, and desktop display allocations.

Calculate the KV cache once and context-window out-of-memory errors stop being mysterious. For an 8B-class transformer with 36 decoder layers and 8 GQA key-value heads of 128 dimensions stored in FP16:

per token = 2 (K and V) × 36 layers × 8 heads × 128 dims × 2 bytes
          = 147,456 bytes ≈ 0.14 MB per token
8K context  ≈ 1.15 GB
32K context ≈ 4.60 GB

That 32K line is why the same 9B model on the same 12 GB Arc B580 runs effortlessly at a 4K context window and runs out of memory at 32K. Always budget VRAM by target context length alongside the weight footprint.

Model class	INT4 weights	+ 8K KV cache	Verdict on 12 GB Arc B580
2.6B dense	~1.7 GB	~2.0 GB	Effortless, room for 64K+ context
4B dense	~2.5 GB	~2.9 GB	Comfortable with long context
8B dense	~4.9 GB	~6.1 GB	Sweet spot for 12 GB VRAM
9B dense	~5.3 GB	~6.5 GB	Sweet spot with ~4 GB spare after runtime
9B dense + DFlash draft	~6.5 GB	~7.7 GB	Fits cleanly with an 8K–16K context
9B dense (INT8)	~9.6 GB	~10.8 GB	Fits on a headless server, zero margin
27B dense (INT4)	~15.0 GB	—	Exceeds 12 GB VRAM — do not run

Best OpenVINO IR models to run on the Intel Arc B580

OpenVINO executes Intermediate Representation (openvino_model.xml and .bin) graphs rather than raw PyTorch safetensors or GGUF files. The pre-converted OpenVINO IR builds below are published under blaj on Hugging Face with INT4 NNCF group quantization, embedded chat templates, stateful beam_idx KV caches, and hidden_states_decoder_layers runtime locators already wired for DFlash speculative decoding. Standard upstream OpenVINO exports work through the exact same server commands.

Use case	Model	Pre-converted OpenVINO IR build	Why it fits the Arc B580
Default 9B text	Qwen3.5-9B-abliterated	blaj/Qwen3.5-9B-abliterated-int4-ov	Highest 9B reasoning and writing quality that still leaves ample 8K KV headroom
Default 9B + DFlash	Qwen3.5-9B-abliterated + DFlash	blaj/Qwen3.5-9B-abliterated-int4-ov + blaj/Qwen3.5-9B-abliterated-DFlash-int4-ov	2.26x speculative speedup inside a ~7.7 GB total VRAM footprint
Vision + language (VLM)	Ornith-1.5-9B	blaj/Ornith-1.5-9B-int4-ov + blaj/Ornith-1.5-9B-DFlash-int4-ov	Multimodal image input plus our fastest 9B DFlash draft pairing (2.93x)
Fast compact 4B	Qwen3-4B	blaj/Qwen3-4B-instruct-int4-ov + blaj/Qwen3-4B-DFlash-int4-ov	Under 3 GB resident, leaving VRAM for 32K–64K context windows
Base 8B completion	Qwen3-8B Base	blaj/Qwen3-8B-Base-int4-ov + blaj/Qwen3-8B-DFlash-int4-ov	Clean 36-layer 8B base model with full DFlash layer annotations
High-speed MoE	LFM2.5-8B-A1B-heretic	blaj/LFM2.5-8B-A1B-heretic-int4-ov	8B total weights in VRAM, ~1B active per token for massive decode speed
Ultra-long context	LFM2.5-2.6B-heretic	blaj/LFM2.5-2.6B-heretic-int4-ov	Tiny ~1.7 GB footprint when a 100K context window matters more than parameter count

Footnote: the one outside model worth naming is Qwen3.6-35B-A3B. It is an MoE with roughly 3B active parameters per token, so it decodes quickly for its total size — but 35B of weights requires 20 GB or more at INT4, so it does not fit in the Arc B580’s 12 GB VRAM without heavy disk or RAM offloading. It becomes a primary native target at 24 GB VRAM or 32 GB shared Core Ultra RAM.

How to set up OpenVINO Model Server on the Intel Arc B580

Running local LLMs on an Arc B580 GPU requires OpenVINO 2025.0 or newer so the Intel Level Zero and OpenCL drivers enumerate Battlemage (BMG-G21) properly. The commands below target OpenVINO Model Server (OVMS) 2026.4.1, which exposes an OpenAI-compatible /v3/chat/completions endpoint on port 8000.

On Linux, require kernel 6.2 or newer, install the Intel GPU compute packages, and ensure your user belongs to the render group:

# as root on Ubuntu / Debian
apt-get update && apt-get install -y \
  ocl-icd-libopencl1 intel-opencl-icd \
  intel-level-zero-gpu level-zero

Docker is the cleanest deployment path on Linux because openvino/model_server:2026.4.1-gpu bundles the matching user-space driver and runtime libraries:

mkdir -p models

docker run -d -p 8000:8000 --rm \
  --user "$(id -u):$(id -g)" \
  -v "$(pwd)/models:/models/:rw" \
  --device /dev/dri \
  --group-add="$(stat -c "%g" /dev/dri/render* | head -n 1)" \
  openvino/model_server:2026.4.1-gpu \
  --model_repository_path /models \
  --source_model blaj/Qwen3.5-9B-abliterated-int4-ov \
  --task text_generation \
  --target_device GPU \
  --rest_port 8000 \
  --model_name qwen9b

On Windows, run the standalone ovms.exe binary distribution rather than mixing pip install openvino into the same environment. Installing the PyPI OpenVINO wheel alongside the Python-enabled OVMS zip puts two competing OpenVINO runtimes on PATH, producing opaque GPU device initialization errors:

mkdir c:\models
ovms --model_repository_path c:\models ^
  --source_model blaj/Qwen3.5-9B-abliterated-int4-ov ^
  --task text_generation --target_device GPU ^
  --rest_port 8000 --model_name qwen9b

Three OVMS CLI behaviors are worth knowing up front:

• --source_model vs --model_path: --source_model accepts a Hugging Face repository ID and downloads it into --model_repository_path on first launch. --model_path points at a local directory you have already downloaded. They are not interchangeable.
• Explicit --target_device GPU: On a desktop with both an Intel Core Ultra or Alder/Raptor Lake iGPU and a discrete Arc B580, specify GPU.0 or GPU.1 explicitly if the integrated GPU enumerates first.
• Config files live in the model directory: Default sampling parameters are read from generation_config.json inside the model folder, and chat templates load from chat_template.jinja or tokenizer_config.json. There is no --generation_config CLI flag in OVMS.

How to enable DFlash speculative decoding on the Arc B580

OpenVINO Model Server supports speculative decoding out of the box with DFlash draft models and Multi-Token Prediction (MTP) heads. Use the 2026.4.1 CLI flags --draft_source_model, --draft_model_path, and --draft_device (older guides sometimes show --draft_model, which OVMS rejects):

docker run -d -p 8000:8000 --rm \
  --user "$(id -u):$(id -g)" \
  -v "$(pwd)/models:/models/:rw" \
  --device /dev/dri \
  --group-add="$(stat -c "%g" /dev/dri/render* | head -n 1)" \
  openvino/model_server:2026.4.1-gpu \
  --model_repository_path /models \
  --source_model blaj/Ornith-1.5-9B-int4-ov \
  --draft_source_model blaj/Ornith-1.5-9B-DFlash-int4-ov \
  --draft_device GPU \
  --task text_generation --target_device GPU \
  --rest_port 8000 --model_name ornith9b

A DFlash draft model reads intermediate decoder hidden states from the target model (hidden_states_decoder_layers) and proposes a block of candidate tokens that the target verifies in a single forward pass. Because the draft and target must share identical vocabulary indexing and hidden-state geometry, only pair a draft with the exact target family it was exported for. For models with a built-in MTP head (openvino_mtp_model.xml inside the target directory), OVMS picks up the head automatically and num_assistant_tokens in the JSON request body controls the speculation depth.

Intel Arc B580 LLM benchmarks and token rates

Single stream, 8K context window, INT4 weights unless noted:

Model	Architecture & precision	Tokens/sec	With DFlash
LFM2.5-2.6B-heretic	2.6B dense, INT4	230	—
Qwen3-4B	4B dense, INT4	150	—
Qwen3-8B	8B dense, INT4	75	—
Qwen3.5-9B-abliterated	9B dense, INT4	68	154
Ornith-1.5-9B	9B dense, INT4, vision	68	199
Ornith-1.5-9B	9B dense, INT8, vision	34	148
LFM2.5-8B-A1B-heretic	8B MoE, 1B active, INT4	600	—

Two rows define how to use the Arc B580 effectively:

• The MoE row (LFM2.5-8B-A1B-heretic at 600 tok/s): All 8 billion parameters reside in the B580’s 12 GB VRAM, but only ~1 billion parameters are activated per generated token. That gives you the memory footprint of an 8B model with the decode speed of a 1B model.
• The DFlash rows (68 → 154 tok/s on Qwen3.5-9B, and 68 → 199 tok/s on Ornith-1.5-9B): Attaching the ~0.9 GB DFlash draft pushes a 9B dense model past a standalone 4B model (199 tok/s vs 150 tok/s) without giving up 9B reasoning or vision quality. Even the heavier INT8 build of Ornith-1.5-9B jumps from 34 tok/s to 148 tok/s with the INT4 DFlash draft attached, recovering more than 4x throughput by verifying multiple tokens per weight read.

Because prompt prefill on the 160 XMX engines is compute-bound while token generation is memory-bandwidth-bound, long prompts process quickly while long responses dominate wall-clock time. DFlash speculative decoding attacks that exact bottleneck by emitting several verified tokens per pass over the target weights.

What doesn’t work on the Intel Arc B580

Resizable BAR is mandatory. Enable Above 4G Decoding and Resizable BAR (ReBAR) in your motherboard BIOS, disable CSM, and boot in UEFI mode. Without ReBAR, the Arc B580 still initializes and runs models without throwing an error, but host-to-VRAM transfers crater and token generation loses a massive share of its throughput.

27B dense models do not fit in 12 GB VRAM. A 27B model at INT4 requires roughly 15 GB for weights alone before allocating a single megabyte of KV cache. Running it on a 12 GB card forces partial CPU/RAM offload over the PCIe 4.0 x8 bus, collapsing decode speed to CPU-bound rates.

INT8 9B leaves zero safety margin for desktop use. At ~9.6 GB of weights plus ~1.15 GB for an 8K KV cache, an INT8 9B model consumes nearly 11 GB before display compositors or a DFlash draft are counted. Stick to INT4 on 12 GB cards and save full INT8 targets for 16 GB+ GPUs or 32 GB Core Ultra systems.

Unbounded context windows cause silent OOMs. Pushing a 9B model to a 32K context window allocates ~4.6 GB of KV cache alongside ~5.3 GB of weights and ~1 GB of draft weights. Cap max_num_seqs and context length to match your actual workload.

Windows WDDM VRAM reservations vary by display setup. Windows reserves a variable slice of VRAM for the desktop compositor and connected monitors. Check actual free device memory in Task Manager or intel_gpu_top on Linux before sizing your KV cache ceiling.

Intel Arc B580 local AI FAQ

Q: Can I run GGUF models with llama.cpp, Ollama, or LM Studio on the Arc B580 instead of OpenVINO IR? Yes. Both the Vulkan and SYCL backends in llama.cpp (as well as LM Studio and Ollama via Vulkan) run GGUF models on the Arc B580 at comparable single-token baseline decode speeds, since reading INT4 weights is governed by the same 456 GB/s GDDR6 bus. OpenVINO IR with OVMS pulls ahead when you want XMX-accelerated prompt prefill, 2.26x–4.35x DFlash speculative decoding, or an OpenAI-compatible serving endpoint.

Q: Does running the Arc B580 in a PCIe 3.0 slot hurt LLM token generation? Once the model weights and KV cache are fully resident inside the card’s 12 GB GDDR6, token generation happens entirely on-card and PCIe link speed barely affects single-user decode. PCIe 3.0 x8 only slows down initial model loading and any workload that spills weights into system RAM.

Q: How many concurrent users can an Arc B580 serve with OVMS? One heavy 9B session comfortably, or two to four concurrent sessions if you drop to a 2.6B or 4B model. Continuous batching in OVMS shares weight reads across parallel requests, but each concurrent request requires its own independent KV cache allocation inside the 12 GB ceiling.

Q: Why is my measured tok/s lower than the table above? The rates above reflect a dedicated GPU with an 8K context window and all weights resident in GDDR6. Expect a 15–25% reduction when driving high-refresh monitors with a desktop browser open, and a 70%+ drop if Resizable BAR is disabled in BIOS or OVMS silently fell back to an integrated GPU or CPU.

Q: Is 12 GB VRAM enough for local vision-language models (VLMs) on the Arc B580? Yes. blaj/Ornith-1.5-9B-int4-ov plus its DFlash draft (blaj/Ornith-1.5-9B-DFlash-int4-ov) and vision encoder fit comfortably alongside an 8K context window. Keep image resolutions reasonable, as high-resolution images tokenize into thousands of visual tokens that fill the KV cache rapidly.

Related guides

• How to Run Local LLMs with OpenVINO on Intel Arc — full overview of the OpenVINO, OVMS, SYCL, and Vulkan stack on Arc GPUs
• Best OpenVINO Models with DFlash Speculative Decoding for Intel Arc and Core Ultra — complete blaj model catalog and hardware tier tables
• How to Convert Models to OpenVINO IR — exporting custom Hugging Face checkpoints to INT4 and INT8 OpenVINO IR
• How to Convert and Run DFlash Models with OpenVINO — exporting and verifying custom DFlash speculative-decoding drafts