intermediate · AI · October 18, 2026 · 8 min read

How to Convert an Abliterated (Uncensored) Model to OpenVINO int8 IR

Turn a BF16 abliterated model from Hugging Face into OpenVINO int8 or int4 IR, including a two-stage conversion and thread cap for large MoE models.

openvino model-conversion quantization abliterated optimum intel-arc

Why abliterated models need a different recipe

Abliterated models — also called uncensored or “heretic” models — are usually published as full BF16 safetensors, not as pre-quantized checkpoints. That matters because the tooling most people reach for first does not help:

GGUF is everywhere, OpenVINO IR is not. Most abliterated releases ship only GGUF quantizations (Q4_K_M, Q6_K, and so on). Those are for llama.cpp, not OpenVINO.
You cannot quantize a GGUF. There is no path from a GGUF file to an int8 OpenVINO IR. You need the BF16 source weights.
The relevant repos are small and unvalidated. A handful of community members do publish -int4-ov and -int8-ov conversions, but they typically have double-digit download counts and no validation artifacts.

So the reliable path is: take the BF16 safetensors, export them yourself, and control the precision.

That path has two shapes depending on model size. A small model converts with one optimum-cli command. A large or mixture-of-experts model will exhaust RAM in a single-command export, and needs the fp16-first two-stage approach described below — along with a cap on the export’s OpenMP thread count.

Get the BF16 source weights

Start by finding a release with full safetensors rather than a GGUF-only repo. The heretic-org namespace is a good signal — it publishes upstream-architecture abliterations with a reproduce/ directory containing the config, requirements, and SHA256 sums used to create the model.

For this walkthrough the target is a 2.6B LFM2.5 model:

huggingface-cli download heretic-org/LFM-2.5-2.6B-heretic \
  --include "*.json" "*.jinja" "*.safetensors" "LICENSE" \
  --exclude "reproduce/*"

Confirm the architecture before converting. The model type has to be one that optimum-intel can export:

import json

cfg = json.load(open("config.json"))
print(cfg["model_type"])       # lfm2
print(cfg["architectures"])    # ['Lfm2ForCausalLM']
print(cfg["num_hidden_layers"])  # 30

Install the conversion toolchain

Conversion runs through optimum-cli, which comes from optimum-intel:

pip install optimum[openvino] nncf

Pin transformers, or the export refuses to run

This is the step that wastes the most time if you skip it. optimum-intel enforces a hard runtime ceiling on transformers at export time, and the error message names the exact version:

ValueError: The current version of Transformers does not allow for the export
of the model. Maximum required is 5.4.0, got: 5.10.4

Note the distinction. optimum-intel’s package metadata declares transformers<5.6, but the runtime gate in optimum/exporters/openvino/convert.py checks for 5.4.0 exactly. Installing 5.5.4 satisfies the metadata and still fails the gate. Pin the literal version:

pip install "transformers==5.4.0"

Verify the toolchain before spending time on a conversion:

optimum-cli --help

If that errors with command not found on a path that clearly exists, the console script’s shebang is stale. This happens whenever a virtualenv is moved or renamed — the entry-point scripts keep pointing at the old interpreter path:

head -1 "$(which optimum-cli)"
# !/home/user/old-path/bin/python   <- stale

# Repair every entry point in the venv at once:
grep -rl 'old-path/bin/python' bin/ | while read -r f; do
  sed -i '1s|.*|#!/absolute/path/to/current/bin/python3|' "$f"
done

Export to int8 OpenVINO IR

The export command takes the source directory and an output directory:

optimum-cli export openvino \
  --model /path/to/LFM-2.5-2.6B-heretic \
  --task text-generation-with-past \
  --weight-format int8 \
  /home/user/Documents/models/lfm2.5-2.6b-heretic-int8-ov

--weight-format int8 performs asymmetric per-channel weight compression through NNCF. The export prints the resulting bitwidth distribution, which is worth reading:

┌───────────────────────────┬─────────────────────────────┐
│ Weight compression mode   │ % all parameters (layers)   │
┝───────────────────────────┿─────────────────────────────┥
│ float                     │ 0% (1 / 190)                │
├───────────────────────────┼─────────────────────────────┤
│ int8_asym, per-channel    │ 100% (189 / 190)            │
└───────────────────────────┴─────────────────────────────┘

A single layer staying in float is normal — it is typically a norm or embedding layer that compression excludes. Anything more than that is worth investigating before you ship the model.

The result is a standard OpenVINO IR layout:

lfm2.5-2.6b-heretic-int8-ov/
├── openvino_model.xml
├── openvino_model.bin
├── openvino_tokenizer.xml
├── openvino_tokenizer.bin
├── openvino_detokenizer.xml
├── openvino_detokenizer.bin
├── config.json
└── chat_template.jinja

TracerWarnings during export are expected for hybrid attention models and are not failures:

TracerWarning: Converting a tensor to a Python boolean might cause the trace
to be incorrect.

Large models: export first, compress second

The single-command export above works for models that fit comfortably in RAM. It does not work for a large MoE model on a 30 GB machine.

A direct --weight-format int8 run on a 16.9 GB BF16 mixture-of-experts model failed here three times. optimum-cli loads the full PyTorch checkpoint and then builds the traced graph, and together they exceed available memory:

Out of memory: Killed process (optimum-cli)
total-vm:82377092kB, anon-rss:28596328kB

28.6 GB of anonymous RSS against 30 GB of RAM plus 32 GB of swap, at which point the kernel kills the process before it writes any output. This is not a problem you can wait out — the memory has to fit.

Split the job in two

Export an fp16 IR first, then compress that IR as a separate step. Compressing an OpenVINO IR never loads PyTorch, so it peaks far lower.

# Stage 1: BF16 weights -> fp16 IR
optimum-cli export openvino \
  --model /path/to/lfm2.5-8b-a1b-heretic \
  --task text-generation-with-past \
  --weight-format fp16 \
  /path/to/lfm2.5-8b-a1b-heretic-fp16

# Stage 2: fp16 IR -> int8 IR
import openvino as ov
import nncf

model = ov.Core().read_model("lfm2.5-8b-a1b-heretic-fp16/openvino_model.xml")
compressed = nncf.compress_weights(
    model,
    mode=nncf.CompressWeightsMode.INT8_ASYM,
    ratio=1.0,
)
ov.save_model(compressed, "lfm2.5-8b-a1b-heretic-int8-ov/openvino_model.xml")

For this 8B MoE model the split looks like:

Stage	Time	Peak RSS
fp16 export	~10 min	fits after thread cap
int8 compression of the IR	29 s	24.9 GB
int4 compression of the IR	39 s	21.1 GB

The payoff is that int8 and int4 both derive from one fp16 IR. You pay the expensive Torch load once.

Cap the OpenMP thread pools

The decisive fix is limiting how many threads the export may use:

export OMP_NUM_THREADS=4
export MKL_NUM_THREADS=4
export OPENBLAS_NUM_THREADS=4
export NUMEXPR_NUM_THREADS=4

By default the export runs with 200+ threads. Each OpenMP worker materializes its own buffers during tracing, so peak resident memory scales with thread count. Capping to 4 reduced the process to 12 threads and the export completed without being killed:

Configuration	Threads	Peak RSS	Outcome
Uncapped, int8	200+	28.6 GB	OOM-killed
Uncapped, fp16	200+	27.4 GB + 27.7 GB swap	no forward progress
Capped, fp16	12	fits	completed in ~10 min

If an export is killed with no Python traceback, check the kernel log for an out-of-memory termination and compare the reported anon-rss against your available memory. If RSS is near your RAM ceiling, lower the thread counts before adding more swap — swap did not save the uncapped runs.

Verify the model generates

Do not assume a successful export means a working model. Load it and run a generation before publishing anything:

import openvino_genai as og

pipe = og.LLMPipeline(
    "/home/user/Documents/models/lfm2.5-2.6b-heretic-int8-ov",
    "GPU",
)
config = og.GenerationConfig(max_new_tokens=64, temperature=0.0)

print(pipe.generate("User: Explain what an abliterated language model is.\nAssistant:", config))

Check that the output is coherent prose in the model’s own voice rather than a repeated or empty string. For an abliterated model, also confirm it responds to the prompt instead of deflecting — that is the actual behavior you converted it for.

Serve it locally

If you already run an OpenAI-compatible server that auto-discovers IR directories, the new model appears with no configuration change. A discovery endpoint confirms it:

curl -s http://127.0.0.1:11436/v1/models

{"object":"list","data":[
  {"id":"gemma4-e4b-int4-ov"},
  {"id":"lfm2.5-2.6b-heretic-int8-ov"},
  {"id":"lfm2.5-8b-a1b-int4-ov"}
]}

With OVMS itself, the modern invocation uses --model_path plus a generated graph rather than the legacy model_config_list JSON:

ovms --configure --model_path /path/to/model --task text_generation
ovms --model_path /path/to/model --task text_generation --rest_port 11436

Choosing int8 over int4

int8 and int4 are different tradeoffs, and for abliterated models int8 is often the right default:

Format	Typical size (2.6B dense)	Measured (8B MoE)	Notes
BF16 source	~5.2 GB	16.9 GB	Reference quality
int8	~2.6 GB	8.0 GB	Minimal quality loss
int4	~1.5 GB	4.3 GB	Smaller, more degradation

Abliteration removes the refusal direction from the weights. Aggressive quantization can erode other capabilities along with it, so int8 keeps more of the original model’s behavior intact. Reach for int4 when memory is the binding constraint, not by default.

MoE models convert proportionally larger because every expert is kept. The 8B mixture-of-experts model above is roughly 3x the parameter count of the 2.6B dense model, and its int8 IR is correspondingly about 3x the size.

Common mistakes

Using a GGUF as the conversion source

GGUF is an inference format, not a conversion input. Start from BF16 safetensors even if a GGUF build is more convenient to download.

Leaving transformers unpinned

The export gate wants 5.4.0 specifically. A newer version that satisfies the declared metadata range will still be rejected at runtime.

Ignoring a broken venv shebang

If a virtualenv has been moved or renamed, every console script in it has a stale interpreter path. The command appears missing even though the file is present.

Publishing without running a generation

An export that completes without error can still produce a model that loops or emits empty output. Always load the IR and generate before you consider it done.

Assuming conversion implies quantization

Converting to IR and compressing weights are separate steps. --weight-format int8 is what actually produces int8 weights; plain export leaves them at source precision.

Adding swap instead of lowering threads

When a large export is OOM-killed, more swap is the intuitive fix and it did not work here. The uncapped run that had 32 GB of swap available still consumed 27.4 GB RSS plus 27.7 GB swap and made no forward progress, because the memory demand scales with thread count. Lower OMP_NUM_THREADS first.

Quick reference

BF16 abliterated weights (safetensors)
     |
     v
pip install "transformers==5.4.0"   <- required pin
     |
     +-- model fits in RAM --> optimum-cli export --weight-format int8
     |
     +-- large / MoE model --> export fp16 IR first (cap OMP threads),
                               then nncf.compress_weights on the IR
     |
     v
openvino_model.xml + .bin (+ tokenizer/detokenizer)
     |
     v
og.LLMPipeline(dir, "GPU")  -> verify generation
     |
     v
Serve via OpenAI-compatible endpoint