// the find
intel/neural-compressor
SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime
Intel's official library for quantizing neural networks — INT8, FP8, INT4, the whole alphabet — across PyTorch, TensorFlow, and now JAX. The primary audience is ML engineers who need to run large models on Intel hardware (Gaudi accelerators, Xeon, Core Ultra) and want production-grade quantization without building it themselves. It also works on NVIDIA and AMD, though with less testing depth.
The AutoRound integration is the real draw: it's one of the better weight-only quantization methods for LLMs and handles tricky models like DeepSeek and Qwen where naive GPTQ falls apart. The MX format support (MXFP8, MXFP4) is ahead of most open-source alternatives, which matters if you're targeting next-gen hardware that supports microscaling. The calibration-based static quantization pipeline is mature — prepare/convert follows PyTorch's own API conventions, so there's minimal impedance mismatch if you already know PyTorch's quantization. Test coverage is reportedly 85% and the CI matrix across frameworks is genuinely comprehensive.
The Intel hardware bias is real: the getting-started example requires a Gaudi2 Docker image, and several features are 'limited testing' on non-Intel hardware — which in practice means you're on your own. NVFP4 and MX quantization are both marked experimental, so don't ship those to prod yet. The library has accumulated a lot of API surface across its version history and older patterns still exist in docs and examples alongside the current ones, making it hard to know which approach to follow. Stars-to-forks ratio (2678/313) suggests more spectators than practitioners, which often means you'll hit edge cases without community precedent.