// the find
m-bain/whisperX
WhisperX: Automatic Speech Recognition with Word-level Timestamps (& Diarization)
WhisperX wraps OpenAI's Whisper with a pipeline that adds word-level timestamps via forced phoneme alignment, VAD-based batching for throughput, and optional speaker diarization via pyannote. It's for anyone who needs accurate per-word timing or speaker labels from audio — subtitle generation, meeting transcription, media indexing. The research paper was accepted at INTERSPEECH 2023.
The 70x realtime speed claim is real — batched inference with the faster-whisper CTranslate2 backend is a meaningful engineering improvement over naive Whisper usage. Word-level alignment via wav2vec2 forced alignment actually works and is noticeably better than Whisper's own timestamp output on real audio. VAD preprocessing before batching is the right call: it reduces hallucination and avoids the sliding-window artifacts you get with buffered transcription. The project is actively maintained (last push yesterday) and the README is honest about the limitations.
Diarization quality is acknowledged as poor in the README itself — overlapping speech is handled badly, and speaker count has to be guessed or set manually. Numeric tokens and special characters (prices, years) can't be aligned at all since wav2vec2 only knows phonemes, so those words silently drop timing. The dependency chain is heavy: CUDA, pyannote (which requires a HuggingFace gated model accept), faster-whisper, torchaudio — getting this stack to run clean on a new machine is a half-day job. The .ass subtitle format was removed in v3 and the tests directory has exactly one test file.