// the find
nyrahealth/CrisperWhisper
Verbatim Automatic Speech Recognition with improved word-level timestamps and filler detection
CrisperWhisper is a fine-tuned variant of Whisper Large v3 that prioritizes verbatim transcription — capturing fillers ('um', 'uh'), stutters, and false starts rather than silently cleaning them up. It pairs that with improved word-level timestamps via a custom attention loss on selected DTW alignment heads. The target audience is anyone who needs forensic-grade transcripts: qualitative researchers, medical transcription, meeting analysis.
The timestamp accuracy improvement is real and measurable — F1 of 0.79 vs 0.66 on AMI IHM is a significant gap, not a rounding win. The attention loss approach is technically sound: training the cross-attention heads directly on timestamped ground truth rather than relying on post-hoc DTW is the right place to intervene. WER on verbatim datasets (AMI at 8.72 vs 16.01) shows the tokenizer retokenization actually works and isn't just marketing. The three-stage training pipeline — tokenizer adaptation, verbatim finetuning, then attention loss — is well thought out and documented.
CC BY-NC 4.0 license is a hard blocker for any commercial use, including SaaS products, and that's buried at the bottom of the README rather than leading with it. The faster-whisper path explicitly warns that timestamp accuracy cannot be guaranteed, which undercuts the main selling point if you need inference speed. Requires a fork of Hugging Face transformers ('nyrahealth/transformers@crisper_whisper') rather than mainline, which is a maintenance liability as transformers keeps moving. The repo is essentially a model card with a thin wrapper — no evaluation scripts, no fine-tuning code, no guidance on what to do if your domain (medical, legal, non-English) doesn't match training data.