// the find
FluidInference/FluidAudio
Frontier CoreML audio models in your apps — text-to-speech, speech-to-text, voice activity detection, and speaker diarization. In Swift, powered by SOTA open source.
FluidAudio is a Swift SDK that runs speech AI entirely on Apple hardware — ASR, TTS, VAD, and speaker diarization — using CoreML with inference routed to the Apple Neural Engine. It wraps several SOTA open-source models (Parakeet, Kokoro, Silero, Pyannote) in a Swift-native API and handles model downloads, format conversion, and ANE scheduling so you don't have to. Aimed at iOS/macOS app developers who want on-device audio AI without shipping a Python runtime or hitting a cloud API.
The ANE routing is the real differentiator — by avoiding GPU/MPS entirely, you get background-capable inference and lower power draw, which matters for ambient/always-on use cases like meeting transcription. The diarization story is unusually complete for an on-device library: three distinct pipeline options (LS-EEND, Sortformer, Pyannote) with honest guidance on when each is the right call. Model catalog breadth is impressive without feeling scattered — multilingual ASR, streaming with EOU detection, voice cloning TTS, and VAD all exist and are documented with benchmarks. The offline-mode enforcement flag and custom registry URL override are practical features most libraries omit, covering real enterprise and privacy-sensitive deployment scenarios.
Android/Windows support is explicitly absent and unscheduled — the Rust wrapper exists but it's a thin shim, not a first-class port, so cross-platform teams will hit a wall. Speaker diarization accuracy degrades noticeably past 4-5 speakers or with heavy overlapping speech; the docs are honest about this but it's a hard limit of the ANE-optimized architectures. The model download-on-first-use pattern means cold start in production is user-visible — there's no built-in mechanism for pre-bundling models at build time beyond the manual loading APIs, which require you to handle HuggingFace asset management yourself. TTS is still beta and English-heavy: Kokoro only covers American English well, PocketTTS multilingual quality varies significantly by language, and prosody control is limited compared to cloud alternatives.