finds.dev← search

// the find

lemire/fastvalidate-utf-8

★ 309 · C · Apache-2.0 · updated Mar 2024

header-only library to validate utf-8 strings at high speeds (using SIMD instructions)

A header-only C library for SIMD-accelerated UTF-8 validation, reaching 0.7 cycles/byte on SSE and 0.45 on AVX. It was a research proof-of-concept that directly led to the simdutf library. The authors say it plainly in the README: this is obsolete and you should use simdutf instead.

The benchmarks are honest and methodical — they compare against a scalar baseline and show exactly where the speedup comes from. The triple license (Apache/Boost/MIT) removes any friction for commercial use. The research paper backing it (Keiser & Lemire, 2021) gives the algorithm a real pedigree, not just a benchmark claim. The ASCII fast-path getting to 0.088 cycles/byte is genuinely useful for systems where most traffic is ASCII.

It is explicitly deprecated by its own author — the README leads with this. x64/SSE-only with no ARM support means it is dead on any modern Apple hardware or cloud ARM instance. The codebase has had essentially zero activity since 2022, so there will be no fixes for edge cases you might find. Anyone who adopts this instead of simdutf is making a maintenance problem for themselves.

View on GitHub →

// want more like this?

We dig through GitHub every week and send a few repos picked for what you actually care about — each with an honest take like this one.

Get finds in your inbox → Search again →