// the find
anza-xyz/jetstreamer
A Solana project geared towards realtime indexing, research, and backfilling with support for all epochs in the history of Solana mainnet, capable of 2.7M TPS+
Jetstreamer streams historical Solana blocks and transactions from Old Faithful's CAR archives over the network and feeds them to a local plugin or Geyser-style consumer. It's aimed at people doing backfills, research, or indexing who need full mainnet history rather than live data.
- The Limitations section says plainly what the project lacks: no account updates, and transactions arrive in their already-executed state, so this replays recorded results rather than executing anything.
- Write durability is spelled out: async_insert with wait_for_async_insert=1, retries with capped backoff for ten minutes, in-flight writes drained on shutdown, and an abort that prints the exact resume slot. The failure path is documented, not left to the reader to discover.
- The firehose connection management (a health-gated ramp-up, recycling of slow connections, and work stealing where the least-progressed thread hands over half its remaining slots) addresses the real problem of CDN throttling over long runs.
- The epoch table separates pre-157 epochs that don't work with modern Geyser from the CU-tracking cutoff at 450, so you can pick a replay window without trial and error.
- The 2.7M TPS figure is a best case: a 64-core machine, a 30 Gbps link, and a local plugin. The README says so, but the number leads the description and most people will see a fraction of it.
- The build is fragile. It needs Clang 16 specifically because of RocksDB, and the recommended path is nix-shell. Anyone on a default Ubuntu or Windows toolchain will spend time on setup before running anything.
- The default ClickHouse mode spawns a bundled server out of the repo's bin/ directory, so a first run takes more than a binary and a DSN. The data lives in that directory, which helps with migration but is easy to lose track of.
- Delivery is at-least-once into ReplacingMergeTree tables, so anyone reading during ingestion has to remember FINAL or tolerate duplicates. Parallel threads also make write order arbitrary, as the README admits, so plugin authors have to design their schemas around both.