// the find
arpitbbhayani/WikiSe
A wikipedia search engine that is completely built in Java and works on Wikipedia XML dumps
A from-scratch Wikipedia search engine built in Java that processes the full English Wikipedia XML dump (~47GB) into a custom inverted index with bit-level compression. It's a university-style systems project — the kind where you learn why Lucene exists. Targeted at developers who want to see how a search index is built without library magic.
The field-scoped query syntax (website:, age:, powers:, nick:) is a nice touch for structured infobox data — it shows the author actually parsed Wikipedia's template markup rather than just tokenizing everything. Bit-level index compression getting 9.7GB down is a real engineering decision, not an afterthought. SAX parsing is the right call for streaming a 47GB XML file without loading it into memory. The secondary index layer (hashing terms to a-z buckets) is a clean two-level lookup that keeps random I/O manageable on spinning rust.
No build system — it's an IntelliJ project with a hardcoded .iml file and no Maven/Gradle, so building it outside the IDE is your problem. Last touched in 2024 but the code feels like a 2015 college assignment: no tests, no ranking algorithm beyond what the README vaguely gestures at, and the 'interesting search' examples (chudail, pratieik) suggest the ranking is basically TF-IDF at best. There's no query parser for boolean logic, phrase search, or anything more expressive than the field prefix trick. The index format is completely proprietary and undocumented, so you can't inspect it, migrate it, or build tooling around it without reading the source.