finds.dev

abacusai/Long-Context

This repository contains code and tooling for the Abacus.AI LLM Context Expansion project. Also included are evaluation scripts and benchmark tasks that evaluate a model’s information retrieval capabilities with context expansion. We also include key experimental results and instructions for reproducing and building on them.

View on GitHub

★ 604PythonApache-2.0updated Nov 2023

// the take

Research code and benchmarks from Abacus.AI's 2023 experiments extending Llama's context window beyond 2048 tokens via RoPE linear scaling and fine-tuning. The main contribution is the Giraffe-13b model (scale=16) and the WikiQA evaluation datasets. This is a research artifact, not a library you'd integrate into a project.

The WikiQA benchmark design is genuinely thoughtful — altering numeric answers to block memorization cheating is the right move and something many evals skip. The zero-shot scale generalization finding (train at 4x, eval at 8x and still get non-zero accuracy) is a concrete and interesting empirical result. Both datasets are released on HuggingFace, so the evaluation methodology is actually reusable. The write-up honestly reports where approaches failed (XPOS never converged) rather than cherry-picking wins.

Last commit November 2023 — RoPE scaling has moved fast since then (YaRN, LongRoPE, and every major model now ships 128k+ context natively), so the practical relevance of extending Llama-13b to 16k is close to zero today. The repo is training scripts and notebooks, not a usable package — no pip install, no clean API, expect friction to get anything running. Presence accuracy (substring match) is a weak eval metric that will over-count partial answers and miss paraphrases. The scale=16 model caps out around 16k tokens in practice despite the math suggesting 32k, and the README admits they don't have a fix for that.

We read the README, the directory tree and a key source file in June 2026. Nobody pays to be listed. How we judge →

// at a glance

Stars
604
Forks
46
Language
Python
License
Apache-2.0
Last commit
17 Nov 2023

// nearby

If Long-Context isn't quite it.

All repos →