// the find
Snowflake-Labs/snowflake-arctic
This is Snowflake's research repo for their Arctic LLM — a 480B parameter Dense-MoE Hybrid model with only 17B active parameters, optimized for enterprise tasks like SQL generation and code. It also contains inference and fine-tuning scripts for Llama 3.1 405B. Aimed at ML engineers who want to run or fine-tune massive models on their own infrastructure.
Arctic's Dense-MoE architecture is genuinely interesting — stacking a 10B dense transformer with a residual 128-expert MoE gives you a model that punches above its active-parameter weight on SQL and code tasks. The training efficiency story holds up: competitive with Llama 3 70B on enterprise benchmarks at 7x lower compute cost. Apache 2.0 license with no gating on weights is the real thing, not watered-down 'open'. The Llama 3.1 405B fine-tuning work — FP8 quantization, ZeRO-3 sharding, single-node support — is practically useful for teams that can't afford a multi-node cluster.
The repo is essentially dormant — last push was August 2024 and Arctic itself has been superseded without any follow-up model work here. The actual training code for Arctic isn't in this repo at all; it's just inference scripts and LoRA fine-tuning wrappers, so calling this a research artifact is generous. Running Arctic requires 8x A100 80GB at minimum, which puts it out of reach for anyone not already operating at Snowflake-tier hardware. Documentation is thin beyond README links to blog posts that may rot.