SmolLM – Fast and Remarkably Powerful
SmolLM introduces a family of small language models with 135M, 360M, and 1.7B parameters trained on a new high-quality dataset called SmolLM-Corpus.
- The 1.7B model uses a more traditional architecture while smaller models adopt Grouped-Query Attention and prioritize depth over width.
- Models trained on curated textbooks outperform web-based textbooks on MMLU and ARC benchmarks.