pythia-70m-deduped

95.6M parameters, text in, text out, 2K tokens of context.

Verified 2026-10-25 Rank #364 in LLMs · score 80.7 method v1 Sources 3 Downloads 30d 1.4M · #92 of 619 tracked

Can I run it?0.1GB at INT4Estimatefits a 16 GB cardmethod

Can I use it?Apache 2.0Permissive; patent grantlicence file2026-10-24

Do people use it?1.4Mdownloads, 30 days#92 of 619 we track · top 15%+2 likes, downloads −3% this week38 likesHugging Face2026-10-24

Worth reading

Picked by rule from 8 tracked pieces: the lab's own post, the most shared, a field report, the latest.

Most discussed · 68 HN pts

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

Pythia is a suite of 16 large language models all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters.

arxiv.org 2023-04-05 68 HN pts · 7 comments 370 words about the pythia family family

Latest · 4 years ago

RedPajama at 440B tokens higher quality than Pythia and StableLM

together.xyz 2023-04-24 6 HN pts · 2 comments about the pythia family family

8 best of 8 tracked

The 2 picks above, plus these 6. Top is the radar's score: discussion, source authority and the lab's own posts, fading with age.

ArticlePythia: Open-source framework for multimodal AI modelscode.fb.com · 2019-05-21 · 66 HN pts
ArticlePythia 1.4B reproduces 3.6% of training samples verbatim given 950-token promptsret2libc.com · 2026-06-09
CodePythia: Interpreting Autoregressive Transformers Across Time and Scalegithub.com · 2023-04-05 · 2 HN pts
ArticleHuggingFace API Tokens were exposed, leaving users vulnerablelasso.security · 2023-12-06 · 2 HN pts
ArticleFacebook Open-Sources Pythia for Vision and Language Multimodal AI Modelsmedium.com · 2019-05-21 · 2 HN pts
ArticlePythia: The first deep learning-based ancient text restoration modelarchaeology.wiki · 2019-10-22 · 1 HN pts

Show all 6

Where it's used

Hugging Face does not say who downloads a model or with what. These are the proxies: API providers listing it, quantized builds, fine-tunes/adapters/merges, and Spaces that load it.

No API provider serves it, so its downloads are people running it themselves: 21 quantized builds and 30 Spaces.

As a service0no API provider listed
On own hardware21quantized builds
Built on28657 fine-tunes · 229 adapters · 0 merges
In apps30Hugging Face Spaces

Top Spaces running it: mlabonne/chessllm ♥ 77

Will it fit?

Weights only, an estimate: leave room for the KV cache. Method

Fits a 16 GB card at INT4

About 0.1 GB at INT4, 0.1 GB at FP8, 0.2 GB at BF16: weights only, so leave room for the KV cache.

Estimate params × bytes per param × 1.2 · how we estimate

The story so far

Weights, every dated article, and repo commits on one rail.

7 dated articles and 9 repo commits so far

Teal is the weights (2023-02-13), amber triangles are 7 dated articles (each links to it), grey ticks are repo commits.

What people built on it

307 derivatives on the Hub. Which build to run, by runtime: the most downloaded in each class.

307 derivatives on the Hub, most of them adapters

From the Hugging Face model tree; quantized builds in teal, fine-tunes, merges and adapters in greys.

GGUF mradermacher/pythia-70m-deduped-i1-GGUF for llama.cpp, Ollama, LM Studio 8.6k downloads GPTQ fbaldassarri/EleutherAI_pythia-70m-deduped-autogptq-int4-gs128-asym for vLLM, SGLang 167 downloads

All variants →

Reference

Everything else, folded: the full tables, every source.

Use-case evidence

1 use case

What it is good at 1 use case

Labels are evidence, not verdicts: measured a third-party leaderboard row for this model; author-reported a number the model author published in the card or eval files; structural a deterministic signal from the name, config or chat template, or a decision-model estimate. Each chip links to its source. All use cases.

Run on a laptop or phonestructuraldecision-model estimate p=0.99 fits: edge

Facts

21 fields

Listed in LLMs

Facts

pythia-70m-deduped has 95.6M parameters. (safetensors metadata, observed 2026-10-24)
Context length is 2,048 tokens (2K). (config.json max_position_embeddings, observed 2026-10-24)
License: Apache 2.0 (friction: none). Permissive; patent grant. (license file, observed 2026-10-24)
Created 2023-02-13, last modified 2023-07-09. (HF API, observed 2026-10-24)

Pipeline tag	text-generation
Library	transformers
Architectures	GPTNeoXForCausalLM
Model type	gpt_neox
Dtype	float16
Params by dtype	U8: 25,165,824 · F16: 70,426,672
Size band	S
MoE	no
Context length	2,048 (config.json max_position_embeddings)
Languages	en
Papers	arXiv:2304.01373, arXiv:2101.00027, arXiv:2201.07311
License	Apache 2.0 (apache-2.0)
Hub snapshot	2026-10-25

Servability

vLLM · SGLang

Servability

vLLM: supported · GPTNeoXForCausalLM · 7050ef881ff0835f6890a47db1859a24c65481d6SGLang: not in registry · 773cb938fe9484a8f7e7e46fb1b1fe96412c59ed

vLLM registry (7050ef881ff0835f6890a47db1859a24c65481d6) · SGLang registry (773cb938fe9484a8f7e7e46fb1b1fe96412c59ed) · Matched by architecture class name against each project's registry on main; 'supported' means the architecture is registered, not that this checkpoint was tested.

Hardware fit

3 estimates

Hardware fit ESTIMATE

Weights-only memory for pythia-70m-deduped: params × bytes/param × 1.2 overhead. KV cache and activations are extra. How this is computed.

BF16 (2 bytes/param)	0.2 GB · Edge (<4 GB)
FP8 (1 byte/param)	0.1 GB · Edge (<4 GB)
INT4 (0.5 bytes/param)	0.1 GB · Edge (<4 GB)

Weights only (params × bytes/param × 1.2 overhead); KV cache and activations extra.

All variants

40 variants

Lineage

Original model Family: EleutherAI/pythia 307 derivatives on the Hub

finetune: 57quantized: 21adapter: 229

Top variants (40 shown, ranked by downloads)

Variant	Relation	Quant	Params	Band	License	Author	Likes	Downloads
mlabonne/chesspythia-70m	finetune	—	70.4M	S	apache-2.0	mlabonne	4	15.2k
mradermacher/pythia-70m-deduped-i1-GGUF	quantized	GGUF	—	—	apache-2.0	mradermacher	0	8.6k
mradermacher/pythia-70m-deduped-GGUF	quantized	GGUF	—	—	apache-2.0	mradermacher	0	6.4k
Xenova/pythia-70m-deduped	quantized	—	—	—	—	Xenova	0	3.4k
mlabonne/grandpythia-200k-70m	finetune	—	70.4M	S	apache-2.0	mlabonne	3	1.9k
tensorblock/pythia-70m-deduped-GGUF	quantized	GGUF	—	—	apache-2.0	tensorblock	0	2k
vineetsharma/databricks-dolly-15k-pythia-70m-deduped-v1	finetune	—	—	—	apache-2.0	vineetsharma	0	1.3k
tim-lawson/mlsae-pythia-70m-deduped-x64-k32-tfm	finetune	—	104M	S	mit	tim-lawson	0	849
jtatman/outputs	finetune	—	70.4M	S	apache-2.0	jtatman	0	751
ddgoede/pythia-70m-ce-matched	finetune	—	70.4M	S	apache-2.0	ddgoede	0	621
agentlans/pythia-70m-wikipedia-paragraphs	finetune	—	70.4M	S	apache-2.0	agentlans	3	319
vineetsharma/databricks-dolly-15k-pythia-70m-deduped	finetune	—	—	—	apache-2.0	vineetsharma	0	428
superlazycoder/chesspythia-70m-random_1M	finetune	—	70.4M	S	apache-2.0	superlazycoder	0	404
tim-lawson/mlsae-pythia-70m-deduped-x1-k32-tfm	finetune	—	71.0M	S	mit	tim-lawson	0	363
tim-lawson/mlsae-pythia-70m-deduped-x32-k32-tfm	finetune	—	87.2M	S	mit	tim-lawson	0	298
tim-lawson/mlsae-pythia-70m-deduped-x1-k32	finetune	—	525K	S	mit	tim-lawson	0	282
tim-lawson/mlsae-pythia-70m-deduped-x4-k32	finetune	—	2.10M	S	mit	tim-lawson	0	282
tim-lawson/mlsae-pythia-70m-deduped-x64-k64-tfm	finetune	—	104M	S	mit	tim-lawson	0	257
tim-lawson/mlsae-pythia-70m-deduped-x4-k32-tfm	finetune	—	72.5M	S	mit	tim-lawson	0	256
tim-lawson/mlsae-pythia-70m-deduped-x2-k32	finetune	—	1.05M	S	mit	tim-lawson	0	233
tim-lawson/mlsae-pythia-70m-deduped-x64-k512-tfm	finetune	—	104M	S	mit	tim-lawson	0	215
tim-lawson/mlsae-pythia-70m-deduped-x32-k32	finetune	—	16.8M	S	mit	tim-lawson	0	209
tim-lawson/mlsae-pythia-70m-deduped-x8-k32	finetune	—	4.19M	S	mit	tim-lawson	0	205
tim-lawson/mlsae-pythia-70m-deduped-x16-k32	finetune	—	8.39M	S	mit	tim-lawson	0	193
tim-lawson/mlsae-pythia-70m-deduped-x256-k32	finetune	—	134M	S	mit	tim-lawson	0	174
franzhanz/pythia-70m-deduped-finetuned-Fox	finetune	—	70.4M	S	apache-2.0	franzhanz	0	173
fbaldassarri/EleutherAI_pythia-70m-deduped-autogptq-int4-gs128-asym	quantized	gptq	70.4M	S	apache-2.0	fbaldassarri	0	167
tim-lawson/mlsae-pythia-70m-deduped-x64-k256-tfm	finetune	—	104M	S	mit	tim-lawson	0	166
franzhanz/pythia-70m-deduped-finetuned-Breitbart	finetune	—	70.4M	S	apache-2.0	franzhanz	0	164
franzhanz/pythia-70m-deduped-finetuned-NYT	finetune	—	70.4M	S	apache-2.0	franzhanz	0	155
franzhanz/pythia-70m-deduped-finetuned-Washpost	finetune	—	70.4M	S	apache-2.0	franzhanz	0	154
franzhanz/pythia-70m-deduped-finetuned-WSJ	finetune	—	70.4M	S	apache-2.0	franzhanz	0	148
tim-lawson/mlsae-pythia-70m-deduped-x64-k32	finetune	—	33.6M	S	mit	tim-lawson	0	143
franzhanz/pythia-70m-deduped-finetuned-Nypost	finetune	—	70.4M	S	apache-2.0	franzhanz	0	134
franzhanz/pythia-70m-deduped-finetuned-USAToday	finetune	—	70.4M	S	apache-2.0	franzhanz	0	133
fbaldassarri/EleutherAI_pythia-70m-deduped-autogptq-int8-gs128-sym	quantized	gptq	70.4M	S	apache-2.0	fbaldassarri	0	130
franzhanz/pythia-70m-deduped-finetuned-MSNBC	finetune	—	70.4M	S	apache-2.0	franzhanz	0	129
franzhanz/pythia-70m-deduped-finetuned-Newsmax	finetune	—	70.4M	S	apache-2.0	franzhanz	0	125
agentlans/pythia-70m-lmsys-prompts	finetune	—	70.4M	S	apache-2.0	agentlans	0	122
weav-geng/llama2-qlora-finetunined-french	adapter	—	—	—	—	weav-geng	0	98

Repository changelog

9 commits

Repository changelog 9 commits

Date	Commit	SHA
2023-07-09	Adding `safetensors` variant of this model (#3)	e93a9fa
2023-06-08	fix batch size and add paper	555800e
2023-04-03	Upload tokenizer.json	979bee2
2023-04-03	Upload special_tokens_map.json	04642c6
2023-04-03	Upload pytorch_model.bin	178954f
2023-04-03	Upload config.json	f289af0
2023-04-03	Upload tokenizer_config.json	a37449d
2023-03-29	create updated readme	b950174
2023-02-13	initial commit	8597c6b

Articles and coverage

8 pieces

Ranked by source signals (HN, Lobsters, authority domains, official posts) and recency. Summaries are generated from a stored copy of the article text; every sentence cites a span of that text, and nothing is shown that failed the citation check.

Analysis7

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

arxiv.org2023-04-05authorityabout the pythia familyHN 68 pts · 7 comments370 words

Pythia is a suite of 16 large language models all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters. The release provides public access to 154 checkpoints for each model alongside tools to download and reconstruct their exact training dataloaders for further study. Case studies include novel results in memorization, term frequency effects on few-shot performance, and reducing gender bias.written, reviewed, cited

Pythia: Open-source framework for multimodal AI models

code.fb.com2019-05-21about the pythia familyHN 66 pts · 18 comments248 words

Pythia is a deep learning framework supporting multitasking in vision and language domains. Built on open-source PyTorch, it incorporates elements from winning entries in the VQA Challenge 2018 and Vizwiz Challenge 2018. The modular design supports distributed training and custom losses to accelerate prototyping and improve reproducibility for the community.written, reviewed, cited

RedPajama at 440B tokens higher quality than Pythia and StableLM

together.xyz2023-04-24about the pythia familyHN 6 pts · 2 comments

Pythia: Interpreting Autoregressive Transformers Across Time and Scale

github.com2023-04-05authorityabout the pythia familyHN 2 pts · 0 comments3,770 words

EleutherAI released the Pythia suite, a collection of autoregressive transformer models designed for interpretability and learning dynamics research. The suite includes eight model sizes trained on the Pile dataset, with each model seeing 299,892,736,000 tokens during training. All models feature 154 checkpoints saved throughout training, and most were trained using mixed precision with fp16, except the 1B model which used bf16 due to an irreconcilable loss spike in fp16.written, reviewed, cited

The 6.9B and 12B models accidentally used a different initialization due to a missing config value.
v0 models of sizes 160m, 410m, and 1.4b were trained with a batch size of 4M tokens across 71500 steps.

HuggingFace API Tokens were exposed, leaving users vulnerable

lasso.security2023-12-06about the pythia familyHN 2 pts · 1 comments1,738 words

Facebook Open-Sources Pythia for Vision and Language Multimodal AI Models

medium.com2019-05-21about the pythia familyHN 2 pts · 0 comments

Pythia: The first deep learning-based ancient text restoration model

archaeology.wiki2019-10-22about the pythia familyHN 1 pts · 0 comments765 words

All coverage tracked for this model (8) — append-only, nothing is ever removed

Title	Source	Published	First seen	Signals
Pythia 1.4B reproduces 3.6% of training samples verbatim given 950-token prompts	ret2libc.com	2026-06-09	2026-10-07	hn
HuggingFace API Tokens were exposed, leaving users vulnerable	lasso.security	2023-12-06	2026-10-07	hn
RedPajama at 440B tokens higher quality than Pythia and StableLM	together.xyz	2023-04-24	2026-10-07	hn
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling	arxiv.org	2023-04-05	2026-10-07	hn
Pythia: Interpreting Autoregressive Transformers Across Time and Scale	github.com	2023-04-05	2026-10-07	hn
Pythia: The first deep learning-based ancient text restoration model	archaeology.wiki	2019-10-22	2026-10-07	hn
Facebook Open-Sources Pythia for Vision and Language Multimodal AI Models	medium.com	2019-05-21	2026-10-07	hn
Pythia: Open-source framework for multimodal AI models	code.fb.com	2019-05-21	2026-10-07	hn

Papers

3 papers

Papers 3

arXiv links found in the model card and its tags. "Peer-reviewed" is shown only with evidence: a published venue in OpenAlex (measured) or the authors' own journal reference on arXiv (reported).

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling this model

Stella Biderman, Hailey Schoelkopf, Quentin Anthony +10 · 2023-04-03 · PDF · HF paper page (9 upvotes)
The Pile: An 800GB Dataset of Diverse Text for Language Modeling cited in card

Leo Gao, Stella Biderman, Sid Black +9 · 2020-12-31 · PDF · HF paper page (10 upvotes)
Datasheet for the Pile cited in card

Stella Biderman, Kieran Bicheno, Leo Gao · 2022-01-13 · PDF · HF paper page

Similar models

3 models

Similar models

distilgpt2 · 88.2M
same size band (S), same task (text-generation), permissive license
gpt2 · 137M
same size band (S), same task (text-generation), permissive license
openai-gpt · 120M
same size band (S), same task (text-generation), permissive license

Sources

3 sources

Sources

Hub snapshot: 2026-10-25 · HF API: 2026-10-24 21:39 UTC · Providers: 2026-10-24 21:44 UTC · Articles: 2026-10-24 21:46 UTC · Summary: 2026-10-07 15:10 UTC · Last verified 2026-10-25

Follow pythia-70m-deduped

Watch this model

/feeds/models/EleutherAI--pythia-70m-deduped.xml(RSS 2.0) · every change to pythia-70m-deduped
/feeds/families/EleutherAI--pythia.xml(RSS 2.0) · every change across the pythia family

Email alerts are not open yet; the feeds above are the same data. See feeds for the item schema.