Ranked by source signals (HN, Lobsters, authority domains, official posts) and recency. Summaries are generated from a stored copy of the article text; every sentence cites a span of that text, and nothing is shown that failed the citation check.
Analysis10
github.com2021-07-04authorityHN front pageHN 636 pts · 146 comments1,329 words
Mesh-Transformer-JAX implements model parallel transformers in JAX using xmap/pjit operators, similar to Megatron-LM, and supports scaling up to approximately 40B parameters on TPUv3s. The library includes an experimental ZeRo style sharding version and released a 6 billion parameter autoregressive text generation model trained on The Pile under the Apache License version 2.0. Using a TPU v3-8, users can fine-tune the model at a rate of ~5000 tokens/second, though the v1 models specifically require jax==0.2.12.written, reviewed, cited
- The device_* scripts only work on a v3-8 and not on larger pods.
- The Megatron-11B model provides no comparable metrics, and several implementations using the released weights do not reproduce the generation quality and evaluations.
textsynth.com2022-04-27HN front pageabout the gpt j familyHN 263 pts · 134 comments51 words
arankomatsuzaki.wordpress.com2021-06-04HN front pageHN 144 pts · 79 comments1,362 words
GPT-J-6B is a 6 billion parameter JAX-based transformer language model that performs nearly on par with 6.7B GPT-3 on zero-shot tasks. The model was trained on 400B tokens from The Pile dataset containing 800GB text with an attention head dimension of 256. Training took roughly five weeks on a TPU v3-256 pod, achieving 5.4 PFLOPs of efficiency.written, reviewed, cited
- Efficient attention mechanisms were not used for simplicity as they would not have significantly improved throughput at this scale.
minimaxir.com2021-06-14HN front pageHN 177 pts · 40 comments3,302 words
GPT-J is a 6 billion parameter model by EleutherAI that outperforms GPT-3 on code generation tasks despite its smaller size. Trained on The Pile, which has a high weight of GitHub and Stack Overflow input, the model often generates correct but convoluted Python code. Running the model requires a TPU to generate text at a reasonable speed.written, reviewed, cited
- The model outputs the alternate solution using bitwise operators for the is_even function.
p3r.one2021-07-04HN front pageabout the gpt j familyHN 118 pts · 39 comments
sematic.dev2023-07-26about the gpt j familyHN 98 pts · 22 comments24 words
exafunction.com2021-09-28about the gpt j familyHN 45 pts · 19 comments74 words
colab.research.google.com2023-03-10about the gpt j familyHN 54 pts · 9 comments1,036 words
tracklify.com2022-03-27about the gpt j familyHN 39 pts · 7 comments
narrativa.com2022-12-14about the gpt j familyHN 15 pts · 5 comments992 words