ai.google.devofficialauthorityabout the gemma 2 family2,266 words
Google released Gemma 2, a family of lightweight, open-weight text-to-text decoder-only models available in English. The 27B variant was trained with 13 trillion tokens using TPUv5p hardware and JAX and ML Pathways software. These models are designed for tasks like question answering and reasoning while remaining small enough to run on laptops or personal cloud infrastructure.written, reviewed, cited
- The 2B model variant was trained using 2 trillion tokens from diverse web documents, code, and mathematics sources.
- Automated techniques filter out personal information and sensitive data to ensure model safety and reliability.
- Gemma 2 IT 27B achieved a 97.22 score on the Winobias 2_2 benchmark for representational harms evaluation.
mediapipe-studio.webapps.google.com2024-03-07officialabout the gemma 2 familyHN 30 pts · 15 comments23 words
developers.googleblog.com2024-08-22officialauthorityabout the gemma 2 familyHN 14 pts · 0 comments891 words
Google released Gemma 2, an open model suite available in 2B, 9B, and 27B parameter sizes that introduces architectural innovations like Alternating Local and Global Attention and Logit Soft-Capping. The 27B variant features 46 layers and a 256000 vocabulary, utilizing Grouped-Query Attention to improve efficiency alongside RMSNorm for Pre and Post-Normalization for training stability. The 2B model demonstrates edge-device capability by outperforming all GPT-3.5 models on the Chatbot Arena.written, reviewed, cited
- The 2B model uses 4 heads for k_proj and v_proj, 8 for q_proj and o_proj with a head size of 256.
- Gemma 2 2B outperforms all GPT-3.5 models on the Chatbot Arena at a size runnable on edge devices.
- Replacing MHA with GQA results in comparable performance while offering parameter efficiency and faster inference times.
