mediapipe-studio.webapps.google.com2024-03-07officialabout the gemma 2 familyHN 30 pts · 15 comments23 words
developers.googleblog.com2024-08-22officialauthorityabout the gemma 2 familyHN 14 pts · 0 comments891 words
Google released Gemma 2, an open model suite available in 2B, 9B, and 27B parameter sizes that introduces architectural innovations like Alternating Local and Global Attention and Logit Soft-Capping. The 27B variant features 46 layers and a 256000 vocabulary, utilizing Grouped-Query Attention to improve efficiency alongside RMSNorm for Pre and Post-Normalization for training stability. The 2B model demonstrates edge-device capability by outperforming all GPT-3.5 models on the Chatbot Arena.written, reviewed, cited
- The 2B model uses 4 heads for k_proj and v_proj, 8 for q_proj and o_proj with a head size of 256.
- Gemma 2 2B outperforms all GPT-3.5 models on the Chatbot Arena at a size runnable on edge devices.
- Replacing MHA with GQA results in comparable performance while offering parameter efficiency and faster inference times.