qwenlm.github.io2024-03-28officialauthorityHN front pageabout the Qwen1.5 familyHN 104 pts · 10 comments1,495 words
Qwen1.5-MoE-A2.7B matches 7B model performance with only 2.7 billion activated parameters using a 64-expert configuration. The model incorporates 4 shared experts alongside 60 routing experts and requires source installation via git for Hugging Face support. It achieves a 75% decrease in training expenses and accelerates inference speed by a factor of 1.74 compared to Qwen1.5-7B.written, reviewed, cited
- The model partitions a single FFN into segments to generate 64 experts without increasing the total parameter count.
- Training from scratch is inefficient, so the team repurposes the existing Qwen-1.8B model to initialize Qwen1.5-MoE-A2.7B.
- The Qwen1.5-MoE-A2.7B-Chat-GPTQ-Int4 quantized variant is available, but AWQ support is currently not supported.
qwenlm.github.io2024-02-05officialauthorityabout the Qwen1.5 familyHN 5 pts · 2 comments3,793 words
Qwen1.5 is a new open-source model series released before the Chinese New Year, offering base and chat models in six sizes ranging from 0.5B to 110B parameters. The Qwen1.5-72B model outperforms Llama2-70B across all benchmarks, and the series features improved multilingual and human-alignment capabilities. All models now uniformly support a context length of up to 32768 tokens, and the Qwen1.5-72B-Chat model is recommended for initial testing.written, reviewed, cited
