arxiv:2303.03915
Albert Villanova del Moral
albertvillanova
AI & ML interests
ML Engineer @ Hugging Face: Agents (Science)
Recent Activity
posted an update about 10 hours ago
Day-0 Mistral Large 4 support in TRL 🚀
- LoRA SFT of the 1T-A52B model on one 8xB300 node, on top of the FP8 checkpoint
- Experts split 8 ways with expert parallelism, everything else sharded with FSDP, all set at load time
- Assistant-only loss out of the box with its training chat template
- ~32.7k tokens per step in ~8 s
Script + Slurm launcher 👇
https://github.com/huggingface/trl/tree/main/examples/sft_mistral_large_4
https://huggingface.co/posts/sergiopaniego/408182860646834 reacted to sergiopaniego's post with 🚀 about 10 hours ago
Mistral Large 4 (Le Chonk 🐈) is now live on the Hub and you can post-train it with TRL today!
SFT + LoRA example ready, runs on a single 8xB300 node
> model: https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B
> TRL example: https://github.com/huggingface/trl/tree/main/examples/sft_mistral_large_4
> blog: https://huggingface.co/blog/mistral-large4 new activity about 10 hours ago
trl-internal-testing/tiny-MistralLarge4ForConditionalGeneration:Sync the chat template with the released checkpointOrganizations
