FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language Paper ⢠2506.20920 ⢠Published Jun 26, 2025 ⢠82
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language Paper ⢠2506.20920 ⢠Published Jun 26, 2025 ⢠82
The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text Paper ⢠2506.05209 ⢠Published Jun 5, 2025 ⢠65
SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model Paper ⢠2502.02737 ⢠Published Feb 4, 2025 ⢠261
SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model Paper ⢠2502.02737 ⢠Published Feb 4, 2025 ⢠261
Towards Best Practices for Open Datasets for LLM Training Paper ⢠2501.08365 ⢠Published Jan 14, 2025 ⢠60
Towards Best Practices for Open Datasets for LLM Training Paper ⢠2501.08365 ⢠Published Jan 14, 2025 ⢠60
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale Paper ⢠2406.17557 ⢠Published Jun 25, 2024 ⢠106
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale Paper ⢠2406.17557 ⢠Published Jun 25, 2024 ⢠106
A Dataset and Strong Baselines for Classification of Czech News Texts Paper ⢠2307.10666 ⢠Published Jul 20, 2023
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only Paper ⢠2306.01116 ⢠Published Jun 1, 2023 ⢠45