ModernBERT pre-trained from scratch on various obfuscated representations of the same 10B tokens.
🔄 In a Training Loop
Daniel Gallagher
DanielGallagherIRE
·
AI & ML interests
I like encoders because they don't talk back.
Recent Activity
liked a model 2 days ago
convaiinnovations/laya updated a dataset 3 days ago
DanielGallagherIRE/FineWeb-Edu-E1-100MOrganizations
A collection of obfuscated version of a 20B-token sample of the FineWeb Edu dataset.
-
DanielGallagherIRE/FineWeb-Edu-10B-Obfuscation
Viewer • Updated • 19.4M • 1.12k -
DanielGallagherIRE/FineWeb-Edu-10B-Nouns-Only
Viewer • Updated • 9.71M • 2.99k -
DanielGallagherIRE/FineWeb-Edu-10B-Shuffled
Viewer • Updated • 9.68M • 1.13k -
DanielGallagherIRE/FineWeb-Edu-10B-PMI-Filtered
Viewer • Updated • 9.71M • 435
ModernBERT pre-trained from scratch on various obfuscated representations of the same 10B tokens.
A collection of obfuscated version of a 20B-token sample of the FineWeb Edu dataset.
-
DanielGallagherIRE/FineWeb-Edu-10B-Obfuscation
Viewer • Updated • 19.4M • 1.12k -
DanielGallagherIRE/FineWeb-Edu-10B-Nouns-Only
Viewer • Updated • 9.71M • 2.99k -
DanielGallagherIRE/FineWeb-Edu-10B-Shuffled
Viewer • Updated • 9.68M • 1.13k -
DanielGallagherIRE/FineWeb-Edu-10B-PMI-Filtered
Viewer • Updated • 9.71M • 435
models 42
DanielGallagherIRE/ModernBERT-O3-Shuffling-SENT
0.2B • Updated • 6
DanielGallagherIRE/ModernBERT-O3-Shuffling-DOC
0.2B • Updated • 6
DanielGallagherIRE/ModernBERT-O3-Baseline
0.2B • Updated • 114
DanielGallagherIRE/ModernBERT-O3-PMI-5
0.2B • Updated • 4
DanielGallagherIRE/ModernBERT-O3-PMI-2.5
0.2B • Updated • 5
DanielGallagherIRE/ModernBERT-O3-PMI-10
0.2B • Updated • 5
DanielGallagherIRE/ModernBERT-O3-Nouns-POS
0.2B • Updated • 11
DanielGallagherIRE/ModernBERT-O3-PMI-7.5
0.2B • Updated • 5
DanielGallagherIRE/ModernBERT-O3-PMI-12.5
0.2B • Updated • 9
DanielGallagherIRE/ModernBERT-O3-Nouns-DEL
0.2B • Updated • 6
datasets 24
DanielGallagherIRE/FineWeb-Edu-E1-100M
Viewer • Updated • 96.3k • 75
DanielGallagherIRE/FineWeb-Edu-Irish
Viewer • Updated • 700k • 175
DanielGallagherIRE/FineWeb-Edu-10B-Shuffled-DOC-Truncated
Viewer • Updated • 11.3M • 238
DanielGallagherIRE/FineWeb-Edu-10B-Shuffled-SENT-Truncated
Viewer • Updated • 11.3M • 593
DanielGallagherIRE/FineWeb-Edu-10B-Obfuscation-Truncated
Viewer • Updated • 11.3M • 531
DanielGallagherIRE/FineWeb-Edu-10B-PMI-Filtered-Delete
Viewer • Updated • 9.71M • 385
DanielGallagherIRE/FineWeb-Edu-20B-E2-OBF-FULL
Viewer • Updated • 9.69M • 431
DanielGallagherIRE/FineWeb-Edu-20B-E2-v3-Obfuscated
Viewer • Updated • 9.69M • 219
DanielGallagherIRE/FineWeb-Edu-20B-E2-v3
Viewer • Updated • 9.69M • 449
DanielGallagherIRE/FineWeb-Edu-20B-E2-v2
Updated • 3