Hugging Face's logo
Activity Feed

AI & ML interests

None defined yet.

Recent Activity
Blyziย  authored a paper 23 days ago
TrustMI: Causally controlling how assistants trust their users
andreagurioli1995ย  submitted a paper about 1 month ago
CodeGraph: Open-Taxonomy Knowledge Graph for Source Code with Wikidata Grounding
eliebakย  authored a paper 2 months ago
Prime Agent: A Self-Improving RLM Harness
View all activity
loubnabnlย 
posted an update about 14 hours ago
view post
Post
106
๐ˆ๐ง๐ญ๐ซ๐จ๐๐ฎ๐œ๐ข๐ง๐  ๐Ÿงฌ ๐…๐ข๐ง๐ž๐ƒ๐๐€: ๐ญ๐ก๐ž ๐›๐ž๐ฌ๐ญ ๐๐š๐ญ๐š๐ฌ๐ž๐ญ ๐ญ๐จ ๐ญ๐ž๐š๐œ๐ก ๐š๐ง๐ฒ ๐ฅ๐š๐ง๐ ๐ฎ๐š๐ ๐ž ๐ฆ๐จ๐๐ž๐ฅ ๐ƒ๐๐€.
Training Qwen3-14B on it produces Carbon-X, the strongest open model on DNA benchmarks, on par with GPT-6.1 Sol.

It contains:
- Midtraining: 300B+ tokens of genomic sequences, papers and web pages mixing DNA with text
- SFT: 200K samples, from variant effect prediction to sequence question answering
- RL: 98K prompts with verifiable rewards

Data, models and the full recipe are open.
HuggingFaceBio/FineDNA

Enjoy this late Halloween treat! ๐ŸŽƒ
Blyziย 
authored a paper 23 days ago
andreagurioli1995ย 
submitted a paper to Daily Papers about 1 month ago
eliebakย 
authored 2 papers 2 months ago
mfrommย 
authored a paper 5 months ago
andreagurioli1995ย 
submitted a paper to Daily Papers 5 months ago
Blyziย 
authored a paper 6 months ago
andreagurioli1995ย 
submitted a paper to Daily Papers 6 months ago
Blyziย 
authored 2 papers 6 months ago
ybelkadaย 
authored a paper 10 months ago
eliebakย 
submitted a paper to Daily Papers 11 months ago
lvwerraย 
authored a paper about 1 year ago
eliebakย 
posted an update about 1 year ago
view post
Post
4,782
Super excited to announce that our research team at Hugging Face will be doing an AMA on reddit r/LocalLLaMA.

Come ask any questions to the team behind SmolLM, FineWeb and more! And who knows, maybe thereโ€™ll be a shiny new release to talk about?

Thursday 4th September, 8AM-11AM PST ๐Ÿค—

science
eliebakย 
posted an update about 1 year ago
view post
Post
804
Motif 2.6B tech report is pretty insane, first time i see a model with differential attention and polynorm trained at scale!

> It's trained on 2.5T of token, with a "data mixture schedule" to continuously adjust the mixture over training.
> They use WSD with a "Simple moving average" averaging the last 6 ckpt every 8B token.
> They trained on Finemath, Fineweb2, DCLM, TxT360.
> Lot of details in the finetuning data they used, for instance they used EvolKit and did some "dataset fusion" to have more compressed knowledge into the data.
> They mention they also tried Normalized GPT, QK-Norm and Cross Layer Attention.

Motif-Technologies/Motif-2.6B
ybelkadaย 
authored 2 papers over 1 year ago
eliebakย 
posted an update over 1 year ago
view post
Post
4,868
Kimi K2 tech report is full of gems as always. Here are my notes on it:

> MuonClip: Pretty crazy how after 70k the training stabilizes and the QK-clip is basically inactive. There is also no loss in perf with QK-clip which is not trivial at all (at small scale but with aggressive threshold). Also a cool explanation of why muon makes the logit explode in appendix E (tl;dr is that muon makes the singular value of the update matrix higher)
> Sparsity scaling laws to justify their ratio, they have a very solid training infra that allows the model to be trained at this sparsity level, they could have increased even more but as sparsity increases the training becomes less efficient.
> They diminish the number of attention heads to make it more efficient for long context since attention heads are a big bottleneck for long context. They also remove 2 of the 3 "first dense" layers in the dsv3 arch.

With the sparsity and attention heads (divided by 2) they achieve 83% increased flops compared to deepseek v3 arch at 128k.

> Data: Rephrasing is KEY. They do a lot more synthetic data generation and rephrase their corpus to have different styles, for longer documents they do it by chunk. I'm (half) surprised by the fact that ONLY 1 epoch (assuming same number of training tokens I think?) of data rephrased 10 times has better accuracy than 10 epochs of the same data rephrased once.
> They do rewriting for Math and Knowledge, for Math they apply the ShallowMath recipe and instruct the model to rephrase in a "learning note" style
> They talk about diversity and probably have some internal stuff/eval to test that, as always still a bit unclear for me how to properly measure that.

The infra is also very nice, quick summary:
> PP=16 (1F1B schedule, a bit custom), EP=16, zero1
> No FP8 computation but for storage of specific layers, selective recomputation for inexpensive block, activation offloading to CPU
lvwerraย 
authored a paper over 1 year ago