question
BTW congratulations on your new model, It's a neat little toy model for sure.
My question is did you do any data preprocessing on the web data since fineweb edu 350BT and HuggingFaceFW/dclm_100BT are quite noisy?
I also have a second question, are you guys planning on using large scale Task-Specific Distillation?
No we didn't actually
BTW congratulations on your new model, It's a neat little toy model for sure.
My question is did you do any data preprocessing on the web data since fineweb edu 350BT and HuggingFaceFW/dclm_100BT are quite noisy?
I also have a second question, are you guys planning on using large scale Task-Specific Distillation?
What are these questions? DCLM and FIneWeb Edu have already been heavily processed; theres no point of processing them more. In fact, they're both model-classified.
Second, task-specific distillation? Why? Whats the point? If you want it to do another task, just fine-tune it. It'll take two seconds.
Also, "a neat little toy"? Its not that neat. SmolLM2, GPTX2.5, MobileLM are all much stronger alternatives. I'm not saying its a waste, but imo it isn't the model I'd use. Furthermore, making text models is completely saturated; and yes that applies to every text model I've made.
First off yes I'm aware its better than webscrapes but I personally have gone through both with scripts and it (atleast the 10bt and 100bt of fineweb and dclm) both have syntax, duplicated text, and other noise so personally I clean them nothing bad just asking.
I'm meaning math, step-by-step instructions, multi turn conversation awareness, ect. Just curious since I personally find it to be very effective.
Its still neat to look at from an enthusiast standpoint. So therefore neat little toy model from what I'm seeing.
First off yes I'm aware its better than webscrapes but I personally have gone through both with scripts and it (atleast the 10bt and 100bt of fineweb and dclm) both have syntax, duplicated text, and other noise so personally I clean them nothing bad just asking.
I'm meaning math, step-by-step instructions, multi turn conversation awareness, ect. Just curious since I personally find it to be very effective.
Its still neat to look at from an enthusiast standpoint. So therefore neat little toy model from what I'm seeing.
Math? FineMath and UltraDataMath.
Step-by-Step instructions? Cosmopedia, Cosmopedia-v2, Tiny-Strange-Textbooks.
Yeah its nice that you clean it but in my opinion, its just a waste of time.
I think that if you give an ablation table comparing DCLM vs your cleaned DCLM, then maybe.
Anyway, its nice to know people have different ways of doing things.
I tries those but it wasnt as effective as directly distilling not to say they are at all bad.
I actually ran into some issues training on them uncleaned more so fineweb. But tbh I don't need to clean it but the data budget I was attempting to work on had 0 room for unwanted noise, but I get your point.
I tries those but it wasnt as effective as directly distilling not to say they are at all bad.
I actually ran into some issues training on them uncleaned more so fineweb. But tbh I don't need to clean it but the data budget I was attempting to work on had 0 room for unwanted noise, but I get your point.
Ohhh. That changed things considerably.
yeah fineweb is really noisy but its edu variant is much better.
The issue with synthetic data isn't really the quality or lack of information, people train models on far worse.
What the issue really seems to be from my perspective is that prompting them is difficult. It's why adding 10% web data massively boosts benchmarks, it acts as a budget "SFT" where the few-shot prompting starts working.