Sarthak @kaytraser

mr-sarthakgupta.github.io Joined April 2015

Tweets

302
Followers

273
Following

3K
Likes

4K

Sarthak @kaytraser

2 weeks ago

@PradyuPrasad checkout SPY Lab and SRI Lab at ETH Zurich and I believe Prof. Maksym would still be doing security research

0 0 6 341 7

View Details

Sarthak @kaytraser

2 months ago

@gauri__gupta @NeoSigmaAI @ritvikkapila amazing work!

0 0 3 180 0

View Details

This blog by Nicolas Carlini is stellar: nicholas.carlini.com/writing/2026/h… Internalizing things based on words is much more difficult to do than internalizing from (bad) experience, but if there is one place you should try hard to learn from as a researcher, it is this post.

1 3 21 2K 8

View Details

Sarthak @kaytraser

3 months ago

@tejalpatwardhan I did feel this some time ago x.com/kaytraser/stat…

Sarthak @kaytraser

7 months ago

@code_star @soldni @eliebakouch @Grad62304977 @samsja19 1. There is a possibility of only repository level filtering when training coding models (this was a choice by deepseekcoder), in this case, they're most likely retaining the different branches. Lets take the example of commaai/openpilot (serving as a typical oss repo here)

1 1 0 546 0

0 0 0 107 0

View Details

Sarthak @kaytraser

3 months ago

@ShashwatGoel7 I found the same, with anchor information, even tiny models are very capable of providing supervision x.com/kaytraser/stat…

Sarthak @kaytraser

5 months ago

what surprised me was that even smaller models one-shotting through I guess, the hard part about making unverifiable domains verifiable isn't about having a strong reasoner model to provide rewards?

2 0 0 232 0

0 0 1 30 0

View Details

Sarthak @kaytraser

4 months ago

@vvvincent_c womdering what those 14 hour tasks are, are they chained tasks or more monolithic?

0 0 0 266 0

View Details

Sarthak @kaytraser

4 months ago

@joel_bkr earlier I used to hink synthetic data would break this logic but seems like there are too many issues with collapse/bad-distributions as we scale that the above intuition still holds

0 0 0 44 0

View Details

Sarthak @kaytraser

4 months ago

@Dorialexander @TheAhmadOsman opus 4.5/4.6 work like magic for generating seed data

0 0 0 41 0

View Details

Sarthak @kaytraser

4 months ago

@sharut_gupta great work! quick question: what is rhe ratio of trainable parameters in the input embeddings vs total model weights that we see here?

0 0 0 48 0

View Details

Sarthak @kaytraser

4 months ago

@bilaltwovec I feel like this thing the equivalent of the "autocomplete phase" we saw in AI coding in automated AI research not sure how long it might take to begin considering abstracting away the underlying research like we're thinking about the future of code right now

0 0 0 74 0

View Details

Sarthak @kaytraser

4 months ago

@khoomeik dataset distillation? I vividly remember those blurry images representative of the entire class, training on just 10 images gave a great performance on imagenet this was a very interesting direction back in the resnet days, wondering where it went

1 0 0 100 0

View Details

Sarthak @kaytraser

4 months ago

@MaziyarPanahi a bit tangential but, have you been using LLMs as judges to supervise the CoTs since CoT supervision would be the primary challenge in this situation

0 0 0 11 0

View Details

Sarthak @kaytraser

4 months ago

@MaziyarPanahi hm, this could definitely be a great thing for cold-starting but how would we then beat those SOTA models?

1 0 1 16 0

View Details

Sarthak @kaytraser

4 months ago

@HaoliYin but the question is what changed eventually x.com/kaytraser/stat…

Sarthak @kaytraser

4 months ago

did synth data generation for the same task in Sept 2024 and today fighting mode collapse was so hard back then and is completely absent now we've came a long way, wondering if it is only because models got larger or did the labs actually get an improved data distribution

1 0 8 11K 5

2 0 10 10K 1

View Details

Sarthak @kaytraser

4 months ago

@sdathath I was considering the situation where pre-training itself might be at fault for mode collapse

0 0 0 60 0

View Details

Sarthak @kaytraser

4 months ago

1 0 8 11K 5

View Details

Sarthak @kaytraser

4 months ago

@sdathath this seems to be more aligned with the task of pure next token generation hence the suspension of being more influenced by changes in pre training

1 0 0 77 0

View Details

Sarthak @kaytraser

4 months ago

@sdathath the reason being that I'm doing generation for a somewhat simple task example: where the earlier models used to fill the names with "john doe" 7/10 times now give a really good diversity of names and this observation goes for most of the peculiarities of the data I know of