tensorholic 🪤 @tensorcel

Into ML & systems performance, learning, and aesthetics. Seeking memetic fitness sf Joined February 2023

Tweets

479
Followers

104
Following

384
Likes

7K

tensorholic 🪤 @tensorcel

3 years ago

@gallabytes @finbarrtimbers for (quite) short sequences it’s faster to recompute

0 0 1 31 0

View Details

I think it's surprising that maximum likelihood training has got us this far, given the train/test distribution shift of other's tokens vs AR sampled tokens. The research community was interested in fixing this sort of thing 2015-2020 but seems like it doesn't matter so much?

0 0 1 56 0

View Details

tensorholic 🪤 @tensorcel

3 years ago

@yacineMTB its mathematically more complicated, less scale-driven, and the outputs are more sus

0 0 2 223 0

View Details

tensorholic 🪤 @tensorcel

3 years ago

only Mariah Carey books spotify streaming revenue as recurring

0 0 0 76 0

View Details

tensorholic 🪤 @tensorcel

3 years ago

it really is a pleasure doing business

0 0 0 29 0

View Details

tensorholic 🪤 @tensorcel

3 years ago

business is when two parties say thank you to each other after an exchange

1 0 0 59 0

View Details

morgan — @morqon

3 years ago

@BrianRoemmele added a line

4 2 45 10K 3

View Details

tensorholic 🪤 @tensorcel

3 years ago

@tszzl those who are sensitive to overwhelming beauty are typically also sensitive to overwhelming ugliness. artists empirically deal with this via drug habits

0 0 5 230 0

View Details

tensorholic 🪤 @tensorcel

3 years ago

@sameQCU argmin.substack.com/p/you-got-a-9-…

1 0 1 8 0

View Details

tensorholic 🪤 @tensorcel

3 years ago

@sameQCU Reminds me of when researchers believed that we had begun overfitting to test sets in vision--so they made a new one--and found: 1. there was no overfitting 2. better models had a smaller gap between original and new. in that case, just better models are just better

2 0 2 39 0

View Details

tensorholic 🪤 @tensorcel

3 years ago

purple is what happens when you begin to show love and compassion to your models

tensorholic 🪤 @tensorcel

3 years ago

today I've learned that one step at learning rate 5e-5 sends a 7b model into a kind of epileptic shock (.9 loss -> 10) from which it never recovers

1 0 1 169 0

0 0 0 86 0

View Details

tensorholic 🪤 @tensorcel

3 years ago

to be clear this is starting from a trained checkpoint

0 0 0 33 0

View Details

tensorholic 🪤 @tensorcel

3 years ago

today I've learned that one step at learning rate 5e-5 sends a 7b model into a kind of epileptic shock (.9 loss -> 10) from which it never recovers

1 0 1 169 0

View Details

tensorholic 🪤 @tensorcel

3 years ago

@nosilverv since LLM's are sampled somewhat greedily, I'd guess the likelihood of that sort of mistake for a token is below the top-p or top-k threshold

0 0 1 5 0

View Details

tensorholic 🪤 @tensorcel

3 years ago

@nosilverv people do that when speaking but rarely when writing on the internet

2 0 2 31 0

View Details

tensorholic 🪤 @tensorcel

3 years ago

we sample LLM's (semi-)greedily because they do not generalize well outside of the distribution they are trained on e.g. correct text with teacher forcing

0 0 0 44 0

View Details

tensorholic 🪤 @tensorcel

3 years ago

@alexeyguzey @norabelrose what makes you yell?

1 0 0 77 0

View Details

tensorholic 🪤 @tensorcel

3 years ago

@norabelrose I'm not sure if these matter & maybe these can be folded into preferences--but RLHF does seem to address a larger class of rewards than dpo does?

1 0 0 48 0

View Details

tensorholic 🪤 @tensorcel

3 years ago

@norabelrose to be more specific I'm referring to rewards which are difficult to describe via rank order over a set of model outputs. eg token-level/process rewards, discrete external reward, maybe you want to fine tune your reward model without collecting more outputs & preferences

1 0 0 25 0

View Details

tensorholic 🪤 @tensorcel

3 years ago

the youtube comment setiment model has become totally goodhardt'd so the top comment is always about someone's dying parent loving the video even if it is a chainsaw operation tutorial