Scaling laws describe how loss changes with scale. Do neurons inside models change predictably too?
We study vision and language models up to 30B params and find systematic scaling in neuron universality, specialization, and selectivity.
Paper+code: avdravid.github.io/rosetta-neuron…
1/n
I thoroughly enjoyed reading this recent paper by @yasamanbb et al (arxiv.org/pdf/2602.15029) that derives analytically why certain latent variables must lead to geometry in word embeddings. (getting Fourier modes even with open boundary but exponential kernel is neat!) I think it would be great to compare this to some of @prfsanjeevarora et al's work on this (eg tinyurl.com/4az4325n)
More broadly, I have been thinking about the right data generating process for language. For vision, we have latent spaces with great manifold structure (eg the SO3 pose of an object) and nonlinear mixing functions. But for language? Are there really any continuous latent variables? What is the "DSprites" of language? Is it all just co-occurrence stats or is there something more in LLM word embeddings?
@KrzakalaF ah, lovely! I quite like this sort of "posit a simpler dynamics and study it" sort of approach. very physicsy! I like Misha Belkin's NFA/RFM stuff (which this reminds me of) for the same reason.
possible path towards an answer to our Open Dir 1??
learningmechanics.pub/openquestions/…
did you know that with a few modifications, you can get the Ising model to simulate cells fighting to the death? one of my favorite side projects of all time:
jamiesimon.io/blog/cell-figh…
Mechanistic interpretability aspires to be the biology of deep learning. @KuninDaniel and @learning_mech say that an emerging theory of deep learning they and their team call 🛠️ learning mechanics 🛠️ will be the physics.
for the bright-eyed and bushy-tailed: there's a Learning Mechanics discord! young academics who want to do research in this area should especially consider joining + starting convos.
discord.gg/GTHfUnf7hz
ditto. props to @justanotherlaw for taking a second look :) (though I also found merit in the criticisms in the first version.) hopeful we can eventually (hopefully soon enough...) make contact w/ AI alignment + governance, whose noble causes we would v much like to aid.
This is a great post and I especially respect the author for updating his view when presented with new information. I strongly encourage young researchers interested in interpretability, science of DL, and safety to look at it.
lesswrong.com/posts/6SRq7mZ9…
It seems we're at a stage where deep learning is evolving from alchemy into an engineering discipline; this is an exciting paper which lays out that a scientific theory is emerging for Deep Learning.
Paper: arxiv.org/abs/2604.21691
Tweet: x.com/learning_mech/…
1/ Deep learning is going to have a scientific theory. We can see the pieces starting to come together, and it's looking a lot like physics!
We're releasing a paper pulling together these emerging threads and giving them a name: learning mechanics.
🔨 arxiv.org/pdf/2604.21691 🔧
yeah, totally!
I once messaged everyone on facebook with my first and last name. I eventually made a big group chat! v ethnically + geographically diverse. probs the closest I've gotten as an adult to meeting a truly random slice of the US.
Aren’t diffusion models explicitly derived from a correspondence with physics and entirely consistent with how physics says you should model systems over a range of scales ( ie mori zwanzig theory: langevin dynamics with a fitted vector field ? ) what more do you want?
Sorry, but these correspondences between AI and physics are vacuous. People have been making them since (at least) the 80s, and they always come to nothing.
11 Followers 699 FollowingA naughty thought a day keeps the stress away. 😜
Robotics, AI, CV, ML, DL, RL, RPA, IoT, Software, Cloud, Blockchain, Data and Automation Expert
371 Followers 960 FollowingAstra Fellow at Constellation right now | PhD student at Warsaw University of Technology and NASK working on AI safety and generative models
355 Followers 278 FollowingStriving to create that in the presence of which we feel more alive
Software gardener/eng: https://t.co/prDUQqX2SM ↝ @swellai @rupa_health Narrative Science @workos
339 Followers 975 FollowingIn pursuit of the Truth without fear of asking questions in this judgemental world.RTs/sharing/likes/comments are result of my struggle for this,not endorsement
2K Followers 866 FollowingRL for domains where human judgment is critical. AI Lead/Cofounder @tacitco. Epeeist. Gardener. Reader. I love low-ego, positive, competent and earnest people.
3K Followers 2K FollowingAssistant Professor / Faculty Fellow @NYUDataScience studying cognition in mind & brain with neural nets, Bayes, other tools | https://t.co/aC3k1IFSl4
3K Followers 170 FollowingI do AI Alignment Research. Formerly @METR_Evals, @redwood_ai; on leave from my PhD at UC Berkeley’s @CHAI_berkeley. Opinions are my own.
1.4M Followers 0 FollowingA universe of atoms, an atom in the universe. Tribute to the great explainer. Tweets about Science and Wisdom. Portrait by L.V Patten.
7K Followers 287 FollowingComputer Science Professor at Northeastern, Ex-Googler. Believes AI should be transparent. @[email protected] @davidbau.bsky.social https://t.co/wmP5LV0pJ4
3K Followers 439 FollowingI like Physics, Statistics, Machine learning, Computer Science & above all playing 🎸. Happy dad 👧 👧. Also professor @ EPFL. Views are my own.
3K Followers 1K FollowingTheoretical neuroscience, theory of neural computation, physics of learning and intelligence. Associate Professor of Applied Mathematics @Harvard SEAS
43K Followers 577 FollowingAssistant prof. @ Stanford; Chief AI Scientist @ MongoDB; Former Co-founder/CEO of Voyage AI
Working on ML, DL, RL, LLMs, and their theory.
7K Followers 735 FollowingScientist @OpenAI. Prev. co-founder @diffeo, acquired by @salesforce // co-authored The Principles of Deep Learning Theory // studied gravity.
12K Followers 2K FollowingInterested in cognition and artificial intelligence. MTS at @AnthropicAI. Previously @DeepMind, cognitive science @StanfordPsych. Tweets are mine.
6K Followers 1K FollowingGroup Leader,
Physics of Intelligence Program at Harvard University
Physics of Artificial Intelligence Group, NTT Research, Inc.
119K Followers 5K FollowingSearching for the numinous
🇦🇺 🇨🇦, currently live in 🇺🇸
Research @AsteraInstitute
https://t.co/maezekzRUb
https://t.co/2dWwZKrvrn