Ben Walker @benjaminwalker
Postdoctoral Researcher with DataSig II @OxUniMaths. Researching Neural DEs and the theory of rough paths. Email: [email protected] benwalker.co.uk University of Oxford Joined February 2022-
Tweets160
-
Followers184
-
Following185
-
Likes103
Unless I’ve missed something, there are still no technical details on how they make the approach subquadratic. Anyone know how they choose which previous tokens a query should attend to without first looking at all previous tokens, or is the subquadratic claim just marketing?
The transformer architecture used for ChatGPT, Gemini, and Claude has defined the last decade of AI. It also introduced a fundamental constraint: compute scales quadratically as context grows. Longer inputs, exponentially higher costs and accuracy that degrades well before the
Using the new GPT-Image-2 to help me express what it feels like to watch a one-off specific instruction survive Codex compaction, and then get passed down forever as legend through each successive compaction
Looks like codex and chatgpt are now down and this is the first time I have seen a foreign language in a response, is this somehow linked?
Codex felt it could only express how totally declarative the dataclass definition should be in Russian
@MingchenZhuge @tydsh @karpathy @DrJimFan @SchmidhuberAI @_akhaliq @hardmaru @zechunliu @YoungXiong1 @HaoZhe65347 @cai_zhipeng What distinguishes a neural computer from a world model? If it learns the dynamics of computation, memory, I/O, and state updates well enough to reproduce the machine’s behaviour, then it seems like a world model where the “world” happens to be a computer.
That explains why gpt-5.4-codex is at capacity
To celebrate 3 million weekly codex users, we are resetting usage limits. We will do this every million users up to 10 million. Happy building!
If only Yule had known about this when he invented autoregressive modelling in 1927, we could have had language models before computers!
Hate to break it to you, but the first LLM was created by Andrey Markov in 1913. he tallied up 20,000 letters from a famous novel and computed p(vowel | vowel) p(consonant | vowel) p(vowel | consonant) p(consonant | consonant) basically 'training' a bigram by hand
Most machine learning is about finding the right feature extractor for a linear readout. Even an LLM.
A visualisation of the idea behind rough path theory: a path is not fully described by its value Each curve has the same area: when the number of circles doubles, their radius is scaled by 2^{-1/2}. The path's value converges to the straight line, but the total area does not
@xlr8harder My chat keeps using a LaTeX macro it invented in a previous chat
ChatGPT found a song I’d been trying to find for ages, then built a playlist around it that was way better than Spotify’s suggestions. How is an LLM better at music recommendation than a direct recommendation system?
Fantastic OxYSS session with Emma Prevot (@OxfordStats) & @benjaminwalker (@OxUniMaths) on the intersection of Causal Inference and Continuous-Time ML. A vibrant discussion on the future of temporal modelling!
New ChatGPT tell: the sentence uses a colon. This is, however, not the only one.
Everything starts to look like a path once you stare at it long enough
@ryu0000000001 A natural next step is developing path-to-path models that can handle irregular or over-sampled inputs. Currently, we use the Log-ODE method, but because it outputs a sequence (the solution only at interval endpoints), the resulting models cannot be stacked.
Take the same underlying path and increase the number of samples. As the sequence length grows, RNNs become difficult to train and Transformers become too expensive. Our continuous-time models instead converge to a continuous hidden-state path.
Want to know more? Log-NCDEs: efficient continuous-time sequence models with strong empirical performance. arxiv.org/abs/2402.18512 SLiCEs: parallel-in-time continuous-time models that don't sacrifice expressivity. arxiv.org/abs/2505.17761
@HochreiterSepp Never liked the term 'Linear RNN', as their updates are nonlinear in (h_t, x_t). The real bottleneck is restrictive structure that prevents hidden-state interactions. Without that restriction, Linear RNNs already have the expressivity for world modelling: arxiv.org/abs/2505.17761
And in a learning setting, it can be understood as a SLiCE with a strong inductive bias for building a contextual memory of a path. Paper: arxiv.org/pdf/2603.19198 3/3
It builds on the exponentially fading memory signature of Eduardo Abi Jaber and Dimitri Sotnikov by moving beyond channel-independent weighting of the past, allowing interactions between channels to shape how history is remembered. 2/3
Congratulations to our PhD student Alex on his first paper! The Exponentially-Weighted Signature. This new SLiCE architecture generalises the signature transform by introducing a trainable continuous-time attention over the history of a path. 1/3
Christoph Reich @ChristophR1996
371 Followers 825 Following @ELLISforEurope Ph.D. Student @tumcvg, @visinf & @Oxford_VGG | https://t.co/yaqdM5V4IW. from @etitdarmstadt & https://t.co/VHpc5R9Fs1. from @CS_TUDarmstadt | Prev. @NECLabsAmerica & @koeppl_lab
Louis @Louis9687221579
73 Followers 3K Following Mainline Economics | Idea page | ramblings of a schizo
Zeyuan Allen-Zhu, Sc.... @ZeyuanAllenZhu
26K Followers 564 Following physics of language models @ Meta (FAIR at MSL, not GenAI or TBD) 🎓：Tsinghua Physics — MIT CSAIL — Princeton/IAS 🏅：IOI x 2 — ICPC — USACO — Codejam — math MCM
Matthew Ford @fordmatt18
4 Followers 63 Following
Kobbitic @kobbitic
6 Followers 1K Following
Matteo Papini @papinimat
530 Followers 731 Following Tenure Track Assistant Professor at @LaStatale Bluesky: @papinimat LinkedIn: https://t.co/hRQmvJewMB
Thomas Lew @thomas__lew
224 Followers 194 Following Research Scientist at @ToyotaResearch. Optimal Control, Machine Learning, Robotics. PhD @Stanford. Previously intern at @Google, @NASAJPL.
Latios @Latios630519
1 Followers 483 Following
cty @cty
52 Followers 1K Following
Tran Dang Hoan @hoanhaiphong
31 Followers 5K Following
Alexis C. @4l3k6_C
118 Followers 3K Following
Cphk @giskard2
10 Followers 215 Following
Alex Yanko 🇺🇦 @LeopolisDream
1K Followers 3K Following ML Team Lead, Former Head of Data and Analytics, working on SaaS projects. Data Science 🔭 Analytics 📈 Engineering.
Manas @trippyhat4
1 Followers 269 Following
Marcus Barnes @MarcusBarnes
1K Followers 6K Following PhD Researcher | LLM4SE & multi-agent AI systems | LLMs for mathematics & autoformalization | Open to research collaborations, industry roles & consulting
Jj @Jj33524298
150 Followers 7K Following
tensor @_1010sor
18 Followers 667 Following first year undergrad in animal husbandry + rhythmic gymnastics @uwaterloo. learning machines
Xero @chaos_xero
0 Followers 2K Following
deliciousSandwich @mrsirrisrm
130 Followers 6K Following
Amey Varhade @ameyvarhade
526 Followers 3K Following Research Fellow @MSFTResearch Previously @IITGuwahati @ibm_in
Raymond Ng @Raymondng_aisg
4 Followers 2K Following
Achilles_of_Myrmidons @AMyrmidons
481 Followers 3K Following
రామయ్య �... @generalusername
400 Followers 1K Following MechE • DatSci • Linguistics • CG Ⱄⰾⱁⰲⱁ Ⱀⰵⰶⰻⱅⱁ
Roger Frigola @RogerFrigola
1K Followers 5K Following Engineer for America's Cup and Formula 1 teams. Machine Learning PhD @Cambridge_Uni. From Barcelona. Flat-sixes and four-cylinder transaxles (sic).
Antibody News @antibodynewshq
259 Followers 5K Following
Joaquin Gajardo @JoaquinGajardoC
10 Followers 294 Following CS and AgTech PhD student @ETH Zürich. Working on ML and 3D vision for agriculture. My mantra: be humble, gentle, help people, and never stop learning :)
SunWukong @Kapishreshtha
75 Followers 4K Following
Kelvin 🦖🤓 @kelvinhan
89 Followers 2K Following #NLProc PhD-ed at @labo_Loria. Currently Research Fellow at @singaporetech. Questions generator. https://t.co/3mUSCnSHTf
Timothy Hitge 🇿�... @tim_hitge
65 Followers 1K Following Research Assistant @imperialcollege. @GoogleDeepMind Scholar. MSc in AI for Science 🧪 @AIMSacza. Prev intern @instadeepai
Zhaoyang Wang @zhaoyangwang_
770 Followers 8K Following Research Scientist & Data Scientist | Foundation Models, Bayesian Optimization, reinforcement learning, and AutoML. Uni of Birmingham.
Akano Lordstrong @Grace_fuelled
211 Followers 6K Following
Corolla @StateOfCorolla
39 Followers 683 Following
Matthew Willetts @matthewjfw3
199 Followers 841 Following Correlation doesn’t imply causation unless I’m doing the regression. prev: Visiting Researcher @turinginst, Research Fellow @uclcs, ML PhD @oxcsml @oxfordstats.
Evelyn @tummycom
850 Followers 7K Following Open Source, Mountain Time. Give me or the universe anonymous feedback: https://t.co/ZaUAOdtsG9
Giosue Migliorini @joh_sweh
53 Followers 587 Following Ph.D. student in stats @UCIrvine | former AI research intern @FlagshipPioneer, @LosAlamosNatLab, @UniBocconi
flight GNC @flight_gnc
253 Followers 3K Following
Sushil Pokhrel @sushilpokhrel
3K Followers 8K Following Biomedical/Materials Engineering researcher, Machine learning, AI + Robotics + CPS, Singularity, etc. Fell in love with ML
Ahmad Baasim Husain @BaasimHusain
22 Followers 4K Following
BOSSMAN @FSucculent
3K Followers 5K Following oil trading robot for @Shell, predicted the 2008 financial crisis, husband and dad.
Jürgen Schmidhuber @SchmidhuberAI
201K Followers 0 Following Introduced basics of: P & T in ChatGPT, very deep learning, meta learning, neural distillation, GANs, etc. Co-authored most-cited AI paper of 20th century
Álvaro Cartea @AlvaroCartea
371 Followers 69 Following Professor of Mathematical Finance, and Director of the Oxford-Man Institute, University of Oxford. Book: Algorithmic and High-Frequency Trading.
Joaquin Gajardo @JoaquinGajardoC
10 Followers 294 Following CS and AgTech PhD student @ETH Zürich. Working on ML and 3D vision for agriculture. My mantra: be humble, gentle, help people, and never stop learning :)
Kelvin 🦖🤓 @kelvinhan
89 Followers 2K Following #NLProc PhD-ed at @labo_Loria. Currently Research Fellow at @singaporetech. Questions generator. https://t.co/3mUSCnSHTf
Timothy Hitge 🇿�... @tim_hitge
65 Followers 1K Following Research Assistant @imperialcollege. @GoogleDeepMind Scholar. MSc in AI for Science 🧪 @AIMSacza. Prev intern @instadeepai
Giosue Migliorini @joh_sweh
53 Followers 587 Following Ph.D. student in stats @UCIrvine | former AI research intern @FlagshipPioneer, @LosAlamosNatLab, @UniBocconi
Nora Hedwig Nordlinde... @HeNordlinder
150 Followers 671 Following Mathematical statistics MSc @Stockholm_Uni . Phylogenetics and deep learning @karolinskainst.
Jorge Bravo Abad @bravo_abad
11K Followers 9K Following AI for Science | Prof. of Physics @UAM_Madrid. Author of "IA y Física": https://t.co/Nxue94kfOG & "Ciencia 5.0": https://t.co/Y3rBUU7Xzg
Philip Schroeder @Philip_MIT
740 Followers 696 Following PhD student at MIT in Computer Science. @MIT_CSAIL @MITEECS @nlp_mit
Fernando Moreno-Pino @fermorenp
228 Followers 2K Following Assistant Professor in Machine Learning at @BristolUni. Research Associate at @Oxford_Man_Inst. Previously PostDoc at @UniofOxford and PhD at @uc3m.
Philipp Nazari @philna00
109 Followers 250 Following PhD Candidate at Max Planck ETH Center for Learning Systems
Teemu Sarapisto @Tsarpf
224 Followers 499 Following CS/ML PhD research in high-dim time-series @helsinkiuni Before: 7y of C++/JS/VR/AR/ML at Varjo, Yle, Reaktor, Automattic ... After dark: synthesizers & 3D gfx
Riccardo Grazzi @riccardograzzi
134 Followers 154 Following Researcher at MSR Cambridge, previously at IIT in Genoa, Italy. Working on principled, efficient optimization for machine learning and on LLMs' expressivity.
Paul Thompson @PTenigma
8K Followers 3K Following Neuroscientist, professor AI guided tour - https://t.co/yyVhX7Jwv2 ENIGMA guided tour - https://t.co/6oEtWdx7np
Leo Gao @nabla_theta
13K Followers 580 Following working on AGI alignment. prev: GPT-Neo, the Pile, LM evals, RL overoptimization, scaling SAEs to GPT-4, interp via circuit sparsity. EleutherAI cofounder.
EXO Labs @exolabs
51K Followers 2 Following Frontier AI on local hardware. EXO 1.0 is now open-source (Apache 2.0): https://t.co/SGGGK784Qp
Alex Cheema @alexocheema
49K Followers 3K Following building @exolabs | prev @UniOfOxford We're hiring: https://t.co/UlkApFndnH
Peter Potaptchik @PPotaptchik
359 Followers 438 Following DPhil student at Oxford https://t.co/JH0l4u7wHv
Sajad Movahedi @Sajad_Movahedi_
42 Followers 126 Following PhD student in machine learning at @ELLISInst_Tue and @MPI_IS with @orvieto_antonio.
Emilian Postolache @EmilianPostola1
601 Followers 660 Following Lead ML Researcher @irisaudiotech | PhD in CS @SapienzaRoma in @GladiaLab | Former @CaFoscari, @SonyCSL, @Dolby and @c4dm
Yuxin Wen @ywen99
622 Followers 870 Following AI Security @OpenAI | PhD @umdcs advised by @tomgoldsteincs
Fu-En (Fred) Yang @FuEnYang1
931 Followers 2K Following Research Scientist @NVIDIAAI | Ph.D. @NTU_TW | Prev. Research Intern @NVIDIAAI | Unifying World, Language & Action for Generalist Robotics
Wei Deng @dwgreyman
414 Followers 625 Following Research Interests: Sampling and Diffusion (Language) Models
Kasper Green Larsen @kasperglarsen
2K Followers 260 Following Professor and Head of Algorithms, Data Structures and Foundations of Machine Learning at Computer Science, Aarhus University
An Thái Lê @an_thai_le
481 Followers 663 Following Assistant Professor for #RobotLearning at VinUniversity | Director of Foundation AI at VinRobotics
Weiyan Shi @shi_weiyan
9K Followers 1K Following Prof @Northeastern | MIT TR-35 | #AI2050 Early Career Fellow | Prev @Columbia @StanfordNLP | Co-created CICERO | human-AI co-evolution + AI safety
Puze LIU @liu_puze
367 Followers 294 Following Curiosity Driven, On Robot Learning. Associate Professor @Tongji_Uni Previous Deputy Head at @DFKI SAIROL, former w/@ias_tudarmstadt and @jan_peters
Neel Jain @neeljain1717
777 Followers 1K Following PhD candidate @umdcs @ml_umd advised by @tomgoldsteincs. Undergrad at Williams College in Math. My views are my own.
tingwu.wang @TingwuWang
1K Followers 583 Following Research scientist in robotics @ GEAR Nvidia. I obtained my PhD from University of Toronto @UofT, Vector Institute @VectorInst 😃
Avi Schwarzschild @A_v_i__S
922 Followers 249 Following Trying to learn about deep learning faster than deep learning can learn about me.
Percy Liang @percyliang
106K Followers 426 Following professor of computer science @Stanford @stanfordnlp, co-founder of @togethercompute, creator of https://t.co/7R5THVogW2, co-founder of @simile_ai, pianist
Samuel Albanie 🇬�... @SamuelAlbanie
8K Followers 1K Following frontier evals lead for gemini @GoogleDeepMind
Visual Inference Lab @visinf
852 Followers 390 Following Visual Inference Lab of @stefanroth at @TUDarmstadt. Research in Computer Vision and Machine Learning.
Shayne Longpre @ShayneRedford
6K Followers 1K Following Lead the Data Provenance Initiative. PhD @MIT. 🇨🇦 Prev: @Google Brain, Apple, Stanford. AI/ML/NLP
Christoph Reich @ChristophR1996
371 Followers 825 Following @ELLISforEurope Ph.D. Student @tumcvg, @visinf & @Oxford_VGG | https://t.co/yaqdM5V4IW. from @etitdarmstadt & https://t.co/VHpc5R9Fs1. from @CS_TUDarmstadt | Prev. @NECLabsAmerica & @koeppl_lab
Hossein Souri @HosseinSouri8
447 Followers 661 Following Senior AI Researcher at @Samsung_RA. CS PhD at @JohnsHopkins, MS at @UofMaryland.
Moran Mizrahi @moranmiz
376 Followers 312 Following PhD in CS @Csehuji (@HyadataLab); Passionate about NLP, Data Science, HCI and Computational Creativity.
Alvaro Arroyo @arroyo_alvr
299 Followers 234 Following PhD ML @UniofOxford ; Transformers & Graph Representation Learning; Previously at @imperialcollege
Jonas Geiping @jonasgeiping
5K Followers 882 Following Machine Learning Researcher in Tübingen at the ELLIS Institute & Max-Planck for Intelligent Systems // Working on Safety & Efficiency of modern ML
Felix Sarnthein @__safelix__
116 Followers 335 Following PhD student in machine learning at @ELLISInst_Tue, @MPI_IS and @CSatETH with @orvieto_antonio. Prev: MSc in CS at @ETH
Yifan Zhang @yifanzhang_
14K Followers 3K Following PhD at @Princeton University, Princeton AI Lab Fellow. RL & LLM Reasoning, Pretraining & Language Modeling. Prev @ Seed @Tsinghua_Uni
Sunny Sanyal @SunnySanyal9
1K Followers 757 Following On Job market | PhD candidate @UTexasECE| Prev Intern @GoogleDeepMind (FAI), @LightningAI & @AmazonScience (Alexa)
Hannah Rose Kirk @hannahrosekirk
4K Followers 755 Following AI researcher trying to make sense of the cyberspace 🤖 Workstream Lead @AISecurityInst. Uni of Ox PhD @oiioxford & Prev @Cambridge_Uni.
ryu @ryu0000000001
360 Followers 214 Following Nothing is boring. No knowledge is irrelevant: only not relevant *yet*. - Jonathan Gorard
ARIA @ARIA_research
17K Followers 56 Following Advanced Research + Invention Agency. Empowering scientists to reach for the edge of the possible.Trends for United States
You might like
