Gregory Clark @_whatcode

1/ We built a compiler that turns arbitrary programs into transformer weights. You give it C code, it produces a standard transformer that executes that program step by step.

A non-hyped explainer of the “cell simulation” paper. The recent study about the “4D” simulation of a minimal cell has been getting a lot of attention on social media. Unfortunately, most posts about it have serious errors. I’ve seen people claim that the model simulates every chemical reaction in the cell, for example, which is not true. Some biomolecules and reactions *are* tracked individually in the simulation, including proteins and RNA (and ribosomes), and the chromosome. But the simulation does not track individual metabolites (like ATP or glucose), water, nucleotide precursors, lipds, and so on. These "other" molecules are represented, instead, as concentrations (using ordinary differential equations). But anyway, here goes my quick explanation: Researchers built a computational model that simulates roughly 100 minutes of biological time, or one cell division, for a single bacterial cell. Each simulation takes 4–6 days to run on two NVIDIA A100 GPUs, and the authors ran it 50 times in replicate. The cell simulation includes some elements of randomness, so each replication attempt leads to a slightly different outcome. When they plotted out these replicates and averaged results, they found that the model could predict a few things without being fitted to experimental data: The simulated cells “divided” every 105 minutes, on average, which matches experimental results; and the mRNA molecules had an average half-life of 3.63 minutes, which is roughly what we’d expect from experiments, too. The cell they are modeling is called JCVI-syn3A, and it is not a naturally-occurring organism. It’s a bacterium that has been engineered, over many years, to have a small genome. It only has 493 genes (compared to 4,000+ for E. coli), all of which are housed on a single chromosome. The Syn3A cell was made by taking a natural organism, called Mycoplasma mycoides, and then slashing out non-essential genes. Its entire proteome, transcriptome, and metabolism have been studied in depth, which is why it’s being used to build these whole-cell simulations. The actual *simulation*, though, is not a single thing! Instead, the authors wrote down all the “stuff” that happens inside a cell (transcription! translation! metabolism! lipid biosynthesis!) and decided which type of mathematical model would be best-suited to describe each thing. Some cell processes were modelled deterministically, others had “spatial” elements, and other parts were relatively random. More specifically, they used four different types of models to build this simulation: 1. A Reaction-Diffusion Master Equation, which was used to model the individual proteins, RNAs, and ribosomes. 2. A Chemical Master Equation, which was used to model things where spatial location doesn’t matter as much (it basically treats the whole cell as one mixed entity); including tRNA charging. 3. Ordinary Differential Equations, which you may be familiar with from Calculus class, were used to model changes in ATP concentration, lipids, and so on. 4. Brownian Dynamics, which simulated the chromosome as a physical chain of beads, where each bead represents 10 base pairs of DNA. The Reaction-Diffusion Equation works like this: Basically, they chopped up the entire digital cell into a 3D grid of cubes. Each cube measures 10 nanometers on each side. The whole cell is about 500 nanometers across, so there are tens of thousands of cubes in the cell's interior. (This is a useful way to coarse grain the simulation; if the cubes were smaller, the simulation would take much longer to run.) Each cube is a little box that contains some number of molecules. At every “step” in the simulation, only one of two things can happen to the molecules in each box: Either they react with a molecule in the same box, or they diffuse (“hop”) to an adjacent box. That’s it; the model is just rolling a die for each molecule at each time step in each box, and using those results to decide how each molecule changes over time. (The reason this spatial model is important is because biology only works if molecules physically bump into each other. And so this spatial grid means that, unlike simpler models, a protein actually has to “diffuse” across boxes in the cell to encounter its reaction partner; only then can it react and do something useful.) So anyway, each of these models is used to represent a different type of molecule. It’s not like there is a single, all-powerful simulation that they are running here; instead, they’re running these four models together, using a script that synchronizes their results with each other. The Reaction-Diffusion equation is the main part of the simulation. It takes time steps of 50 microseconds of biological time. Every 12.5 milliseconds of biological time — meaning every 250 RDME steps — the simulation pauses so that the other models can synchronize based on the latest state of the simulation. The Brownian Dynamics part runs on a completely separate GPU, and only updates every four seconds of biological time. So that's the gist here. But let's also be honest about what this simulation does NOT do: - It does not include polysomes, which are a cluster of ribosomes that all latch onto a single mRNA and translate at the same time. Polysomes are really common inside of cells, but this simulation assumes that each mRNA can only be translated by one ribosome at a time. - It does not include polycistronic transcription. In bacteria, genes are often grouped next to each other on the chromosome and thus “transcribed” (or turned into mRNA) all at once, together. The majority of genes in E. coli, for example, are arranged in these operons, and the authors of this paper acknowledge that many Syn3A genes are likely co-transcribed the same way. But the simulation doesn't capture it. - The authors manually tuned many parameters to get the model to make predictions that more closely resemble experiments. Earlier simulations were waaaayyyyy off from experimental results. For example, they adjusted the ratio of mRNA binding rates to ribosomes versus degradosomes because, in earlier simulations, mRNA was being degraded too quickly, before ribosomes could translate it, causing most proteins to be severely underproduced. - In the Brownian Dynamics model, the authors added a “fake” 12 pN physical force to push the two daughter chromosomes apart during division, because the real biological mechanism for chromosome partitioning in Syn3A is not known. - And some other things. That being said: This model is really cool! I love papers like this! I'm enamored by scientists who choose really difficult problems (like simulating an entire cell) and actually go after it and make progress! This paper is amazing because it shows us what we are able to simulate well, and what we don't yet understand, and to figure out which experiments we ought to perform to reconcile the two. So instead of framing this paper as "OH MY GOSH SCIENTISTS FIGURED OUT HOW TO SIMULATE AN ENTIRE CELL!" we should frame it as proof that there is still plenty of room at the bottom, many measurements to be made, and many avenues to explore as we seek to understand biology better.

@pickover @ErrantScience I misread "denial" as "dental", which didn't seem incorrect..

I think I've uncovered the next Turing test for AI video: writing. I've tried this prompt dozens of times on every model: "man writes 'hi' in chalk on blackboard" None can do it. Veo 2 (below) gets the closest. It's actually frustrating to watch!

I drew these strawberries with mathematical equations.

@jeffdavismd @AndrewYang Nobody wants men to compete in women's sports; that's a weird spin that the right pretends the left has. Trump has ties to project 2025, which literally aims to deconstruct parts of the constitution.

@jeffdavismd @AndrewYang Wut. Trump pushed for stronger boarder policies to be blocked earlier this year. Trump has encouraged Putin in his war on Ukraine. Trumps proposals to increase tariffs on imports would cause more inflation than Kamala's proposed policies. Trumps tax cuts only favor the rich.

Check out my membership card from patreon.com/slow_spokes!

@010010110000110 Mine would look about the same.

What if we could run Transformer models without worrying about context length? With our new work Unlimiformer, you can jailbreak your current models to use unlimited length inputs! Preprint: arxiv.org/abs/2305.01625 Thread 🧵 (1/6)

Not easy, but if we can figure out how to extend this to LLMs the impact would be huge. Imagine having access to models that take 5 minutes to ponder each response but the output is as good as a model that's 1,000x larger and trained for 1,000x longer than GPT-4

Igor Babuschkin @ibab

3 years ago

I keep revisiting this great paper from @andy_l_jones: “Scaling scaling laws with board games”. It shows how training compute and inference compute of MCTS can be traded off against each other. 10x more MCTS steps is almost the same as training 10x more. arxiv.org/abs/2104.03113

12 68 511 332K 276

Did you know that you can estimate correspondences with Stable Diffusion in an unsupervised manner? Our new method surpasses weakly supervised methods and closes the gap to strongly supervised methods. Our new paper shows you how! Paper: arxiv.org/abs/2305.15581 See thread ⬇️

New preprint on scGPT: first foundation large language model for single cell biology, pretrained on 10 million cells. Just as text is made of words, cells are characterized by genes Some thoughts on how cool this is & why it challenges the status quo of single cell analysis🧵🧵

Meta-Learning Task Generator: randomly generate an infinity of simple meta-reinforcement learning tasks. github.com/ThomasMiconi/M… The space covered includes bandit tasks, the Harlow task, the 2-step task, mazes, and others. Also, it's easy to represent graphically:

keyboard_arrow_left Previous keyboard_arrow_right Next

Our paper on understanding AlphaZero♟️is now published at PNAS! pnas.org/doi/10.1073/pn… The paper "studies" AZ's internals and its behaviors in collaborations with @DeepMind and world chess champion @32gcfhkmm. What did we learn? 🧵

It's totally okay to not have any research ideas that excite you from time to time. Instead of having a structure that rewards continuous publication of mediocre ideas, researchers should be allowed to have "downtime," where they teach, mentor, write, and freely think.

People will soon realize that automated prompt engineering is akin to latent variable inference. Then they will realize that latent variable inference is akin to reasoning/planning. Then they will realize that transformers were missing that piece all along.

Hear about our unpublished speculations and wild kanjectures with @kanjun and @joshalbrecht in this conversation 👇👇 It was so much fun to chat, thanks for having me! 😊

Imbue @imbue_ai

4 years ago

Learn about supermasks, iterative learning, fortuitous forgetting, and much more with @oh_that_hat from Mila! generallyintelligent.ai/podcast/2022-1…

1 3 14 0 2

📢📢📢 𝗔𝘁𝘁𝗲𝗻𝘁𝗶𝗼𝗻 𝗕𝗲𝗮𝘁𝘀 𝗖𝗼𝗻𝗰𝗮𝘁𝗲𝗻𝗮𝘁𝗶𝗼𝗻 𝗳𝗼𝗿 𝗖𝗼𝗻𝗱𝗶𝘁𝗶𝗼𝗻𝗶𝗻𝗴 𝗡𝗲𝘂𝗿𝗮𝗹 𝗙𝗶𝗲𝗹𝗱𝘀 We ran A LOT of experiments to find the best way to make neural fields generalize... so you don’t have to! arxiv.org/abs/2209.10684

What would happen if climate-related tipping points, were pushed beyond their limit? Host @maiyamay_ looks at major tipping points, and what will happen if triggered. Some damages would include massive droughts, heat waves, food shortages, and much more. to.pbs.org/3D4WJcL