matt turk @TurkMatthew

ML researcher @withprotegeai prev: ML @cleanlabAI @goodwatercap, Quant @coinbase & @goldmansachs, EECS @ucberkeley datalab.withprotege.ai/people?person=… New York, NY Joined March 2012

Tweets

1K
Followers

719
Following

2K
Likes

10K

matt turk @TurkMatthew

3 days ago

Not banned in Greece

0 0 2 37 0

View Details

val @valtryeks

4 days ago

kinda messed up that brunson isn’t gonna be able to get into the parade because of the no bag policy

51 730 20K 381K 409

View Details

Game of Thrones @GameOfThrones

a week ago

Jalen Brunson, First of His Name, Breaker of Double Teams, King of New York, Most Valuable Player.

382 16K 102K 3.1M 4K

View Details

Real App @realapp

a week ago

Jalen Brunson just had the highest scoring Finals series by a point guard in NBA HISTORY.

106 2K 14K 388K 649

View Details

Trung Phan @TrungTPhan

a week ago

LinkedIn, X, Instagram, Reddit

469 5K 54K 2.0M 3K

View Details

matt turk @TurkMatthew

a week ago

@TheRealDanSaedi @mattjoseph27 @OpenAI 🔥

1 0 3 62 0

View Details

matt turk @TurkMatthew

2 weeks ago

@TheRealDanSaedi Incredible launch. Excited to see what is next!

1 0 7 68 0

View Details

Congrats on the launch to my friends @mattjoseph27 @TheRealDanSaedi @_jcfe these guys are a top tier team and have built something truly game changing for marketing. So proud of their hard work so far.

Daniel A. Saedi (DataManDan) @TheRealDanSaedi

2 weeks ago

AI can now read your customers' minds. We raised a $20M Series A lead by 8VC & Lingotto to build this. Introducing Minerva, built in collaboration with OpenAI:

562 384 2K 1.3M 2K

2 0 7 466 0

View Details

Protege @withprotegeai

3 weeks ago

We're excited to see @DataLabResearch @TurkMatthew’s new paper, “Counterfactual Evaluation Reveals Hidden Capability Profiles in Clinical LLMs and Agents,” accepted to the inaugural RLEval Workshop at ACM CAIS 2026 and selected for an invited talk. 🔍️ The guiding research question: "When clinically important patient facts change, does the model appropriately change its recommendations?" 👉️ If you change a key clinical detail that changes the case context, the model should change its recommendation. But if a clinically meaningful fact changes and the model doesn't update its recommendation, that counterfactual test exposes the gap. The model wasn't truly reasoning through the specific case in front of it — it was relying on surface patterns or the general shape of the case rather than the patient's actual circumstances. 🚨 What Matt found: "Producing the right answer and responding appropriately to new information are distinct capabilities - and future evaluation frameworks need both." ‼️ Why this matters: This connects directly to one of the hardest problems in benchmark design. It's not enough to measure whether a model arrives at the right answer. We also need to know whether it would arrive at a different answer when the underlying facts change. This is the kind of benchmark-design question that @engyziedan's @DataLabResearch focus on. Better benchmarks require more than held-out datasets. They require realistic, ground-truth evaluation frameworks that can distinguish between a model that genuinely updates its reasoning and one that reaches the correct answer for the wrong reasons. Congratulations to @TurkMatthew, and we're excited about the cutting-edge benchmark and evaluation work happening across the Protege DataLab.

0 2 7 181 0

View Details

matt turk @TurkMatthew

3 weeks ago

@arxiv @CAISconf You can see the workshop and accepted papers here: rl-eval.github.io

0 0 1 51 0

View Details

matt turk @TurkMatthew

3 weeks ago

Excited to share that my paper, "Counterfactual Evaluation Reveals Hidden Capability Profiles in Clinical LLMs and Agents", is now available on @arxiv (link in the comments). The paper was accepted to the inaugural RLEval Workshop at @CAISconf- a workshop focused on methods and reinforcement learning environments for evaluating AI agents - and was selected for an invited talk based on reviewer ratings. LLM evaluation is difficult. Models that look equally capable on traditional benchmarks can behave very differently when the underlying facts change. Most current benchmarks focus on whether a model's output looks correct. But in real-world settings, especially in healthcare, what often also matters is whether the model appropriately updates its recommendations when the underlying facts change. In this work, I introduce the Causal Sensitivity Score (CSS), a pre-registered counterfactual evaluation framework designed to measure exactly that: "When clinically important patient facts change, does the model appropriately change its recommendations?" Across six frontier models and several hundred oncology tumor board cases, I found that models with very similar performance on standard coverage-based metrics often behave dramatically differently under counterfactual interventions. In fact, model rankings were nearly reversed depending on whether you measured coverage or responsiveness. The paper also shows that these findings transfer to tool-using agents, revealing failure modes that remain hidden under conventional evaluation approaches. The broader takeaway is that producing the right answer and responding appropriately to new information are distinct capabilities - and future evaluation frameworks should measure both. This work is also closely aligned with the mission of our DataLab at @withprotegeai: building rigorous datasets, benchmarks, and evaluation frameworks that better reflect how AI systems perform in the real world. As AI moves into increasingly complex, high-stakes domains, measuring what models know is important - but we also need to measure how they respond when reality changes. I'm grateful to my colleagues and collaborators, especially @engyziedan and Wes Hopkins, as well as the medical professionals who helped validate the results.

3 1 6 364 0

View Details

matt turk @TurkMatthew

3 weeks ago

@arxiv @CAISconf Link to my paper: arxiv.org/abs/2605.30590

0 0 1 44 0

View Details

matt turk @TurkMatthew

3 weeks ago

@HarryStebbings @nico_laqua Grindslop

0 0 4 144 0

View Details

Zachary Lipton @zacharylipton

4 weeks ago

deep learning research was the original vibe math

14 23 310 74K 31

View Details

Mayor Zohran Kwame Mamdani @NYCMayor

4 weeks ago

.@NYCSanitation I'd like to report a sweep

1K 31K 288K 11.2M 6K

View Details

matt turk @TurkMatthew

4 weeks ago

Noah @NoahKingJr

4 weeks ago

Are people on LinkedIn aware that one day we are all going to die

149 1K 11K 467K 377

0 0 0 97 0

View Details

Trace Cohen @Trace_Cohen

4 weeks ago

NYC summers hate your weekends. I analyzed 3 years of Central Park rain data to see if the feeling was real. It was. 27 of 38 summer weekends had measurable rain. That’s 71%. Friday was the rainiest day of the week, with rain on 43.6% of summer Fridays. Basically worse than a coin flip for your evening plans. Sunday had the highest rainfall volume of any day, averaging 0.17 inches. It may not rain every Sunday, but when it does, it commits. Thursday was objectively the best day to be outside in NYC: lowest precipitation, clearest skies, least weekend-related misery. The wildest stat: 70.8% of rainy Sundays were preceded by a gross Friday or Saturday. The weekend basically telegraphs its own downfall. I built a full dashboard using NWS Central Park data with every weekend tracked and every raindrop counted:

89 481 4K 1.1M 1K

View Details

matt turk @TurkMatthew

4 weeks ago

@dair_ai Way too noisy of a process to forecast

0 0 0 66 0

View Details

matt turk @TurkMatthew

a month ago

Top-tier read calling out performative grind culture. “Great work has always demanded sacrifice and often brutal hours and I'm not disputing this. What I'm disputing is the direction. These people, many of them friends, have more economic freedom than any class in history and they've chosen, freely, to simulate the conditions of a Chinese assembly line and call it virtue.”

Will Manidis @WillManidis

a month ago

x.com/i/article/2056…

178 332 3K 1.5M 4K

0 0 1 149 0

View Details

OpenAI @OpenAI

a month ago

Today, we share a breakthrough on the planar unit distance problem, a famous open question first posed by Paul Erdős in 1946. For nearly 80 years, mathematicians believed the best possible solutions looked roughly like square grids. An OpenAI model has now disproved that belief, discovering an entirely new family of constructions that performs better. This marks the first time AI has autonomously solved a prominent open problem central to a field of mathematics.