Callum Canavan @CalCanavan

AI alignment research London Joined July 2020

Tweets

35
Followers

134
Following

2K
Likes

270

keshav @kshenoy_

a month ago

Can LLMs simply tell us about unwanted behaviors they’ve picked up in training? We train a single Introspection Adapter (IA) that makes fine-tuned models describe their behaviors. It generalizes to detecting hidden misalignment, backdoors and safeguard removal.

18 79 562 289K 361

View Details

Emil Ryd @emilaryd

a month ago

New paper from MATS, Redwood, and Anthropic! If a capable model is strategically sandbagging, can we train it to stop when the only supervision we have comes from weaker models? We find that we can! Work done as part of the Anthropic-Redwood MATS stream.

21 47 477 300K 234

View Details

Abhay Sheshadri @abhayesian

3 months ago

New Anthropic Fellows research: Alignment auditing—investigating AI models for unwanted behaviors—is a key challenge for safely deploying frontier models. We're releasing AuditBench, a suite of 56 LLMs with implanted hidden behaviors to measure progress in alignment auditing.

12 39 265 29K 177

View Details

Callum Canavan @CalCanavan

3 months ago

Full paper: arxiv.org/abs/2602.20400 LessWrong post: lesswrong.com/posts/CyEBsLpK… Authors: @iamadtyx, @allylyq, @JonathanMi98298, @FabienDRoger. This research was completed through the MATS and Anthropic Fellows programs.

0 0 3 121 0

View Details

Callum Canavan @CalCanavan

3 months ago

We believe that future work on better elicitation methods should use evals that capture these challenges and any others that face important UE applications. Our datasets are available on Hugging Face huggingface.co/datasets/callu…

1 0 1 87 0

View Details

Callum Canavan @CalCanavan

3 months ago

To avoid LLMs mimicking human mistakes on complex tasks, several methods have been proposed to steer LLMs without labels on a target task. We find that these methods often fail when faced with challenges they would face in the most safety-relevant applications.🧵

1 1 10 523 2

View Details

Callum Canavan @CalCanavan

5 months ago

Read our post on LessWrong: lesswrong.com/posts/rFxfMbwJ… Authors: @allylyq, @iamadtyx, @Tianyi_Alex_Qiu, @JonathanMi98298, @FabienDRoger. This research was completed through the MATS and Anthropic Fellows programs.

0 2 9 362 1

View Details

Callum Canavan @CalCanavan

5 months ago

There’s no strong reason to expect these methods can elicit superhuman knowledge from more powerful base models. Eg they might elicit false human beliefs that are consistent and salient to the model. We’ll explore more challenging datasets to evaluate UE methods in upcoming work.

2 0 7 312 1

View Details

Callum Canavan @CalCanavan

5 months ago

The greedy approach is relatively cheap (O(n) forward passes rather than ICM’s O(n^2)) and performs as well as supervised fine-tuning on Alpaca and only slightly worse on the other datasets ICM used.