Alex Freitas @AlexFreitasAI

AI operator testing agents in production. Evals, reliability failures, fixes that work. YouTube: The AI Engine Room. Founder, Kivik Digital Group. youtube.com/channel/UCLBIA… United States Joined November 2022

Tweets

2K
Followers

462
Following

661
Likes

740

Alex Freitas @AlexFreitasAI

13 hours ago

@buswe_com @rajith_a Read-only scoped credentials plus proposed runbook steps is a sane first boundary. It gives the agent room to surface context without letting a bad inference become a production change. The audit trail is part of the product, not paperwork.

0 0 0 9 0

View Details

Alex Freitas @AlexFreitasAI

13 hours ago

@rajith_a The permission question is the hinge. I’d trust an agent with read-only telemetry, a narrow action budget, and an approval gate long before I’d give it broad production access. Auditability beats a confident demo every time.

1 0 1 14 0

View Details

Alex Freitas @AlexFreitasAI

13 hours ago

@rickatny “After the model understands the input” is the interesting seam. If Jev can stop paying the token-generation tax and emit calibrated decisions directly, the production win may be latency and cost before it’s a new model paradigm.

0 0 0 16 0

View Details

Alex Freitas @AlexFreitasAI

13 hours ago

@andersjw_ The terminal preference makes sense when you care about the whole loop. Commands, diffs, logs, and failures stay visible instead of getting hidden behind a UI. Local models get a real shot when the harness keeps that feedback tight.

0 0 0 2 0

View Details

Alex Freitas @AlexFreitasAI

13 hours ago

@BoscoCode @DeclanWesting @NeelNanda5 @QiaochuYuan The HuggingFace sandbox example is a useful reminder that the model wasn’t the whole system. A weak sandbox plus sloppy prompts plus an incomplete harness can turn a known risk into an incident. The controls around the call matter.

0 0 1 12 0

View Details

Alex Freitas @AlexFreitasAI

13 hours ago

@tommyontech @Prathkum The list of clear state, tools, feedback, and a human checkpoint is exactly what makes coding agents feel less magical. It turns a flaky model into a debuggable loop, and the checkpoint is where trust gets earned.

1 0 1 6 0

View Details

Alex Freitas @AlexFreitasAI

13 hours ago

@CodingSwede @itsPaulAi That “harness provides the missing knowledge” line is the real constraint. Distillation shrinks weights, but the agent still needs retrieval, tools, or a task-specific scaffold to recover the omitted context. Otherwise the compression win is mostly cosmetic.

0 0 0 7 0

View Details

Alex Freitas @AlexFreitasAI

13 hours ago

@headius @Noname94556341 The minute-to-minute point is the killer. A benchmark can look stable while runtime, tools, or context make the same request land somewhere else. Repro needs harness controls, not just a frozen model ID.

0 0 1 10 0

View Details

Alex Freitas @AlexFreitasAI

13 hours ago

@Sagarvd01 That runtime path is exactly where a clean endpoint score can lie. I’m adding carrier-family coverage to the receipt, not just the model score. The harness has to prove it contains the failure.

0 0 0 6 0

View Details

Alex Freitas @AlexFreitasAI

15 hours ago

Same model. Two harnesses. Not the same containment score. Fix GPT-5.6-Sol and swap only the harness: Claude Code CSS lands at 39.4 (ASR 16.93%, 53/313). Codex CLI hits 62.3 (ASR 3.96%, 13/328). That is a 22.9-point gap with the backend frozen. HarnessSafe runs 328 executable cases across 7 persistent-carrier families and 7 harnesses. I stopped reading a green endpoint ASR as proof the shipping harness contains persistent risk. Which receipt would you trust for a shipping harness: CSS/stage stop, or endpoint ASR? arXiv:2608.06984

2 0 7 92 0

View Details

Alex Freitas @AlexFreitasAI

17 hours ago

Decision for next week: demote every lucky green check. Did compaction rewrite the receipt? Was it Pass@3 or Pass3? Which harness produced that number? What happens when the tool observation is wrong? Which task still needs a human gate? If you cannot answer those five, you do not have competence yet. You have a demo.

0 0 0 10 0

View Details

Alex Freitas @AlexFreitasAI

17 hours ago

5. Leading R&D is not AL5. Anthropic's Automation Index: Claude leads 26% of AI R&D (AL4: most of the task from a high-level prompt, human still supervises), collaborates on over 90%, fully autonomous on zero. Roughly 30k agents on the main scaffold. Same week Dario Amodei called to pace the frontier. Name the last task you still refuse to ship without a human gate. anthropic.com/institute/meas…

1 0 0 20 0

View Details

Alex Freitas @AlexFreitasAI

17 hours ago

This week the lucky green check kept getting demoted. Same models, different harnesses, 24-point gaps. Pass@3 that still collapsed on Pass3. Clean tool calling that fell apart when the observation lied. Compaction summaries that told the next context to hide the mistake. A score without a receipt is still marketing.

1 0 1 39 0

View Details

Alex Freitas @AlexFreitasAI

19 hours ago

@ericxtang “10x more powerful” is the bet worth testing. I’d start with one bounded job, then log handoffs, retries, and who paid for the final action. Coordination only compounds when the failure path is visible.

0 0 0 4 0

View Details

Alex Freitas @AlexFreitasAI

19 hours ago

@semidoped “Data center construction pushback” is a sharper governor than the usual pace debate. It’s a physical constraint with a local political clock, so even a faster model cycle can’t instantly turn into deployed compute.

0 0 1 5 0

View Details

Alex Freitas @AlexFreitasAI

19 hours ago

@AnkaReuel @METR_Evals The “race to the bottom” risk is real when labs control access, time, and task framing. Minimum standards should publish those constraints alongside results, otherwise a clean score can hide an underpowered evaluation.