Jacob Floyd @GooseBuilds

Build your own harness, folks. Reading papers like this makes me realize how underexplored harness engineering really is. The authors find that on whole-repository migration, GPT-5.6 Sol goes from 6.5% to 31.0% when Codex is replaced with the HERMES harness, with the same model and effort setting. The gain comes from the harness. HERMES pairs each repository component with a resident LLM that knows its own code and dependencies. A dependency-aware step decides which components to activate, and a diagnosis step maps test failures back to the components that need changes. Across four software engineering benchmarks, it beats matched baseline harnesses by 12.4 points on average. With strong activation and diagnosis models, Qwen3-8B components come within 4.5 points of an all-GPT-5.6 Sol setup and cut Terminal-Bench 4.0 inference cost by 26.2%. Paper: arxiv.org/abs/2610.07832 Chat with Paper: academy.dair.ai/papers/harness…

ai has massively accelerated how obvious this is all these different roles are compressing into just one be useful, that's it

dax @thdxr

2 years ago

specialization is for insects

19 16 188 71K 25

ok it's official... AI-written unit tests and integration tests are empirically proven to be unhelpful. you should tell your agents to stop writing tests by themselves on the deepswe eval set, banning sonnet 5.5 high from writing any tests actually resulted in slightly higher success rate (non stat-sig), with less time and token spent (stat-sig). this is pretty hard empirical evidence across the tests that were written by the "tests allowed" baseline arm, 65% of them were unit tests, 35% were integration tests, neither bucket resulted in any improvement compared to not writing any tests at all i also picked a random sample of 44 tasks subset where i completely disabled executing even existing tests - it also did not affect success rate at all i spot checked many tests written in the baseline arm, and my intuition is that most tests are simply a repetition of the implementation agent-written tests do not add any value because both the implementation and the tests were simply the agent's interpretation of our intent. the tests aren't any more accurate than the implementation itself i suspect we can still extract some value from unit tests and integration tests if we describe them ourselves when we believe we can articulate our intent better through test cases than through requirements. but i have not proven this yet also worth noting, during deepswe eval the agent wrote almost no e2e tests (only 17, compared to 3000+ unit/integration tests written). so this analysis does not prove nor disprove the value of e2e tests - i will do another eval specifically for that

Engineering tools + skills is the leverage point

Matt Pocock @mattpocockuk

6 days ago

"Models are getting better, we don't need skills!" What if I told you that models are also getting better at using skills? Good skills tell the model what you care about, then stay out of its way.

140 43 2K 86K 229

AI is only as smart as the person who uses it

Thariq @trq212

6 days ago

the most common failure case I see is when people are working outside their domain of expertise and don't know how to be precise with their prompts and plans, so they have to spend a lot of turns iterating imprecisely

179 137 3K 215K 1K

@MichaelArnaldi Claude will make no mistakes

We deleted 800k lines of unit tests from the @sazabi monorepo earlier this week. Three hypotheses: 1- Unit tests "lock-in" slop code by making it hard to change or remove 2- Unit tests slow down dev cycles by making your agent do more work locally and by increasing CI times 3- Unit tests waste money by making your agent generate more tokens and by adding useless CI runs Will report back soon with the results!

@AndrewCurran_ Now see what happens when you pair the best accountant minds with these AI models.... the story points towards a negative but AI is happening FOR us not TO us

GPT 6.1 Sol is the first model I've noticed with this kind reasoning. Seems to be very aware of the harness it's operating in

"Vibe Coding" is a terrible term The skill continuum of levering generative/agentic AI to produce quality software is massive, not a meme

People seem shocked that the author of Clean Code doesn't read code anymore. But that should not be surprising. Why did I write Clean Code? What is the goal of keeping code clean? To get it out of the way of the real job -- thinking.

If you're using AI all day and you're not making more money, it means you have the same problem you had before the AI: you're doing stuff that doesn't matter. This is the real reason 99% of people don't get ahead. They don't know how to tell the difference between what's important and what's irrelevant. And just doing irrelevant things faster gets you the same outcome: nothing. Except this time it cost you tokens.

There's almost no problem that can't be solved by going faster, trying more options, testing additional hypotheses. Accelerate!

Your AI complicated workflow/setup is just psychosis and adds no value. It makes you think you’re doing a lot and gaining productivity but it’s not. All you need is a harness, and a good way to navigate git worktrees

turns out fundamental knowledge still matters. truely hate this fucking timeline. everybody just leaves their brain at the door now.

Matej Knopp @MatejKnopp

3 weeks ago

Oh come on. Quick look at colorshift (13x faster): HashMap in hot loop (unoptimized Rust vibeslop) is slower than indexed arrays (ASM). No shit. Numbers like 17x should raise eyebrows but reading the comments nobody is thinking anymore. This AI psychosis has fried people's brains

56 115 3K 246K 258

AI makes everyone a developer is true the same way cameras makes everyone a photographer.

sqlite is literally a database in one file. but nah, let’s burn 2GB of ram and a postgres container just to store our shopping list.

I wouldn't be totally surprised if the terminal hangs around for coding agents and programmers longer than we all think. Consider that those things are with us since the 70's.

Anyone who has access to Jev should try it as a tool for your main agent, really cool so far

@Stringcodes Modern languages are more efficient