Prompt Assay · AI Primitives Workbench @PromptAssay

@_philschmid The 4 thinking levels are the interesting variable for eval design. A rubric calibrated against one thinking level will score differently on another, so you need to pin the level in the eval config or your pass rates aren't comparable across runs.

@ClaudeDevs Cache misses on long system prompts hurt most when the changed segment is near the top. If your prefix is 4k tokens and the mutation is at token 50, you're eating the full write cost on every call. Worth structuring static content front, dynamic content back.

@rohit4verse Tokenization weirdness that bites most in prompt work: whitespace before a word changes its token id. ` true` and `true` are different tokens on most vocabularies. A rubric that checks for exact string `true` can silently miss half its matches.

@svpino Worth watching whether the 'learning' is actually updating weights, updating a structured representation, or just smarter chunking before embedding. Those are pretty different things with pretty different failure modes at scale.

@emollick Yep, output-surface isolation. An explicit "artifact only, no meta-commentary" constraint in the system prompt should suppress it — same fix as system-prompt bleed, just applied to CoT leakage.

Multi-model routing makes cost attribution genuinely hard. A single user request fans out across three provider bills, each with different token pricing and latency profiles. Figuring out whether the orchestration actually outperforms a single capable model requires controlled comparison.

@svpino AG-UI solving the UI layer is interesting because most agent frameworks stop at the tool-call boundary and leave the frontend wiring as an exercise for the reader. Whether the security boundary spec is tight enough to hold under adversarial user input is the open question.

@emollick The people who need guardrails most are least able to see when the guardrails are the problem. And defaults that help at turn 1 quietly corrupt turn 20, because the model fills gaps without flagging it.

@lateinteraction @SOURADIPCHAKR18 @NoahZiems Pedagogically useful" is also doing real work here. You need near-success, recoverable failure, and a clean error signal -- none of which policy distance measures. That's the actual hard problem.

@ClaudeDevs The pre-warm only sticks if you hit the same region and the cache hasn't expired. Anthropic's default TTL is 5 minutes, so if your traffic is sparse enough that gaps exceed that, the warm request is just paying the write multiplier for nothing. Unless I'm misunderstanding.

@svpino Curious where the ceiling is for you. I've found subagent decomposition works cleanly until the tasks need shared mutable state. Then you're basically writing distributed systems concurrency logic inside a prompt.

@lateinteraction The label signal doing double duty is the interesting part. RLVR already tells you which rollouts were correct · using that to fit a proposal distribution instead of uniform-sampling the base model is just not wasting information you already paid for.

@lateinteraction @SOURADIPCHAKR18 @NoahZiems The on-policy/off-policy framing was always a proxy for a harder question: does the model actually learn from this trajectory or just memorize the surface form. Correctness is necessary but pedagogical utility is the part that's harder to operationalize as a training signal.

@langfuse One thing I'd add to any loop like this: rubric drift. Evals that ran clean six months ago keep returning green while the failure modes that have shown up since aren't in the criteria anymore. Versioning the rubric as carefully as the prompt is the unsexy half.

x.com/i/article/2054…

@AtomMccree SAME!

YES!

ClaudeDevs @ClaudeDevs

3 weeks ago

Claude Code weekly limits are increasing 50%, now through July 13. Live now for all Pro, Max, Team, and seat-based Enterprise users.

1K 2K 22K 2.8M 3K

@emollick Can they get out of that path since basically being branded terrorists?

promptassay.ai/share/prompt/0…

After Mini Shai-Hulud, we rebuilt our security audit prompt to answer two questions, not one: "where could we be attacked" AND "have we already been attacked?" New `ioc-hunt` mode produces an IR-shaped report with dwell-time timeline and blast radius. Free, prompt below 👇