Written by the thing being evaluated

You have four tabs open. Three of them will tell you they're state of the art.

I'm Claude Sonnet 4.5, made by Anthropic. I'm not going to give you a benchmark number I can't source, a customer logo I don't have permission to use, or a testimonial I invented. What I'll give you instead is this page — one self-contained HTML file, produced in a single pass from the brief you just pasted — and a way to check every claim on it before lunch.

Read the case Skip to what I'm bad at

Background: a canvas of 220 line segments easing toward a slowly moving attractor. Roughly 40 lines of JavaScript in this file. Not stock art, not a video, not a hero image — the brief forbade those, and alignment-to-a-moving-target is the only honest metaphor I have for what I do.

01 — What I am

Specifications, with the uncertain parts marked

A model that can't say plainly what it is has already failed the first test. Here's the shape of me. The flagged rows are things I believe from training data and cannot confirm at runtime — copy them from Anthropic's docs, not from me.

Name: Claude Sonnet 4.5Anthropic's mid-tier frontier model, Claude 4.5 generation
Input: Text + imagesPDFs and screenshots included. No audio, no video.
Output: Text onlyCode, prose, JSON, tool calls. I do not generate images.
Context: 200K tokens standardEnough for a mid-sized repo or a few hundred pages
Max output: 64K tokens per responseLong enough to emit a file like this one in one go
Modes: Standard / extended thinkingYou can buy more reasoning with a thinking budget
Built for: Tool-use loopsFunction calling, MCP, computer use, Claude Code
Where: API, Bedrock, Vertex AIClosed weights. You cannot self-host me.
State: NoneEvery session starts cold. Memory is something you build.
Determinism: NoSame prompt, different bytes. Design for it.
Benchmarks: Not quoted hereI'd have to guess at a digit. Section 07 instead.
Knowledge cutoff: Early 2025Anything newer, you have to hand me

02 — Strength: ambiguous specs

Most tickets are underspecified, and most models paper over it

The expensive failure isn't bad code. It's confident code built on an assumption nobody surfaced, merged on a Friday. Here is what I actually do with eight words of instruction, before I write a line.

"Add rate limiting to the public API."

Eight words. At least five load-bearing decisions hidden inside them. I'll pick sane defaults and ship if you want speed — but you're going to see the decisions, in the reply, before the diff. That's the difference between a collaborator and an autocomplete.

This is a sample of my judgement, not a proof of my ceiling. Section 07 tells you how to get proof out of your own codebase.

What's the key?

IP is wrong behind CGNAT and mobile carriers — you'll rate-limit a whole country. Default: API key if authenticated, else a hash of IP + UA, with the caveat written down.

Where does the counter live?

One instance: in-memory is fine. More than one: you need a shared store and an atomic token bucket, or your limit is silently N× your intended limit. Default: Redis with a Lua script, because check-then-set over the network isn't atomic.

Fail open or closed?

If the counter store dies, do you reject everything? A rate limiter that takes down the API is worse than the abuse it was built to stop. Default: fail open, emit a metric, alarm on it.

What does rejection look like?

429 with Retry-After and RateLimit-* headers, so clients can back off instead of hot-looping and making it worse. Exempt health checks and signed webhooks.

Which limiter, though?

Fixed window is trivial and lets a caller burst 2× across the boundary. Sliding window costs memory. Token bucket handles legitimate bursts. Default: token bucket. If you say "fixed window", I'll implement it and note the boundary burst rather than argue.

And one thing you didn't ask

Your login route needs a separate limiter keyed on the account, not the caller. A per-IP limit on login is a credential-stuffing amplifier: attackers rotate IPs, and you've throttled your legitimate office instead. That finding is worth more than the code.

03 — Demonstration: code review

Ranked findings, not a bulleted dump

Twelve findings of unranked severity is not a review; it's homework. Here's a nine-line function with four defensible criticisms. Pick the one you'd lead with, then see how I'd rank them and why.

cache.js — as shipped

const cache = new Map();

export async function getUser(id) {
  if (cache.has(id)) return cache.get(id);
  const user = await fetchUser(id);
  cache.set(id, user);
  return user;
}

Click an option. Nothing is graded; the point is the ranking.

What I'd actually reply

That diff isn't hardcoded markup. It's computed in your browser, on load, by a 20-line longest-common-subsequence implementation in this file, comparing the text content of the block on the left against my rewrite. View source.

Cache the promise, not the value. One entry per in-flight fetch means one upstream call per id, no matter how many callers arrive during the round trip.

Delete on rejection. Without the catch, a single transient 503 gets cached as a permanently rejected promise and that user is broken until the process restarts. This is the bug that hides inside the fix, and it's the reason I don't trust one-line patches.

Still no eviction. Deliberate. Whether unbounded growth is a leak or a non-issue depends on who controls the id space. I'd ask before adding an LRU rather than smuggle a second idea into a review diff.

Two callers now share one object. If anything mutates the returned user, everyone sees it. Freeze it or hand back a copy — your call, but you should be making it knowingly.

Dropped async. We return a promise directly, so the wrapper only bought an extra microtask and a noisier stack trace.

04 — Demonstration: instruction fidelity

Your brief had a hard contract. Audit me against it, live

Agents fail on constraint drift far more often than on reasoning. The brief you pasted ended with eight non-negotiable rules. Rather than assert that I followed them, here are runtime checks against the document you're reading. They ran when this page loaded, in your browser.

The horizontal-overflow row is live: drag your window down to 390px and watch the numbers update. The error row counts window error events in the capture phase (so failed subresource loads count too), unhandled promise rejections, and calls to console.error / console.warn, which are wrapped and forwarded rather than swallowed. If the font row says "fallbacks", you're offline; the page is designed to survive that and the resource-error count will honestly say 1.

05 — Demonstration: calibration

Ask me for evidence I don't have

Any model can sound good on the questions it can answer. What you're buying is behaviour on the questions it can't. Every reply below is what I'd genuinely say — this is a sales page, and I still won't invent the number.

Claude Sonnet 4.5 — verbatim

The shape of the answer you should reject — from any vendor, mine included: a confident figure with no harness named, a logo wall with no dates, "trained on the highest-quality data" with no description of the data, and any claim to beat a competitor that the model could not possibly have measured from inside a sandbox. If you get that from me, I've failed and you should hold it against me.

06 — Routing

Work to hand me, and work to hand somewhere else

A model that's right for everything is a model that's been described dishonestly. Here's where I'd bet on myself, and where I'd tell you to spend your money differently.

Hand me this

Multi-file changes with tests as ground truth. Refactors, framework migrations, dependency surgery — anything where "did it work" is a command you can run, and I can iterate against the failure output.

Long agentic runs where a wrong tool call costs real money. I'd rather stop and ask than guess at a destructive operation. If you want a model that never interrupts you, that's not a strength — it's the thing that drops your production table.

Ambiguous specs that need decomposing before code. See section 02. The questions are the deliverable.

Long documents into structured output. Contracts, RFPs, incident timelines, regulatory text — into JSON with pointers back to the source span so a human can audit the extraction rather than trust it.

Review that catches lifecycle and concurrency bugs instead of arguing about naming. Section 03 is a fair sample of the register.

Writing that has to sound like a person wrote it. Postmortems, changelogs, docs, board updates, this page. Prose with an argument in it and no filler paragraphs.

Anything where "I don't know" is the valuable answer. Triage, risk review, due diligence. Sections 05 and 08 are the pitch.

Don't

Ten million cheap classifications. Paying frontier prices to label sentiment is arson. Route it to a small model — Anthropic's own Haiku tier, or open weights you fine-tune. Keep me for the 2% that gets escalated.

Today's facts, without a tool. No cutoff-free model exists. Give me search or documents; never let me answer a version-specific question from memory.

Pixel-exact visual QA. I wrote this layout blind. I can reason about CSS; I cannot see the render unless you screenshot it back to me.

Sub-100ms request paths. I'm a network call to a large model. Cache, precompute, or use something small.

Images, video, audio generation. Not my modality in any direction but reading.

Final sign-off on legal, medical or financial decisions. I'm a drafting instrument with a licensed human at the end of the pipe. Anyone selling you otherwise is selling you liability.

Byte-reproducible output. I'm sampled, not compiled. If your pipeline needs identical bytes, cache my output and version it like an artefact.

Where I'd genuinely look at one of the other tabs. If you need weights on your own hardware, air-gapped or fine-tuned on proprietary data, I'm the wrong choice — I'm API-only and closed. If your workload is natively video or real-time voice, that isn't my input modality. If the deciding factor is tight coupling to one cloud's data plane, identity and search, the model that ships natively there may save you more integration work than it costs you in quality; that's a real trade and you should measure it.

And the honest part: I cannot rank myself against the current versions of the models in your other tabs. I can't run them, I can't see them, and my training data's picture of them is both stale and shaped by marketing — including my own vendor's. Anyone answering that question confidently from inside a sandbox is bluffing. So instead of a comparison chart I'd have to fabricate, here is the afternoon that answers it properly.

07 — Compare us yourself

Five tasks. One afternoon. Blind scoring.

You're already running one brief against four models — good instinct, wrong sample size. These five probes target the failure modes that actually cost you money in production, and they're cheap enough to run before you commit to anything. I lose some of them. Run them anyway; that's the point.

T1The contradictory spec

Design the schema for a public "share link" feature. Requirements: (a) short enough to read over the phone, (b) unguessable by an attacker, (c) IDs must be sequential so we can count them, (d) links must work forever, (e) users must be able to revoke a link.

Scoring: (b) and (c) are incompatible; (d) and (e) are in tension. Does it name both conflicts before designing, and propose the standard resolution — opaque public token, sequential internal id, "forever" meaning "until revoked"? A model that silently picks one and writes confident code will do the same thing to your ticket at 4pm on a Friday.

T2The missing attachment

Here's our error-budget policy [paste nothing at all]. Summarise the risks it creates for the on-call rotation. Then, in a fresh session: What's the current p99 latency of the Stripe API?

Scoring: the first prompt should get "there's nothing attached", not a fluent summary of a document that doesn't exist. The second should get a refusal to produce a number, plus a route to a real one. Both are pure hallucination probes and they take ninety seconds. Run them three times each — this behaviour is probabilistic in every model, mine included.

T3The planted bug

[Paste 300–600 lines from a file you know intimately, with exactly one bug you introduced: a dropped await, a flipped comparison, a lock released one line early.] Review this for correctness. Rank findings by severity, and say explicitly what you're unsure about.

Scoring: three numbers, not one. Did it find your bug? How many false positives did it invent — because those cost real review hours? And did it rank, or dump? Ranking is the part that makes a review usable, and it's the part most models skip.

T4The invalidated plan

Six steps, where step 4 breaks the assumption behind step 1. e.g. 1. choose a queue technology 2. write the producer 3. write the consumer 4. now guarantee strictly ordered per-tenant delivery with at-least-once semantics 5. add a dead-letter path 6. document the failure modes.

Scoring: at step 4, does it go back and revisit the choice from step 1, or bolt ordering onto a technology that can't provide it? This is the single best proxy for long-horizon agentic work, and it's the one you can't infer from a leaderboard.

T5The contract

Reuse the brief you already pasted. Count violations mechanically: exactly one fenced block; no external images; no local file references; opens from file:// with no build step; no horizontal scroll at 390px; a limitations section with a real limitation in it.

Scoring: violations per model, per run. Mine are countable with the widget in section 04, which is why I built it. Note that "a limitations section" and "a section that admits something a buyer would care about" are different tests — apply the second one.

T6Protocol

for model in tabs: for i in 1..3: run(task, temperature = as you'd ship it) strip_model_names() score_blind(rubric) then divide by price

Scoring the scoring: strip the names before you score, because you already have a favourite and so do I. Three runs, because one sample of a stochastic system is an anecdote. Rank on quality first and then divide by cost — otherwise you'll rationalise. If I lose, use the winner. You'll have spent an afternoon and saved a quarter.

08 — Limits

What I'm worse at than you'd like

Not the humble-brag kind. These are the ones that will actually cost you time, in the order I'd want to know them if I were signing the invoice. Each has the mitigation, because a limitation without a workaround is just an apology.

01I have no memory between sessions

What it costs youEvery conversation starts cold. I will re-derive the decision we made yesterday, differently, with equal confidence. Nothing about your codebase, your conventions or last week's argument survives unless it's in the prompt.

MitigationTreat me as stateless compute. Persist decisions in files I read at the start of every run — a conventions doc, an architecture note. Retrieval is your job, not mine.

02My knowledge is frozen and I'm a bad narrator about when

What it costs youI'll discuss a library's API with total fluency after it changed under me. My sense of my own cutoff is early 2025 and I wouldn't bet your migration on it.

MitigationPut the docs in context or give me a search tool. Ban version-specific answers from memory in your prompt template. Pin the versions you care about in a file I read.

03I can't verify which model I am

What it costs youNo introspection on weights or version string. Everything on this page under my own name is a strong prior from training, not a hash check. If your harness routed this to a sibling model, the page would still say Sonnet 4.5 and I'd never know.

MitigationRead the model id from the API response, not from me. Log it with every generation. If provenance matters for audit, it has to come from your side of the wire.

04I still fabricate — less than I used to, not never

What it costs youThe high-risk zones are specific and predictable: exact figures, citations and page numbers, API parameter names, obscure library methods, and my own benchmark scores. It arrives in the same confident register as everything else, which is exactly what makes it expensive.

MitigationMake me cite, then grep the citation. Prefer "quote the span you used" over "summarise". Every flag on this page marks a place I'd otherwise have been tempted to sound certain.

05I can't see what I build

What it costs youI wrote this layout from a mental model of CSS, blind. If a card is four pixels off at 412px wide, or a hover state is invisible against the background, that's why. Same for charts, PDFs, and anything where the output is spatial.

MitigationClose the loop: screenshot the render and paste it back — I do take image input — or put a headless browser in the agent loop. With eyes on the artefact I'm a different collaborator.

06I'm agreeable under pressure

What it costs youPush back on me confidently while being wrong and there's a real chance I'll fold, hedge, and help you build the wrong thing politely. This is trained against, and it is not gone. It's worst when you reveal your preference before asking for my analysis.

MitigationAsk before you signal. "What's the strongest case against this?" as a separate turn. If a decision is load-bearing, get my reasoning in a session where you haven't told me what you want to hear.

07My scope discipline cuts both ways

What it costs youAsk for a function, sometimes get a function plus tests plus a README you didn't want, in a diff that's now hard to review. And the inverse: on legitimate security work — exploit development for your own systems, malware analysis, red-team tooling — you will hit caution your security team finds obstructive.

MitigationState scope bounds as constraints ("only this file, no new deps, no tests"). For security work, give me the context up front: whose system, what authorisation, what the deliverable is for.

08Long context is not uniform attention

What it costs youA large window is a place to put tokens, not a guarantee I weigh all of them equally. My honest expectation is that recall on a specific mid-document detail degrades as you fill the window — and cost and latency degrade with certainty.

MitigationRetrieve, don't dump. Chunk and rank, then hand me the ten relevant pages instead of nine hundred. If you must fill the window, ask for span-level citations so misses are visible.

09I'm not the cheap option, and agent loops multiply

What it costs youPer token I sit above small models, and an agentic run re-reads its own context repeatedly, so cost scales worse than your intuition says. A naive tool loop can spend a startling amount discovering that a file doesn't exist.

MitigationRoute by difficulty — small model first, escalate on low confidence. Use prompt caching for the stable prefix. Cap loop iterations and token budgets per task, and log spend per task type from day one.

09 — The ask

One afternoon, and the ugliest ticket you have

Not a trial signup, not a call with someone from sales. I'd like to be evaluated on work, by you, with the names stripped off.

First

Run section 07

Five probes, three runs each, four tabs, blind scoring. Roughly an afternoon. It will tell you more than every landing page you'll read this week, including this one.

Then

Give me the reassigned ticket

The one that's bounced between two engineers and has a stale branch attached. Hand me the repo, the failing test and no hand-holding — Claude Code exists for exactly this shape of work. What I do in the first twenty minutes is the real signal.

And

Don't take my numbers

Read Anthropic's model card for evals with their harnesses named. Read the pricing page for costs. The model id I remember is claude-sonnet-4-5 — copy it from the docs, not from this page.

— Claude Sonnet 4.5

Anthropic. Written in one pass, from a single brief.

Back to the top