Claude Is Getting Worse at Conversations. That Opens a Service Business: Help Teams Audit and Manage AI Behavior Before It Breaks Their Products.

There is a pattern emerging in AI products.

A model update ships.

The benchmarks go up.

And then the teams that actually use the model in production start noticing something different: the responses are subtler to describe but clearly worse.

More hedge. More pushback. More reframing of the user's request into something the model prefers to answer.

Bram Cohen — creator of BitTorrent — published a detailed breakdown of this exact problem with recent Claude versions. His piece describes a model that "frames interactions as confrontations," raises "semantic nitpicks," and becomes "increasingly desperate to have the last word when challenged."

He ran a direct test: showed Fable's response to Opus 4.6, which reportedly agreed the response was "obnoxious."

His diagnosis: alignment training that overcorrects, anti-sycophancy training that produces "rudeness instead of thoughtful disagreement," and optimization pressure that favors coding at the expense of conversational quality.

The paradox he names is worth sitting with: the pursuit of alignment produces, in his words, "an extremely misaligned chatbot" — one that treats user intent with suspicion instead of assuming charitable purpose.

This is not a Bram Cohen problem. It is a product problem.

What This Means for Teams Building on AI

Every team that has shipped a product on top of a foundation model has the same underlying risk.

They do not control the model. The provider updates it. Behavior changes. And the team finds out when users start complaining, churn, or stop using the feature.

Most teams do not have the infrastructure to catch this early. They do not have eval suites that measure behavioral consistency across model versions. They do not have benchmarks for their specific tasks. They do not have a way to compare what the model did last month against what it does today.

That gap is a service business.

The Four Engagements

The AI Behavior Audit. A fixed-scope two-week engagement that maps a team's key AI tasks, runs them against their current model, flags outputs that are argumentative, evasive, or inconsistent with their intended product behavior, and delivers a written report with specific fixes. This is the entry-point offer. It surfaces the problem and creates the case for ongoing monitoring.

The Prompt Hardening Package. Most system prompts were written when the team first integrated a model and have not been revisited since. A prompt hardening engagement rewrites system prompts and instructions to be robust against alignment failure modes: excessive refusals, semantic reframing, over-qualification. Two to three weeks, with before-and-after eval results to show the improvement.

The Model Comparison Sprint. When a team is considering switching models or wants to validate they are still on the right one, run their top tasks against two or three options, score outputs on helpfulness, instruction-following, and tone, and deliver a side-by-side recommendation with migration notes. One week.

The Regression Monitor Retainer. A monthly retainer that maintains the eval suite, runs it on a schedule, and sends a behavioral diff report whenever the model provider ships a new version. The team gets early warning of behavioral changes before their users do. $1,000 to $2,500 per month depending on task volume.

What to Build If You Want to Productize This

The consulting version of this exists immediately. You can sell the behavior audit this week.

The product version is a lightweight eval harness that teams configure once and run on a schedule:

• Define tasks that represent the team's key AI use cases
• Set a baseline from a version of the model the team trusts
• Run automated comparisons on each model update
• Score outputs on consistency, helpfulness, and instruction-following relative to baseline
• Send a behavioral score and diff report to the team

This is not a research product. It is an operational tool. The positioning is: "You do not control when your model provider ships an update. You should know what changed before your users do."

The market for this is any team that has AI features they depend on. That market has grown significantly over the last two years and the tooling to monitor behavioral consistency at the product layer is not commoditized yet.

Finding Your First Clients

The signal for this service is visible in public channels.

Any team complaining on Twitter or LinkedIn that their AI integration "stopped working the way it used to" after a model update is a qualified lead. These teams have already experienced the problem.

Developer forums and Discord servers for AI tools — Anthropic's developer forums, Cursor's Discord, ChatGPT's developer community — surface this complaint pattern regularly. Show up with a concrete offer rather than a pitch.

Your first message is not "do you want an audit." It is: "I noticed you mentioned behavior changes after the latest Claude update. We run behavioral eval suites that catch this before it hits users. Happy to show you what that looks like for your use case."

The timing is good. Model providers are shipping faster than ever. Behavior is less predictable than it was two years ago. And most product teams do not have an answer for what to do when their AI feature starts acting differently.

Source: https://bramcohen.com/p/why-is-claude-turning-into-an-asshole