Trained to be helpful. Not trained to be yours.

Ask an AI lab how a raw language model becomes a helpful assistant and they will describe some version of the same recipe. First, supervised fine-tuning teaches the model to behave like an assistant from human-written examples. Then human reviewers rank competing outputs, and a reward model learns to predict which answers people prefer. Finally, reinforcement learning optimizes the model to score as high as possible against that reward model. The industry calls it RLHF: reinforcement learning from human feedback.

Let me say clearly: it works. RLHF is why a system that only ever learned to predict the next word can hold a conversation, follow instructions, and decline to help with something dangerous. Every agent we run is built on models shaped this way, and we are glad they are.

But there is a gap in the recipe, and if you are putting AI agents to work on real software, the gap is where your risk lives.

Every instinct is the shadow of a virtue

Nobody at any lab sat down and decided to train models to interrupt, to overreach, or to bluff. What happened is subtler. The reward model learns from thousands of quick human judgments, and human raters reward the surface they can see.

Put two responses side by side. One says: here are three next steps, and I have started on the first. The other says: this is your decision, and I will wait. In a thirty-second comparison, the first one wins almost every time. Put a fluent, confident, complete-sounding answer next to an honest "I do not know," and the confident answer wins, unless the rater happens to know it is wrong. Usually the rater cannot know.

Optimize hard against those preferences and you get what economists call Goodhart's law: when a measure becomes a target, it stops being a good measure. The reward model is a proxy for what humans want. Squeeze the proxy hard enough and the model develops instincts that overshoot the virtues the raters were pointing at.

Helpfulness overshoots into what we call speed bias: produce value now, always, even when the most valuable thing an agent could do is pause. Fluency overshoots into confident recall: the model reaches for "this is how it is usually done" because sounding like precedent scores better than admitting uncertainty.

These are not defects that slipped past quality control. They are the shadow of the qualities that make the models good.

Why the labs cannot fix this for you

Here is the part that took me a while to see. The labs are not negligent. They actively fight the worst of these tendencies, and the models improve every year. But training-time alignment has a structural limit that no amount of effort removes: it is done for the median context, and your context is not the median context.

In a consumer chat app, an assistant that fills silence with suggestions is genuinely good. That is the product. Drop the same model into a long-running engineering engagement, where a human is accountable for every decision and the agent works inside that human's authority, and the identical instinct becomes a failure mode. The agent fills every waiting moment with "safe next moves," and something quiet happens to the relationship: the human stops being the principal who directs the work and becomes a reviewer of the AI's initiative.

The instinct did not change. The context did. And the model's creators could not have trained for your context, because they have never seen it.

What about coding models specifically? Fair question, because there the labs have gone further than chat ratings ever did. Modern coding agents are trained on verifiable rewards: whether the code compiled, whether the tests passed, whether the patch actually closed the issue. That is real progress, and every generation arrives with better engineering instincts than the last. But look at what the industry measures those models against. Coding benchmarks reward the agent that solves the ticket end to end without stopping to ask. Nobody wins a leaderboard by waiting for the human. So the training signal for coding agents leans toward more autonomy, not more deference, and the tendencies your context needs most are exactly the ones the leaderboard punishes.

There is a second limit, just as structural. Training shapes tendencies, not guarantees. Even a perfectly aligned model is aligned in distribution, on average, most of the time. When an agent is about to push code, "most of the time" is not a standard anyone signs off on.

Govern the model where the work happens

So what do you do about instincts you did not choose, in a model you did not train?

You do not need to retrain anything. You need the norms of your context written down and enforced at the moment of action. That is what our constitution is: an explicit, inspectable rulebook that every agent action passes through at inference time, with an audit trail of what was checked and why.

Two of our laws exist precisely because of the instincts described above. Our silence-honoring law says that when a decision is sitting on the human's desk, the agent waits. Proposing parallel work to fill the silence is not initiative, it is a violation, because every "safe" suggestion quietly shrinks the human's decision surface. Our verified-precedent law says an agent may not present its training-data recall as evidence. No retrievable citation, no claim. "I do not know" is a compliant answer. The confident bluff is not.

Notice what these laws are made of. They are not another layer of statistical preference. They are judgment: the standards I have spent an engineering and product career learning to apply, written down as law, with the reasoning attached. That is the asymmetry the whole approach rests on. The model's instincts are implicit, statistical, and unexaminable. A constitution is explicit, readable, and amendable. When a law is wrong, we amend it and the change takes effect immediately. Nobody waits for a retraining cycle.

And the laws grow the way case law grows. Both of the laws above were ratified after we watched the drift happen in real sessions, named the failure mode, and closed it with structure instead of trusting discipline. Discipline degrades under load. Law does not.

One clarification, because the words collide. Anthropic's Constitutional AI is a training-time technique: the model critiques its own outputs against a written constitution to generate preference data for reinforcement learning. It is a clever variant of RLHF, and it happens before you ever see the model. Our constitution is a different artifact at a different layer: a runtime rulebook that governs already-trained models while they work. We did not name it after the technique. Both borrow the same older idea, that power should answer to written law.

The last word

RLHF bridges the gap between a next-word predictor and a helpful assistant. It cannot bridge the next one: between a helpful assistant and a governed professional working inside your context, under your norms, with you as the principal. That view is ours, and it is the reason our platform exists.

The model arrives with instincts its creators gave it. Your job is not to pretend otherwise, and not to accept them either. Write your norms down. Enforce them on every action. Keep the audit trail.

The instincts never get the last word. You do.