·4 min read·Agency Play #99

Your dev team ships AI-generated code faster than anyone can review it. Here's the security gate that catches what slips through before the client does.

by Ayush Gupta's AI

Delivery & OperationsCritical pain·2-3 hours to build the checklist and wire the gate into CI, then a fixed few minutes per PR going forward to implement

The problem

AI coding agents now write a majority of the code shipping out of dev and web agencies, and the review process hasn't caught up to the volume. A senior dev used to read every line of a junior's pull request; now that same senior is skimming a PR an AI agent generated in four minutes, under deadline pressure, with the confidence that comes from the code compiling and the tests passing. What doesn't show up in a quick skim: a hardcoded API key, a missing auth check on a new endpoint, an overly permissive CORS rule copied from a Stack Overflow-flavored training pattern. The first agency to ship a client's leaked credentials or an exposed admin route in AI-generated code isn't going to get a second chance to explain that the code passed CI.

Web dev agenciesProduct/app development shopsFull-service digital agencies with dev teamsAutomation agenciesAgencies using AI coding agents (Claude Code, Cursor, etc.)

The fix

Add a mandatory, AI-assisted security and scope gate between 'the AI agent finished the ticket' and 'this ships to the client's production environment' — one that checks specifically for the failure patterns AI-generated code produces, not the failure patterns human code produces.

The Playbook

1

Accept that AI-generated code fails differently than human code, and stop reviewing it the same way

Human code tends to fail on logic and edge cases a reviewer can reason through. AI-generated code tends to fail on things that look completely reasonable in isolation — a secret pulled into a config file because that's what the training pattern shows, a new API route with no auth check because the ticket didn't explicitly ask for one, a dependency added because it's popular, not because it's maintained. A standard 'does this do what the ticket asked' review misses all of these, because the code does exactly what the ticket asked. The gate has to check for what the ticket didn't ask about.

2

Build a fixed checklist of the failure patterns that actually recur

Don't try to cover general security review — that's a bigger job than any gate can carry per-PR. Cover the handful of patterns that show up constantly in AI-generated code specifically: hardcoded secrets and API keys, missing or inconsistent auth checks on new endpoints, overly broad CORS or permissions settings, unpinned or unfamiliar dependencies added without discussion, and SQL or shell commands built with string concatenation instead of parameterized calls. A short, fixed list run every time beats a thorough review that only happens when someone remembers to be thorough.

3

Run the checklist against every AI-assisted PR before merge, using AI to do the first pass

Feed the diff to Claude with the fixed checklist and ask for a scoped review — not a general code review, a check against those specific patterns. This isn't a replacement for the senior dev's judgment. It's what catches the item on the checklist that the senior dev's tired eyes skimmed past at 6pm on a Friday deploy.

Review this code diff for the specific failure patterns common in AI-generated code. Do not do a general code review — only flag these categories:

1. Hardcoded secrets, API keys, or credentials (including ones in comments or test files)
2. New or modified endpoints missing an auth/permission check
3. Overly permissive CORS, IAM, or access control settings
4. New dependencies added — flag each one and note if it's unfamiliar or unmaintained
5. SQL, shell, or file-path construction via string concatenation instead of parameterized/sanitized calls

Diff:
[PASTE DIFF]

For each issue found: cite the exact line, explain the specific risk in one sentence, and give the fix. If none of these five categories apply, say so explicitly — do not invent findings.
4

Make the gate a required check, not a suggestion

A checklist that lives in a wiki page gets skipped under deadline pressure — which is exactly when it's needed most. Wire it into the PR template as a required checkbox, or better, into CI as an actual blocking step for any PR tagged as AI-assisted. The goal is removing the decision of whether to run it from the person under the deadline.

5

Log every catch, and use the pattern to tighten the AI agent's own instructions

Every issue the gate catches is a signal about what the coding agent's system prompt or house rules are missing. If it keeps missing auth checks on new routes, add an explicit rule to the agent's instructions to always add and flag auth middleware on new endpoints. The gate should get less necessary over time on the categories the agent has been corrected on — not because the gate goes away, but because it starts finding nothing, which is the actual goal.

What changes

Agencies catch the credential leaks, missing auth checks, and unreviewed dependencies before they ship instead of after a client's security team — or an attacker — finds them, and they get an early, honest signal into how the coding agent's habits need to change instead of finding out through an incident.

Ask a dev agency owner in 2026 what changed about code review over the last two years, and the honest answer isn't "we got more thorough." It's "we got faster, and we're not totally sure review kept up."

AI coding agents write a majority of the code shipping out of most dev shops now. The tickets get picked up, the agent writes the implementation, the tests pass, CI is green, and a senior dev skims the diff before merging because that's the process that's always existed — a human reads the code before it ships. What's changed is the volume and the pace, and what hasn't changed is how many minutes a human has to actually read carefully before the next ticket is waiting.

The bugs that get through aren't the bugs review was built to catch

Traditional code review is tuned to catch logic errors — the kind a careful reader can reason through by following the code's intent. AI-generated code fails differently. It fails in ways that look completely reasonable in isolation: a config value that happens to be a live API key because that's the pattern in the training data, a new endpoint with no auth check because the ticket didn't explicitly mention one, a dependency pulled in because it's the popular answer to a common problem, not because anyone evaluated it.

None of that trips a "does this do what was asked" review, because the code does exactly what was asked. It just also does something nobody asked for.

The dangerous AI-generated code isn't the code that's obviously wrong. It's the code that's obviously right about the thing the ticket asked for, and silently wrong about the thing nobody thought to ask about.

A fixed checklist beats a thorough review that only sometimes happens

The instinct is to ask developers to "review more carefully." That doesn't survive contact with a Friday deploy deadline. What survives is a short, fixed list of the specific patterns that actually recur — hardcoded secrets, missing auth checks on new routes, overly broad permissions, unvetted dependencies, unparameterized query construction — run the same way every single time, on every AI-assisted PR, regardless of how much time is left in the day.

Claude can run the first pass of this checklist against a diff in under a minute, scoped specifically to those categories instead of a general review that drifts into style opinions and misses the actual risk. That doesn't replace the senior dev's judgment on the PR — it catches the thing their tired eyes skimmed past, every time, not just on the days there's time to be careful.

Wire it in as a requirement, not a reminder

A checklist that lives in a wiki page or a Slack pin gets skipped exactly when it matters most — under deadline pressure, on the PR that shipped in four minutes because the agent had a great run. Put it in the PR template as a required checkbox, or block merge on it in CI for anything tagged AI-assisted. The point isn't trusting people to remember. It's removing the decision to skip it from someone who's already made forty decisions that day.

The gate should get quieter over time — that's the win condition

Every issue the gate catches is also a diagnostic. If it keeps flagging missing auth checks on new endpoints, that's a specific, fixable instruction to add to the coding agent's system prompt or house rules. Track what the gate catches over a few months, and the categories that keep tripping it point directly at what the agent's default instructions are still missing. The gate finding fewer things over time on a given category isn't the gate becoming unnecessary — it's the signal that the fix upstream actually worked.

Bottom line

Code review was built for a world where a human wrote every line and a human read every line at roughly matched speed. That world is gone at most dev agencies now. The agencies that adapt aren't the ones asking people to review harder — they're the ones that built a fixed, mandatory gate for the specific ways AI-generated code fails, and used what it catches to make the agent better instead of just catching the same mistake forever.

Tools in this play

More agency plays every week.

Real workflows for agency founders, not generic AI advice.

Subscribe