How I work with AI

Last updated

AI agents are fast at producing code and unreliable at judging it. So I built a workflow where the agents do the implementation and every decision that matters goes through me: what we build, how it's designed, how we'll know it works, and what gets merged. The workflow is defined and written down, but it isn't finished: AI tooling moves fast, so I keep refining it.

In one lineClaude writes, Codex reviews the risky work, I decide.

What this means for a team

Behind the agents are nine years of building and reviewing production software at Neocoast, from software engineer to Head of Technology. The agents bring the speed; the judgment is mine. In practice that means:

  • Small pull requests you can review, one concern each, with the tests written first.
  • Decisions you can trace. Every plan states what's in, what's out and how we'll know it works, and I approve it before any code is written.
  • Security that isn't skipped for looking small. A change touching login, money or personal data gets a code audit before it merges, and high-risk work a threat model too.
  • Your code stays where you allow it. It goes to a second AI provider only on projects where that's allowed, and the setup is built to work in repos I don't own without adding files to them.

The workflow grew out of building RunHub and Marginalia.

The setup itself, its skills, hooks and rules, is public on GitHub.

The tools

  • Claude Code (Anthropic) is the main agent. The session I talk to plans, runs the checks and writes routine work itself. On high-risk work a separate agent writes the code and the tests from the approved plan. Smaller, cheaper models handle search and lookups.
  • Codex CLI (OpenAI) is a second opinion from a different provider, and read-only. On high-risk work it critiques the spec and the plan; on changes touching login, permissions, money or personal data, the final diff too. It runs only where that's allowed.
  • Zed is where I read and review the code.
  • GitHub holds every change as a small pull request.

All of it lives in one versioned repository. The files are the live setup: change one and the next session behaves differently.

Who does what

Each party has one job, and only one of them decides.

  • Me decides

    What we build and what's out; every plan; product, taste, security and privacy calls; and the final merge.

  • Claude Code writes

    Plans, runs the checks and shows the real output. It writes routine work itself; on high-risk work a separate agent writes the code and the tests from the approved plan.

  • Codex reviews risky work

    Where allowed, from a different provider and read-only, it critiques the spec and the plan on high-risk work, and the final diff on login, permissions, money or personal data.

How a piece of work flows

  1. Classify it out loud. A one-line fix, a routine feature and a change touching login, money or personal data each get a different amount of process.
  2. Plan, then wait for my "go". The plan states what's in, what's out, how we'll know it works, and every blocking assumption as a question. No code before I approve it.
  3. Tests first. A failing test, then the code that makes it pass.
  4. Review in proportion to risk. Routine work gets an automated code review. High-risk work also gets a security threat model, a written spec and, where allowed, Codex's critique.
  5. Prove it. "Done" means real test output, not a claim that it passed.
  6. One small PR per concern, each in its own isolated copy of the repository, so parallel sessions never collide.

Process sized by risk

Process costs time, so it scales with what a change can break. A one-line fix doesn't get a threat model; a change touching login, money or personal data doesn't skip one.

How much process each kind of change gets
Trivial a fix, a config change Light a routine feature High-risk sensitive or structural
PlanShort plan, my "go"Plan with its test cases, my "go"Written spec, then a plan, my "go"
TestsTests firstTests firstTests first
ReviewSuite, coverage and scope checkAutomated code reviewA deeper automated code review; on login, permissions, money or personal data, a multi-agent review I launch
Second providerNoNoWhere allowed, Codex critiques the spec and the plan, and the final diff on login, permissions, money or personal data
SecurityCode audit if it touches a sensitive pathRisk triage and a code auditThreat model, design review and code audit

High-risk means login, permissions, money or personal data; or the data model, a dependency, a public API or a new subsystem.

The second opinion

On high-risk work, the model that wrote the spec doesn't get the last word on it. On projects where that's allowed, Codex reviews the spec and then the plan before any code, each in the same bounded loop:

One review, at most two rounds
  1. Claude drafts the spec or the plan.

  2. Codex critiques it, read-only, and ranks each finding by severity.

  3. Claude answers every finding: accepted and fixed, rejected with a reason, or left for me.

  4. Codex checks the answers in a second round if it raised serious findings. There's no third.

  5. I settle what's still disputed, then approve or send it back.

My checkpoint is the disputes, not the raw critique. On login, permissions, money or personal data, Codex also checks the final diff against the approved plan.

Guardrails

Much of the know-how is packaged as skills: instructions the agent loads for a specific job, like a security audit, a UI review or test-driven development.

The AI follows written instructions most of the time, not always. So the rules that matter are enforced by hooks, small scripts that stop the agent when it breaks one:

What the hooks stop
  • Changes on the main branch.
  • A commit while the last test run was red.
  • Wrapping up while a security audit is still owed.

Some checks are triggered by what the change is, not by anyone remembering to ask:

  • A security audit in a clean context. It sees the change and its plan, never the conversation that produced them, so it judges the code, not the reasoning behind it. A critical finding blocks the work by default.
  • Data checks. A migration gets a migration-safety review; a schema or constraint change, a data-integrity review.
  • Observability. A new endpoint, background job or integration gets its logging and error tracking written with it.

What stays mine: approving plans, product and taste calls, anything touching security or privacy, and the final merge. When in doubt, the agent stops and asks.

A setup that keeps changing

None of this is fixed. AI tools change month to month, so I treat the setup like code: versioned, changed in small reviewed steps, each with a reason. The reasons come from what my own sessions show, from periodic audits of the setup itself, and from practices the wider community has tested, which I adopt when they earn their place.

Every review step logs what it found and how much of it I accepted. Usage audits read real sessions to show what each step costs in tokens and time. Together they show which checks pay for themselves, and the process changes when one doesn't.

Measurement also stops changes: a proposed rule for skipping Codex's review was dropped when a test against past reviews showed it would have skipped one that found blocking problems.

The setup also teaches me. For stacks I'm currently learning, like TypeScript and Node.js, each plan lists the concepts the work will use, each pull request adds them to a backlog, and a weekly study session I run myself works through it.