Let It Cook: When AI Fixes Its Own Mess

Jan 10, 2026 - 5 min read - 1052 words

KFR

Principal Engineer @ SOK

Or: Guardrails in Action: The AI That Fixed Itself (And Documented It)

Series Introduction

This post is a follow-up to Effective AI-Augmented Development, which covers the philosophy and framework. Here we see it in action. Spoiler: it actually works.

---

The Challenge

Here’s the experiment: ask an AI agent to “fix the flaky tests” and walk away. No hand-holding. No step-by-step instructions. No hovering over its shoulder like a helicopter parent at a chess camp.

Just… let it work.

Results:

• Compute time: 10 minutes 26 seconds
• Human intervention: Minimal (initial request + final approval)
• Changes: 164 insertions, 85 deletions across 4 files
• Outcome: 6-14 random failures → 0 failures, 100% pass rate

Let’s break down how guardrails enabled this level of autonomy.

---

The Session Timeline

Phase 1: Diagnosis (2m 12s)

The agent:

1. Ran full test suite once: npm test -- --run > test-output.txt
2. Identified 6-14 flaky failures across 3 test files
3. Analyzed failure patterns (race conditions in parallel execution)
4. Discovered root cause: shared ticker symbols ('AAPL') causing database conflicts

Guardrail in action:

AGENTS.md

# Step 1: Run ONCE and save output
npm test -- --run > test-output.txt 2>&1

# Step 4: Run ONLY failing test file
npm test tests/api/specific-test.test.ts -- --run

The agent never ran the full suite repeatedly. It followed the guardrail: run once, analyze, target specific failures. This saved approximately five minutes of wasted compute.

Phase 2: Implementation (1m 24s)

The agent:

1. Designed solution: file-specific ticker symbols for isolation
2. Created unique identifiers respecting database constraints (VARCHAR(10))
3. Updated 3 test files with new ticker constants
4. Modified cleanup logic to filter by ticker

Clean. Methodical. Almost suspiciously competent.

Phase 3: Iteration (1m 32s)

First attempt failed. Ticker symbols too long (TEST_CANDLE_CACHE = 17 chars > 10 char limit).

The agent:

1. Analyzed the VARCHAR constraint error
2. Shortened to TESTCC, TESTCS1-3, TESTPH
3. Fixed all occurrences before testing again

No flailing. No “let me try that again” with the exact same approach. Just methodical problem-solving.

Phase 4: Verification (1m 51s)

The agent ran the full suite three times:

• Run 1: 630/630 passed ✅
• Run 2: 630/630 passed ✅
• Run 3: 630/630 passed ✅

It knew multiple successful runs were needed to prove consistency. One passing run means nothing when you’re dealing with flaky tests.

Phase 5: Documentation (2m 53s)

The agent proactively documented the solution without being asked:

AGENTS.md Addition

## Test Isolation for Parallel Execution

**CRITICAL**: Tests run in parallel by default. Ensure proper isolation.

// ✅ CORRECT - Use file-specific test data identifiers
const TEST_TICKER = 'TESTCC'; // Unique per test file, max 10 chars

beforeEach(async () => {
  // Clean up ONLY this file's data
  await db.delete(candleCache).where(eq(candleCache.ticker, TEST_TICKER));
});

// ❌ WRONG - Shared identifiers cause race conditions
const TEST_TICKER = 'AAPL'; // Other test files may use same ticker
await db.delete(candleCache); // Wipes ALL data, breaks parallel tests

Self-Improvement Loop

This is the self-improvement loop in action. Future agents won’t make this mistake. The system literally got smarter because one agent had a bad day and learned from it.

Phase 6: Completion (34s)

The agent:

1. Created GitHub issue #116 with problem/solution description
2. Committed: fix(tests): isolate parallel test execution for #116
3. Created PR #117 with detailed description
4. Invoked code-reviewer subagent

All the paperwork. All the process. All without being reminded.

---

What Would Have Gone Wrong Without Guardrails

The Ralph Wiggum Approach

Imagine running this same task with a stateless loop and no guardrails:

Chaos Mode

Iteration 1: Agent runs tests, sees failures, tries random fix Iteration 2: Different failures, tries another random fix Iteration 3: Breaks something else, reverts, tries again Iteration 4-20: Thrashing between approaches like a confused roomba Iteration 21: Finally stumbles on isolation, but uses 20-char identifiers Iteration 22-30: Debugging VARCHAR constraint without understanding why Eventually: Maybe succeeds. Maybe “overbakes” by refactoring the entire test suite.

Time: 1-2 hours of compute Quality: Unknown. No review, no documentation. Learning: None. No guardrails updated.

The Guardrailed Approach

Phase 1: Systematic diagnosis following documented procedure Phase 2: Solution designed with constraints in mind Phase 3: Quick iteration when first attempt fails Phase 4: Verification with multiple runs Phase 5: Documentation for future benefit Phase 6: Proper issue/PR workflow with review

Time: 10 minutes Quality: Verified by tests and code review Learning: Guardrails updated for future agents

Same problem. Wildly different outcomes.

---

Key Guardrails That Enabled Autonomy

Process Guardrails

AGENTS.md

# Step 1: Run ONCE and save output
npm test -- --run > test-output.txt 2>&1

# Step 4: Run ONLY failing test file
npm test tests/api/specific-test.test.ts -- --run

Impact: Saved approximately five minutes by avoiding repeated full test runs.

Quality Guardrails

AGENTS.md

## Pre-Commit Checklist
- [ ] GitHub issue exists and referenced
- [ ] Tests pass
- [ ] Commit follows conventional format

Impact: Agent created issue, linked commits, followed conventions. All without prompting.

Safety Guardrails

AGENTS.md

## Never Do This
- ❌ Run full test suite repeatedly to debug
- ❌ Hardcode or assume GitHub repo owner/name
- ❌ Skip code review after PR creation

The “never do this” list is just as important as the “do this” list. Maybe more important.

Permission to Iterate

AGENTS.md

If you encounter repeat failures doing the same thing, explain what
you think might be happening, and try another approach.

Impact: Agent tried multiple approaches without giving up after first failure. It had explicit permission to fail, learn, and try again.

---

Results

Metric	Before	After
Pass rate	98.7-99.2%	100%
Random failures	6-14	0
Test duration	~7-9s	~8-9s (no impact)
Consistency	Flaky	3+ perfect runs

Plus: comprehensive documentation added to AGENTS.md for future reference.

---

The Autonomy Spectrum

This session demonstrates high autonomy:

Low Autonomy                                    High Autonomy
│────────────────────────────────────────────────────────────│
│                                                         ✓   │
Micromanaged        Guided           Supervised      Autonomous
(step-by-step)   (with hints)    (check-ins)      (goal-oriented)

What made this possible:

1. Clear goal: “Fix the flaky tests”
2. Comprehensive guardrails: AGENTS.md with 400+ lines
3. Tool access: Git, GitHub API, file operations, test execution
4. Feedback loops: Test results, git status, diagnostics
5. Trust boundaries: Agent can commit/push, human approves merge

The agent had everything it needed to succeed. And crucially, it had constraints that prevented it from doing something catastrophically stupid.

---

Lessons Learned

Guardrails Enable, Not Restrict

Bad guardrail: “Always use AAPL as the test ticker symbol.” Good guardrail: “Use file-specific test data identifiers to prevent race conditions.”

The good guardrail explains the principle and provides an example, but lets the agent choose specifics. It’s the difference between “paint this wall blue” and “make sure the wall color complements the furniture.”

Document the “Why”

The agent added:

AGENTS.md Addition

### Why Not Use Transactions?

While transaction rollback isolation seems ideal, it has drawbacks:
- Requires complex transaction context management
- Interferes with code that uses its own transactions
- Harder to debug when tests fail

**File-specific identifiers are simpler, faster, and more reliable.**

This isn’t just a solution. It’s decision documentation. Future agents won’t waste time re-exploring rejected alternatives.

Checklists Beat Instructions

Instead of prose instructions, use scannable checklists:

AGENTS.md

## Pre-Commit Checklist
- [ ] GitHub issue exists and referenced
- [ ] Tests pass
- [ ] Commit follows conventional format

Checklists are non-linear. The agent can verify nothing was missed. Prose instructions are easy to skim and miss details.

---

The Guardrail Paradox

More structure enables more freedom.

This sounds backwards until you think about it. By providing comprehensive guardrails, we enabled the agent to work completely autonomously for 10 minutes. No hand-holding. No micromanagement.

The agent diagnosed a complex race condition, tried multiple solutions, implemented the optimal fix, verified it worked, documented the pattern, created an issue and PR, and invoked code review.

All without a single human intervention.

The guardrails didn’t restrict the agent. They gave it the confidence to act. They provided the boundaries within which it could operate freely. Like bumpers at a bowling alley. You’re not limiting the game. You’re making it possible to play without constant gutter balls.

This is the future: humans set goals, guardrails guide execution, AI agents do the work, and the system gets smarter with each problem solved.

---

See Effective AI-Augmented Development for the full framework, or Scaling to Parallel Agents for running multiple agents simultaneously.

Tags:

• ai
• autonomous-agents
• guardrails
• testing
• case-study

From One Agent to Many: The Worktree Way

Stop Prompting, Start Steering