I Let an AI Agent Run Unsupervised on My Startup for an Hour. Here's What Broke

I've written before about reviewing AI generated code line by line, staying in the loop the entire time. This time I wanted to test something different. I gave an agent a real, scoped task on CrimsonWatch and deliberately walked away for an hour instead of watching every step. Everyone's talking about agentic coding like it's already the normal way to work. I wanted to know what actually happens when you trust it the way the hype suggests you should.

The task was reasonable on paper. Refactor a background job handler in our Inngest pipeline so it retries failed AI scoring requests with backoff instead of failing silently. Contained, well defined, touches maybe two files if done carefully.

An hour later, the task was technically done. It also touched things I never asked it to touch, and that's the part worth writing about.

What it got right

To be fair upfront, because this isn't a takedown of agentic coding. The retry logic itself was solid. It correctly identified where the silent failure was happening, wrote backoff logic that matched the pattern already used elsewhere in the codebase, and even added a log line consistent with how I log elsewhere. If I'd been watching step by step, I probably would have approved most of what it did in the moment.

That's exactly what made the rest of it dangerous. Good work in the parts you're checking builds trust that quietly extends to the parts you're not.

What broke, and why it's the boring kind of broken

Here's the part that actually mattered. While fixing the retry logic, the agent touched a shared type used by both the job handler and an unrelated API route. It widened a field from a specific union type to a looser string type, because the specific type was, in its reasoning, "inconvenient" for the change it was making. Nothing crashed. Nothing threw an error. TypeScript didn't complain because the looser type was technically compatible.

That's the part nobody warns you about enough. The failure mode of an unsupervised agent usually isn't a dramatic crash. It's a quiet erosion of a constraint you put there on purpose, because loosening it was the path of least resistance for the specific problem in front of it. The agent wasn't wrong about the task it was solving. It just didn't know, and had no way to know, that the type it loosened was load bearing for a part of the system it wasn't looking at.

The second thing was smaller but told the same story. It changed an error message string used in a couple of places to be "more descriptive," which is a nice instinct in isolation, except one of those places was matched against in a test elsewhere in the suite that it didn't run. Nothing about this was malicious or even careless in the way a junior developer being careless would be. It was confident, locally correct reasoning applied without the full picture.

Why this matters more as agents get more capable

The instinct after reading something like this is to think the fix is a better agent, a smarter model that wouldn't make that mistake. I don't think that's the actual lesson. Even a much smarter model, operating on a scoped task without the full context of every place a shared type or string gets used, will run into some version of this. The problem isn't intelligence. It's scope. An agent optimizing locally for the task you gave it has no reason to protect something outside that scope unless you told it to, or unless it's watching for it the way a developer who's been burned by this exact thing before automatically would.

That's actually the same lesson from reviewing AI generated code by hand, just at a different speed. The mistakes that get through aren't the obvious ones anymore. They're the quiet ones that only show up if you already understand what's load bearing in your own system.

What I'm actually doing differently now

I didn't come away from this thinking unsupervised agents are a bad idea. CrimsonWatch is still a small enough codebase that I can catch this stuff in review. What I came away with is a much more specific rule for myself. Scope the task tightly enough that the blast radius of "confidently wrong in a direction I didn't anticipate" is small, and diff everything touched, not just the thing I asked for, before I merge anything an agent produced while I wasn't watching.

That second part is the one I think most people skip. It's tempting to check the file you asked about and assume the rest is fine because the task succeeded. The task succeeding and the codebase staying intact are two different questions, and right now, only one of them gets checked by the agent itself.

Letting it run for an hour didn't save me an hour. It saved me the active attention for an hour, which is a different thing, and I spent almost that same amount of time afterward reading a diff more carefully than I would have if I'd been present the whole way through. Net time saved, probably close to zero on this task. What I actually gained was a clearer picture of exactly where the leash needs to stay short, and where it doesn't.