Don't automate the part that needs you

The pitch sounds right. Wire your AI into everything the company produces, the meetings, the tickets, the messages, the code, and let it keep its own picture of how the company works current. A system that reads what’s happening and updates itself. A loop that closes on its own, with nobody maintaining it.

I’d take half of that and leave the other half.

The reading half is the good half. An AI that can see what’s actually going on beats one working from a document somebody wrote in January and never opened again. That part is the whole promise, and it’s real. The writing half is where it turns. The moment the system updates its own picture of the company with nobody approving the change, it starts to drift. Quietly, the way these things always do.

I’ve written about this at the file level. Context isn’t one thing. Some of it barely moves: who you are, how the company makes decisions, what it won’t compromise on. Some of it changes by the week. The slow-moving part is the part you guard, because when it drifts you don’t catch it in the moment. You catch it a month later, when everything the AI produces is a little off and you can’t say why. Identity drifts in silence.

Now run that at company scale. A loop rewriting the company’s own sense of itself, unsupervised, is the same failure with more surface area. It reads one tense thread and decides the strategy changed. It sees a single refund handled a new way and writes that down as the policy. None of it wrong enough to flag, all of it compounding. Six months in, the context the AI runs on describes a company that doesn’t quite exist.

The fix is one gate in the right place. Let the system read everything and draft everything, the meeting summary, the updated decision, the new version of how something gets done. Then a person looks and presses yes. Not on every change. On the ones that touch what the company actually is, the slow cold things that drift without a sound. The fast stuff, what shipped today, what’s blocked this week, let it flow. Match the gate to what’s at stake.

That’s the closed loop worth building. The system does the reading and the drafting, which is most of the work and the part you’d never want to do by hand. You keep the one call that can’t be handed off: whether the new version is still true. The loop closes through a person, on purpose.

Automate the reading. Keep the judgment. A loop like this is only worth running if someone still owns what goes into it.