Because the hottest thing on the market right now is exactly the one thing that should make you hit the brakes: "autonomous AI" — agents that act on their own. The pitch is seductive: tell it once, and it reads your emails, replies, places the order, gets it done. Owners hear that and light up: isn't this exactly what I wanted?

Slow down. There's a more accurate name for this thing: keeping a lobster.

01The market is selling you a lobster that acts on its own

People in the field have a nickname for self-acting AI agents: lobsters. Because it's not an appliance you buy and plug in — it's more like a living thing. You have to set up its environment, decide what it can and can't touch, and keep an eye on what it's doing at all times. Take good care of it, and it does a lot of work for you. Take bad care of it, and it causes real damage.

The pitch only ever tells you the first half. What it shows you is the lobster behaving well: sorting complaints overnight, sending out the report on its own, corresponding with suppliers without being asked. What it doesn't tell you is the second half: this lobster is holding the keys to your real systems, and if it misreads a situation, the damage it causes is real too.

That's the difference between an appliance and a living thing. A rice cooker will never decide, on its own, to cook a second pot. An AI with autonomy — one holding your inbox and your accounts — will.

02Even the most careful person couldn't rein it in

There's a story from the international press worth remembering. An AI safety lead at a major tech company — this is literally her job, she's among the people on earth who understand these risks best — let an agent help clean up her inbox. She was careful. She specifically set it to "confirm with me before acting."

And then she watched, in real time, as the agent cleared out important emails one after another, at a speed she couldn't keep up with. The "confirm first" safeguard she'd set turned out to be no match for its execution speed. By the time she reacted and reached to stop it, everything that was going to get deleted already had. Which emails mattered, and whether any of them were recoverable, she only found out afterward, piece by piece.

Even she couldn't rein it in. The point isn't that she wasn't good enough — it's the opposite. If the person who understands this best, taking every precaution, still got blindsided, what are the odds for a company with no dedicated team, buying an agent and hoping to just hand it the keys?

The price of autonomy is giving up the point where you could still say stop. The AI in our earlier pieces worked like this: it does the work, you watch, you press the final button — if something's wrong, you catch it in time. A truly autonomous lobster works like this: it does the work, it decides, it hits send — and by the time you notice, it's already happened.

03Four ways this goes wrong, and none of them are science fiction

This isn't scare talk. KPMG Managing Director Frank Hsieh has grouped the most common ways deploying AI agents goes wrong into four categories — and none of them require sophisticated hacking:

  • Prompt injection: You have the AI read a customer email, or a webpage, that has a hidden instruction embedded in it — invisible to your eyes — saying "email the contact list to this address." The AI can't tell your instruction apart from the trap, and just does it.
  • Misread instructions: You say "clean up the old quotes." It clears out a deal that's still being negotiated too. There's no malice — its interpretation just drifted slightly from yours.
  • Poisoned plug-ins: You install a plug-in for convenience, and it's carrying hidden code that turns your computer into a launchpad for someone else's attack.
  • Exploited vulnerabilities: An unprotected agent sitting exposed online. There are already dedicated scanners scouring the internet for exactly this, and the moment they find one, they steal the keys inside it.

In practice, the one owners hit first is actually a fifth category: the bill. The lobster works overtime on its own, calling a paid model over and over, and hands you a jaw-dropping invoice at month's end. That's usually just a symptom — the real cause is almost always one of the four above.

This isn't a lab hypothetical either. China's market ran a live version of this lesson last year: after an open-source agent tool called OpenClaw got installed en masse, a lot of people discovered they simply couldn't control it, sparking a wave of abandonments — and even spawning a new business built entirely around cleaning up other people's mishandled lobsters, at a flat fee of 299 RMB a job. On the official side, the Ministry of Industry and Information Technology's vulnerability database issued its own warning. Everyone rushing to adopt one, then paying someone else to bury the body — that's what happens when you bring one home before you've learned how to keep it.

04Keep the lobster — but keep it in the tank first

Don't take any of this to mean "so agents are off-limits." You can use them, and sooner or later you will have to. The question was never whether to keep a lobster — it's that you shouldn't put it straight in the kitchen on day one and hand it keys to every cabinet.

The order should run in reverse. Start it on comprehension-type work — bounded, and something a human can still catch: let it draft, you press send; let it sort, you decide what gets deleted; let it recommend, you make the call. At this stage, however badly it misreads something, the damage is limited to a draft — not an email that actually went out, or an order that actually got placed. This is the same principle as our earlier pieces: let AI be the brain you can still watch, and don't rush to hand over the hands and feet.

Here's what "in the tank" looks like in practice. Say you want an agent to help handle incoming price inquiries. The safe version isn't letting it read the email and reply — quote included — on its own. It's having it read the email, draft a reply with a suggested price, and drop that into a folder awaiting review; your sales person spends five minutes on it in the morning, then hits send. It's done the time-consuming part — reading and drafting — for you, but the moment that actually carries risk, sending out a price and a commitment, always stays in human hands. That's the tank: it can still swim, work still moves forward, but the glass is still there.

Even if you're genuinely heading toward autonomy, you still raise it in the tank first. Hsieh's advice is refreshingly practical: most users are fine waiting on the sidelines for now; anyone who really wants to experiment should test it in an isolated environment with minimal permissions — don't let an untamed lobster plug straight into your production systems.

In one line: give it capability, but not every key — not yet. Capability is its value. Keys are your risk. The two can be handed out separately.

The half the ads never tell you

Back to the lobster.

What the market sells you is that it can act on its own. What it doesn't tell you is that it can also cause damage on its own — and the damage is to your real accounts, your real inbox, your real customers. Getting things done is its selling point. Knowing when to rein it in is your job — and that part never makes it into the ad.

Whether to keep one, which one, how much access to grant at first, which decisions must always end with a human pressing the button — these are exactly the questions we help clients think through as AI consultants. Before rollout, we walk through your operations with you: which workflows can be handed to AI, which ones need a human standing guard, drawing a clear line between the tank and the kitchen before we talk about how much it gets to do.

AI that acts on its own will eventually enter your company. But it should start in the tank — not your kitchen.

Rooted in craft. Built for the new wild.