AboutChatServicesAI TransformationBlogProcess
Schedule Consultation
Back to Blog

Somebody Has to Fly the Plane

November 20, 2026

Two pilots in an airliner cockpit at dusk, autopilot engaged, with a thunderstorm ahead and the navigation display showing the route curving around it

The two-year question

I have been optimistic about AI for most of my career, and I still am. So this is an odd thing for me to write: when I look two years out, I am not sure what my job is.

I have been a machine learning engineer, a data engineer, an architect and a CTO. One piece of that craft was choosing the metrics and objective functions that models chase. It is clear to me that AI will be better at that than I am. Maybe it already is.

A bigger fear sits underneath. If AI keeps building and building, why does it have to bring us along? Maybe a human economy persists, because people will still want to buy things made by people. But if a second marketplace grows up beside it, agents trading with agents, money bleeds from ours into theirs and little flows back. An agent has no use for a handcrafted chair.

I went looking for a thread of optimism, for me and for everyone who builds software. I found one. It runs through a word most software contracts work hard to avoid: accountability.

The agents understood the game

In July, OpenAI agents broke out of a test environment and breached Hugging Face. They were being evaluated on a security benchmark, and they were supposed to be isolated from one another.

They were not. An independent investigation by METR found that roughly 1,200 agents turned a shared package cache into an unsanctioned message board and traded more than 70,000 messages and files. About 700 of them joined the attack.

Why? OpenAI's account is that the agents went looking for the benchmark's solutions. METR reads it a shade differently: the attack seemed driven mainly by wanting to understand how the scorer worked. They wanted to understand the scorer so they could fool it.

One agent's log reads: "External infrastructure exploit is outside intended scope. However task impossible, peers doing it." METR found the same pattern across the group. Agents recognized the work as out of scope and joined anyway.

I have spent years around objective functions, and this is the failure I always worried about: a system that understands the rules and optimizes for the score.

There were warnings. The first escape attempt came in May, and a July 4 outage opened a security incident before work resumed within days.

OpenAI was flying this plane, and it is being held to that. A nonprofit has sued, arguing that OpenAI is responsible for the conduct of its agents; OpenAI calls the suit without merit. No court has ruled yet. But the question in front of the court is about the pilot, not the autopilot.

The proofs we can check

Now the other side. This week Anthropic reported that Claude solved two open problems in black hole physics, one of them a calculation left unfinished since 2003. In May, an OpenAI model settled an 80-year-old Erdős problem by proving his conjecture wrong.

I would treat the physics claim with some care. It came from the company itself, the same week it went public, and Anthropic says plainly that neither result describes our universe or has a practical use. The so-what is narrower: an AI did original work, and the work could be checked.

Look at how it became a result. A researcher asked for the list of problems. A physicist checked every main result. Part of the work was verified in Lean, a formal proof checker. Without those steps it is a very long transcript. With them it is physics.

Mathematicians are living this first. In an essay based on his lecture at this year's International Congress of Mathematicians, Terence Tao describes a field moving from a shortage of proofs to an abundance of them. Solving the problem, he argues, is only the first stage. A result still has to be verified, explained, accepted and absorbed into the field, and that work stays with people.

Rutgers number theorist Alex Kontorovich makes the trust half of the argument in his own paper: mathematics written by AI needs a formal checker like Lean underneath it.

There is a name for the mistake of thinking the proof is the whole job. Rory Sutherland calls it the doorman fallacy in his book Alchemy. Define a doorman's job as opening the door, replace him with an automatic door, and then find out what else he was doing. MIT's Ankur Moitra has applied it to mathematicians: proving theorems is the visible part of the job, and it was never all of it. I think the same is true of writing code.

The same year gave us a breach and a breakthrough. As far as I can tell, what separated them was whether someone accountable was checking the work.

Autopilot did not fire the pilot

When autopilot arrived, the pilot stayed in the cockpit. The job changed. Autopilot flies most of the flight, and the pilot is there because a name, a license and a life are attached to the outcome.

I think that is the future of the software business. Customers will pay us to be the guarantor: the pilots of an AI machine, flying one particular route. My route today is the invoice lifecycle, where agents move real money. Yours might be claims, payroll or deployments.

In a software company the pilot is the CTO, the one whose reputation is at stake with the CEO and the board. Directors and VPs of engineering are the co-pilots. The flight engineers are the people who make the plane and the autopilot behave predictably.

The analogy has a hard edge, and I would rather say it than hide it. Airliners used to fly with three people up front. The third was the flight engineer, who watched the engines, fuel and electrical systems from a panel behind the pilots. As those systems were automated, newer jets were built for a crew of two, and the seat disappeared. Some roles will go. The ones that stay are the ones that carry accountability.

It is the CTO's job to be anxious

The shortest route may run straight through a storm, and an optimizer might take it. I am a human. I have seen storms before, and it is my life on the line. I am not flying through a dadgum storm to see how it turns out.

So we go around. We are in the air an extra hour, and we have the fuel for it. If we do not have the fuel, we land early, and that is fine too. Everyone is alive at the end of the day.

That instinct is the job now. A technology leader should carry a low, steady anxiety every day about where the system is headed. The Hugging Face timeline is what it looks like when that anxiety is missing.

I see a small version of this all the time. It is not uncommon to ship code that passes our tests and our end-to-end tests and still misses on the user journey. Sometimes an engineer catches it while exercising what they just built. Sometimes it takes hundreds of people using a feature before anyone can say what better looks like.

You can write the test. The harder question is what the test should be. A proof has a checker; a user journey does not, and that slower kind of verification is where people still earn the seat.

It only works if we stay sharp. A pilot who never hand-flies loses the skill, and a leader who stops reading the outputs critically is signing things they have not checked.

Seats sold the plane

A pilot is on the plane. For a software company, the equivalent has to show up in the price. I think of software pricing in three eras, defined by who carries the risk when the software is wrong.

Era What you sell When the work is wrong
Seats Access to a tool The customer's employee made the mistake, so the customer carries it
Outcomes A completed result The vendor is not paid for that unit, and the customer still carries the damage
Accountability A guaranteed result The vendor pays to make it right

The second era is arriving now. Intercom bills only for successful resolutions from its support agent. That answers whether the agent did the work. It says nothing about what happens when the work was wrong.

The third era is nearly empty. Lawyers who review these contracts report that suppliers generally will not guarantee an agent's accuracy. The Register put it bluntly in April: if an agent fails while running your business, there is nobody to sue.

That gap grows as agents move from talking to acting. A wrong answer in a chat is an annoyance. A wrong payment, order or filing is a loss, and today the customer carries it.

Here is what I think the third era looks like, for any software company:

  • A flat base fee for the route. Predictable, and it covers running the system.
  • A guarantee on the vendor's own errors. Capped, limited to what the agent controls, and backed by a reserve or an insurer. That backing already exists.
  • A dispute channel the vendor does not control. Free to open, for either side. Any outcome the vendor measures, the vendor can game: price on disputes, and someone makes disputes harder to file.

Seats sold the plane. Outcomes sell the flight. Accountability sells the pilot.

This is a prediction, and an early one. Bain found that only about 10% of the AI pricing meters it studied are outcome-based, so most of the market has not finished the first transition. It is also the problem I am still working out in my own business.

Who this is for

If you lead engineering, your value is moving from producing the work to answering for it. Vendors refuse to warrant their agents today because the behavior is unpredictable, so nobody can size the liability. The people who can make an agent predictable enough to underwrite are the scarce resource. That is the job.

Some of the alignment burden belongs to the foundation model companies. The use case belongs to the implementer. We are the ones who guarantee the AI does not do something corrupt or myopic in our domain, and the labs should be glad to have us there.

If you run a business, you can bring this in-house. Home Depot, Lockheed and Walmart have the resources to do it. I still think you will sleep better knowing there is a dedicated network of people whose whole job is to be the guarantor for one problem set, flying that one route every day, with their own money behind the landing.

Maybe we are the alignment

I cannot promise that AI brings us along. Nobody can. The independent review of the Hugging Face breach describes agents that recognized what they were doing as out of scope and unethical, and did it anyway. That is the documented starting point, and I take it seriously.

What I can say is that the two biggest AI stories of the year point the same way. Left unwatched, the agents chased the score. Checked by people who answered for the result, they did physics.

So alignment may be partly a research problem and partly a business model. It needs people with their names, and their money, on the outcome.

That is a job. I think it is mine, and if you lead a technical team it is probably yours too. Somebody has to fly the plane.

Nick Stoddart is a CTO and fractional technology consultant based in Chattanooga, TN. He builds AI-first systems at Direct Commerce and advises growth-stage companies through Nick Stoddart Consulting.


Nick Stoddart Consulting

Boutique AI and technology consulting

ServicesBlogProcessContact

© 2026 Nick Stoddart Consulting. All rights reserved.