“1,000 agents at your fingertips”
“Can read a 2,000 page legislation instantly”
“Cut your customer service backlog by 90%”
I have had discussions with enterprise leaders that lack a framework to evaluate AI and so aren’t sure if they are missing out or if the whole thing is nonsense.
Can’t blame them, remember Web3? Remember that was a thing? What about when Robotic Process Automation was going to change every thing in 2017, what happened there?
Goal
By the end of this post, you will understand the practical limits of AI so you can better evaluate opportunities, see through the hype, understand what it means to be AI-Native, and how to get there.
Plan
- We will highlight some subtle limits with thought experiments.
- This hopefully lands us at a credible speed limit of AI,
- Defines an AI-Native enterprise, and finally, describes how
- AI Product Engineers can get an enterprise to move at the speed limit of AI.
Brain Pain
What are some hard technical concepts that you have had to learn recently or tried to teach someone recently?
Really think about it or this whole post won’t work…
What made it difficult?
A lot of the times it is the amount of different concepts you have to hold in your head at once.
Sometimes it is a lack of context or a base of understanding which needs to be built.
It isn’t easy, and it doesn’t feel great, it is difficult.
Learning is not supposed to be fun. It doesn’t have to be actively not fun either, but the primary feeling should be that of effort.
Andrej Karpathy
Grokking Speed
Below is a Necker cube. It is an optical illusion.
You can make the cube have the purple face closest to you,
or the grey face closest to you.
Switch between the two. Grey → Purple, Purple → Grey.
Now switch between the grey face being closest to you, to the purple face being closest to you, but at the tempo of the 1977 hit song “Stayin’ Alive” by the Bee Gees.
Can you do it?
I can’t switch it that fast. I can “feel” my brain trying to arrange itself to get me to view I want, but it isn’t instant.
Why is it so hard to switch between these two understandings of the cube?
We have perfect context and information, there is no text or difficult concepts that need understanding, and yet even with this simple task there is some cognitive limit to how quickly we can process it.
Assignments
What is the last complex deliverable your boss has asked you to make and how did that process go?
Did you nail it the first time?
Or did it take a lot of iterations because of requirements that weren’t included with the first request.
Or your boss didn’t really even know what they were looking for until they saw your output.
Turns out Prompt Engineering has been around a long time in enterprise. AGI has been solved internally in every enterprise for as long as they have been around.
Humans are AGI that we prompt to produce output for us, and clearly some are better at it than others!
Delegating
How many direct reports can you manage for effective outputs?
5? 7?
Maybe you are Jensen Huang and you are able to handle 60 direct reports.
Why not 100?
What is holding you back, what is the issue with managing that many reports?
I personally cannot articulate that many goals, and review that much output in any meaningful way, I don’t have the bandwidth.
What Would You Do?
You have an attic full of books.
One of them has a subtle message that tells you where $1m is (you are almost positive it is in there).
You have a nephew Dan who is extremely fast and very helpful but sloppy at times with his work.
The $1m expires in 1 week…
What would you do?
Me? I would want to check them myself. I want to be part of the process to make sure the books don’t have the message if it turns out it is not there. I wouldn’t trust Dan’s output without being along for the ride.
This is a different review than just checking the answer. Some class of problems you can just review the output to see if its right.
A math problem may take a lot of steps to solve but once you get to the answer like x = 2, you can just plug it back into the equation to see if its right.
Some problems you can just review the output, some problems you need to review the process.
Who Is Held Accountable?
Frank creates an AI agent, Ralph, that reads his payroll information and generates tax forms and submits them to the IRS.
The IRS sends a letter that Frank has not paid his taxes.
Who is going to jail, human Frank or AI agent Ralph?
Frank! Frank is going to jail!
last thing,
What are the Subjective Steps?
Some steps are deterministic (will always happen, like an if/then), and some steps are subjective (requires judgement).
Follow along with a generic financial enterprise’s loan application processing flow and determine which steps are deterministic and which are subjective.
- Risk thresholds and fraud rules are defined by a human based on regulatory requirements and business appetite.
Answer
Subjective – Regulatory requirements are written in human language and are not easily turned into machine rules. Business appetite is also subjective about what “aggressive” actually means in practice.
- When a transaction occurs, it is run through the fraud detection system.
Answer
Deterministic – If transaction occurs, then run through fraud.
- AI model analyzes transaction against customer’s historical behavior patterns.
Answer
Deterministic – Kind of a trick question. The subjective part happened earlier: when humans decided what features to include, what training data to use, how to label fraud vs. not fraud, and what thresholds to set. But the model’s execution itself is deterministic.
- Rules engine flags transactions exceeding hard limits (e.g., >$10k, foreign country).
Answer
Deterministic – These are hard coded rules.
- AI assigns a risk score based on anomaly detection.
Answer
Deterministic – same input always produces same score (unless using GenAI and the output is non-deterministic).
- Transactions above the risk threshold are routed to a fraud analyst for review.
Answer
Deterministic – threshold comparison is a fixed rule
- Analyst determines whether to approve, block, or escalate the transaction.
Answer
Subjective – human uses judgment to interpret context and edge cases.
Take-Aways (finally are getting to the point)
- Humans have a max cognition speed even with perfect information
- Defining what is “correct” is non-trivial and is dependent on the goal
- Some reviews require process review (not just output review)
- Computers cannot be held accountable, only humans
- Business activities have subjective and deterministic steps
So the Speed Limit of AI Is…
Human stakeholders understanding defined goals
+
Approving subjective decisions with appropriate context
=
AI-Native Enterprise
If you remove all deterministic steps, have AI pre-generate subjective decisions for easy human review with appropriate context, you are moving at the speed limit of AI.
The only lag in this process is your ability to review/approve subjective decisions, and define concrete goals (which itself is a subjective decision).
How AI Product Engineers Get Enterprises Moving at the Speed Limit of AI
So we understand the speed limit of AI, but how do you get there?
You can boil down the work into 4 steps:
- Understand strategy and define the goals of a system/process/solution/company
- Understand the full system (teams, dependencies, constraints)
- Determine which steps require human review
- Which require process review
- Which require output review
- Design solutions that optimize for human review steps
- User interfaces that provides relevant context
- AI output that can easily be updated
- Create data flywheel that turns subjective decisions deterministic
The output is a system that collects subjective human reasoning and applies it to future decisions- requiring fewer and fewer human interventions- turning service into software.
Which step is the most painful one?
Without a doubt it is Step 2, understanding the full system.
Garry Tan describes this work as “ethnographers spelunking”.
To build these systems someone has to understand everything.
This is where the dream of the “AI-Native Enterprise” usually hits a brick wall. It starts with an executive demanding we “AI-enable the XYZ process.” Then, the responsibility begins to slide sideways because no one wants to own the outcome.
- The Product Manager is now on the hook for a goal they don’t understand. They write some high-level feature like “Automate XYZ process” and then tells a Business Analyst to “go gather requirements and write up the user stories.”
- The Business Analyst sits down with a Subject Matter Expert (SME), but the SME doesn’t know how to speak “Developer.” They describe their day-to-day, but they don’t know what AI can actually do, so they leave out the most important details.
- The Business Analyst just writes down exactly what the SME said—meaning they end up documenting a broken, 20-year-old manual process that shouldn’t even exist anymore.
- The Engineer looks at these “requirements” and sees total ambiguity. They don’t want to build the wrong thing, so they throw the ticket back and tell the PM: “We need clearer specs before we can ship a single line of code.”
The result? You spend six months and $2M to build a high-tech version of a process that was already garbage. Everyone stayed in their silo, no one looked at the whole system, and nothing actually moved forward.
Enter the AI Product Engineer.
As Ronnie Coleman said: “Everybody wants to be a bodybuilder, but don’t nobody want to lift no heavy-ass weights.”
Mapping the “messy real world” is the heavy-ass weight. Most people want to “do AI” from the safety of their silo.
An AI Product Engineer is someone that is competent in 4 domains:
- Full stack engineering (wireframes to deployment)
- Applied AI (knows key ML/GenAI techniques and gotchas)
- Consulting (can deep-dive a domain and come up with coherent business logic)
- Program management (handle dozens of stake holders in engineering, product, go-to-market, SMEs, etc)
Basically a technical AI founder that can build AI-native products and processes for an enterprise from 0 to 1 and beyond.
Consulting is critical. Extracting tribal knowledge into something consumable is the whole trick, and it absolutely sucks to do.
So, What Should You Do?
“AI-Native Enterprise” isn’t a software you buy. It is a process of untangling your own business logic.
Your rules are hidden inside the heads of 50 different employees, spread across legacy codebases, and buried in unwritten “tribal knowledge.”
Leaders: Find Ronnie
Identify the people in your organization who are willing to do the “ethnography.” Find the engineers who leave their IDEs to sit with an accountant for three hours to understand why a specific tax form is rejected.
Give them the air cover to fail, iterate. Don’t measure them by lines of code; measure them by how deterministic decisions they have removed and many subjective decisions they have successfully turned into deterministic software.
Ronnie: Lift Weights
Accept that “lifting the heavy-ass weights” means dealing with ambiguity. It means accepting that Step 2 (Understanding the System) is the job.
- Don’t ask for a spec; ask to shadow the user.
- Don’t build a chatbot; build a context engine.
- Don’t try to automate the human out of the loop; build a system that gives the human superpowers when they are in the loop.
Leave a Reply