A global manufacturer spends eighteen months and $2.3 million building an AI system to automate order processing. Demo’s flawless. Leadership loves it. Rollout approved.
Just a few weeks into production, everything started to unravel. The AI system made a shocking mistake, automatically approving a massive $400,000 order for a customer who was already three months behind on their payments. At the same time, it flagged a completely normal shipment as “high risk” and brought fulfillment to a grinding halt for their biggest client. When the VP of Operations demanded to know who was responsible for these decisions, nobody could give him a clear answer. It was as if the AI had acted on its own, without any human oversight or accountability. The whole situation was a mess, and it was clear that something had gone terribly wrong with the system.
Here’s the thing, the AI wasn’t wrong. It was doing exactly what it was built to do, making decisions based on patterns in data. The problem? Nobody had thought through which decisions it should make. Or which ones it absolutely shouldn’t because it requires human oversight.
This isn’t just an isolated incident, it’s a widespread phenomenon that’s occurring all around us, at this very moment. And what’s really interesting is that it highlights a crucial aspect that often gets overlooked in most discussions about artificial intelligence.
The real question isn’t whether to deploy AI. It’s whether you’re ready to fundamentally rewire how your organization makes decisions.
The companies that figure this out will pull ahead. The ones that don’t will spend a lot of money learning expensive lessons.
The Decision Engine Problem
You’ve heard the pitch. It’s in every boardroom. “Let’s deploy AI to automate this process.”
That sounds good. It sounds like things are moving forward, evolving. But there’s a big problem with that way of thinking. It assumes the system is actually working properly, and all we need to do is make it go faster.
Most companies are trying to use AI, but it’s not working out. Researchers at MIT found that almost all AI projects in big businesses fail. It’s not that the AI itself is the problem, it’s just that it can’t learn from the way things are already being done. AI tools are great for individuals, but when you try to use them in a big company, they don’t work very well. They can’t adapt to the existing workflows, so they just don’t work. What’s happening is that companies are trying to add AI to their old processes, instead of changing the way they do things to make AI work. This is what MIT calls “the learning gap”.
From my experience working with clients, it seems that the issue of integrating new technologies is actually a matter of figuring out how to make decisions. Companies are not just having trouble incorporating AI into their daily operations, but they are also struggling with a more fundamental question. What’s the best way to make decisions in our organization? Where should humans be involved in the decision-making process?
At its core, every organization is, in part, a machine that makes decisions. Think about it, every day there are tons of choices being made, like whether to approve or reject orders, how to handle exceptions, and which customers to prioritize. These decisions happen thousands of times a day and ultimately decide whether the business succeeds or fails. It’s all about making the right calls, and it’s what sets the winners apart from the losers.
These decision patterns were created a long time ago, in a different time with different technology. The people who set them up are no longer with the company, and the knowledge about how they work is scattered and hard to find. It’s hidden in old ERP configurations, complicated approval processes, and informal conversations that happen in the hallways. Nobody has written it down or challenged it, so it just keeps going on as it always has.
And now we’re layering AI on top of all that and wondering why stuff breaks.
McKinsey’s 2025 State of AI report makes the point clearly. Workflow redesign drives more EBIT impact from AI than anything else they tested. More than model selection. More than data quality. Yet only 21% of organizations actually redesign their workflows when they deploy AI.
The other 79% are just automating their existing problems faster.
Why Full Autonomy Is a Bad Bet
The market is slowly figuring out the most successful AI deployments aren’t the most automated ones. They’re the most intentionally designed.
Think about what actually happens when you give an AI agent full autonomy in a messy business environment. It optimizes for the metrics you handed it. But real business decisions aren’t just about metrics. They involve context and relationships and exceptions and judgment calls that don’t exist anywhere in the training data.
You know that customer who’s really late with their payment, 90 days overdue? It’s possible they’re going through some changes and you’ve already sorted out a new payment plan with them. And what about that shipment that’s been flagged as high risk? Maybe your most important customer just asked to change the delivery address because they’re opening a new location. The thing is, artificial intelligence doesn’t have any way of knowing these details, it just can’t, at least not yet.
AI systems are great at recognizing patterns, identifying unusual activity, and processing large amounts of data quickly. They can do all this much faster than any human team. However, they have a significant limitation, they can’t know what they don’t know. In complicated business situations, the things you’re not aware of often have a bigger impact than the things you do know. This is because unknown factors can affect your decisions and outcomes in unexpected ways, making it crucial to consider both the known and unknown elements when making business decisions.
By 2027, Gartner expects that more than 40% of projects involving AI that can act on its own will be canceled. But this won’t be because the AI models didn’t work as planned. The real reason is that many organizations are skipping a crucial step, which is deciding when to let AI make decisions independently and when human judgment should still be involved. This is the hard part, and it’s essential to get it right. If organizations don’t take the time to figure this out, their AI projects are likely to fail. It’s not just about having the right technology, but also about understanding how to use it effectively and responsibly. AI agents work. But automation without orchestrating human oversight is just chaos moving faster.
What Orchestration Actually Means
So what does it mean to orchestrate decisions?
It’s not just about dividing tasks between AI and humans, it’s about creating a harmonious workflow. True orchestration is when all the different parts of an organization, including lines of business, subject matter experts, processes, and decision-making, come together in a way that makes sense. Each person knows what they’re responsible for and what’s expected of them. The right information gets to the right people at the right time, and it’s clear who’s accountable for what. This way, everyone is on the same page and working towards the same goal.
It starts with a question most organizations have never actually sat down and answered. For each decision in a critical workflow, who should make it? With what information? And what happens when someone gets it wrong?
At Duro, we call this mapping the “decision architecture.” It’s the invisible structure underneath how choices move through your organization. When we map it for clients, we almost always find the same thing. Either the current setup was designed for a completely different era, or it’s a patchwork of processes and stitched together tech that’s creating drag everywhere. The organizations winning with AI right now aren’t just buying new technology. They’re building new ways of working that actually leverage what humans and machines each do best. Here’s the framework we use.
You may know this as Human-in-the-Loop. We call it Human-in-Control.
Human-in-Control AI Orchestration
This isn’t about humans babysitting every AI action. That defeats the whole point. It’s deliberate design. Each player positioned where they actually create value. AI handles gathering, pattern recognition, execution. Humans own judgment, context, accountability.
We’re not limiting AI here. We’re building something smarter than either humans or machines could pull off alone.
Three Forces Making This Urgent
This isn’t just another hype cycle. Three real forces are converging that make intentional AI design necessary, not optional.
Regulators Want Answers
In the healthcare industry, we’ve seen a lot of changes in care management and compliance. The FDA’s January 2026 guidance on AI-enabled clinical decision support made one thing explicit. Clinicians must be able to independently review the basis for AI recommendations. If a software program doesn’t allow that, it’s still treated as a medical device and has to follow all the same rules. Software companies need to make sure their products are transparent and allow doctors to review what AI systems recommend. The pattern is clear. Regulators aren’t asking if you use AI. They want to know who’s accountable when it goes wrong. “The AI decided” doesn’t cut it anymore.
Boards Want ROI
One venture partner called 2026 “the show me the money year for AI.” Boards stopped counting pilots. They’re counting dollars now.
This pressure actually helps. It means AI projects need clear ownership, measurable outcomes, real accountability. All of which require knowing exactly where humans stay in control. The era of impressive demos that go nowhere is ending. What’s replacing it is a harder conversation about which investments actually move the business.
Governance Separates Winners from Losers
The gap between a good demo and a production system isn’t technical. It’s organizational. You can get an AI agent to 80% accuracy over a weekend. Getting to the 99% you need for real deployment takes governance. Identity management. Audit trails. Escalation paths. Human checkpoints.
Smart organizations treat governance as competitive advantage, not compliance overhead. If you can deploy AI responsibly, you can deploy it faster.
What This Looks Like When It Works
Theory only gets you so far. Here’s what happened with two recent clients.
Manufacturing: Purposeful Pup
We’ve all seen this problem before. A company like Purposeful Pup has its critical data spread out across different systems like Oracle, Microsoft Dynamics, and Jira. As a result, their team wastes a huge amount of time, around 35 to 45 hours every week, just trying to manually match up orders, fulfillment, and payments. It’s a real challenge and every decision they want to make requires searching through multiple systems, which is not only time consuming but also prone to errors. Things can easily fall through the cracks, and before you know it, cash gets stuck and everything comes to a standstill.
The obvious move was to automate everything. Let AI handle prioritization, escalation, follow ups.
We did something different. We mapped how decisions actually flowed through their operation and designed the orchestration around it. Some decisions needed human judgment, like credit holds, exception approvals, customer escalations. Others could safely go to AI, like aggregating data, flagging patterns, updating statuses.
What we built is an AI agent that pulls order data from all three systems, flags problems, and pushes recommendations into Slack where the team already works. Humans still approve the actions. The agent recommends. People decide.
We’ve seen some great results. Things are getting resolved 40 percent faster, and we’re making 20 percent reduction in fulfillment errors. Plus, we’re collecting cash 3 to 5 days sooner. But here’s the thing. It’s not because we’ve replaced humans with machines. Instead, we’ve taken away all the boring, repetitive tasks that were taking up their time, and freed them up to focus on the things that really matter, the things that require their judgment and expertise. By getting the busy work out of the way, we’re letting humans do what they do best.
Healthcare: Care Management
We’ve worked on healthcare projects where human-in-control isn’t optional. It’s how care management has to work.
In care management, AI agents coordinate enrollment, flag high risk members, prep service plans. But clinical decisions stay with the care team. What intervention to recommend. Which members to prioritize. How to handle the edge cases.
AI catches patterns humans would miss. Humans bring judgment AI can’t replicate. Together, the orchestration delivers what neither could do alone. Less manual reconciliation, earlier risk detection, better HEDIS compliance, real cost savings.
This isn’t AI replacing people. It’s AI and humans amplifying each other.
The Playbook
If you’re looking at AI agent solutions or trying to unstick a pilot that stalled, here’s what we’ve learned works.
1. Map the Decisions, Not Just the Process
Before building anything, figure out where decisions actually happen. Not the official flowchart. The real one. Who makes calls today? What information do they actually use? What happens when they’re wrong?
This stage is what we refer to as the “Ignite” phase. It’s a critical point where many projects unfortunately take shortcuts, but the truth is, you can’t really revamp something that you haven’t taken the time to fully understand and map out first.
2. Design the Human-in-Control Moments
You don’t need a person to check every single decision. A lot of them are straightforward. But when it comes to big, important things like money, talking to customers, following rules, or dealing with situations that require validation, it’s usually best to have a human take a look.
Be intentional about it. Build your AI agent architecture around these moments. Make them fast and contextual, and don’t force people to dig for the information they need. Sometimes that means layering AI into existing tools. Sometimes it’s a new app that solves a specific problem.
3. Bake Governance In
Identity, permissions, audit trails, escalation paths. These aren’t things you add later. They’re what separate a demo from something you can actually run in production.
Simple test. If you can’t answer “who’s accountable when this breaks?” you’re not ready to deploy.
4. Start Small, Then Scale
Pick one high value workflow. Ship it. Prove the ROI. Build the muscle. Then grow from there. This isn’t being timid. It’s the pattern behind every successful deployment we’ve seen. Companies that try to do everything at once end up with nothing.
What It Comes Down To
After years of helping organizations get past the demo phase into AI that actually works, here’s what we’ve learned.
The 95% failure rate isn’t a technology problem. It’s an orchestration problem.
The organizations winning with AI agents in 2026 did the hard work first. They orchestrated their lines of business, their experts, and their processes around a redesigned decision architecture. Governance built in from day one.
McKinsey’s data backs this up. High performers are nearly three times more likely to fundamentally redesign workflows than everyone else. That’s the gap. Not models. Not data quality. Orchestration.
Human-in-control isn’t a compromise. It’s not limiting what AI can do. It’s the design principle that separates demos that impress boards from systems that actually transform businesses.
The question isn’t whether AI will change how your organization works. It’s whether you’ll shape that change or just react to it.
At Duro, we create AI solutions that bring together human and machine intelligence right from the beginning. If you’re feeling stuck or unsure about where to start, try AI One Step. It’s a 30-minute session that helps you cut through the confusion and find the best place to begin.
Check out our approach on our website at duro.design/aiagentsolutions.