Picture the typical mid-market executive right now. There's a ChatGPT tab open in at least three browser windows across the office. Marketing is using one AI writing tool, sales is using another, and someone in finance just signed up for a third that IT doesn't know about yet. The CEO gets asked about AI at every board meeting and answers confidently: "Yes, we're using it." And technically, that's true.
But using AI and scaling AI are two entirely different things. A new report from Kaufman Rossin makes the gap impossible to ignore. In The State of Artificial Intelligence in the Mid-Market, the top 50 CPA and advisory firm surveyed mid-market companies and found that 94% are already using generative AI. Only 2% have operationalized it at scale.
94% adoption. 2% operationalization. That's not a technology gap. It's an organizational one.
The Illusion of Progress
The Kaufman Rossin data shows that most mid-market companies are confusing activity with progress. The report found that 83% of companies have moved past basic experimentation — they're running structured pilots or embedding AI into specific processes. That sounds encouraging until you see that only 2% have made it to the finish line.
The gap between "running pilots" and "scaled operations" isn't a small jump. It's where most implementation efforts go to die.
What this looks like on the ground is recognizable to anyone who's spent time inside a mid-size organization. Marketing picked up a generative AI tool for content drafts. The sales team is using a separate AI tool for prospecting. Someone in operations found an AI scheduling assistant they love. Each of these decisions was made independently, often by individual employees, with no coordination around data access, vendor agreements, or compliance.
According to the report, this decentralized approach is "overwhelming executives and complicating enterprise-wide strategy." That's a diplomatic way of saying: the tools are multiplying faster than the organization can govern them, and the executive team is stuck playing catch-up.
The problem with this kind of fragmented adoption isn't just inefficiency. It's the false confidence it creates. When every department claims to be "using AI," it's easy for leadership to assume real progress is underway. In reality, they've got dozens of disconnected experiments generating noise and mounting technical debt, with no shared infrastructure to build on.
Why This Is Worse Than Not Adopting at All
Every AI tool a department adopts independently eventually needs to be integrated, replaced, or decommissioned. When that reckoning comes — and it always does — the cost in time, money, and disruption is far higher than if you'd made deliberate tooling decisions from the start. That's the technical debt side of the equation.
The governance side is equally serious. When employees pick their own AI tools, corporate data flows through systems IT has never reviewed. RSM's 2026 Cybersecurity Special Report found that only 35% of middle market executives use formal AI governance frameworks, despite the fact that one in four of those same companies experienced a ransomware attack in the previous year. The disconnect is striking: 96% of executives said they were confident in their cybersecurity posture, while 18% had suffered a data breach. "Shadow AI," where employees use unauthorized tools outside formal security frameworks, is a direct consequence of uncoordinated adoption.
But false confidence might be the most damaging consequence of all. When leadership believes AI adoption is happening because they see activity across departments, they underinvest in the underlying infrastructure — data governance, integration architecture, talent development — that would actually enable scale. The tools multiply. The real work doesn't happen. And the organization keeps reporting progress to the board while the gap quietly widens.
The Barriers the Report Identifies
The Kaufman Rossin report points to three primary obstacles. None of them are surprising. Each one is harder to solve than it looks.
1. The AI Skills Gap
Finding qualified AI talent is still one of the biggest bottlenecks for mid-market companies. There's a shortage of technical talent who can build and deploy AI systems, and separately, a shortage of operational literacy at the management level — executives who understand enough about AI to make good strategic decisions.
RSM's January 2026 survey found that mid-market firms are investing in AI skills training even as hiring challenges persist. The lesson isn't just to hire AI specialists, though that helps. It's to build AI literacy across the leadership team so that strategic decisions aren't delegated entirely to vendors with their own interests.
2. Cybersecurity Concerns
Risk management considerations are slowing AI deployment, and for good reason. When AI tools process sensitive business data — customer records, financial information, proprietary operational data — the security implications are real.
The RSM cybersecurity numbers tell a clear story: only 35% of middle market companies use formal AI governance frameworks, but adoption is accelerating. "Organizations are accelerating AI adoption, but many don't yet have a clear destination or a governance model to guide them," said Daniel Gabriel, principal with RSM US LLP. Without defined frameworks, companies are essentially betting that their informal controls are sufficient. It's a bet that gets harder to justify by the quarter.
3. Legacy System Integration
This is the barrier that trips up even well-intentioned AI programs. Mid-market companies often run on ERP systems and core infrastructure that was never designed to communicate with modern AI tools. Connecting new AI capabilities to old operational systems requires integration work that's technically complex, expensive, and time-consuming.
Shearer's Foods, a snack manufacturer operating across 16 plants in North America, ran the same ERP system for more than 20 years. When they finally moved to modernize, they chose RSM as their implementation partner and Boomi as their integration layer. The ERP itself wasn't the problem; the data needed to flow between systems before anything more sophisticated could happen.
Justin Tapp, integration architect at Shearer's Foods, described the shift from handwritten code to a managed integration platform as "a fundamental change in how the IT team approaches scale and flexibility." The path forward wasn't buying more tools. It was building the integration layer that lets data move reliably between systems.
That's the part most AI implementation plans skip.
What "Getting the House in Order" Actually Means
Suzanne McIvor, director of strategic business alliances at RSM Canada, put it plainly in a conversation at Boomi World 2026: "What we're finding really is that when we get under the hood, while there are business use cases, they don't always come with business impact. It's really about the data activation, the data governance, getting the house in order."
That phrase captures the part of AI implementation that never makes the press release. Before an organization can extract enterprise-wide value from AI, it needs to answer some basic questions: Where does our data live? Is it clean and consistent? Who has access to what? How do our systems share information?
Most mid-market companies can't answer these questions cleanly. Not because they're poorly run, but because those questions haven't mattered as much until now. AI changes the calculus.
The Kaufman Rossin report maps four stages of AI maturity, which are worth knowing because most executives overestimate where their organizations sit:
- Dabblers: using tools ad hoc, no coordinated strategy
- Testers: running structured pilots to evaluate specific applications
- Builders: scaling what works and building the infrastructure underneath it
- Operators: fully deployed, with ROI they can actually measure and defend
The vast majority of mid-market companies are currently somewhere between Dabbler and Tester. The jump to Builder requires intentional investment in infrastructure. Not more pilots.
The ROI Problem No One Wants to Admit
The Kaufman Rossin report found that quantifying the financial return on AI investments "continues to challenge nearly all organizations." Among companies using AI, the most commonly cited benefit is time savings — which is real, but notoriously difficult to translate into a number that satisfies a board.
Companies are increasing AI spending — most plan to do so, the report found — but they can't clearly articulate the return. That's defensible in the short term when AI feels like a competitive necessity. It becomes untenable when the board starts asking hard questions and the honest answer is, "We're confident it's working."
The path to defensible ROI runs through the same groundwork as the path to scale: clear use cases tied to specific business outcomes, data infrastructure that makes measurement possible, and governance structures that create accountability. Without those, AI spend is a line item with no owner and no success criteria.
What the 2% Did Differently
The Kaufman Rossin framework outlines four pillars for a durable AI program: use cases and pilots, data and platform strategy, governance, and people and culture. In practice, sequencing matters as much as the pillars themselves.
Start with data readiness, not deployment. Before adding more AI tools, audit what you have. Where does your critical business data live? Can it be accessed programmatically? Is it consistent across systems? Nobody puts data audits on a roadmap slide, but everything else sits on top of this work. Companies like Shearer's Foods spent significant time and investment on integration infrastructure before unlocking anything that looks like AI at scale.
Build governance before you need it. The instinct is to address governance after problems emerge. That's backwards. A lightweight governance framework — a defined inventory of approved AI tools, clear data handling policies, and assigned ownership — is far easier to build proactively than reactively after a breach or compliance violation. Only 35% of mid-market companies have formal frameworks in place. The ones building them now will be ahead of those scrambling to catch up later.
Tie pilots to P&L outcomes, not activity metrics. Time saved is not a business case. When evaluating AI pilots, define the financial outcome before the build starts. What does success look like in dollars, margin points, or customer retention? If the pilot can't articulate a clear financial thesis, it's research, not an investment. Nearly half of RSM's mid-market clients are private equity-backed, which means ROI accountability isn't optional. Even if your organization isn't PE-backed, applying that same discipline pays off when the board demands justification.
Make organizational alignment a hard requirement. The AI programs that reach scale share one trait: cross-functional ownership. That means executive-level alignment on which use cases get investment, who owns outcomes, and how success gets measured — scoped before the pilots launch, not after they stall.
The 92% still between experimenting and building aren't failing from lack of ambition or budget. They're failing because the conventional wisdom around AI adoption — move fast, pilot everything, iterate — is reasonable advice that stops working exactly when you need it most.
The moment scale becomes the goal, the playbook changes. Data readiness first. Governance as a prerequisite. ROI defined before you spend, not rationalized afterward.
The gap between 94% and 2% isn't a technology problem. It's a strategy problem. The companies that close it won't be the ones with the biggest AI budgets. They'll be the ones who stopped piloting long enough to actually build something.