AI team structure is one of those decisions that looks boring and matters a lot. Centralized or embedded or hybrid — each pattern has a right size, a right company shape, and a failure mode. Teams who pick the wrong pattern for their stage spend a year fixing it. This post is the structure decision framework, with the patterns that actually work at different scales.
Under 50 engineers: embedded
One to three AI engineers, sitting inside product teams, shipping AI features that matter for the product. No separate AI team. No platform team. The AI engineers are full-stack AI — they do prompts, evals, infrastructure, integrations, on-call. They're the bridge between the AI ecosystem and the product.
Works well: founder-driven companies, single product focus, AI as a central differentiator. Fails: when AI features proliferate across multiple product areas without shared infrastructure, technical debt accumulates quickly.
Red flag to graduate: you're copying the same eval setup into three different teams, or you're having the same retrieval tuning conversation every quarter. Time to add a platform team.
50-500 engineers: hybrid
A small platform team (3-8 engineers) owns shared infrastructure: eval tooling, observability, model routing, the vendor integration layer, security/compliance wrappers. Embedded AI engineers in product teams build features using this platform. Products move fast; platform prevents chaos.
This is the structure most of our mid-size clients end up at. Platform team reports to CTO or VP Eng, not to a product VP. Embedded pods report into product organizations. Platform is service-oriented: it publishes libraries and APIs, runs office hours, and doesn't gatekeep product team decisions.
Common failure: platform team becomes a bottleneck because they start reviewing every AI feature. Avoid: platform should provide guardrails (libraries, defaults, policies) not approvals. If every PR touches platform review, the structure is broken.
500+ engineers: federated
AI Center of Excellence sets standards, drives research, and handles cross-cutting concerns (governance, compliance, vendor relationships). Platform teams (plural — ML platform, inference platform, eval platform) own specific infrastructure surfaces. Domain AI teams (inside business units) ship products. A central research function (small) pursues longer-term bets.
This structure serves acquisition-heavy companies, multi-business-unit firms, and industries with domain-specific AI (healthcare, financial services) where domain teams need deep specialization.
Common failure: CoE becomes a governance layer that slows everything down while delivering no platform value. Governance without delivery is bureaucracy. CoE needs to ship either platforms or standards that measurably reduce cost/risk for domain teams.
Common mistakes across all structures
Creating 'an AI team' without defining what it does. The vague mandate produces politicking. Define the charter: what does this team own, what does it not own, what does success look like?
Separating AI engineers from product engineers. The discipline is converging — AI engineering is software engineering with LLM tooling. Total isolation produces ivory-tower AI that doesn't fit product realities.
Hiring an AI VP without defining the accountability. 'Head of AI' should own specific product or platform outcomes, not be an advisory role.
Underinvesting in platform while over-staffing feature teams. The feature teams will rebuild the same primitives; you pay platform costs with interest.
Reporting lines matter
Platform teams should report to engineering leadership, not to product. Their job is to serve the engineering organization. Embedded AI engineers should report to the product team they're embedded in — not dotted-line into a central AI function. Dotted lines produce conflicting priorities.
The exception: domain AI teams in regulated industries often need dotted-line accountability to compliance or risk functions. Make the accountability explicit, not implicit.