Problem & Constraints
As conversational AI began appearing across enterprise platforms, each vendor increasingly brought its own assistant and its own approach to enterprise knowledge. Making those assistants useful often meant making company information available within each vendor's environment, creating the potential for the same knowledge to be replicated across multiple platforms with different governance and lifecycle models.
At the same time, the employee experience could not be organized around where information happened to live. A restaurant employee might ask about training, a recipe, an operating procedure, or a problem that crossed multiple knowledge domains. The architecture needed to support both narrowly focused assistants and broader experiences capable of reasoning across those domains without requiring the user or conversational client to know where the answer should come from.
- Enterprise data control
- Enterprise knowledge needed to remain within company-controlled platforms rather than being replicated into each assistant provider's environment.
- Identity and authorization
- Internal and externally delivered experiences needed to use Microsoft Entra ID company credentials and RBAC-controlled access.
- Multiple knowledge domains
- Training, recipes, and operational knowledge originated from different enterprise sources and needed to remain independently manageable.
- Flexible assistant scope
- The architecture needed to support both single-skill assistants and broader multi-skill experiences without forcing every implementation into an agentic model.
- Cross-domain reasoning
- Broader assistants needed to determine when a request required one knowledge skill, several skills, or no retrieval at all.
- Experience independence
- Enterprise knowledge and orchestration could not be tightly coupled to a particular bot, vendor assistant, or conversational interface.
- Centralized response generation
- Retrieval, reasoning, and synthesis needed to remain within enterprise-controlled services, returning a completed response rather than exposing the underlying retrieval architecture to the client.
Architecture Overview
The architecture separated conversational experiences from the enterprise capabilities used to answer them. Enterprise-controlled clients could connect directly to purpose-built backends, while clients outside company control were required to enter through Azure API Management. Each backend was configured with only the capabilities appropriate to its use case, from a focused assistant using a single skill to a broader competency composed of multiple skills. Microsoft Entra ID and role-based access control provided a common identity and authorization model across those experiences.
Skills abstracted the underlying knowledge and retrieval mechanisms from both the conversational client and the orchestration layer. Unstructured and semi-structured knowledge could be indexed through Azure AI Search, while structured information could be queried directly when appropriate. For multi-domain experiences, an Azure OpenAI-powered agent could reason over the request and invoke one or more available skills before synthesizing a complete response; focused implementations could use a simpler retrieval path without agentic orchestration. This allowed the same architectural model to support both narrowly scoped assistants and more intelligent cross-domain experiences without exposing the underlying enterprise data architecture to the client.
Key Architecture Decisions
Separate experience from intelligence
Keep retrieval, orchestration, and response generation independent of the conversational client so different experiences can consume the same enterprise capabilities.
Design around capabilities, not assistants
Configure backends around the skills and orchestration required for a use case rather than tying intelligence to a particular assistant or channel.
Compose bounded skills into competencies
Represent knowledge domains as reusable skills that can operate independently or be combined into broader business competencies.
Match orchestration to the use case
Use simple, focused retrieval when sufficient and agentic orchestration when requests require reasoning across multiple skills.
Keep enterprise knowledge behind the boundary
Return completed responses to conversational clients while keeping retrieval, source interaction, and synthesis within enterprise-controlled services.
Govern at the trust boundary
Allow enterprise-controlled clients direct access while requiring clients outside company control to enter through Azure API Management.
Preserve enterprise identity
Use Microsoft Entra ID and RBAC across conversational channels so authentication and authorization remain enterprise-controlled.
Abstract retrieval from the knowledge source
Let skills hide whether knowledge is retrieved through Azure AI Search or queried directly from structured enterprise data.
Engineering Approach
The architecture combined Azure-native AI, search, identity, and integration services behind a set of reusable enterprise capabilities. Rather than prescribing one implementation pattern for every assistant, the approach allowed backends to be configured with the skills, knowledge sources, and orchestration appropriate to the use case. The same model could therefore support straightforward retrieval experiences and more sophisticated agentic interactions while preserving common boundaries for identity, knowledge access, and externally exposed APIs.
- Application & Services
- Azure Functions · Azure OpenAI · Azure Bot Service · RESTful APIs · Conversational Clients
- Data & Messaging
- Azure AI Search · Domain-Specific Search Indexes · Structured Data Access · Unstructured Data · Semi-Structured Data
- Patterns & Principles
- Retrieval-Augmented Generation (RAG) · Agentic RAG · Skills & Competencies · Capability Abstraction · Composable Capabilities · Separation of Concerns · Loose Coupling
- Release & Reliability
- Independent Skill Evolution · Backend Isolation · Focused vs. Agentic Orchestration · Extensible Capability Model
- Security
- Microsoft Entra ID · Role-Based Access Control (RBAC) · Azure API Management · Token-Based Authentication · Governed External Access
Results & Evolution
The architecture established a reusable model for connecting conversational experiences to enterprise knowledge without coupling those experiences to individual source systems or requiring enterprise content to be replicated across assistant platforms. The Skills and Competencies model provided a common abstraction for composing knowledge capabilities, while supporting both focused retrieval and broader agentic reasoning within the same overall architecture.
AI architecture has evolved rapidly since this design was developed in early 2025. The emergence of standardized protocols such as Model Context Protocol, together with expanding enterprise AI gateway and agent capabilities, creates new options for exposing, governing, and orchestrating enterprise capabilities. Rather than retrofitting those technologies into the original design, I revisited the same architectural problem using the capabilities available today.
Same problem. New architecture
See how I would approach these requirements today using MCP and the Microsoft IQ enterprise intelligence layer. Explore the current-day architecture →
Architectural Perspective
My approach to architecture is to create abstractions and clear boundaries that contain complexity and allow systems to evolve. Here, conversational experiences were separated from the intelligence behind them, while bounded skills could operate independently or compose into broader competencies. The goal was not to make every experience agentic, but to use the simplest orchestration appropriate to the problem while preserving security, governance, and extensibility.