Building Production AI with Microsoft Semantic Kernel
Why Semantic Kernel matters for enterprise AI — and how we use it to build reliable, multi-model AI systems that survive production traffic.
Most AI projects die at the prototype stage. The demo works beautifully — a clean chat interface, impressive responses, maybe a file upload feature. Then production arrives: concurrent users, session management, model failover, cost controls, security requirements, and the realization that fetch('/api/chat') isn't an architecture.
At Acubent, we've standardized on Microsoft Semantic Kernel as our AI orchestration layer. Here's why, and how we use it.
The Problem with Direct API Integration
Calling OpenAI's API directly works for prototypes. For production systems, you need:
- Multi-model routing — Different tasks benefit from different models. Classification might use a fast, cheap model while complex reasoning needs a more capable one.
- Plugin architecture — AI assistants need to do things: query databases, call APIs, process files. Semantic Kernel's plugin system provides structured tool use.
- Prompt management — System prompts aren't strings in code files. They're versioned, testable configuration.
- Provider abstraction — Your client shouldn't care whether you're using OpenAI, Azure OpenAI, or Gemini. The interface should be consistent.
Our ChatCompletionService Pattern
We've built a reusable ChatCompletionService on Semantic Kernel that provides:
- A unified interface for chat completions across providers
- Intelligent model routing based on task type
- Streaming response support for responsive UX
- Plugin registration for extending AI capabilities
- Token usage tracking for cost management
This service powers ChatSphere, our production AI chat platform, and Butler Bot, our service automation assistant.
Session Memory Architecture
Conversation continuity requires more than sending the last few messages. Our architecture uses IMemoryCache for development and Redis-ready distributed caching for production:
User Message → Session Cache Lookup → Context Assembly → AI Router → Semantic Kernel → Stream Response → Cache Update
The session cache stores conversation history, uploaded file context, and user preferences — ensuring the AI has full context without re-processing documents on every message.
Lessons from Production
Three things we've learned building AI for production:
- Streaming is non-negotiable — Users expect token-by-token responses. Batch responses feel broken in 2024.
- File intelligence needs pipelines — PDF parsing, OCR, and document extraction should happen before the AI sees the content, not during the conversation.
- Security can't be an afterthought — OAuth, passkey support, and identity-ready backends should be in the foundation, not bolted on later.
Semantic Kernel doesn't solve all of these — but it provides the orchestration layer that makes solving them systematically possible.