Solution
AI infrastructure and model deployment
Design and deployment of CPU or GPU inference servers, model storage and access APIs. Integration of AI models with software and data platforms, including connections to cloud services.
Architecture and implementation
Compute and model serving
Deployment of compute resources and model serving software.
APIs and gateways
Interfaces connecting applications to local models and hosted AI services.
Deployment and operations
Environment automation and AI service monitoring integration.
Design decisions
Deployment of AI models on CPU or GPU inference servers, with internal APIs and model storage. The architecture also specifies where indexes and logs are stored, which external services are called and how updates are downloaded.
Illustrative implementation
A local model can be exposed through an internal API and connected to a document index, with latency, concurrency and memory measurements.
Engineering outputsInference environment, API, access configuration and capacity measurements
Related projects
A private platform connects a research agent to a document index and a local Llama model, with retrieval filtered by access permissions. For example, it can support review of a contract file by locating relevant passages in the firm’s documents and reference texts, then preparing an answer with source references.
Docker
.NET
Microsoft Entra ID
Infomaniak
LangChain
Linux
Llama (Meta)
Model Context Protocol
Microsoft 365
PostgreSQL
Python
Components and flows
- Extraction of Microsoft 365 documents and reference collections, followed by text and vector indexing in PostgreSQL
- LangChain agent using a Llama model and read-only MCP search tools, with permission filtering and bounded tool calls
- Available context checked before generation, with references presented in a .NET application
Development and integration
- Inference servers and GPU compute
- Open models and cloud services
- Containers, storage and orchestration
- Gateways, metrics and application integration
Systems involved
- vLLM, Ollama and model APIs
- Azure, Google Cloud and private infrastructure
- Databases, indexes and applications
Service lineArtificial intelligence →
To go deeper
Evaluating open-weight model quality, availability and operating cost
Public rankings identify candidates. Selection depends on the licence, performance on the task and the resources required to operate the model.
Three configurations for sizing local AI
Memory, throughput and concurrent demand guide the hardware choice. Three profiles help prepare a trial and compare total operating cost.
Deploying internal AI with open models and suitable infrastructure
Required quality, data constraints and operating capacity determine the deployment model. The model and its environment need to be evaluated together.
Discuss your project
A discussion of your project’s design, development and production deployment.