Deploy AI Agents with Docker Compose

Why Docker Compose for AI Agents

AI agent systems are inherently multi-service architectures. A typical production agent requires at minimum an agent runtime process, a language model endpoint (either a local model server or an API proxy), a database for conversation history and agent state, and often additional services like a vector store for RAG retrieval, a message queue for task distribution, and monitoring infrastructure for observability. Running all of these services manually, managing their startup order, configuring their network connections, and keeping their configurations synchronized is the kind of operational work that consumes entire engineering days and introduces subtle bugs that only appear in production.

Docker Compose solves this by encoding your entire multi-service architecture in a declarative YAML file. Every service, its configuration, its dependencies, its resource limits, and its network connections are defined in one place. When you run docker compose up, Compose reads the file, creates isolated containers for each service, establishes network connections between them, mounts persistent storage volumes, and starts everything in the correct order based on dependency declarations and health checks. When you run docker compose down, it tears everything down cleanly. This deterministic lifecycle management eliminates the "it works on my machine" problem that plagues multi-service systems.

The containerization model that Docker provides is particularly well suited to AI workloads. AI agents depend on specific versions of Python, specific versions of ML libraries like PyTorch or TensorFlow, specific CUDA toolkit versions for GPU access, and specific model weights that can be several gigabytes in size. These dependencies are fragile, version-sensitive, and often conflict with other software on the same machine. Containers encapsulate all of these dependencies into isolated images where they cannot interfere with each other or with the host system. You can run an agent that requires CUDA 12.4 alongside another that requires CUDA 11.8 without any conflict, because each container carries its own complete dependency tree.

Docker Compose has also evolved specifically to support AI workloads. Docker introduced a top-level models element in the Compose specification that lets you declare AI models as first-class infrastructure resources. Docker Model Runner provides native GPU access for running local models without the overhead of putting the model inside a container. These features, combined with the existing Compose capabilities for service orchestration, health checking, resource management, and networking, make Docker Compose the most practical tool for deploying agent systems that need more than a single process but less than a full Kubernetes cluster.

Architecture Patterns for Containerized Agents

The way you structure your Compose services depends on whether your agents use cloud-hosted models, locally-hosted models, or a combination of both. Each pattern has different resource requirements, cost profiles, and operational characteristics.

Pattern 1: API-backed agents. The simplest pattern runs your agent runtime in a container that calls external model APIs like OpenAI, Anthropic, or Google. The Compose file includes the agent service, a database for state persistence, and any tools the agent needs access to. This pattern requires minimal local resources because the compute-intensive model inference happens on the provider servers. It is the right choice when you prioritize simplicity, when your agents need access to the most capable frontier models, or when you want to avoid the capital expense and operational burden of running GPUs locally. The tradeoff is ongoing API costs that scale linearly with usage, dependency on external service availability, and data leaving your infrastructure for every inference call.

Pattern 2: Local model servers. This pattern runs a model server like Ollama, vLLM, or Text Generation Inference alongside your agent in the same Compose stack. The model server loads the model into GPU memory and exposes an OpenAI-compatible API endpoint on the internal Docker network. Your agent service connects to it by service name, making configuration trivial. This pattern eliminates per-call API costs, keeps all data on your infrastructure, and gives you full control over model selection and quantization. The tradeoff is significant upfront hardware investment, operational responsibility for model serving infrastructure, and typically lower capability than frontier cloud models. You also need to manage GPU memory carefully, since large models can consume 20 to 80 GB of VRAM depending on parameter count and quantization level.

Pattern 3: Hybrid routing. The most flexible pattern uses a routing layer that directs requests to either local or cloud models based on task complexity, cost thresholds, or latency requirements. Simple tasks like classification, extraction, or short-form generation route to a local model that handles them cheaply and quickly. Complex tasks like multi-step reasoning, code generation, or long-context analysis route to a frontier cloud model that handles them more accurately. This pattern requires more Compose services and routing logic, but it optimizes the cost-capability tradeoff for workloads where agent tasks vary significantly in difficulty.

Pattern 4: Multi-agent orchestration. Production agent systems often involve multiple specialized agents that collaborate on complex tasks. A research agent gathers information, an analysis agent evaluates it, a writing agent produces output, and a review agent checks quality. In Compose, each agent can be a separate service with its own configuration, model access, and tool permissions. A message broker like Redis or RabbitMQ handles inter-agent communication, and a shared database maintains the overall task state. This pattern enables horizontal scaling of individual agents based on workload, independent deployment and updating of each agent, and clear separation of concerns that simplifies debugging and monitoring.

Anatomy of an Agent Compose File

A production-grade Compose file for AI agents typically includes five categories of services: the agent runtime, the model layer, the data layer, the communication layer, and the observability layer. Understanding how these fit together helps you design Compose files that are maintainable, scalable, and operationally sound.

The agent runtime service is the core of your stack. It contains the application code that implements your agent logic, including prompt templates, tool definitions, memory management, and orchestration flow. This service typically builds from a custom Dockerfile based on a Python or Node.js base image, installs your application dependencies, and runs your agent entry point. It declares dependencies on the model layer and data layer services so Compose starts them first. Environment variables configure the model endpoint URL, database connection strings, API keys, and agent-specific parameters like temperature, max tokens, and tool timeout values.

The data layer provides persistence and retrieval. At minimum this includes a database for conversation history and agent state, commonly PostgreSQL for relational data or Redis for fast key-value access. Agent systems that use retrieval-augmented generation also include a vector database like Qdrant, Weaviate, Milvus, or Chroma for storing and searching document embeddings. Each database runs as a separate Compose service with its own persistent volume, health check, and resource limits. Using named volumes rather than bind mounts ensures that data survives container restarts and that volume management is handled by Docker rather than the host filesystem.

The communication layer becomes necessary when you run multiple agents or need asynchronous task processing. Redis, RabbitMQ, or NATS provide message queuing that decouples task submission from task execution. This decoupling lets you scale agent workers independently of the service that receives requests, handle traffic spikes by buffering tasks in the queue, and recover from agent crashes without losing pending work. The communication layer is optional for single-agent systems but essential for any production deployment that handles concurrent requests or coordinates multiple agents.

The observability layer provides visibility into what your agents are doing and how well they are performing. At minimum this includes structured logging that captures agent decisions, tool calls, model interactions, and error events. More mature deployments add distributed tracing with Jaeger or Zipkin to follow request flows across services, metrics collection with Prometheus to track latency, throughput, and resource utilization, and dashboards with Grafana to visualize system health. These observability tools run as additional Compose services and receive data from your agent runtime through standard protocols like OpenTelemetry.

GPU and Hardware Acceleration

Running local AI models requires GPU access, and Docker has matured significantly in how it handles GPU passthrough to containers. The NVIDIA Container Toolkit (formerly nvidia-docker) lets Docker containers access NVIDIA GPUs on the host system. In your Compose file, you declare GPU reservations in the deploy.resources.reservations section, specifying how many GPUs each service needs and optionally which specific GPU devices to use. Docker handles the runtime configuration that exposes the GPU drivers, CUDA libraries, and device files inside the container.

GPU memory management is the most critical operational concern for containerized AI workloads. A 7-billion parameter model loaded in 16-bit precision requires approximately 14 GB of VRAM. A 70-billion parameter model requires approximately 140 GB in 16-bit or 35 to 40 GB in 4-bit quantization. When multiple services share a GPU, their combined memory usage cannot exceed the physical VRAM, or the out-of-memory killer will terminate one or more processes without warning. Compose resource limits let you set memory reservations and caps for each service, but VRAM limits require either model server configuration or careful planning of which models run on which GPUs.

Docker Model Runner offers an alternative approach to GPU access that avoids putting the model inside a container entirely. Model Runner runs models natively on the host GPU and exposes them through an OpenAI-compatible API endpoint. Your Compose services connect to this endpoint by host network address. This approach provides better GPU performance because it eliminates the container abstraction layer for model inference, simpler GPU driver management because the host system handles drivers directly, and easier model management because models are downloaded and cached at the host level rather than baked into container images. The tradeoff is that Model Runner is Docker-specific and does not provide the same portability as a fully containerized model server.

For teams that do not have local GPUs, Compose integrates with cloud GPU providers through Docker offload feature. Running docker compose --offload up sends your stack to a cloud environment with GPU access, while logs and results stream back to your local terminal. This gives you the same Compose workflow without requiring GPU hardware on your development machine. The cloud execution is billed by usage, making it practical for development and testing workloads where you need occasional GPU access but cannot justify purchasing dedicated hardware.

Persistent Data and State Management

AI agents generate and depend on several types of persistent data that must survive container restarts, image updates, and stack redeployments. Conversation histories record every interaction and enable agents to maintain context across sessions. Agent state includes task progress, decision logs, and intermediate results for multi-step workflows. Model weights, while typically read-only, are large files that you want to cache locally rather than re-downloading on every container start. Vector database indices store document embeddings that may have taken hours to generate and cannot be recreated quickly.

Docker volumes are the correct mechanism for all of these persistence needs. Named volumes are managed by Docker, stored outside the container filesystem, and survive container removal and recreation. They also support Docker volume drivers, which let you back volumes to network storage, cloud block storage, or distributed filesystems without changing your application code. In your Compose file, you declare named volumes in the top-level volumes section and mount them into containers in each service volumes configuration.

Bind mounts, the other option for persistent storage, map a specific host directory into the container. They are useful during development when you want to edit code on the host and have changes reflected immediately in the container, but they create portability problems because they depend on the host filesystem layout. For production deployments, named volumes are almost always the better choice. The exception is model weight storage, where bind mounts to a fast NVMe drive can provide better I/O performance for loading large model files.

Backup strategy for agent data should account for the different consistency requirements of each data type. Relational databases like PostgreSQL support point-in-time recovery with WAL archiving and provide strong transactional consistency guarantees. Vector databases vary widely in their backup capabilities, with some supporting snapshot-based backups and others requiring you to stop writes during backup to ensure consistency. Redis data, if used for ephemeral caching, may not need backup at all, but if used for persistent task queues, requires RDB snapshots or AOF persistence configured at the server level.

Networking and Service Discovery

Docker Compose creates a dedicated bridge network for each stack, and every service in the stack is automatically addressable by its service name within that network. When your agent service needs to connect to a PostgreSQL database, it uses postgres as the hostname (matching the service name in the Compose file), not localhost or a hard-coded IP address. This built-in service discovery eliminates DNS configuration, load balancer setup, and manual IP management for internal communication between your agent stack services.

Network isolation is both a convenience and a security feature. Services in one Compose stack cannot reach services in another stack by default. This prevents accidental cross-contamination between environments, limits the blast radius of security breaches, and ensures that development and production stacks running on the same machine cannot interfere with each other. When you intentionally need cross-stack communication, you create an external network that both stacks join, giving you explicit control over which services can reach which other services.

Port mapping controls which services are accessible from outside the Docker network. Most services in an agent stack should not be externally accessible. Your databases, message brokers, model servers, and internal tools should communicate only on the internal Docker network. Only the entry point to your agent system, typically an API gateway or web interface, should map a port to the host. This minimizes your attack surface and prevents unauthorized access to internal services. In the Compose file, services that only need internal access declare no ports mapping but remain fully accessible to other services in the stack through the internal network.

For multi-agent systems where agents communicate through message passing, the message broker service becomes the central network hub. Redis Pub/Sub, RabbitMQ, or NATS handle message routing between agents without requiring direct network connections between agent containers. This message-based architecture is more resilient than direct HTTP calls between agents because the broker buffers messages during transient failures, enables fan-out patterns where one message triggers multiple agent actions, and provides natural backpressure when downstream agents cannot keep up with incoming work.

Security Considerations

Containerized AI agents introduce security concerns at multiple levels: the container runtime, the model access layer, the tool execution layer, and the data storage layer. Addressing security at each level prevents the most common attack vectors and compliance failures in production agent deployments.

Container images should follow the principle of minimal surface area. Use slim or distroless base images that contain only the runtime your application needs. Never run agent processes as root inside the container, because a container escape vulnerability combined with root privileges gives an attacker full host access. Specify a non-root user in your Dockerfile and ensure your application runs under that user. Pin your base image versions to specific digests rather than floating tags to prevent supply chain attacks where a compromised tag points to a malicious image.

Secrets management is critical for agent systems because they typically need API keys for model providers, database passwords, and credentials for external tools. Never embed secrets in your Compose file, Dockerfile, or application code. Docker Compose supports external secret files and environment variable files that keep credentials out of version control. For production deployments, use Docker secrets or an external secrets manager like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault. Your agent runtime should read secrets from environment variables or mounted files at startup, never from hard-coded strings in source code.

Tool execution is the highest-risk component of an AI agent system. When an agent calls tools that execute code, access filesystems, make network requests, or interact with databases, it is operating with the permissions of the container it runs in. Limit these permissions aggressively. Use Docker security options to drop unnecessary Linux capabilities, restrict system calls with seccomp profiles, and mount filesystems read-only where the agent does not need write access. If your agent executes arbitrary code as a tool, run that execution in a separate, heavily sandboxed container with no network access, no volume mounts, and strict resource limits.

Network security within the Docker stack matters even though the services are on an isolated bridge network. Use encrypted connections (TLS) for communication between services that handle sensitive data, especially between the agent runtime and any database that stores user conversations or personal information. While the Docker network provides isolation from external traffic, it does not encrypt internal traffic by default, meaning any container that is compromised could sniff network traffic between other services on the same bridge.

From Development to Production

One of Docker Compose greatest strengths for AI agent deployment is the ability to use the same fundamental architecture from development through production, adjusting only the operational parameters. Your development Compose file runs all services on a single machine with relaxed resource limits, debug logging, and hot-reload capabilities. Your production Compose file runs the same services with strict resource limits, structured production logging, health checks, restart policies, and optimized configurations.

Compose supports multiple file overrides that make this workflow practical. A base compose.yaml defines your services, their relationships, and their default configurations. A compose.override.yaml file adds development-specific settings like bind mounts for code hot-reloading, exposed debug ports, and verbose logging. A compose.prod.yaml file adds production settings like resource limits, restart policies, production environment variables, and health check configurations. You select which files to use at runtime: docker compose up automatically loads the base and override files for development, while docker compose -f compose.yaml -f compose.prod.yaml up loads the base and production files for deployment.

Health checks are essential for production agent deployments because AI workloads have unique failure modes that differ from traditional web services. A model server might accept TCP connections but fail to return coherent responses because the model weights are corrupted. An agent might respond to health check endpoints but hang indefinitely on actual tasks because it is stuck in a reasoning loop. Effective health checks for agent systems go beyond basic port connectivity and verify that the service can actually perform its core function. For model servers, this means verifying that a short inference request completes successfully. For agent runtimes, this means verifying that the agent can reach its required dependencies and process a minimal test task.

Restart policies determine how Compose handles service failures. The unless-stopped policy restarts services automatically after crashes, which is appropriate for stateless services like API gateways and monitoring tools. For stateful services like databases and model servers, you may want the on-failure policy with a maximum retry count, because repeated crashes of a stateful service often indicate a data corruption or resource exhaustion issue that automatic restarts will not fix. For agent services, consider the failure mode: if an agent crashes because of a transient error like a network timeout, automatic restart is appropriate. If it crashes because of a logic error in its reasoning, restarting just reproduces the crash.

Docker Compose vs Alternatives

Docker Compose occupies a specific niche in the deployment tooling landscape, positioned between manual process management and full container orchestration. Understanding where Compose fits helps you decide when it is the right tool and when you should consider alternatives.

Compared to bare-metal deployment, where you install and run services directly on the host operating system, Compose provides dependency isolation, reproducibility, and simplified lifecycle management. Bare-metal deployment gives you maximum performance because there is no container overhead, direct hardware access without passthrough layers, and simplicity for single-service deployments. For AI workloads specifically, bare-metal deployment can provide 5 to 15 percent better GPU performance because it eliminates the container runtime overhead on GPU operations. However, bare-metal deployment makes multi-service management exponentially more complex as your stack grows, and it makes reproducibility across environments nearly impossible without extensive configuration management tooling.

Compared to Kubernetes, Compose is dramatically simpler for small to medium deployments. Kubernetes provides automatic scaling, rolling deployments, self-healing, service mesh, and a rich ecosystem of operators and controllers. It also introduces significant operational complexity: etcd management, API server configuration, node management, networking plugins, storage provisioners, and a steep learning curve for teams that have not operated Kubernetes before. For agent deployments running on fewer than 10 to 15 machines, Compose provides sufficient orchestration without the overhead. When your agent system grows to require automatic horizontal scaling across many machines, geographic distribution, or multi-tenant isolation, Kubernetes becomes worth the investment.

Docker Compose also integrates with cloud container services that provide managed infrastructure without the operational burden of Kubernetes. Docker native integrations with Google Cloud Run and Azure Container Apps let you deploy the same Compose file to serverless container platforms that handle scaling, load balancing, and infrastructure management automatically. This path gives you a Compose-based development workflow with cloud-managed production infrastructure, which is an attractive middle ground for teams that need scalability but do not want to operate Kubernetes.

Explore This Topic