Inference Builder is a tool that automatically generates inference pipelines and integrates them into either a microservice or a standalone application. It takes an inference configuration file and an OpenAPI specification (when integrated with an HTTP server) as inputs, and may also require custom code snippets in certain cases.
The output of the tool is a Python package that can be used to build a microservice container image with a customized Dockerfile.
The Inference Builder consists of three major components:
- Code templates: These are reusable modules for various inference backends and frameworks, as well as for API servers. They are optimized and tested, making them suitable for any model with specified inputs, outputs, and configuration parameters.
- Common inference flow: It serves as the core logic that standardizes the end-to-end inference process—including data loading and pre-processing, model inference, post-processing, and integration with the API server. It supports pluggable inference backends and frameworks, enabling flexibility and performance optimization.
- Command line tool: It generates a source code package by combining predefined code templates with the Common Inference Flow. It also automatically produces corresponding test cases and evaluation scripts to support validation and performance assessment.
Visit our documentation for more details:
First, be sure your system meets the requirement.
| Operating System | Python | CPU | GPU1 |
|---|---|---|---|
| Ubuntu 24.04 | 3.12 | x86, aarch64 | Nvidia ADA, Hopper, Blackwell |
Next, follow these steps to get started:
sudo apt update
sudo apt install python3.12-venv python3.12-devDocker environment must be properly set up with below packages for building and running the examples:
- Docker: Installation Guide
- Docker Compose: Installation Guide
- NVIDIA Container Toolkit: Installation Guide
Ensure nvidia runtime added to /etc/docker/daemon.json to run GPU-enabled containers
{
"runtimes": {
"nvidia": {
"args": [],
"path": "nvidia-container-runtime"
}
}
}The docker group must exist in your system, please check if it has been created using getent group docker. Your current user must belong to the docker group, If not, run the command below, then log out and back in for the group change to take effect.
sudo usermod -aG docker $USERNote for TEGRA users: If you're using a TEGRA device, you'll also need to install the Docker buildx plugin:
sudo apt install docker-buildxNote for Jetson Orin users: External storage integration is required to support the execution of the DeepStream microservice on the Jetson Orin platform.
Download and install the NGC CLI from the NGC page and follow the NGC CLI Guide to set up the tool.
cd <deepstream-repo>/tools/inference_builderpython3 -m venv .venv
source .venv/bin/activate
pip3 install -r requirements.txtNow you can try our examples to learn more. These examples span all supported backends and demonstrate their distinct inference flows.
Compared to manually crafting inference source code, Inference Builder offers developers the following advantages:
- Separation of concerns: Introduces a new programming paradigm that decouples inference data flow and server logic from the model implementation, allowing developers to focus solely on model behavior.
- Backend flexibility: Standardizes data flow across different inference backends, enabling developers to switch to the optimal backend for their specific requirement without rewriting the entire pipeline.
- Hardware acceleration: Automatically enables GPU-accelerated processing to boost performance.
- Streaming support: Provides built-in support for streaming protocols such as RTSP with minimal configuration.
- Standardized testing: Automates and standardizes test case generation to simplify validation and evaluation workflows.
Inference Builder includes MCP (Model Context Protocol) integration, enabling AI-assisted pipeline generation directly within Cursor or Claude Code. With MCP, you can use natural language to generate inference pipelines, build Docker images, and explore sample configurations.
For detailed tool reference and advanced usage, see mcp/README-MCP.md.
Before setting up MCP, ensure you have completed the Getting Started steps above.
Ensure that you are in the Inference Builder component directory
(tools/inference_builder/ from the DeepStream repository root).
Use the provided management script from the component directory:
# Start (runs in background)
MCP_API_KEY=MY_SECRET ./mcp/server_manager.sh start --port 8000
# Check status
./mcp/server_manager.sh status
# Tail logs
./mcp/server_manager.sh logs
# Stop
./mcp/server_manager.sh stopOmit MCP_API_KEY to allow unauthenticated access. Use --workspace-root to set a custom directory for per-session workspaces.
Add the following to ~/.cursor/mcp.json (global) or <project>/.cursor/mcp.json (project-specific):
{
"mcpServers": {
"deepstream-inference-builder": {
"url": "http://<host>:8000/mcp",
"headers": { "Authorization": "Bearer MY_SECRET" }
}
}
}Then navigate to File > Preferences > Cursor Settings > MCP. A green status icon next to deepstream-inference-builder confirms the connection:
Run the following command to register the server:
# With MCP_API_KEY enabled on the server
export MCP_API_KEY=MY_SECRET
codex mcp add deepstream-inference-builder \
--url http://<host>:8000/mcp \
--bearer-token-env-var MCP_API_KEY
# Without MCP_API_KEY
codex mcp add deepstream-inference-builder \
--url http://<host>:8000/mcpVerify the connection is configured:
codex mcp listRun the following command to register the server:
# User-level (available in all projects)
claude mcp add --transport http --scope user \
deepstream-inference-builder http://<host>:8000/mcp \
--header "Authorization: Bearer MY_SECRET"
# Project-level
claude mcp add --transport http --scope project \
deepstream-inference-builder http://<host>:8000/mcp \
--header "Authorization: Bearer MY_SECRET"Omit --header if no API key was set on the server. Then verify the connection by running /mcp in the Claude Code console:
❯ deepstream-inference-builder · ✔ connected
Invoke the tools by mentioning "deepstream inference builder" in your prompt:
- "Show me what sample configurations are available from the inference builder?"
- "Generate a DeepStream object detection pipeline using the inference builder with PeopleNet transformer model from NGC."
| Tool | Description |
|---|---|
generate_inference_pipeline |
Generate inference pipelines from YAML configuration files |
build_docker_image |
Build Docker images from generated pipelines |
docker_run_image |
Run Docker images for testing and troubleshooting |
prepare_model_repository |
Download models from NGC or HuggingFace and prepare repositories |
generate_nvinfer_config |
Generate DeepStream nvinfer runtime configuration files |
| Resource | Description |
|---|---|
docs://README.md |
Project documentation |
docs://mcp/README-MCP.md |
MCP integration documentation |
schema://config.schema.json |
Configuration schema reference |
samples://config/* |
Sample pipeline configurations |
samples://dockerfile/* |
Sample Dockerfiles |
samples://processor/* |
Sample preprocessors and postprocessors |
You can now try out the example prompts using either Cursor or Claude Code.
This repository ships the Inference Builder skill at
skills/inference-builder/. Claude Code discovers
it through the root .claude/skills link, and other compatible agents can copy that
directory into their user-level skills directory. The skill resolves the in-tree CLI
and schemas from the DeepStream repository root.
You can now try out the example prompts using Claude Code.
Inference Builder follows the DeepStream repository's development and licensing policy.
The project is not accepting external code contributions; see the root
README.md for details.
The project is under active development and the following features are expected to be supported in the near future:
- Support for more backends and frameworks such as SGLANG.
- Support for more model types such as speech models.
Footnotes
-
If you only generate the inference pipeline without running it, no GPU is required. ↩