Computer Vision Setup
1. Choose a Capture Tool
| Tool | Best fit |
|---|---|
| DS Capture | General DirectShow devices |
| KS Capture | Compatible Windows Kernel Streaming devices |
| MF Capture | Media Foundation devices, including common Elgato hardware |
| Magewell Capture | Magewell Pro capture cards |
| OpenCV Capture | Simple DirectShow/OpenCV compatibility path |
| Dataset Playback | Saved images and video frames for testing without a capture card |
| OBS Capture | An OBS scene or source through the Helios OBS integration |
Select the device, frame rate, resolution, and format shown by that tool. Start it and confirm Write FPS remains stable. If a requested combination fails, choose a format/resolution the device actually exposes.
If colors look washed out or too dark, start with an SDR source and check the capture device’s color/range settings. Native capture paths do not guarantee HDR-to-SDR tone mapping. Reconnect capture after changing source settings; see Capture Plugins.
2. Configure Compute and Python
For CV Python, choose a valid Python 3.10+ environment under Preferences → Python. Use Manage Python Environments if you need to create one or install a script’s packages.
For detection, pose, segmentation, OCR, or TensorRT engine building, choose Preferences → Hardware → Compute GPU. DirectML and TensorRT availability depends on the detected GPU and installed environment. DirectML uses the FP16 or FP32 types declared by the ONNX model automatically; there is no separate precision switch.
3. Start a CV Script
- CV Python: select a
.pyscript. The panel must show a valid Python environment and active video input. - CV C++: select a supported CV DLL. The panel must show active video input.
Use Restart after changing a script or when you need to reset its state.
4. Inspect the Result
Open OpenGL Display for the hardware-accelerated preview or Video Display for the CPU-rendered preview. Lower the display rate if previewing consumes resources needed by capture or inference.
Use Fuser output when the script draws to the Fuser target and you want those overlays on a selected display.
5. Connect Controller Output
Computer vision does not require controller output unless the script also publishes data or affects an output workflow. When it does:
| Step | Action |
|---|---|
| 1 | Start the physical input tool if the script reads controller state. |
| 2 | Start the compatible processing or output tool that uses the script's result. |
| 3 | Start the selected hardware, virtual, or Remote Play destination. |
| 4 | Verify the final result in Device Monitor. |
Model Compatibility
Support depends on the model’s export and selected GPU runtime; a family name alone does not guarantee compatibility. Use the model task declared by its author: detect, pose, or segment. OCR uses its separate OCR workflow. Classification, oriented bounding boxes, and arbitrary pose skeletons are not supported by the object-model API.
| Model family | Supported use | Export requirements |
|---|---|---|
| YOLO | Detection, 17-keypoint pose, segmentation | Use a supported ONNX export for the intended task; objectness-form YOLOv5 detection needs model_family: "yolov5" |
| RT-DETR | Detection | Concatenated exports need model_family: "rtdetr" |
| RF-DETR | Detection and segmentation | Keep the exported dets, labels, and, for segmentation, masks output names |
An ONNX file must include its tensor data; external tensor sidecars and raw PyTorch checkpoints are not loadable model files. TensorRT uses engines built through Helios (.ecache/.eecache); DirectML uses the source weights (.onnx/.ennx). Do not substitute a raw TensorRT .engine file.
For authors importing through MCP, set model_family only for the two cases above. Standard Ultralytics 17-keypoint pose exports do not need an extra class count. Pose exports that include objectness need a positive class_count equal to the exporter’s complete class count. Get that value from the model author; it is not the number of classes you want to display.
Class IDs remain those supplied by the model. For example, official RF-DETR COCO exports use class 1 for person, while common YOLO COCO exports use 0. Check custom models’ own class lists before configuring priorities.
TensorRT requires FP32 input and output tensors even when internal computation uses FP16. DirectML accepts FP16 or FP32 inputs and outputs automatically. If loading fails, verify the task, export type, class metadata, and GPU choice in Output Panel. Updating class recommendations does not require rebuilding weights or engines; reload the model to use changed metadata.
Segmentation Settings
For a segmentation model, Model Options offers per-class mask visibility, color, opacity, threshold, and BBox or Mask anchoring. Start with the author’s recommendations.
- Color, opacity, and visibility affect the preview only; hiding a mask does not stop results.
- Threshold controls which pixels belong to the mask and can change the anchor.
- BBox positions the anchor relative to the detection box.
- Mask positions Anchor Y between the mask’s first and last occupied rows and Anchor X across the selected row’s foreground span. A midpoint can lie over a hole; it is not snapped to a foreground pixel. Empty masks are excluded in this mode.
These settings do not require an engine rebuild. Script authors can set the same options through Python Inference or C++ Inference.
Dataset Collection
Use Tools → Dataset Capture to save raw or cropped frames. Time and button triggers work with raw video; class-ID triggers require running inference.
Performance Checklist
- Keep capture resolution and FPS no higher than the workflow needs.
- Use a tight inference or vision ROI when possible.
- Reduce display preview rate before reducing capture rate.
- Avoid running multiple capture tools for the same source.
- Build TensorRT engines on the GPU and environment where they will be used.