Python Inference

helios.inference gives CV Python scripts access to compatible models installed in Helios. Helios uses the Compute GPU selected in Preferences.

Start a Model

from helios import inference


class CVWorker:
    def __init__(self, width, height):
        models = inference.list_models()
        if not models:
            raise RuntimeError("No inference models are installed")

        self.engine = inference.create_inference_engine(models[0]["uuid"])
        self.engine.set_confidence_threshold(0.45)
        self.engine.set_nms_threshold(0.50)
        self.engine.start()

    def process(self, frame):
        result = self.engine.wait_for_results_raw()
        if result is None:
            return

        count = int(result["num_detections"])
        detections = result["detections"][:count]
        if count:
            first = detections[0]
            class_id = int(first["class_id"])
            confidence = float(first["confidence"])
            center_x = float(first["center_x"])
            center_y = float(first["center_y"])

Create the engine in __init__ and reuse it. wait_for_results_raw() waits indefinitely by default so the result stays paired with the current frame. Pass timeout_ms=0 or another non-negative value only when the script intentionally accepts abandoning that frame's result after the deadline.

An engine started in CVWorker.__init__() has no result for the first process() call and returns None; its first result can arrive on the next frame. The same applies when starting an engine during process(). Handle None as no completed result, as in the example. It is different from a completed result with zero detections and does not by itself indicate a model-load failure.

Structured Result View

When a result is available, wait_for_results_raw() returns a zero-dimensional NumPy structured view with named fields:

Only detections[:num_detections] is valid. Detection, pose, and segmentation share one global limit of 64 published detections per frame, not 64 per class.

Each detection has these named fields:

apparent_distance is a dimensionless, size-derived relative distance: larger values represent a smaller, apparently farther detection. It is not a physical distance. Configure the expected height-to-width ratio globally or for selected classes:

self.engine.set_apparent_distance_aspect_ratio(
    expected_height_per_width=2.5,
    class_ids=None,
)

The default expected ratio is 2.5. Use model metadata to map class ids to labels.

The result and nested arrays are borrowed reusable views. Read or copy the values needed for the current callback and do not retain those views across later frames.

Segmentation Results

Segmentation models provide the structured detection result plus one binary mask per published detection. After a successful wait_for_results_raw(), call:

mask_result = self.engine.segmentation_results_raw()
if mask_result is not None:
    (
        mask_frame_sequence,
        mask_format,
        mask_count,
        mask_width,
        mask_height,
        mask_row_bytes,
        mask_plane_stride,
        source_width,
        source_height,
        transform,
        masks,
    ) = mask_result

Mask i corresponds to detection i. masks is a read-only borrowed view containing mask_count packed mask planes. Within a plane, row y starts at y * mask_row_bytes; pixel x is bit x & 7 of byte x >> 3. Use mask_plane_stride to move between masks. The transform tuple maps mask coordinates to the source frame. Consume the view immediately and do not retain it across frames.

Model and Session Functions

The session exposes model_type, model_info, loaded_model_description, task, and is_paused. It supports load_model(uuid), unload_model(), pause(), resume(), stop(), and destroy().

One session holds one object model. Create additional sessions for additional models and account for the added CPU/GPU load. Helios supports up to 30 simultaneously started inference sessions in total, and each session supports up to 16 OCR regions. Calling pause() or resume() during process() changes inference beginning with the next frame; a wait during that transition returns immediately.

Every model must declare its task exactly as detect, pose, or segment.

Constants

Use the named constants rather than relying on their numeric values:

Filtering and ROI

Normalized positions use 0.0..1.0. New sessions begin with the model's recommended ROI and class priority when that metadata exists. Explicit ROI or priority calls replace the recommendation until its corresponding set_use_recommended_...(True) call is used again.

Search Activation

Use set_search_buttons(...) to run inference only for configured controller conditions. It accepts simple button indices, (button_index, state) conditions, or up to eight OR clauses. State 1 means pressed and 0 means released.

Use a stick-magnitude trigger with:

self.engine.set_search_stick(1, radius=25)                  # magnitude above 25
self.engine.set_search_stick(2, radius=25, less_than=True)  # magnitude below 25
self.engine.set_search_stick(None)                          # disable stick trigger

stick is 1 or 2; radius is required and accepts 0..100. Search buttons and the stick are alternatives, so either can activate inference. With neither configured, inference runs on every frame.

Drawing

set_draw_detections(True) enables boxes. Related settings include box thickness, confidence text, ROI drawing, and BGR colors. Pose sessions also provide keypoint and skeleton controls. Drawing is independent of returned results: hiding a box does not remove its result.

Segmentation masks draw by default. Use set_draw_segmentation_masks(enabled, class_ids=None), set_segmentation_mask_opacity(opacity, class_ids=None), and set_color_segmentation_mask(b, g, r, class_ids=None) to control their display without changing raw masks.

Apparent-distance text is off by default:

self.engine.set_draw_apparent_distance(True)

It draws the rounded dimensionless apparent_distance value when enabled independently. Confidence drawing also includes apparent distance and, for the selected target, available motion speed; an unavailable speed appears as V--%/s.

Segmentation Settings

Omit class_ids for a global setting or supply the intended class ID or IDs. Per-class settings override global settings until changed or cleared with clear_class_overrides(...).

Use set_anchor_point(...) for the anchor fractions and pixel offsets. With "mask", Y selects a row across the occupied mask height and X selects a fraction across that row’s foreground span. The nearest occupied row is used; X can lie over a hole. An empty mask excludes the detection. With "bbox", anchor coordinates remain relative to the box and an empty mask is allowed.

The threshold changes published masks and mask-derived anchors. Color, opacity, and drawing settings affect visualization only. These settings also appear in Model Options.

Priorities, Anchors, and Optional Prediction

Use recommended class priorities unless the script needs a different ordering. Only the highest-priority tier represented among retained candidates is published; lower tiers do not fill unused slots. Classes grouped in the winning tier have equal priority and share the 64-result capacity.

Static and pose-derived anchor calls, polar-origin settings, ignore regions, and target-delay settings adjust which result geometry is returned. Optional anchor prediction is configured through set_anchor_prediction(...); its output and status are reported through the named prediction fields. Use it only when the script can provide reliable, correctly timestamped measurements. The full call reference is included under the installed SDK's docs directory.

Use set_apparent_distance_zoom(zoom_factor) to compensate relative distance for external camera or application zoom: 1.0 is normal view and 2.0 is 2× magnification. The value must be finite and greater than zero. Do not add compensation for Helios ROI cropping. Apparent distance is relative, not metres.

set_anchor_prediction_distance_lead(milliseconds_per_distance_unit) adds optional distance-based prediction time. It accepts finite 0..1000 and defaults to 0. Calibrate it for the application; it is not a physical-distance or gravity model.

get_detection_motion(detection) accepts a current raw detection row and returns velocity_x, velocity_y, speed, and prediction_flags. Motion is relative to the background in frame-heights per second, positive right/down. Check the VALID flag (4); flag 64 marks an estimate continuing without a current camera measurement. Warmup, disabled prediction, and nonselected detections return zero values. Copy values you need beyond the callback.

Camera Motion

Enable set_camera_motion_enabled(True) only when needed, then read get_camera_motion(). It returns start_timestamp_ns, end_timestamp_ns, displacement_x/y, variance_x/y, and valid_axes.

Displacement is background movement divided by frame height, positive right/down; camera movement has the opposite sign. Variance uses squared normalized units. Validity bit 1 is X and bit 2 is Y. Check each axis, check timestamp freshness, and avoid processing the same interval twice. Disabled, stopped, paused, or unavailable observations have zero validity; enabling or resuming requires a fresh frame pair. This is general scene motion, not a measurement of one specific cause.

For text recognition, see the OCR reference.

Model Precision

DirectML accepts object models with FP16 or FP32 inputs and outputs and uses their declared types automatically. No user precision toggle is needed. The model description reports its input/output precision. TensorRT precision is selected when building an engine; its model inputs and outputs must remain FP32. See Model Compatibility for supported families and export requirements.

Troubleshooting

• Confirm the selected model appears in list_models() and declares detect, pose, or segment.
• Confirm the Compute GPU is available in Helios Preferences.
• Start the session before waiting for results and keep engine creation out of process(frame).
• Read only detections[:num_detections] and do not retain borrowed result or mask views.
• A hard runtime failure raises inference.InferenceCoreError; do not treat it as an ordinary zero-detection frame.
• For INSTALLATION_INVALID (error code 18), install the latest complete official Helios release and check whether security software quarantined a packaged file.
• If stick-gated inference never runs, confirm the stick is 1 or 2 and its radius is 0..100.
• Report a persistent setup failure once instead of printing it every frame.
• Check the Output Panel for model or GPU errors.