videoflow.processors.vision package

The videoflow.processors.vision package contains processors that are used in the computer vision domain, such as detectors, classifiers, trackers, pose estimation, etc.

Submodules

videoflow.processors.vision.annotators module

class videoflow.processors.vision.annotators.BoundingBoxAnnotator(class_labels_path: str | None = None, class_labels_dataset: str | None = 'coco', box_color: tuple = (255, 225, 0), box_thickness: int = 2, text_color: tuple = (255, 255, 0), nb_tasks: int = 1, **kwargs)

Bases: ImageAnnotator

Draws bounding boxes on images.

• Arguments:

  • class_labels_path: path to pbtxt file that defines the labels indexes

  • class_labels_dataset: If class_labels_path is None, then we use this attribute to download the file from the releases folder. Currently supported datasets are: coco, oidv4, pascal and kitti.

  • box_color: color to use to draw the boxes

  • box_thickness: thickness of boxes to draw

  • text_color: color of text to draw

supported_datasets = ['coco', 'oidv4', 'pascal', 'kitti', 'faces']

class videoflow.processors.vision.annotators.ImageAnnotator(nb_tasks: int = 1, **kwargs)

Bases: ProcessorNode

Interface for all image annotators. All image annotators receive as input an image and annotation metadata, and return as output a copy of the image with the drawings representing the metadata.

process(im: ndarray, annotations: Any) → ndarray

Returns a copy of im visually annotated with the annotations defined in annotations

class videoflow.processors.vision.annotators.SegmenterAnnotator(class_labels_path: str | None = None, class_labels_dataset: str | None = 'coco', transparency: float = 0.5, nb_tasks: int = 1, **kwargs)

Bases: ImageAnnotator

Draws bounding boxes on images. - Arguments:

• class_labels_path: path to pbtxt file that defines the labels indexes

• class_labels_dataset: If class_labels_path is None, then we use this attribute to download the file from the releases folder. Currently supported datasets are: coco, oidv4, pascal and kitti.

• transparency: A value between 0 and 1 for the transparency of the mask

colors = [(255, 0, 0), (0, 255, 0), (0, 0, 255), (255, 255, 0), (255, 0, 255), (0, 255, 255), (255, 127.5, 0), (255, 0, 127.5), (127.5, 255, 0), (0, 255, 127.5), (127.5, 0, 255), (0, 127.5, 255)]

supported_datasets = ['coco', 'oidv4', 'pascal', 'kitti']

class videoflow.processors.vision.annotators.TrackerAnnotator(box_color: tuple = (255, 225, 0), box_thickness: int = 2, text_color: tuple = (255, 255, 255), nb_tasks: int = 1, **kwargs)

Bases: ImageAnnotator

Draws bounding boxes on images with track id.

videoflow.processors.vision.counters module

videoflow.processors.vision.detectors module

Collection of object detection processors

class videoflow.processors.vision.detectors.ObjectDetector(nb_tasks: int = 1, device_type: str = 'cpu', name: str | None = None, partition_by: str | None = None, join_policy: JoinPolicy | dict | None = None, gpu_count: int = 1, gpu_resource_name: str | None = None, **kwargs: Any)

Bases: ProcessorNode

Abstract class that defines the interface of object detectors

process(im: ndarray) → ndarray

• Arguments:

  • im (np.ndarray): (h, w, 3)

• Returns:

  • dets: np.ndarray of shape (nb_boxes, 6) Specifically (nb_boxes, [ymin, xmin, ymax, xmax, class_index, score])

    The box coordinates are returned unnormalized (values NOT between 0 and 1, but using the original dimension of the image)

videoflow.processors.vision.pose module

videoflow.processors.vision.segmentation module

class videoflow.processors.vision.segmentation.Segmenter(nb_tasks: int = 1, device_type: str = 'cpu', name: str | None = None, partition_by: str | None = None, join_policy: JoinPolicy | dict | None = None, gpu_count: int = 1, gpu_resource_name: str | None = None, **kwargs: Any)

Bases: ProcessorNode

Abstract class that defines the interface to do image segmentation in images

process(im: ndarray) → ndarray

• Arguments:

  • im (np.ndarray): (h, w, 3)

• Returns:

  • masks: np.ndarray of shape (nb_masks, h, w)

  • classes: np.ndarray of shape (nb_masks,)

  • scores: np.ndarray of shape (nb_masks,)

videoflow.processors.vision.trackers module

class videoflow.processors.vision.trackers.BoundingBoxTracker(device_type: str = 'cpu', name: str | None = None, **kwargs: Any)

Bases: OneTaskProcessorNode

Tracks bounding boxes from one frame to another. It keeps an internal state representation that allows it to track across frames.

process(dets: ndarray) → ndarray

• Arguments:

  • dets: np.ndarray of shape (nb_boxes, 6) Specifically (nb_boxes, [ymin, xmin, ymax, xmax, class_index, score])

• Returns:

  • tracks: np.ndarray of shape (nb_boxes, 5) Specifically (nb_boxes, [ymin, xmin, ymax, xmax, track_id])

videoflow.processors.vision.transformers module

class videoflow.processors.vision.transformers.CropImageTransformer(crop_dimensions: ndarray | None = None, **kwargs)

Bases: ProcessorNode

• Arguments:

  • crop_dimensions: np.ndarray of shape (nb_boxes, 4) second dimension entries are [ymin, xmin, ymax, xmax] or None

• Raises:

  • ValueError:

    • If any of crop_dimensions less than 0

    • If ymin > ymax or xmin > xmax

process(im: ndarray, crop_dimensions: ndarray | None) → List[ndarray]

Crops image according to the coordinates in crop_dimensions. If those coordinates are out of bounds, it will raise errors

• Arguments:

  • im (np.ndarray): shape of (h, w, 3)

  • crop_dimensions: np.ndarray of shape (nb_boxes, 4) second dimension entries are [ymin, xmin, ymax, xmax] or None

• Raises:

  • ValueError:

    • If any of crop_dimensions less than 0

    • If any of crop_dimensions out of bounds

    • If ymin > ymax or xmin > xmax

• Returns:

  • list of np.ndarrays: Returns a list of cropped images of the same size as crop_dimensions

class videoflow.processors.vision.transformers.MaskImageTransformer(**kwargs)

Bases: ProcessorNode

process(im: ndarray, mask: ndarray) → ndarray

Masks an image according to given masks

• Arguments:

  • im (np.ndarray): shape of (h, w, 3)

  • mask (np.ndarray): (h, w) of type np.float32, with values between zero and one

• Raises:

  • ValueError:

    • If mask does not have same height and width as im

class videoflow.processors.vision.transformers.ResizeImageTransformer(maintain_ratio: bool = False, **kwargs)

Bases: ProcessorNode

process(im: ndarray, new_size: tuple[int, int]) → ndarray

Resizes image according to coordinates in new_size

• Arguments:

  • im (np.ndarray): shape of (h, w, 3)

  • new_size (tuple): (new_height, new_width)

• Raises:

  • ValueError:

    • If new_height or new_width are negative