videoflow.utils package

Submodules

videoflow.utils.downloader module

videoflow.utils.downloader.get_file(fname: str, origin: str | list, untar: bool = False, md5_hash: str | None = None, file_hash: str | None = None, cache_subdir: str = 'models', hash_algorithm: str = 'auto', extract: bool = False, archive_format: str | list | None = 'auto', cache_dir: str | None = None) → str[source]

Copied from Keras repository.

Downloads a file from a URL if it not already in the cache. By default the file at the url origin is downloaded to the cache_dir ~/.videoflow, placed in the cache_subdir models, and given the filename fname. The final location of a file example.txt would therefore be ~/.videoflow/models/example.txt. Files in tar, tar.gz, tar.bz, and zip formats can also be extracted. Passing a hash will verify the file after download. The command line programs shasum and sha256sum can compute the hash.

  • Arguments:
    • fname: Name of the file. If an absolute path /path/to/file.txt is

      specified the file will be saved at that location.

    • origin: Original URL of the file.

    • untar: Deprecated in favor of ‘extract’.

      boolean, whether the file should be decompressed

    • md5_hash: Deprecated in favor of ‘file_hash’.

      md5 hash of the file for verification

    • file_hash: The expected hash string of the file after download.

      The sha256 and md5 hash algorithms are both supported.

    • cache_subdir: Subdirectory under the Keras cache dir where the file is

      saved. If an absolute path /path/to/folder is specified the file will be saved at that location.

    • hash_algorithm: Select the hash algorithm to verify the file.

      options are ‘md5’, ‘sha256’, and ‘auto’. The default ‘auto’ detects the hash algorithm in use.

    • extract: True tries extracting the file as an Archive, like tar or zip.

    • archive_format: Archive format to try for extracting the file.

      Options are ‘auto’, ‘tar’, ‘zip’, and None. ‘tar’ includes tar, tar.gz, and tar.bz files. The default ‘auto’ is [‘tar’, ‘zip’]. None or an empty list will return no matches found.

    • cache_dir: Location to store cached files, when None it

      defaults to the [Keras Directory](/faq/#where-is-the-keras-configuration-filed-stored).

  • Returns

    • Path to the downloaded file

videoflow.utils.downloader.validate_file(fpath: str, file_hash: str, algorithm: str = 'auto', chunk_size: int = 65535) → bool[source]

Validates a file against a sha256 or md5 hash.

  • Arguments:

    • fpath: path to the file being validated

    • file_hash: The expected hash string of the file. The sha256 and md5 hash algorithms are both supported.

    • algorithm: Hash algorithm, one of ‘auto’, ‘sha256’, or ‘md5’. The default ‘auto’ detects the hash algorithm in use.

    • chunk_size: Bytes to read at a time, important for large files.

  • Returns:

    Whether the file is valid

videoflow.utils.generic_utils module

class videoflow.utils.generic_utils.DelayedInterrupt(signals: int | list[int] | tuple[int, ...])[source]

Bases: object

class based on: http://stackoverflow.com/a/21919644/487556

It delays interrupts until the code exits the entered block

class videoflow.utils.generic_utils.DelayedKeyboardInterrupt[source]

Bases: DelayedInterrupt

class videoflow.utils.generic_utils.Progbar(target: int | None, width: int = 30, verbose: int = 1, interval: float = 0.05, stateful_metrics: Iterable[str] | None = None)[source]

Bases: object

Displays a progress bar.

  • Arguments

    • target: Total number of steps expected, None if unknown.

    • width: Progress bar width on screen.

    • verbose: Verbosity mode, 0 (silent), 1 (verbose), 2 (semi-verbose)

    • stateful_metrics: Iterable of string names of metrics that should not be averaged over time. Metrics in this list will be displayed as-is. All others will be averaged by the progbar before display.

    • interval: Minimum visual progress update interval (in seconds).

add(n: int, values: list | None = None) → None[source]
update(current: int, values: list | None = None) → None[source]

Updates the progress bar.

  • Arguments

    • current: Index of current step.

    • values: List of tuples: (name, value_for_last_step). If name is in stateful_metrics, value_for_last_step will be displayed as-is. Else, an average of the metric over time will be displayed.

videoflow.utils.graph module

videoflow.utils.graph.flatten(items: Iterable) → list[source]

Returns flattened iterable from any nested iterable

videoflow.utils.graph.has_cycle(producers: Sequence[Node]) → bool[source]

Used to detect if the graph is not acyclical. Returns true if it finds a cycle in the graph. It begins exploring the graph from producers down all the way to consumers.

videoflow.utils.graph.topological_sort(producers: Sequence[Node]) → list[Node][source]

Creates a topological sort of the computation graph.

  • Arguments:

    • producers: a list of producer nodes, that is, nodes with no parents.

  • Returns:

    • stack: a list of nodes in topological order. If a node A appears before a node B on the list, it means that node A does not depend on node B output

videoflow.utils.parsers module

videoflow.utils.parsers.parse_label_map(path_to_labels: str) → dict[int, str][source]
  • Arguments:

    • path_to_labels (str): path to pbtx file

  • Returns:

    • dict of form { id(int) : label(str)}

videoflow.utils.plugins module

Third-party registration via importlib.metadata entry points.

Videoflow’s extension points are plain dict/list registries with an explicit register_*() function, pre-seeded with the built-ins. That covers the common case, where the code doing the registering is imported anyway (a component module the worker loads via VF_NODE_CLASS registers its own payload codecs on import).

It does not cover the case where nothing imports the extension first — a blob store selected by URL scheme in a config file, say, or a host-side videoflow debug decode that must understand a vendor payload type without knowing which package defines it. For those, a package declares an entry point and videoflow imports it on demand.

Stdlib only, by design: an extension mechanism that needs its own dependency is a worse trade than the if/elif it replaces.

videoflow.utils.plugins.load_plugin_group(group: str) → None[source]

Imports every extension registered in the importlib.metadata entry-point group group, so its register_*() side effects take hold. Idempotent: a group is scanned at most once per process.

Each entry point may resolve to either a module (imported for its top-level register_*() calls) or a callable (imported and then called). A plugin that raises is logged and skipped rather than propagated — one broken third-party package must not stop a worker from starting, since the registration it was providing may not even be needed by this flow.

  • Arguments:

    • group: entry-point group name, e.g. 'videoflow.blob_stores'.

videoflow.utils.system module

Host GPU inspection.

The one place videoflow enumerates physical GPU ordinals. Two faces of the multi-GPU visibility contract (RFC 0003) live here: granted_gpus() is the component-facing side — inside a worker, the visible devices are exactly the granted devices, numbered 0..n-1 — and visible_physical_gpus() is the engine-facing side, the pool of physical ordinals the local engine partitions across workers (engines/local.py). Kubernetes workers normally don’t need either (the device plugin already masks the container to its grant), but non-CUDA runtimes and native wrappers can call granted_gpus() instead of parsing CUDA_VISIBLE_DEVICES themselves.

videoflow.utils.system.NVIDIA_SMI_QUERY = ('--query-gpu=index,uuid,name,memory.total,memory.used', '--format=csv,noheader,nounits')

one CSV row per physical GPU, memory in MiB (nounits). -L adds the MIG UUIDs.

Type:

The nvidia-smi query the local allocation backend inventories from

videoflow.utils.system.NVIDIA_SMI_TIMEOUT_SECONDS = 5

Bound on the nvidia-smi probe — a wedged driver otherwise hangs the local engine’s launch path indefinitely. Normal responses take well under 1s.

videoflow.utils.system.apply_mask(devices: Sequence[DeviceIdentity], mask: Sequence[str] | None) → list[DeviceIdentity][source]

The devices a CUDA_VISIBLE_DEVICES mask exposes, in mask order, as the CUDA runtime resolves it: an integer entry is a physical ordinal, a GPU-… entry a card UUID (a unique prefix suffices), a MIG-… entry a MIG instance UUID. The runtime’s rules, as observed on driver R595 (plan Phase 4, ALLOC-015): enumeration stops at the first entry that names no device — an unknown UUID, an out-of-range ordinal, or an entry of the other form in a mixed list — and a mask naming one device twice is invalid as a whole (cudaErrorInvalidDevice: nothing is exposed). None (unset) exposes every physical card; MIG instances are never exposed implicitly.

videoflow.utils.system.get_gpus_available_to_process() → list[int]

Deprecated pre-RFC-0003 name for granted_gpus().

videoflow.utils.system.get_number_of_gpus() → int[source]

The number of physical GPUs in the system, 0 when nvidia-smi is missing, failing or hung. Counts the GPU <n>: lines of nvidia-smi -L, so MIG instances are not enumerated: a MIG-enabled card counts once, even though its compute is only reachable through its MIG devices, which videoflow does not address locally (see visible_physical_gpus()).

videoflow.utils.system.get_system_gpus() → set[int][source]

Returns the ids of gpus in the machine as a set of integers

videoflow.utils.system.granted_gpus() → list[int][source]

The calling process’s granted devices as valid CUDA indices [0..n-1] (RFC 0003). Correct in a pod (the device plugin renumbers the grant) and locally (CUDA renumbers the CUDA_VISIBLE_DEVICES mask), so a component can both check len(granted_gpus()) against a hard device minimum in open() and place work with .to(f'cuda:{i}') for each entry i.

Under an integer mask (or none) this counts whole physical cards. Under a UUID mask it counts the mask’s resolved entries — one per GPU-… card and one per MIG-… instance, in mask order, which is what CUDA enumerates (whether a driver exposes more than one MIG instance to a process is the driver’s call; the count reports what the mask names, and runtime.gpucheck reports what was delivered).

videoflow.utils.system.host_devices_observed() → Known[list[DeviceIdentity]] | Unknown[source]

Every physical GPU (and MIG instance) of this host, with UUIDs and memory — or Unknown when nvidia-smi is absent, failing or hung. Never “no GPUs” for a failed read: the local allocation backend refuses to grant on an unobserved host rather than under-deliver silently (ALLOC-014).

videoflow.utils.system.mask_entries(value: str | None) → list[str] | None[source]

The entries of a CUDA_VISIBLE_DEVICES value (None when unset; [] for the hide-all idiom '').

videoflow.utils.system.parse_smi_list(text: str) → list[DeviceIdentity][source]

nvidia-smi -L as identities: one per physical GPU, and one per MIG instance (mig_uuid set, mig_profile its 1g.24gb-style name, uuid its parent card’s). A MIG-enabled card’s compute is reachable only through its instances, which is why they are enumerated here and not by get_number_of_gpus.

videoflow.utils.system.parse_smi_query(csv_text: str) → list[DeviceIdentity][source]

nvidia-smi --query-gpu=index,uuid,name,memory.total,memory.used rows as identities (memory in bytes).

videoflow.utils.system.parse_smi_used(csv_text: str) → dict[str, int][source]

uuid -> memory.used in bytes from the same query, for peak-memory admission.

videoflow.utils.system.visible_devices() → list[DeviceIdentity][source]

The devices the calling process can address, honouring an inherited CUDA_VISIBLE_DEVICES by ordinal, card UUID or MIG UUID (RFC 0006 plan Phase 4; the identity-carrying counterpart of visible_physical_gpus). Empty when the host cannot be observed: callers that must tell “no GPUs” from “no answer” use host_devices_observed.

videoflow.utils.system.visible_physical_gpus() → list[int][source]

The physical device ordinals visible to the calling process, sorted: the system’s GPUs intersected with CUDA_VISIBLE_DEVICES (unset ⇒ all of them). A mask with GPU-…/MIG-… entries is resolved the way the CUDA runtime resolves it (apply_mask: identities via nvidia-smi -L, and nothing past the first entry that names no device), with a warning naming the entries the runtime would not expose. Engine-facing: this is the pool the local engine partitions into per-worker CUDA_VISIBLE_DEVICES masks (engines/local.py). These are not valid cuda:i indices under a CUDA_VISIBLE_DEVICES mask — CUDA renumbers the masked devices to 0..n-1; components should use granted_gpus() instead.

MIG caveat: a MIG-enabled card contributes a single ordinal — its instances are not enumerated, and an integer CUDA_VISIBLE_DEVICES entry cannot address a MIG slice (slices are pinned by MIG-<uuid> or granted by the Kubernetes device plugin), so the local engine never partitions below a whole card.

videoflow.utils.transforms module

videoflow.utils.transforms.resize_add_padding(im: ndarray, t_h: int, t_w: int) → ndarray[source]

Resizes an image to a target size, adding padding if necessary to maintain the aspect ratio - Arguments:

  • im (np.ndarray): shape (h, w, 3)

  • t_h (int): target height

  • t_w (int): target width