videoflow.runtime package

Submodules

videoflow.runtime.assetcheck module

Worker-side verification of a node’s declared assets (plan Phase 4, RUN-033).

A model file on a hostPath exists on the host that has it, not on the host a rescheduled pod landed on; a claim or a download can put the bytes everywhere, but only a digest says they are the declared bytes. So before a node is opened the worker walks node.required_assets() and refuses — with ResourceUnavailable, naming the host, the path and the fix — to process with a missing or different asset. Nothing here fetches or distributes: the deployment (a PVC, an image layer, prepare.py) owns distribution; the runtime only verifies identity.

class videoflow.runtime.assetcheck.AssetReport(path: str, expected: str, actual: str | None, portable: bool, status: str)

Bases: object

actual: str | None

expected: str

path: str

portable: bool

status: str

videoflow.runtime.assetcheck.check_assets(requirements: Sequence[AssetRequirement]) → list[AssetReport]

One report per requirement; never raises — verify_assets decides.

videoflow.runtime.assetcheck.sha256_of(path: str) → str

videoflow.runtime.assetcheck.verify_assets(requirements: Sequence[AssetRequirement], node_name: str = 'node') → list[AssetReport]

• Raises:

  • ResourceUnavailable: an asset is missing on this host or its bytes are not the declared ones. The remedy distinguishes a local-only asset (pin the worker to the host that has it, or make it portable) from a portable one (the distribution delivered other bytes).

videoflow.runtime.assetcheck.verify_node_assets(node: Node) → list[AssetReport]

The worker’s entry point: verify_assets over the node’s declaration.

videoflow.runtime.gpucheck module

Worker-side verification of a GPU grant: what the node was granted versus what the process can actually see, decided before the node is opened.

The allocator (a Kubernetes device plugin, the local backend, one day a DRA driver) decides which devices a worker gets; this module never allocates. It answers the component-side questions RFC 0006’s plan (Phase 4, RUN-039/043/044) puts to a worker:

• How many usable devices did I receive, by which identities? The DeliveredGrant the launcher recorded (VF_GPU_GRANT_JSON) when there is one; otherwise the CUDA-visible namespace enumerated through the inherited CUDA_VISIBLE_DEVICES mask by ordinal, card UUID or MIG UUID — never by counting integers in the mask, which is how a valid UUID grant used to read as “no GPUs”.

• Is a short grant fatal? Only when the node says so (gpu_fallback = 'none'); by default a shortfall is reported, the CPU fallback is named, and the node opens.

• Does the grant have the topology the node requires? requires_peer_access is verified with nvidia-smi topo -p2p r (driver-level P2P capability, no CUDA dependency); a count-only grant cannot satisfy it silently.

What could not be observed stays “unknown” in the report: a missing nvidia-smi in a CPU image is not evidence of zero devices.

class videoflow.runtime.gpucheck.GrantReport(requested: int, delivered: tuple[~videoflow.backends.allocation.DeviceIdentity, ...], source: str, execution: str, exclusive: bool, mask: str | None, peer_access: ~videoflow.backends.outcomes.Known[bool] | ~videoflow.backends.outcomes.Unknown | None = None, problems: tuple[str, ...] = (), evidence: ~typing.Mapping[str, ~typing.Any] = <factory>)

Bases: object

• Arguments:

  • requested: the node’s gpu_count (0 for a CPU node).

  • delivered: the devices the worker can use, in CUDA index order — the grant’s when it was observed, else the enumerated namespace.

  • source: grant (a launcher-recorded DeliveredGrant), enumeration (nvidia-smi through the mask) or unobserved (neither answered).

  • execution: gpu, cpu (declared fallback taken) or unknown.

  • exclusive: whether the launcher promised exclusive devices (never assumed).

  • peer_access: Known(True/False) once verified, Unknown otherwise.

  • problems: every shortfall found, in words; empty when the grant is whole.

delivered: tuple[DeviceIdentity, ...]

evidence: Mapping[str, Any]

exclusive: bool

execution: str

mask: str | None

peer_access: Known[bool] | Unknown | None = None

problems: tuple[str, ...] = ()

requested: int

source: str

to_dict() → dict[str, Any]

videoflow.runtime.gpucheck.enumerate_devices(environ: Mapping[str, str] = environ({'ACCEPT_EULA': 'Y', 'ACTIONS_ID_TOKEN_REQUEST_TOKEN': 'eyJhbGciOiJSUzI1NiIsImtpZCI6IjM4ODI2YjE3LTZhMzAtNWY5Yi1iMTY5LThiZWI4MjAyZjcyMyIsInR5cCI6IkpXVCIsIng1dCI6InlrTmFZNHFNX3RhNGsyVGdaT0NFWUxrY1lsQSJ9.eyJJZGVudGl0eVR5cGVDbGFpbSI6IlN5c3RlbTpTZXJ2aWNlSWRlbnRpdHkiLCJhYyI6Ilt7XCJTY29wZVwiOlwicmVmcy9oZWFkcy9tYXN0ZXJcIixcIlBlcm1pc3Npb25cIjozfV0iLCJhY3NsIjoiMTAiLCJhdWQiOiJ2c286Y2ZmMGE0YWEtZjA3My00ZGJjLTliODQtM2IxNzFmYjRlYTM3IiwiYmlsbGluZ19vd25lcl9pZCI6Ik9fa2dET0F3bC1ydyIsImV4cCI6MTc4OTQyMjk4MSwiaHR0cDovL3NjaGVtYXMubWljcm9zb2Z0LmNvbS93cy8yMDA4LzA2L2lkZW50aXR5L2NsYWltcy9wcmltYXJ5c2lkIjoiZGRkZGRkZGQtZGRkZC1kZGRkLWRkZGQtZGRkZGRkZGRkZGRkIiwiaHR0cDovL3NjaGVtYXMueG1sc29hcC5vcmcvd3MvMjAwNS8wNS9pZGVudGl0eS9jbGFpbXMvc2lkIjoiZGRkZGRkZGQtZGRkZC1kZGRkLWRkZGQtZGRkZGRkZGRkZGRkIiwiaWF0IjoxNzg5NDAwNzgxLCJpc3MiOiJodHRwczovL3Rva2VuLmFjdGlvbnMuZ2l0aHVidXNlcmNvbnRlbnQuY29tIiwiam9iX2lkIjoiZTcxMDY3NWEtZWQ1NC01NTMzLWIwYWYtMDFkZjE1MjY1ODNhIiwiam9iX3dvcmtmbG93X3JlZiI6InZpZGVvZmxvdy92aWRlb2Zsb3cvLmdpdGh1Yi93b3JrZmxvd3MvZG9jcy55bWxAcmVmcy9oZWFkcy9tYXN0ZXIiLCJqb2Jfd29ya2Zsb3dfc2hhIjoiYmFjYThmYTkwMWQ4OGIwN2YwYzBlM2YwMGE0ZjhhNDI2MzYxMTU1MSIsIm5hbWVpZCI6ImRkZGRkZGRkLWRkZGQtZGRkZC1kZGRkLWRkZGRkZGRkZGRkZCIsIm5iZiI6MTc4OTQwMDQ4MSwib2lkY19leHRyYSI6IntcImFjdG9yXCI6XCJqYWRpZWxhbVwiLFwiYWN0b3JfaWRcIjpcIjEwMjYwNjdcIixcImJhc2VfcmVmXCI6XCJcIixcImNoZWNrX3J1bl9pZFwiOlwiMTA0MDQ0Mjc2NzcyXCIsXCJldmVudF9uYW1lXCI6XCJwdXNoXCIsXCJoZWFkX3JlZlwiOlwiXCIsXCJqb2Jfd29ya2Zsb3dfcmVmXCI6XCJ2aWRlb2Zsb3cvdmlkZW9mbG93Ly5naXRodWIvd29ya2Zsb3dzL2RvY3MueW1sQHJlZnMvaGVhZHMvbWFzdGVyXCIsXCJqb2Jfd29ya2Zsb3dfc2hhXCI6XCJiYWNhOGZhOTAxZDg4YjA3ZjBjMGUzZjAwYTRmOGE0MjYzNjExNTUxXCIsXCJyZWZcIjpcInJlZnMvaGVhZHMvbWFzdGVyXCIsXCJyZWZfcHJvdGVjdGVkXCI6XCJmYWxzZVwiLFwicmVmX3R5cGVcIjpcImJyYW5jaFwiLFwicmVwb3NpdG9yeVwiOlwidmlkZW9mbG93L3ZpZGVvZmxvd1wiLFwicmVwb3NpdG9yeV9pZFwiOlwiMTgxNTU0OTM5XCIsXCJyZXBvc2l0b3J5X293bmVyXCI6XCJ2aWRlb2Zsb3dcIixcInJlcG9zaXRvcnlfb3duZXJfaWRcIjpcIjUwOTUzOTAzXCIsXCJyZXBvc2l0b3J5X3Zpc2liaWxpdHlcIjpcInB1YmxpY1wiLFwicnVuX2F0dGVtcHRcIjpcIjFcIixcInJ1bl9pZFwiOlwiMzQ4NjQzNDM5NjBcIixcInJ1bl9udW1iZXJcIjpcIjExXCIsXCJydW5uZXJfZW52aXJvbm1lbnRcIjpcImdpdGh1Yi1ob3N0ZWRcIixcInNoYVwiOlwiYmFjYThmYTkwMWQ4OGIwN2YwYzBlM2YwMGE0ZjhhNDI2MzYxMTU1MVwiLFwid29ya2Zsb3dcIjpcIkRvY3NcIixcIndvcmtmbG93X3JlZlwiOlwidmlkZW9mbG93L3ZpZGVvZmxvdy8uZ2l0aHViL3dvcmtmbG93cy9kb2NzLnltbEByZWZzL2hlYWRzL21hc3RlclwiLFwid29ya2Zsb3dfc2hhXCI6XCJiYWNhOGZhOTAxZDg4YjA3ZjBjMGUzZjAwYTRmOGE0MjYzNjExNTUxXCJ9Iiwib2lkY19zdWIiOiJyZXBvOnZpZGVvZmxvdy92aWRlb2Zsb3c6cmVmOnJlZnMvaGVhZHMvbWFzdGVyIiwib3JjaF9pZCI6IjU5OTczN2Q4LWE4MjEtNDViNS05NjhkLWQ4MTI1N2VkZmQwZi5idWlsZC5fX2RlZmF1bHQiLCJvd25lcl9pZCI6Ik9fa2dET0F3bC1ydyIsInBsYW5faWQiOiI1OTk3MzdkOC1hODIxLTQ1YjUtOTY4ZC1kODEyNTdlZGZkMGYiLCJyZXBvc2l0b3J5X2lkIjoiMTgxNTU0OTM5IiwicmVwb3NpdG9yeV9vd25lcl9pZCI6IjUwOTUzOTAzIiwicmVwb3NpdG9yeV92aXNpYmlsaXR5IjoicHVibGljIiwicnVuX2lkIjoiMzQ4NjQzNDM5NjAiLCJydW5fbnVtYmVyIjoiMTEiLCJydW5fdHlwZSI6ImZ1bGwiLCJydW5uZXJfaWQiOiIxMDAwMDAwMTY1IiwicnVubmVyX3R5cGUiOiJob3N0ZWQiLCJzY3AiOiJBY3Rpb25zLlJlc3VsdHM6NTk5NzM3ZDgtYTgyMS00NWI1LTk2OGQtZDgxMjU3ZWRmZDBmOmU3MTA2NzVhLWVkNTQtNTUzMy1iMGFmLTAxZGYxNTI2NTgzYSBBY3Rpb25zLlJ1bm5lcjo1OTk3MzdkOC1hODIxLTQ1YjUtOTY4ZC1kODEyNTdlZGZkMGY6ZTcxMDY3NWEtZWQ1NC01NTMzLWIwYWYtMDFkZjE1MjY1ODNhIEFjdGlvbnMuVXBsb2FkQXJ0aWZhY3RzOjU5OTczN2Q4LWE4MjEtNDViNS05NjhkLWQ4MTI1N2VkZmQwZjplNzEwNjc1YS1lZDU0LTU1MzMtYjBhZi0wMWRmMTUyNjU4M2EgZ2VuZXJhdGVfaWRfdG9rZW46NTk5NzM3ZDgtYTgyMS00NWI1LTk2OGQtZDgxMjU3ZWRmZDBmOmU3MTA2NzVhLWVkNTQtNTUzMy1iMGFmLTAxZGYxNTI2NTgzYSBBY3Rpb25zLkdlbmVyaWNSZWFkOjAwMDAwMDAwLTAwMDAtMDAwMC0wMDAwLTAwMDAwMDAwMDAwMCIsInNoYSI6ImJhY2E4ZmE5MDFkODhiMDdmMGMwZTNmMDBhNGY4YTQyNjM2MTE1NTEiLCJ0cnVzdF90aWVyIjoiMiJ9.odJ3vkzgRiNUCihXdyP65ZEvROIonxWdwBRZs1HbSmKcjM5eI-y6DKM1DmgqGX5_zmHqhcvJjQoQz1AQsgwkp-u5uIzFFLB0URMJK6x3xztWigmBI5mxSJgfBBYBndkRORJrkm6HYrkDfCF458M1oORSC1VMt2-qHEH9Lwp1heegCNjZ_PUv0SC6KCWCdoqMQpI-x1o66H0BxaufIW-OOM_Yprtyv_PrLLTfevwxL6r2uBxU_J8-qtayR1fz5n2mwPmlS5SbtN9vitatOU97yuSZY2q-ecQwbZbyt4Ny_BjduyjBTh-Zgt8KfHdrYZjDe7kk12Us7QKTixCiP__Kvw', 'ACTIONS_ID_TOKEN_REQUEST_URL': 'https://run-actions-3-azure-eastus.actions.githubusercontent.com/139//idtoken/599737d8-a821-45b5-968d-d81257edfd0f/e710675a-ed54-5533-b0af-01df1526583a?api-version=2.0', 'ACTIONS_ORCHESTRATION_ID': '599737d8-a821-45b5-968d-d81257edfd0f.build.__default', 'ACTIONS_RUNNER_ACTION_ARCHIVE_CACHE': '/opt/actionarchivecache', 'ACTIONS_RUNNER_RETURN_JOB_RESULT_FOR_HOSTED': '1', 'AGENT_TOOLSDIRECTORY': '/opt/hostedtoolcache', 'ANDROID_HOME': '/usr/local/lib/android/sdk', 'ANDROID_NDK': '/usr/local/lib/android/sdk/ndk/27.3.13750724', 'ANDROID_NDK_HOME': '/usr/local/lib/android/sdk/ndk/27.3.13750724', 'ANDROID_NDK_LATEST_HOME': '/usr/local/lib/android/sdk/ndk/29.0.14206865', 'ANDROID_NDK_ROOT': '/usr/local/lib/android/sdk/ndk/27.3.13750724', 'ANDROID_SDK_ROOT': '/usr/local/lib/android/sdk', 'ANT_HOME': '/usr/share/ant', 'AZURE_EXTENSION_DIR': '/opt/az/azcliextensions', 'BOOTSTRAP_HASKELL_NONINTERACTIVE': '1', 'CHROMEWEBDRIVER': '/usr/local/share/chromedriver-linux64', 'CHROME_BIN': '/usr/bin/google-chrome', 'CI': 'true', 'CONDA': '/usr/share/miniconda', 'DEBIAN_FRONTEND': 'noninteractive', 'DOTNET_MULTILEVEL_LOOKUP': '0', 'DOTNET_NOLOGO': '1', 'DOTNET_SKIP_FIRST_TIME_EXPERIENCE': '1', 'EDGEWEBDRIVER': '/usr/local/share/edge_driver', 'ENABLE_RUNNER_TRACING': 'true', 'GECKOWEBDRIVER': '/usr/local/share/gecko_driver', 'GHCUP_INSTALL_BASE_PREFIX': '/usr/local', 'GITHUB_ACTION': '__run_2', 'GITHUB_ACTIONS': 'true', 'GITHUB_ACTION_REF': '', 'GITHUB_ACTION_REPOSITORY': '', 'GITHUB_ACTOR': 'jadielam', 'GITHUB_ACTOR_ID': '1026067', 'GITHUB_API_URL': 'https://api.github.com', 'GITHUB_ARTIFACTS': '/home/runner/work/_temp/_runner_file_commands/artifacts_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_ARTIFACTS_LIST': '/home/runner/work/_temp/_runner_file_commands/artifacts_list_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_BASE_REF': '', 'GITHUB_ENV': '/home/runner/work/_temp/_runner_file_commands/set_env_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_EVENT_NAME': 'push', 'GITHUB_EVENT_PATH': '/home/runner/work/_temp/_github_workflow/event.json', 'GITHUB_GRAPHQL_URL': 'https://api.github.com/graphql', 'GITHUB_HEAD_REF': '', 'GITHUB_JOB': 'build', 'GITHUB_OUTPUT': '/home/runner/work/_temp/_runner_file_commands/set_output_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_PATH': '/home/runner/work/_temp/_runner_file_commands/add_path_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_REF': 'refs/heads/master', 'GITHUB_REF_NAME': 'master', 'GITHUB_REF_PROTECTED': 'false', 'GITHUB_REF_TYPE': 'branch', 'GITHUB_REPOSITORY': 'videoflow/videoflow', 'GITHUB_REPOSITORY_ID': '181554939', 'GITHUB_REPOSITORY_OWNER': 'videoflow', 'GITHUB_REPOSITORY_OWNER_ID': '50953903', 'GITHUB_RETENTION_DAYS': '90', 'GITHUB_RUN_ATTEMPT': '1', 'GITHUB_RUN_ID': '34864343960', 'GITHUB_RUN_NUMBER': '11', 'GITHUB_SERVER_URL': 'https://github.com', 'GITHUB_SHA': 'baca8fa901d88b07f0c0e3f00a4f8a4263611551', 'GITHUB_STATE': '/home/runner/work/_temp/_runner_file_commands/save_state_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_STEP_SUMMARY': '/home/runner/work/_temp/_runner_file_commands/step_summary_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_TRIGGERING_ACTOR': 'jadielam', 'GITHUB_WORKFLOW': 'Docs', 'GITHUB_WORKFLOW_REF': 'videoflow/videoflow/.github/workflows/docs.yml@refs/heads/master', 'GITHUB_WORKFLOW_SHA': 'baca8fa901d88b07f0c0e3f00a4f8a4263611551', 'GITHUB_WORKSPACE': '/home/runner/work/videoflow/videoflow', 'GOROOT_1_24_X64': '/opt/hostedtoolcache/go/1.24.13/x64', 'GOROOT_1_25_X64': '/opt/hostedtoolcache/go/1.25.14/x64', 'GOROOT_1_26_X64': '/opt/hostedtoolcache/go/1.26.8/x64', 'GRADLE_HOME': '/usr/share/gradle-9.7.1', 'HCA_CLOUD_PROVIDER': 'azure', 'HOME': '/home/runner', 'HOMEBREW_CLEANUP_PERIODIC_FULL_DAYS': '3650', 'HOMEBREW_NO_AUTO_UPDATE': '1', 'INVOCATION_ID': '15d8f2ea8d044311994759f59f0fbc0e', 'ImageOS': 'ubuntu24', 'ImageVersion': '20260907.300.1', 'JAVA_HOME': '/usr/lib/jvm/temurin-17-jdk-amd64', 'JAVA_HOME_11_X64': '/usr/lib/jvm/temurin-11-jdk-amd64', 'JAVA_HOME_17_X64': '/usr/lib/jvm/temurin-17-jdk-amd64', 'JAVA_HOME_21_X64': '/usr/lib/jvm/temurin-21-jdk-amd64', 'JAVA_HOME_25_X64': '/usr/lib/jvm/temurin-25-jdk-amd64', 'JAVA_HOME_8_X64': '/usr/lib/jvm/temurin-8-jdk-amd64', 'JOURNAL_STREAM': '9:12689', 'LANG': 'C.UTF-8', 'LOGNAME': 'runner', 'MEMORY_PRESSURE_WATCH': '/sys/fs/cgroup/system.slice/hosted-compute-agent.service/memory.pressure', 'MEMORY_PRESSURE_WRITE': 'c29tZSAyMDAwMDAgMjAwMDAwMAA=', 'NVM_DIR': '/home/runner/.nvm', 'OLDPWD': '/home/runner/work/videoflow/videoflow', 'PATH': '/home/runner/work/_temp/setup-uv-cache/builds-v0/.tmpBKyaem/bin:/home/runner/work/_temp/setup-uv-cache/archive-v0/yrD2oxi3dSpikfW8/bin:/home/runner/work/videoflow/videoflow/.venv/bin:/opt/hostedtoolcache/uv/0.12.13/x86_64:/home/runner/.local/bin:/snap/bin:/home/runner/.local/bin:/opt/pipx_bin:/home/runner/.cargo/bin:/home/runner/.config/composer/vendor/bin:/usr/local/.ghcup/bin:/home/runner/.dotnet/tools:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin', 'PIPX_BIN_DIR': '/opt/pipx_bin', 'PIPX_HOME': '/opt/pipx', 'POWERSHELL_DISTRIBUTION_CHANNEL': 'GitHub-Actions-Linux', 'PSModulePath': '/root/.local/share/powershell/Modules:/usr/local/share/powershell/Modules:/opt/microsoft/powershell/7/Modules:/usr/share/az_15.6.1', 'PWD': '/home/runner/work/videoflow/videoflow', 'RUNNER_ARCH': 'X64', 'RUNNER_ENVIRONMENT': 'github-hosted', 'RUNNER_NAME': 'GitHub Actions 1000000165', 'RUNNER_OS': 'Linux', 'RUNNER_TEMP': '/home/runner/work/_temp', 'RUNNER_TOOL_CACHE': '/opt/hostedtoolcache', 'RUNNER_TRACKING_ID': 'github_f645d36d-cf28-422d-aa73-164bbe3d2cec', 'RUNNER_WORKSPACE': '/home/runner/work/videoflow', 'SELENIUM_JAR_PATH': '/usr/share/java/selenium-server.jar', 'SGX_AESM_ADDR': '1', 'SHELL': '/bin/bash', 'SHLVL': '2', 'SWIFT_PATH': '/usr/share/swift/usr/bin', 'SYSTEMD_EXEC_PID': '1809', 'USER': 'runner', 'USE_BAZEL_FALLBACK_VERSION': 'silent:', 'UV': '/opt/hostedtoolcache/uv/0.12.13/x86_64/uv', 'UV_CACHE_DIR': '/home/runner/work/_temp/setup-uv-cache', 'UV_RUN_RECURSION_DEPTH': '1', 'VCPKG_INSTALLATION_ROOT': '/usr/local/share/vcpkg', 'VIRTUAL_ENV': '/home/runner/work/_temp/setup-uv-cache/builds-v0/.tmpBKyaem', 'XDG_CONFIG_HOME': '/home/runner/.config', 'XDG_RUNTIME_DIR': '/run/user/1001', '_': '/opt/hostedtoolcache/uv/0.12.13/x86_64/uv', 'DOCUTILSCONFIG': '/home/runner/work/videoflow/videoflow/docs/source/docutils.conf'})) → Known[list[DeviceIdentity]] | Unknown

The devices this process can address through its mask, or Unknown when the host cannot be read.

videoflow.runtime.gpucheck.parse_p2p_matrix(text: str) → dict[tuple[int, int], str]

nvidia-smi topo -p2p r as (i, j) -> status (OK, NS, CNS, GNS, X).

videoflow.runtime.gpucheck.peer_access_observed(devices: tuple[DeviceIdentity, ...]) → Known[bool] | Unknown

Whether every pair of the granted physical devices reports P2P read capability OK in nvidia-smi topo -p2p r. Unknown when the matrix cannot be read or a device has no host ordinal (a MIG slice never has peer access; two slices of one card are not peers).

videoflow.runtime.gpucheck.report_backend(requested: int, environ: Mapping[str, str] = environ({'ACCEPT_EULA': 'Y', 'ACTIONS_ID_TOKEN_REQUEST_TOKEN': 'eyJhbGciOiJSUzI1NiIsImtpZCI6IjM4ODI2YjE3LTZhMzAtNWY5Yi1iMTY5LThiZWI4MjAyZjcyMyIsInR5cCI6IkpXVCIsIng1dCI6InlrTmFZNHFNX3RhNGsyVGdaT0NFWUxrY1lsQSJ9.eyJJZGVudGl0eVR5cGVDbGFpbSI6IlN5c3RlbTpTZXJ2aWNlSWRlbnRpdHkiLCJhYyI6Ilt7XCJTY29wZVwiOlwicmVmcy9oZWFkcy9tYXN0ZXJcIixcIlBlcm1pc3Npb25cIjozfV0iLCJhY3NsIjoiMTAiLCJhdWQiOiJ2c286Y2ZmMGE0YWEtZjA3My00ZGJjLTliODQtM2IxNzFmYjRlYTM3IiwiYmlsbGluZ19vd25lcl9pZCI6Ik9fa2dET0F3bC1ydyIsImV4cCI6MTc4OTQyMjk4MSwiaHR0cDovL3NjaGVtYXMubWljcm9zb2Z0LmNvbS93cy8yMDA4LzA2L2lkZW50aXR5L2NsYWltcy9wcmltYXJ5c2lkIjoiZGRkZGRkZGQtZGRkZC1kZGRkLWRkZGQtZGRkZGRkZGRkZGRkIiwiaHR0cDovL3NjaGVtYXMueG1sc29hcC5vcmcvd3MvMjAwNS8wNS9pZGVudGl0eS9jbGFpbXMvc2lkIjoiZGRkZGRkZGQtZGRkZC1kZGRkLWRkZGQtZGRkZGRkZGRkZGRkIiwiaWF0IjoxNzg5NDAwNzgxLCJpc3MiOiJodHRwczovL3Rva2VuLmFjdGlvbnMuZ2l0aHVidXNlcmNvbnRlbnQuY29tIiwiam9iX2lkIjoiZTcxMDY3NWEtZWQ1NC01NTMzLWIwYWYtMDFkZjE1MjY1ODNhIiwiam9iX3dvcmtmbG93X3JlZiI6InZpZGVvZmxvdy92aWRlb2Zsb3cvLmdpdGh1Yi93b3JrZmxvd3MvZG9jcy55bWxAcmVmcy9oZWFkcy9tYXN0ZXIiLCJqb2Jfd29ya2Zsb3dfc2hhIjoiYmFjYThmYTkwMWQ4OGIwN2YwYzBlM2YwMGE0ZjhhNDI2MzYxMTU1MSIsIm5hbWVpZCI6ImRkZGRkZGRkLWRkZGQtZGRkZC1kZGRkLWRkZGRkZGRkZGRkZCIsIm5iZiI6MTc4OTQwMDQ4MSwib2lkY19leHRyYSI6IntcImFjdG9yXCI6XCJqYWRpZWxhbVwiLFwiYWN0b3JfaWRcIjpcIjEwMjYwNjdcIixcImJhc2VfcmVmXCI6XCJcIixcImNoZWNrX3J1bl9pZFwiOlwiMTA0MDQ0Mjc2NzcyXCIsXCJldmVudF9uYW1lXCI6XCJwdXNoXCIsXCJoZWFkX3JlZlwiOlwiXCIsXCJqb2Jfd29ya2Zsb3dfcmVmXCI6XCJ2aWRlb2Zsb3cvdmlkZW9mbG93Ly5naXRodWIvd29ya2Zsb3dzL2RvY3MueW1sQHJlZnMvaGVhZHMvbWFzdGVyXCIsXCJqb2Jfd29ya2Zsb3dfc2hhXCI6XCJiYWNhOGZhOTAxZDg4YjA3ZjBjMGUzZjAwYTRmOGE0MjYzNjExNTUxXCIsXCJyZWZcIjpcInJlZnMvaGVhZHMvbWFzdGVyXCIsXCJyZWZfcHJvdGVjdGVkXCI6XCJmYWxzZVwiLFwicmVmX3R5cGVcIjpcImJyYW5jaFwiLFwicmVwb3NpdG9yeVwiOlwidmlkZW9mbG93L3ZpZGVvZmxvd1wiLFwicmVwb3NpdG9yeV9pZFwiOlwiMTgxNTU0OTM5XCIsXCJyZXBvc2l0b3J5X293bmVyXCI6XCJ2aWRlb2Zsb3dcIixcInJlcG9zaXRvcnlfb3duZXJfaWRcIjpcIjUwOTUzOTAzXCIsXCJyZXBvc2l0b3J5X3Zpc2liaWxpdHlcIjpcInB1YmxpY1wiLFwicnVuX2F0dGVtcHRcIjpcIjFcIixcInJ1bl9pZFwiOlwiMzQ4NjQzNDM5NjBcIixcInJ1bl9udW1iZXJcIjpcIjExXCIsXCJydW5uZXJfZW52aXJvbm1lbnRcIjpcImdpdGh1Yi1ob3N0ZWRcIixcInNoYVwiOlwiYmFjYThmYTkwMWQ4OGIwN2YwYzBlM2YwMGE0ZjhhNDI2MzYxMTU1MVwiLFwid29ya2Zsb3dcIjpcIkRvY3NcIixcIndvcmtmbG93X3JlZlwiOlwidmlkZW9mbG93L3ZpZGVvZmxvdy8uZ2l0aHViL3dvcmtmbG93cy9kb2NzLnltbEByZWZzL2hlYWRzL21hc3RlclwiLFwid29ya2Zsb3dfc2hhXCI6XCJiYWNhOGZhOTAxZDg4YjA3ZjBjMGUzZjAwYTRmOGE0MjYzNjExNTUxXCJ9Iiwib2lkY19zdWIiOiJyZXBvOnZpZGVvZmxvdy92aWRlb2Zsb3c6cmVmOnJlZnMvaGVhZHMvbWFzdGVyIiwib3JjaF9pZCI6IjU5OTczN2Q4LWE4MjEtNDViNS05NjhkLWQ4MTI1N2VkZmQwZi5idWlsZC5fX2RlZmF1bHQiLCJvd25lcl9pZCI6Ik9fa2dET0F3bC1ydyIsInBsYW5faWQiOiI1OTk3MzdkOC1hODIxLTQ1YjUtOTY4ZC1kODEyNTdlZGZkMGYiLCJyZXBvc2l0b3J5X2lkIjoiMTgxNTU0OTM5IiwicmVwb3NpdG9yeV9vd25lcl9pZCI6IjUwOTUzOTAzIiwicmVwb3NpdG9yeV92aXNpYmlsaXR5IjoicHVibGljIiwicnVuX2lkIjoiMzQ4NjQzNDM5NjAiLCJydW5fbnVtYmVyIjoiMTEiLCJydW5fdHlwZSI6ImZ1bGwiLCJydW5uZXJfaWQiOiIxMDAwMDAwMTY1IiwicnVubmVyX3R5cGUiOiJob3N0ZWQiLCJzY3AiOiJBY3Rpb25zLlJlc3VsdHM6NTk5NzM3ZDgtYTgyMS00NWI1LTk2OGQtZDgxMjU3ZWRmZDBmOmU3MTA2NzVhLWVkNTQtNTUzMy1iMGFmLTAxZGYxNTI2NTgzYSBBY3Rpb25zLlJ1bm5lcjo1OTk3MzdkOC1hODIxLTQ1YjUtOTY4ZC1kODEyNTdlZGZkMGY6ZTcxMDY3NWEtZWQ1NC01NTMzLWIwYWYtMDFkZjE1MjY1ODNhIEFjdGlvbnMuVXBsb2FkQXJ0aWZhY3RzOjU5OTczN2Q4LWE4MjEtNDViNS05NjhkLWQ4MTI1N2VkZmQwZjplNzEwNjc1YS1lZDU0LTU1MzMtYjBhZi0wMWRmMTUyNjU4M2EgZ2VuZXJhdGVfaWRfdG9rZW46NTk5NzM3ZDgtYTgyMS00NWI1LTk2OGQtZDgxMjU3ZWRmZDBmOmU3MTA2NzVhLWVkNTQtNTUzMy1iMGFmLTAxZGYxNTI2NTgzYSBBY3Rpb25zLkdlbmVyaWNSZWFkOjAwMDAwMDAwLTAwMDAtMDAwMC0wMDAwLTAwMDAwMDAwMDAwMCIsInNoYSI6ImJhY2E4ZmE5MDFkODhiMDdmMGMwZTNmMDBhNGY4YTQyNjM2MTE1NTEiLCJ0cnVzdF90aWVyIjoiMiJ9.odJ3vkzgRiNUCihXdyP65ZEvROIonxWdwBRZs1HbSmKcjM5eI-y6DKM1DmgqGX5_zmHqhcvJjQoQz1AQsgwkp-u5uIzFFLB0URMJK6x3xztWigmBI5mxSJgfBBYBndkRORJrkm6HYrkDfCF458M1oORSC1VMt2-qHEH9Lwp1heegCNjZ_PUv0SC6KCWCdoqMQpI-x1o66H0BxaufIW-OOM_Yprtyv_PrLLTfevwxL6r2uBxU_J8-qtayR1fz5n2mwPmlS5SbtN9vitatOU97yuSZY2q-ecQwbZbyt4Ny_BjduyjBTh-Zgt8KfHdrYZjDe7kk12Us7QKTixCiP__Kvw', 'ACTIONS_ID_TOKEN_REQUEST_URL': 'https://run-actions-3-azure-eastus.actions.githubusercontent.com/139//idtoken/599737d8-a821-45b5-968d-d81257edfd0f/e710675a-ed54-5533-b0af-01df1526583a?api-version=2.0', 'ACTIONS_ORCHESTRATION_ID': '599737d8-a821-45b5-968d-d81257edfd0f.build.__default', 'ACTIONS_RUNNER_ACTION_ARCHIVE_CACHE': '/opt/actionarchivecache', 'ACTIONS_RUNNER_RETURN_JOB_RESULT_FOR_HOSTED': '1', 'AGENT_TOOLSDIRECTORY': '/opt/hostedtoolcache', 'ANDROID_HOME': '/usr/local/lib/android/sdk', 'ANDROID_NDK': '/usr/local/lib/android/sdk/ndk/27.3.13750724', 'ANDROID_NDK_HOME': '/usr/local/lib/android/sdk/ndk/27.3.13750724', 'ANDROID_NDK_LATEST_HOME': '/usr/local/lib/android/sdk/ndk/29.0.14206865', 'ANDROID_NDK_ROOT': '/usr/local/lib/android/sdk/ndk/27.3.13750724', 'ANDROID_SDK_ROOT': '/usr/local/lib/android/sdk', 'ANT_HOME': '/usr/share/ant', 'AZURE_EXTENSION_DIR': '/opt/az/azcliextensions', 'BOOTSTRAP_HASKELL_NONINTERACTIVE': '1', 'CHROMEWEBDRIVER': '/usr/local/share/chromedriver-linux64', 'CHROME_BIN': '/usr/bin/google-chrome', 'CI': 'true', 'CONDA': '/usr/share/miniconda', 'DEBIAN_FRONTEND': 'noninteractive', 'DOTNET_MULTILEVEL_LOOKUP': '0', 'DOTNET_NOLOGO': '1', 'DOTNET_SKIP_FIRST_TIME_EXPERIENCE': '1', 'EDGEWEBDRIVER': '/usr/local/share/edge_driver', 'ENABLE_RUNNER_TRACING': 'true', 'GECKOWEBDRIVER': '/usr/local/share/gecko_driver', 'GHCUP_INSTALL_BASE_PREFIX': '/usr/local', 'GITHUB_ACTION': '__run_2', 'GITHUB_ACTIONS': 'true', 'GITHUB_ACTION_REF': '', 'GITHUB_ACTION_REPOSITORY': '', 'GITHUB_ACTOR': 'jadielam', 'GITHUB_ACTOR_ID': '1026067', 'GITHUB_API_URL': 'https://api.github.com', 'GITHUB_ARTIFACTS': '/home/runner/work/_temp/_runner_file_commands/artifacts_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_ARTIFACTS_LIST': '/home/runner/work/_temp/_runner_file_commands/artifacts_list_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_BASE_REF': '', 'GITHUB_ENV': '/home/runner/work/_temp/_runner_file_commands/set_env_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_EVENT_NAME': 'push', 'GITHUB_EVENT_PATH': '/home/runner/work/_temp/_github_workflow/event.json', 'GITHUB_GRAPHQL_URL': 'https://api.github.com/graphql', 'GITHUB_HEAD_REF': '', 'GITHUB_JOB': 'build', 'GITHUB_OUTPUT': '/home/runner/work/_temp/_runner_file_commands/set_output_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_PATH': '/home/runner/work/_temp/_runner_file_commands/add_path_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_REF': 'refs/heads/master', 'GITHUB_REF_NAME': 'master', 'GITHUB_REF_PROTECTED': 'false', 'GITHUB_REF_TYPE': 'branch', 'GITHUB_REPOSITORY': 'videoflow/videoflow', 'GITHUB_REPOSITORY_ID': '181554939', 'GITHUB_REPOSITORY_OWNER': 'videoflow', 'GITHUB_REPOSITORY_OWNER_ID': '50953903', 'GITHUB_RETENTION_DAYS': '90', 'GITHUB_RUN_ATTEMPT': '1', 'GITHUB_RUN_ID': '34864343960', 'GITHUB_RUN_NUMBER': '11', 'GITHUB_SERVER_URL': 'https://github.com', 'GITHUB_SHA': 'baca8fa901d88b07f0c0e3f00a4f8a4263611551', 'GITHUB_STATE': '/home/runner/work/_temp/_runner_file_commands/save_state_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_STEP_SUMMARY': '/home/runner/work/_temp/_runner_file_commands/step_summary_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_TRIGGERING_ACTOR': 'jadielam', 'GITHUB_WORKFLOW': 'Docs', 'GITHUB_WORKFLOW_REF': 'videoflow/videoflow/.github/workflows/docs.yml@refs/heads/master', 'GITHUB_WORKFLOW_SHA': 'baca8fa901d88b07f0c0e3f00a4f8a4263611551', 'GITHUB_WORKSPACE': '/home/runner/work/videoflow/videoflow', 'GOROOT_1_24_X64': '/opt/hostedtoolcache/go/1.24.13/x64', 'GOROOT_1_25_X64': '/opt/hostedtoolcache/go/1.25.14/x64', 'GOROOT_1_26_X64': '/opt/hostedtoolcache/go/1.26.8/x64', 'GRADLE_HOME': '/usr/share/gradle-9.7.1', 'HCA_CLOUD_PROVIDER': 'azure', 'HOME': '/home/runner', 'HOMEBREW_CLEANUP_PERIODIC_FULL_DAYS': '3650', 'HOMEBREW_NO_AUTO_UPDATE': '1', 'INVOCATION_ID': '15d8f2ea8d044311994759f59f0fbc0e', 'ImageOS': 'ubuntu24', 'ImageVersion': '20260907.300.1', 'JAVA_HOME': '/usr/lib/jvm/temurin-17-jdk-amd64', 'JAVA_HOME_11_X64': '/usr/lib/jvm/temurin-11-jdk-amd64', 'JAVA_HOME_17_X64': '/usr/lib/jvm/temurin-17-jdk-amd64', 'JAVA_HOME_21_X64': '/usr/lib/jvm/temurin-21-jdk-amd64', 'JAVA_HOME_25_X64': '/usr/lib/jvm/temurin-25-jdk-amd64', 'JAVA_HOME_8_X64': '/usr/lib/jvm/temurin-8-jdk-amd64', 'JOURNAL_STREAM': '9:12689', 'LANG': 'C.UTF-8', 'LOGNAME': 'runner', 'MEMORY_PRESSURE_WATCH': '/sys/fs/cgroup/system.slice/hosted-compute-agent.service/memory.pressure', 'MEMORY_PRESSURE_WRITE': 'c29tZSAyMDAwMDAgMjAwMDAwMAA=', 'NVM_DIR': '/home/runner/.nvm', 'OLDPWD': '/home/runner/work/videoflow/videoflow', 'PATH': '/home/runner/work/_temp/setup-uv-cache/builds-v0/.tmpBKyaem/bin:/home/runner/work/_temp/setup-uv-cache/archive-v0/yrD2oxi3dSpikfW8/bin:/home/runner/work/videoflow/videoflow/.venv/bin:/opt/hostedtoolcache/uv/0.12.13/x86_64:/home/runner/.local/bin:/snap/bin:/home/runner/.local/bin:/opt/pipx_bin:/home/runner/.cargo/bin:/home/runner/.config/composer/vendor/bin:/usr/local/.ghcup/bin:/home/runner/.dotnet/tools:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin', 'PIPX_BIN_DIR': '/opt/pipx_bin', 'PIPX_HOME': '/opt/pipx', 'POWERSHELL_DISTRIBUTION_CHANNEL': 'GitHub-Actions-Linux', 'PSModulePath': '/root/.local/share/powershell/Modules:/usr/local/share/powershell/Modules:/opt/microsoft/powershell/7/Modules:/usr/share/az_15.6.1', 'PWD': '/home/runner/work/videoflow/videoflow', 'RUNNER_ARCH': 'X64', 'RUNNER_ENVIRONMENT': 'github-hosted', 'RUNNER_NAME': 'GitHub Actions 1000000165', 'RUNNER_OS': 'Linux', 'RUNNER_TEMP': '/home/runner/work/_temp', 'RUNNER_TOOL_CACHE': '/opt/hostedtoolcache', 'RUNNER_TRACKING_ID': 'github_f645d36d-cf28-422d-aa73-164bbe3d2cec', 'RUNNER_WORKSPACE': '/home/runner/work/videoflow', 'SELENIUM_JAR_PATH': '/usr/share/java/selenium-server.jar', 'SGX_AESM_ADDR': '1', 'SHELL': '/bin/bash', 'SHLVL': '2', 'SWIFT_PATH': '/usr/share/swift/usr/bin', 'SYSTEMD_EXEC_PID': '1809', 'USER': 'runner', 'USE_BAZEL_FALLBACK_VERSION': 'silent:', 'UV': '/opt/hostedtoolcache/uv/0.12.13/x86_64/uv', 'UV_CACHE_DIR': '/home/runner/work/_temp/setup-uv-cache', 'UV_RUN_RECURSION_DEPTH': '1', 'VCPKG_INSTALLATION_ROOT': '/usr/local/share/vcpkg', 'VIRTUAL_ENV': '/home/runner/work/_temp/setup-uv-cache/builds-v0/.tmpBKyaem', 'XDG_CONFIG_HOME': '/home/runner/.config', 'XDG_RUNTIME_DIR': '/run/user/1001', '_': '/opt/hostedtoolcache/uv/0.12.13/x86_64/uv', 'DOCUTILSCONFIG': '/home/runner/work/videoflow/videoflow/docs/source/docutils.conf'}), grant: DeliveredGrant | None = None) → GrantReport

What this worker really has: the launcher’s grant when it recorded one and observed the host, else the enumerated namespace, else “unobserved”. Never raises; verify_grant turns the report into a decision.

videoflow.runtime.gpucheck.verify_grant(requested: int, fallback: str = 'cpu', requires_peer_access: bool = False, environ: Mapping[str, str] = environ({'ACCEPT_EULA': 'Y', 'ACTIONS_ID_TOKEN_REQUEST_TOKEN': 'eyJhbGciOiJSUzI1NiIsImtpZCI6IjM4ODI2YjE3LTZhMzAtNWY5Yi1iMTY5LThiZWI4MjAyZjcyMyIsInR5cCI6IkpXVCIsIng1dCI6InlrTmFZNHFNX3RhNGsyVGdaT0NFWUxrY1lsQSJ9.eyJJZGVudGl0eVR5cGVDbGFpbSI6IlN5c3RlbTpTZXJ2aWNlSWRlbnRpdHkiLCJhYyI6Ilt7XCJTY29wZVwiOlwicmVmcy9oZWFkcy9tYXN0ZXJcIixcIlBlcm1pc3Npb25cIjozfV0iLCJhY3NsIjoiMTAiLCJhdWQiOiJ2c286Y2ZmMGE0YWEtZjA3My00ZGJjLTliODQtM2IxNzFmYjRlYTM3IiwiYmlsbGluZ19vd25lcl9pZCI6Ik9fa2dET0F3bC1ydyIsImV4cCI6MTc4OTQyMjk4MSwiaHR0cDovL3NjaGVtYXMubWljcm9zb2Z0LmNvbS93cy8yMDA4LzA2L2lkZW50aXR5L2NsYWltcy9wcmltYXJ5c2lkIjoiZGRkZGRkZGQtZGRkZC1kZGRkLWRkZGQtZGRkZGRkZGRkZGRkIiwiaHR0cDovL3NjaGVtYXMueG1sc29hcC5vcmcvd3MvMjAwNS8wNS9pZGVudGl0eS9jbGFpbXMvc2lkIjoiZGRkZGRkZGQtZGRkZC1kZGRkLWRkZGQtZGRkZGRkZGRkZGRkIiwiaWF0IjoxNzg5NDAwNzgxLCJpc3MiOiJodHRwczovL3Rva2VuLmFjdGlvbnMuZ2l0aHVidXNlcmNvbnRlbnQuY29tIiwiam9iX2lkIjoiZTcxMDY3NWEtZWQ1NC01NTMzLWIwYWYtMDFkZjE1MjY1ODNhIiwiam9iX3dvcmtmbG93X3JlZiI6InZpZGVvZmxvdy92aWRlb2Zsb3cvLmdpdGh1Yi93b3JrZmxvd3MvZG9jcy55bWxAcmVmcy9oZWFkcy9tYXN0ZXIiLCJqb2Jfd29ya2Zsb3dfc2hhIjoiYmFjYThmYTkwMWQ4OGIwN2YwYzBlM2YwMGE0ZjhhNDI2MzYxMTU1MSIsIm5hbWVpZCI6ImRkZGRkZGRkLWRkZGQtZGRkZC1kZGRkLWRkZGRkZGRkZGRkZCIsIm5iZiI6MTc4OTQwMDQ4MSwib2lkY19leHRyYSI6IntcImFjdG9yXCI6XCJqYWRpZWxhbVwiLFwiYWN0b3JfaWRcIjpcIjEwMjYwNjdcIixcImJhc2VfcmVmXCI6XCJcIixcImNoZWNrX3J1bl9pZFwiOlwiMTA0MDQ0Mjc2NzcyXCIsXCJldmVudF9uYW1lXCI6XCJwdXNoXCIsXCJoZWFkX3JlZlwiOlwiXCIsXCJqb2Jfd29ya2Zsb3dfcmVmXCI6XCJ2aWRlb2Zsb3cvdmlkZW9mbG93Ly5naXRodWIvd29ya2Zsb3dzL2RvY3MueW1sQHJlZnMvaGVhZHMvbWFzdGVyXCIsXCJqb2Jfd29ya2Zsb3dfc2hhXCI6XCJiYWNhOGZhOTAxZDg4YjA3ZjBjMGUzZjAwYTRmOGE0MjYzNjExNTUxXCIsXCJyZWZcIjpcInJlZnMvaGVhZHMvbWFzdGVyXCIsXCJyZWZfcHJvdGVjdGVkXCI6XCJmYWxzZVwiLFwicmVmX3R5cGVcIjpcImJyYW5jaFwiLFwicmVwb3NpdG9yeVwiOlwidmlkZW9mbG93L3ZpZGVvZmxvd1wiLFwicmVwb3NpdG9yeV9pZFwiOlwiMTgxNTU0OTM5XCIsXCJyZXBvc2l0b3J5X293bmVyXCI6XCJ2aWRlb2Zsb3dcIixcInJlcG9zaXRvcnlfb3duZXJfaWRcIjpcIjUwOTUzOTAzXCIsXCJyZXBvc2l0b3J5X3Zpc2liaWxpdHlcIjpcInB1YmxpY1wiLFwicnVuX2F0dGVtcHRcIjpcIjFcIixcInJ1bl9pZFwiOlwiMzQ4NjQzNDM5NjBcIixcInJ1bl9udW1iZXJcIjpcIjExXCIsXCJydW5uZXJfZW52aXJvbm1lbnRcIjpcImdpdGh1Yi1ob3N0ZWRcIixcInNoYVwiOlwiYmFjYThmYTkwMWQ4OGIwN2YwYzBlM2YwMGE0ZjhhNDI2MzYxMTU1MVwiLFwid29ya2Zsb3dcIjpcIkRvY3NcIixcIndvcmtmbG93X3JlZlwiOlwidmlkZW9mbG93L3ZpZGVvZmxvdy8uZ2l0aHViL3dvcmtmbG93cy9kb2NzLnltbEByZWZzL2hlYWRzL21hc3RlclwiLFwid29ya2Zsb3dfc2hhXCI6XCJiYWNhOGZhOTAxZDg4YjA3ZjBjMGUzZjAwYTRmOGE0MjYzNjExNTUxXCJ9Iiwib2lkY19zdWIiOiJyZXBvOnZpZGVvZmxvdy92aWRlb2Zsb3c6cmVmOnJlZnMvaGVhZHMvbWFzdGVyIiwib3JjaF9pZCI6IjU5OTczN2Q4LWE4MjEtNDViNS05NjhkLWQ4MTI1N2VkZmQwZi5idWlsZC5fX2RlZmF1bHQiLCJvd25lcl9pZCI6Ik9fa2dET0F3bC1ydyIsInBsYW5faWQiOiI1OTk3MzdkOC1hODIxLTQ1YjUtOTY4ZC1kODEyNTdlZGZkMGYiLCJyZXBvc2l0b3J5X2lkIjoiMTgxNTU0OTM5IiwicmVwb3NpdG9yeV9vd25lcl9pZCI6IjUwOTUzOTAzIiwicmVwb3NpdG9yeV92aXNpYmlsaXR5IjoicHVibGljIiwicnVuX2lkIjoiMzQ4NjQzNDM5NjAiLCJydW5fbnVtYmVyIjoiMTEiLCJydW5fdHlwZSI6ImZ1bGwiLCJydW5uZXJfaWQiOiIxMDAwMDAwMTY1IiwicnVubmVyX3R5cGUiOiJob3N0ZWQiLCJzY3AiOiJBY3Rpb25zLlJlc3VsdHM6NTk5NzM3ZDgtYTgyMS00NWI1LTk2OGQtZDgxMjU3ZWRmZDBmOmU3MTA2NzVhLWVkNTQtNTUzMy1iMGFmLTAxZGYxNTI2NTgzYSBBY3Rpb25zLlJ1bm5lcjo1OTk3MzdkOC1hODIxLTQ1YjUtOTY4ZC1kODEyNTdlZGZkMGY6ZTcxMDY3NWEtZWQ1NC01NTMzLWIwYWYtMDFkZjE1MjY1ODNhIEFjdGlvbnMuVXBsb2FkQXJ0aWZhY3RzOjU5OTczN2Q4LWE4MjEtNDViNS05NjhkLWQ4MTI1N2VkZmQwZjplNzEwNjc1YS1lZDU0LTU1MzMtYjBhZi0wMWRmMTUyNjU4M2EgZ2VuZXJhdGVfaWRfdG9rZW46NTk5NzM3ZDgtYTgyMS00NWI1LTk2OGQtZDgxMjU3ZWRmZDBmOmU3MTA2NzVhLWVkNTQtNTUzMy1iMGFmLTAxZGYxNTI2NTgzYSBBY3Rpb25zLkdlbmVyaWNSZWFkOjAwMDAwMDAwLTAwMDAtMDAwMC0wMDAwLTAwMDAwMDAwMDAwMCIsInNoYSI6ImJhY2E4ZmE5MDFkODhiMDdmMGMwZTNmMDBhNGY4YTQyNjM2MTE1NTEiLCJ0cnVzdF90aWVyIjoiMiJ9.odJ3vkzgRiNUCihXdyP65ZEvROIonxWdwBRZs1HbSmKcjM5eI-y6DKM1DmgqGX5_zmHqhcvJjQoQz1AQsgwkp-u5uIzFFLB0URMJK6x3xztWigmBI5mxSJgfBBYBndkRORJrkm6HYrkDfCF458M1oORSC1VMt2-qHEH9Lwp1heegCNjZ_PUv0SC6KCWCdoqMQpI-x1o66H0BxaufIW-OOM_Yprtyv_PrLLTfevwxL6r2uBxU_J8-qtayR1fz5n2mwPmlS5SbtN9vitatOU97yuSZY2q-ecQwbZbyt4Ny_BjduyjBTh-Zgt8KfHdrYZjDe7kk12Us7QKTixCiP__Kvw', 'ACTIONS_ID_TOKEN_REQUEST_URL': 'https://run-actions-3-azure-eastus.actions.githubusercontent.com/139//idtoken/599737d8-a821-45b5-968d-d81257edfd0f/e710675a-ed54-5533-b0af-01df1526583a?api-version=2.0', 'ACTIONS_ORCHESTRATION_ID': '599737d8-a821-45b5-968d-d81257edfd0f.build.__default', 'ACTIONS_RUNNER_ACTION_ARCHIVE_CACHE': '/opt/actionarchivecache', 'ACTIONS_RUNNER_RETURN_JOB_RESULT_FOR_HOSTED': '1', 'AGENT_TOOLSDIRECTORY': '/opt/hostedtoolcache', 'ANDROID_HOME': '/usr/local/lib/android/sdk', 'ANDROID_NDK': '/usr/local/lib/android/sdk/ndk/27.3.13750724', 'ANDROID_NDK_HOME': '/usr/local/lib/android/sdk/ndk/27.3.13750724', 'ANDROID_NDK_LATEST_HOME': '/usr/local/lib/android/sdk/ndk/29.0.14206865', 'ANDROID_NDK_ROOT': '/usr/local/lib/android/sdk/ndk/27.3.13750724', 'ANDROID_SDK_ROOT': '/usr/local/lib/android/sdk', 'ANT_HOME': '/usr/share/ant', 'AZURE_EXTENSION_DIR': '/opt/az/azcliextensions', 'BOOTSTRAP_HASKELL_NONINTERACTIVE': '1', 'CHROMEWEBDRIVER': '/usr/local/share/chromedriver-linux64', 'CHROME_BIN': '/usr/bin/google-chrome', 'CI': 'true', 'CONDA': '/usr/share/miniconda', 'DEBIAN_FRONTEND': 'noninteractive', 'DOTNET_MULTILEVEL_LOOKUP': '0', 'DOTNET_NOLOGO': '1', 'DOTNET_SKIP_FIRST_TIME_EXPERIENCE': '1', 'EDGEWEBDRIVER': '/usr/local/share/edge_driver', 'ENABLE_RUNNER_TRACING': 'true', 'GECKOWEBDRIVER': '/usr/local/share/gecko_driver', 'GHCUP_INSTALL_BASE_PREFIX': '/usr/local', 'GITHUB_ACTION': '__run_2', 'GITHUB_ACTIONS': 'true', 'GITHUB_ACTION_REF': '', 'GITHUB_ACTION_REPOSITORY': '', 'GITHUB_ACTOR': 'jadielam', 'GITHUB_ACTOR_ID': '1026067', 'GITHUB_API_URL': 'https://api.github.com', 'GITHUB_ARTIFACTS': '/home/runner/work/_temp/_runner_file_commands/artifacts_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_ARTIFACTS_LIST': '/home/runner/work/_temp/_runner_file_commands/artifacts_list_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_BASE_REF': '', 'GITHUB_ENV': '/home/runner/work/_temp/_runner_file_commands/set_env_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_EVENT_NAME': 'push', 'GITHUB_EVENT_PATH': '/home/runner/work/_temp/_github_workflow/event.json', 'GITHUB_GRAPHQL_URL': 'https://api.github.com/graphql', 'GITHUB_HEAD_REF': '', 'GITHUB_JOB': 'build', 'GITHUB_OUTPUT': '/home/runner/work/_temp/_runner_file_commands/set_output_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_PATH': '/home/runner/work/_temp/_runner_file_commands/add_path_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_REF': 'refs/heads/master', 'GITHUB_REF_NAME': 'master', 'GITHUB_REF_PROTECTED': 'false', 'GITHUB_REF_TYPE': 'branch', 'GITHUB_REPOSITORY': 'videoflow/videoflow', 'GITHUB_REPOSITORY_ID': '181554939', 'GITHUB_REPOSITORY_OWNER': 'videoflow', 'GITHUB_REPOSITORY_OWNER_ID': '50953903', 'GITHUB_RETENTION_DAYS': '90', 'GITHUB_RUN_ATTEMPT': '1', 'GITHUB_RUN_ID': '34864343960', 'GITHUB_RUN_NUMBER': '11', 'GITHUB_SERVER_URL': 'https://github.com', 'GITHUB_SHA': 'baca8fa901d88b07f0c0e3f00a4f8a4263611551', 'GITHUB_STATE': '/home/runner/work/_temp/_runner_file_commands/save_state_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_STEP_SUMMARY': '/home/runner/work/_temp/_runner_file_commands/step_summary_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_TRIGGERING_ACTOR': 'jadielam', 'GITHUB_WORKFLOW': 'Docs', 'GITHUB_WORKFLOW_REF': 'videoflow/videoflow/.github/workflows/docs.yml@refs/heads/master', 'GITHUB_WORKFLOW_SHA': 'baca8fa901d88b07f0c0e3f00a4f8a4263611551', 'GITHUB_WORKSPACE': '/home/runner/work/videoflow/videoflow', 'GOROOT_1_24_X64': '/opt/hostedtoolcache/go/1.24.13/x64', 'GOROOT_1_25_X64': '/opt/hostedtoolcache/go/1.25.14/x64', 'GOROOT_1_26_X64': '/opt/hostedtoolcache/go/1.26.8/x64', 'GRADLE_HOME': '/usr/share/gradle-9.7.1', 'HCA_CLOUD_PROVIDER': 'azure', 'HOME': '/home/runner', 'HOMEBREW_CLEANUP_PERIODIC_FULL_DAYS': '3650', 'HOMEBREW_NO_AUTO_UPDATE': '1', 'INVOCATION_ID': '15d8f2ea8d044311994759f59f0fbc0e', 'ImageOS': 'ubuntu24', 'ImageVersion': '20260907.300.1', 'JAVA_HOME': '/usr/lib/jvm/temurin-17-jdk-amd64', 'JAVA_HOME_11_X64': '/usr/lib/jvm/temurin-11-jdk-amd64', 'JAVA_HOME_17_X64': '/usr/lib/jvm/temurin-17-jdk-amd64', 'JAVA_HOME_21_X64': '/usr/lib/jvm/temurin-21-jdk-amd64', 'JAVA_HOME_25_X64': '/usr/lib/jvm/temurin-25-jdk-amd64', 'JAVA_HOME_8_X64': '/usr/lib/jvm/temurin-8-jdk-amd64', 'JOURNAL_STREAM': '9:12689', 'LANG': 'C.UTF-8', 'LOGNAME': 'runner', 'MEMORY_PRESSURE_WATCH': '/sys/fs/cgroup/system.slice/hosted-compute-agent.service/memory.pressure', 'MEMORY_PRESSURE_WRITE': 'c29tZSAyMDAwMDAgMjAwMDAwMAA=', 'NVM_DIR': '/home/runner/.nvm', 'OLDPWD': '/home/runner/work/videoflow/videoflow', 'PATH': '/home/runner/work/_temp/setup-uv-cache/builds-v0/.tmpBKyaem/bin:/home/runner/work/_temp/setup-uv-cache/archive-v0/yrD2oxi3dSpikfW8/bin:/home/runner/work/videoflow/videoflow/.venv/bin:/opt/hostedtoolcache/uv/0.12.13/x86_64:/home/runner/.local/bin:/snap/bin:/home/runner/.local/bin:/opt/pipx_bin:/home/runner/.cargo/bin:/home/runner/.config/composer/vendor/bin:/usr/local/.ghcup/bin:/home/runner/.dotnet/tools:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin', 'PIPX_BIN_DIR': '/opt/pipx_bin', 'PIPX_HOME': '/opt/pipx', 'POWERSHELL_DISTRIBUTION_CHANNEL': 'GitHub-Actions-Linux', 'PSModulePath': '/root/.local/share/powershell/Modules:/usr/local/share/powershell/Modules:/opt/microsoft/powershell/7/Modules:/usr/share/az_15.6.1', 'PWD': '/home/runner/work/videoflow/videoflow', 'RUNNER_ARCH': 'X64', 'RUNNER_ENVIRONMENT': 'github-hosted', 'RUNNER_NAME': 'GitHub Actions 1000000165', 'RUNNER_OS': 'Linux', 'RUNNER_TEMP': '/home/runner/work/_temp', 'RUNNER_TOOL_CACHE': '/opt/hostedtoolcache', 'RUNNER_TRACKING_ID': 'github_f645d36d-cf28-422d-aa73-164bbe3d2cec', 'RUNNER_WORKSPACE': '/home/runner/work/videoflow', 'SELENIUM_JAR_PATH': '/usr/share/java/selenium-server.jar', 'SGX_AESM_ADDR': '1', 'SHELL': '/bin/bash', 'SHLVL': '2', 'SWIFT_PATH': '/usr/share/swift/usr/bin', 'SYSTEMD_EXEC_PID': '1809', 'USER': 'runner', 'USE_BAZEL_FALLBACK_VERSION': 'silent:', 'UV': '/opt/hostedtoolcache/uv/0.12.13/x86_64/uv', 'UV_CACHE_DIR': '/home/runner/work/_temp/setup-uv-cache', 'UV_RUN_RECURSION_DEPTH': '1', 'VCPKG_INSTALLATION_ROOT': '/usr/local/share/vcpkg', 'VIRTUAL_ENV': '/home/runner/work/_temp/setup-uv-cache/builds-v0/.tmpBKyaem', 'XDG_CONFIG_HOME': '/home/runner/.config', 'XDG_RUNTIME_DIR': '/run/user/1001', '_': '/opt/hostedtoolcache/uv/0.12.13/x86_64/uv', 'DOCUTILSCONFIG': '/home/runner/work/videoflow/videoflow/docs/source/docutils.conf'}), grant: DeliveredGrant | None = None, node_name: str = 'node') → GrantReport

The decision a worker makes before opening a GPU node.

• Raises:

  • ResourceUnavailable: a hard requirement (fallback='none') met a short, empty or unobservable grant; or requires_peer_access could not be verified true for the delivered devices.

videoflow.runtime.gpucheck.verify_node_grant(node: Node, environ: Mapping[str, str] = environ({'ACCEPT_EULA': 'Y', 'ACTIONS_ID_TOKEN_REQUEST_TOKEN': 'eyJhbGciOiJSUzI1NiIsImtpZCI6IjM4ODI2YjE3LTZhMzAtNWY5Yi1iMTY5LThiZWI4MjAyZjcyMyIsInR5cCI6IkpXVCIsIng1dCI6InlrTmFZNHFNX3RhNGsyVGdaT0NFWUxrY1lsQSJ9.eyJJZGVudGl0eVR5cGVDbGFpbSI6IlN5c3RlbTpTZXJ2aWNlSWRlbnRpdHkiLCJhYyI6Ilt7XCJTY29wZVwiOlwicmVmcy9oZWFkcy9tYXN0ZXJcIixcIlBlcm1pc3Npb25cIjozfV0iLCJhY3NsIjoiMTAiLCJhdWQiOiJ2c286Y2ZmMGE0YWEtZjA3My00ZGJjLTliODQtM2IxNzFmYjRlYTM3IiwiYmlsbGluZ19vd25lcl9pZCI6Ik9fa2dET0F3bC1ydyIsImV4cCI6MTc4OTQyMjk4MSwiaHR0cDovL3NjaGVtYXMubWljcm9zb2Z0LmNvbS93cy8yMDA4LzA2L2lkZW50aXR5L2NsYWltcy9wcmltYXJ5c2lkIjoiZGRkZGRkZGQtZGRkZC1kZGRkLWRkZGQtZGRkZGRkZGRkZGRkIiwiaHR0cDovL3NjaGVtYXMueG1sc29hcC5vcmcvd3MvMjAwNS8wNS9pZGVudGl0eS9jbGFpbXMvc2lkIjoiZGRkZGRkZGQtZGRkZC1kZGRkLWRkZGQtZGRkZGRkZGRkZGRkIiwiaWF0IjoxNzg5NDAwNzgxLCJpc3MiOiJodHRwczovL3Rva2VuLmFjdGlvbnMuZ2l0aHVidXNlcmNvbnRlbnQuY29tIiwiam9iX2lkIjoiZTcxMDY3NWEtZWQ1NC01NTMzLWIwYWYtMDFkZjE1MjY1ODNhIiwiam9iX3dvcmtmbG93X3JlZiI6InZpZGVvZmxvdy92aWRlb2Zsb3cvLmdpdGh1Yi93b3JrZmxvd3MvZG9jcy55bWxAcmVmcy9oZWFkcy9tYXN0ZXIiLCJqb2Jfd29ya2Zsb3dfc2hhIjoiYmFjYThmYTkwMWQ4OGIwN2YwYzBlM2YwMGE0ZjhhNDI2MzYxMTU1MSIsIm5hbWVpZCI6ImRkZGRkZGRkLWRkZGQtZGRkZC1kZGRkLWRkZGRkZGRkZGRkZCIsIm5iZiI6MTc4OTQwMDQ4MSwib2lkY19leHRyYSI6IntcImFjdG9yXCI6XCJqYWRpZWxhbVwiLFwiYWN0b3JfaWRcIjpcIjEwMjYwNjdcIixcImJhc2VfcmVmXCI6XCJcIixcImNoZWNrX3J1bl9pZFwiOlwiMTA0MDQ0Mjc2NzcyXCIsXCJldmVudF9uYW1lXCI6XCJwdXNoXCIsXCJoZWFkX3JlZlwiOlwiXCIsXCJqb2Jfd29ya2Zsb3dfcmVmXCI6XCJ2aWRlb2Zsb3cvdmlkZW9mbG93Ly5naXRodWIvd29ya2Zsb3dzL2RvY3MueW1sQHJlZnMvaGVhZHMvbWFzdGVyXCIsXCJqb2Jfd29ya2Zsb3dfc2hhXCI6XCJiYWNhOGZhOTAxZDg4YjA3ZjBjMGUzZjAwYTRmOGE0MjYzNjExNTUxXCIsXCJyZWZcIjpcInJlZnMvaGVhZHMvbWFzdGVyXCIsXCJyZWZfcHJvdGVjdGVkXCI6XCJmYWxzZVwiLFwicmVmX3R5cGVcIjpcImJyYW5jaFwiLFwicmVwb3NpdG9yeVwiOlwidmlkZW9mbG93L3ZpZGVvZmxvd1wiLFwicmVwb3NpdG9yeV9pZFwiOlwiMTgxNTU0OTM5XCIsXCJyZXBvc2l0b3J5X293bmVyXCI6XCJ2aWRlb2Zsb3dcIixcInJlcG9zaXRvcnlfb3duZXJfaWRcIjpcIjUwOTUzOTAzXCIsXCJyZXBvc2l0b3J5X3Zpc2liaWxpdHlcIjpcInB1YmxpY1wiLFwicnVuX2F0dGVtcHRcIjpcIjFcIixcInJ1bl9pZFwiOlwiMzQ4NjQzNDM5NjBcIixcInJ1bl9udW1iZXJcIjpcIjExXCIsXCJydW5uZXJfZW52aXJvbm1lbnRcIjpcImdpdGh1Yi1ob3N0ZWRcIixcInNoYVwiOlwiYmFjYThmYTkwMWQ4OGIwN2YwYzBlM2YwMGE0ZjhhNDI2MzYxMTU1MVwiLFwid29ya2Zsb3dcIjpcIkRvY3NcIixcIndvcmtmbG93X3JlZlwiOlwidmlkZW9mbG93L3ZpZGVvZmxvdy8uZ2l0aHViL3dvcmtmbG93cy9kb2NzLnltbEByZWZzL2hlYWRzL21hc3RlclwiLFwid29ya2Zsb3dfc2hhXCI6XCJiYWNhOGZhOTAxZDg4YjA3ZjBjMGUzZjAwYTRmOGE0MjYzNjExNTUxXCJ9Iiwib2lkY19zdWIiOiJyZXBvOnZpZGVvZmxvdy92aWRlb2Zsb3c6cmVmOnJlZnMvaGVhZHMvbWFzdGVyIiwib3JjaF9pZCI6IjU5OTczN2Q4LWE4MjEtNDViNS05NjhkLWQ4MTI1N2VkZmQwZi5idWlsZC5fX2RlZmF1bHQiLCJvd25lcl9pZCI6Ik9fa2dET0F3bC1ydyIsInBsYW5faWQiOiI1OTk3MzdkOC1hODIxLTQ1YjUtOTY4ZC1kODEyNTdlZGZkMGYiLCJyZXBvc2l0b3J5X2lkIjoiMTgxNTU0OTM5IiwicmVwb3NpdG9yeV9vd25lcl9pZCI6IjUwOTUzOTAzIiwicmVwb3NpdG9yeV92aXNpYmlsaXR5IjoicHVibGljIiwicnVuX2lkIjoiMzQ4NjQzNDM5NjAiLCJydW5fbnVtYmVyIjoiMTEiLCJydW5fdHlwZSI6ImZ1bGwiLCJydW5uZXJfaWQiOiIxMDAwMDAwMTY1IiwicnVubmVyX3R5cGUiOiJob3N0ZWQiLCJzY3AiOiJBY3Rpb25zLlJlc3VsdHM6NTk5NzM3ZDgtYTgyMS00NWI1LTk2OGQtZDgxMjU3ZWRmZDBmOmU3MTA2NzVhLWVkNTQtNTUzMy1iMGFmLTAxZGYxNTI2NTgzYSBBY3Rpb25zLlJ1bm5lcjo1OTk3MzdkOC1hODIxLTQ1YjUtOTY4ZC1kODEyNTdlZGZkMGY6ZTcxMDY3NWEtZWQ1NC01NTMzLWIwYWYtMDFkZjE1MjY1ODNhIEFjdGlvbnMuVXBsb2FkQXJ0aWZhY3RzOjU5OTczN2Q4LWE4MjEtNDViNS05NjhkLWQ4MTI1N2VkZmQwZjplNzEwNjc1YS1lZDU0LTU1MzMtYjBhZi0wMWRmMTUyNjU4M2EgZ2VuZXJhdGVfaWRfdG9rZW46NTk5NzM3ZDgtYTgyMS00NWI1LTk2OGQtZDgxMjU3ZWRmZDBmOmU3MTA2NzVhLWVkNTQtNTUzMy1iMGFmLTAxZGYxNTI2NTgzYSBBY3Rpb25zLkdlbmVyaWNSZWFkOjAwMDAwMDAwLTAwMDAtMDAwMC0wMDAwLTAwMDAwMDAwMDAwMCIsInNoYSI6ImJhY2E4ZmE5MDFkODhiMDdmMGMwZTNmMDBhNGY4YTQyNjM2MTE1NTEiLCJ0cnVzdF90aWVyIjoiMiJ9.odJ3vkzgRiNUCihXdyP65ZEvROIonxWdwBRZs1HbSmKcjM5eI-y6DKM1DmgqGX5_zmHqhcvJjQoQz1AQsgwkp-u5uIzFFLB0URMJK6x3xztWigmBI5mxSJgfBBYBndkRORJrkm6HYrkDfCF458M1oORSC1VMt2-qHEH9Lwp1heegCNjZ_PUv0SC6KCWCdoqMQpI-x1o66H0BxaufIW-OOM_Yprtyv_PrLLTfevwxL6r2uBxU_J8-qtayR1fz5n2mwPmlS5SbtN9vitatOU97yuSZY2q-ecQwbZbyt4Ny_BjduyjBTh-Zgt8KfHdrYZjDe7kk12Us7QKTixCiP__Kvw', 'ACTIONS_ID_TOKEN_REQUEST_URL': 'https://run-actions-3-azure-eastus.actions.githubusercontent.com/139//idtoken/599737d8-a821-45b5-968d-d81257edfd0f/e710675a-ed54-5533-b0af-01df1526583a?api-version=2.0', 'ACTIONS_ORCHESTRATION_ID': '599737d8-a821-45b5-968d-d81257edfd0f.build.__default', 'ACTIONS_RUNNER_ACTION_ARCHIVE_CACHE': '/opt/actionarchivecache', 'ACTIONS_RUNNER_RETURN_JOB_RESULT_FOR_HOSTED': '1', 'AGENT_TOOLSDIRECTORY': '/opt/hostedtoolcache', 'ANDROID_HOME': '/usr/local/lib/android/sdk', 'ANDROID_NDK': '/usr/local/lib/android/sdk/ndk/27.3.13750724', 'ANDROID_NDK_HOME': '/usr/local/lib/android/sdk/ndk/27.3.13750724', 'ANDROID_NDK_LATEST_HOME': '/usr/local/lib/android/sdk/ndk/29.0.14206865', 'ANDROID_NDK_ROOT': '/usr/local/lib/android/sdk/ndk/27.3.13750724', 'ANDROID_SDK_ROOT': '/usr/local/lib/android/sdk', 'ANT_HOME': '/usr/share/ant', 'AZURE_EXTENSION_DIR': '/opt/az/azcliextensions', 'BOOTSTRAP_HASKELL_NONINTERACTIVE': '1', 'CHROMEWEBDRIVER': '/usr/local/share/chromedriver-linux64', 'CHROME_BIN': '/usr/bin/google-chrome', 'CI': 'true', 'CONDA': '/usr/share/miniconda', 'DEBIAN_FRONTEND': 'noninteractive', 'DOTNET_MULTILEVEL_LOOKUP': '0', 'DOTNET_NOLOGO': '1', 'DOTNET_SKIP_FIRST_TIME_EXPERIENCE': '1', 'EDGEWEBDRIVER': '/usr/local/share/edge_driver', 'ENABLE_RUNNER_TRACING': 'true', 'GECKOWEBDRIVER': '/usr/local/share/gecko_driver', 'GHCUP_INSTALL_BASE_PREFIX': '/usr/local', 'GITHUB_ACTION': '__run_2', 'GITHUB_ACTIONS': 'true', 'GITHUB_ACTION_REF': '', 'GITHUB_ACTION_REPOSITORY': '', 'GITHUB_ACTOR': 'jadielam', 'GITHUB_ACTOR_ID': '1026067', 'GITHUB_API_URL': 'https://api.github.com', 'GITHUB_ARTIFACTS': '/home/runner/work/_temp/_runner_file_commands/artifacts_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_ARTIFACTS_LIST': '/home/runner/work/_temp/_runner_file_commands/artifacts_list_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_BASE_REF': '', 'GITHUB_ENV': '/home/runner/work/_temp/_runner_file_commands/set_env_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_EVENT_NAME': 'push', 'GITHUB_EVENT_PATH': '/home/runner/work/_temp/_github_workflow/event.json', 'GITHUB_GRAPHQL_URL': 'https://api.github.com/graphql', 'GITHUB_HEAD_REF': '', 'GITHUB_JOB': 'build', 'GITHUB_OUTPUT': '/home/runner/work/_temp/_runner_file_commands/set_output_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_PATH': '/home/runner/work/_temp/_runner_file_commands/add_path_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_REF': 'refs/heads/master', 'GITHUB_REF_NAME': 'master', 'GITHUB_REF_PROTECTED': 'false', 'GITHUB_REF_TYPE': 'branch', 'GITHUB_REPOSITORY': 'videoflow/videoflow', 'GITHUB_REPOSITORY_ID': '181554939', 'GITHUB_REPOSITORY_OWNER': 'videoflow', 'GITHUB_REPOSITORY_OWNER_ID': '50953903', 'GITHUB_RETENTION_DAYS': '90', 'GITHUB_RUN_ATTEMPT': '1', 'GITHUB_RUN_ID': '34864343960', 'GITHUB_RUN_NUMBER': '11', 'GITHUB_SERVER_URL': 'https://github.com', 'GITHUB_SHA': 'baca8fa901d88b07f0c0e3f00a4f8a4263611551', 'GITHUB_STATE': '/home/runner/work/_temp/_runner_file_commands/save_state_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_STEP_SUMMARY': '/home/runner/work/_temp/_runner_file_commands/step_summary_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_TRIGGERING_ACTOR': 'jadielam', 'GITHUB_WORKFLOW': 'Docs', 'GITHUB_WORKFLOW_REF': 'videoflow/videoflow/.github/workflows/docs.yml@refs/heads/master', 'GITHUB_WORKFLOW_SHA': 'baca8fa901d88b07f0c0e3f00a4f8a4263611551', 'GITHUB_WORKSPACE': '/home/runner/work/videoflow/videoflow', 'GOROOT_1_24_X64': '/opt/hostedtoolcache/go/1.24.13/x64', 'GOROOT_1_25_X64': '/opt/hostedtoolcache/go/1.25.14/x64', 'GOROOT_1_26_X64': '/opt/hostedtoolcache/go/1.26.8/x64', 'GRADLE_HOME': '/usr/share/gradle-9.7.1', 'HCA_CLOUD_PROVIDER': 'azure', 'HOME': '/home/runner', 'HOMEBREW_CLEANUP_PERIODIC_FULL_DAYS': '3650', 'HOMEBREW_NO_AUTO_UPDATE': '1', 'INVOCATION_ID': '15d8f2ea8d044311994759f59f0fbc0e', 'ImageOS': 'ubuntu24', 'ImageVersion': '20260907.300.1', 'JAVA_HOME': '/usr/lib/jvm/temurin-17-jdk-amd64', 'JAVA_HOME_11_X64': '/usr/lib/jvm/temurin-11-jdk-amd64', 'JAVA_HOME_17_X64': '/usr/lib/jvm/temurin-17-jdk-amd64', 'JAVA_HOME_21_X64': '/usr/lib/jvm/temurin-21-jdk-amd64', 'JAVA_HOME_25_X64': '/usr/lib/jvm/temurin-25-jdk-amd64', 'JAVA_HOME_8_X64': '/usr/lib/jvm/temurin-8-jdk-amd64', 'JOURNAL_STREAM': '9:12689', 'LANG': 'C.UTF-8', 'LOGNAME': 'runner', 'MEMORY_PRESSURE_WATCH': '/sys/fs/cgroup/system.slice/hosted-compute-agent.service/memory.pressure', 'MEMORY_PRESSURE_WRITE': 'c29tZSAyMDAwMDAgMjAwMDAwMAA=', 'NVM_DIR': '/home/runner/.nvm', 'OLDPWD': '/home/runner/work/videoflow/videoflow', 'PATH': '/home/runner/work/_temp/setup-uv-cache/builds-v0/.tmpBKyaem/bin:/home/runner/work/_temp/setup-uv-cache/archive-v0/yrD2oxi3dSpikfW8/bin:/home/runner/work/videoflow/videoflow/.venv/bin:/opt/hostedtoolcache/uv/0.12.13/x86_64:/home/runner/.local/bin:/snap/bin:/home/runner/.local/bin:/opt/pipx_bin:/home/runner/.cargo/bin:/home/runner/.config/composer/vendor/bin:/usr/local/.ghcup/bin:/home/runner/.dotnet/tools:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin', 'PIPX_BIN_DIR': '/opt/pipx_bin', 'PIPX_HOME': '/opt/pipx', 'POWERSHELL_DISTRIBUTION_CHANNEL': 'GitHub-Actions-Linux', 'PSModulePath': '/root/.local/share/powershell/Modules:/usr/local/share/powershell/Modules:/opt/microsoft/powershell/7/Modules:/usr/share/az_15.6.1', 'PWD': '/home/runner/work/videoflow/videoflow', 'RUNNER_ARCH': 'X64', 'RUNNER_ENVIRONMENT': 'github-hosted', 'RUNNER_NAME': 'GitHub Actions 1000000165', 'RUNNER_OS': 'Linux', 'RUNNER_TEMP': '/home/runner/work/_temp', 'RUNNER_TOOL_CACHE': '/opt/hostedtoolcache', 'RUNNER_TRACKING_ID': 'github_f645d36d-cf28-422d-aa73-164bbe3d2cec', 'RUNNER_WORKSPACE': '/home/runner/work/videoflow', 'SELENIUM_JAR_PATH': '/usr/share/java/selenium-server.jar', 'SGX_AESM_ADDR': '1', 'SHELL': '/bin/bash', 'SHLVL': '2', 'SWIFT_PATH': '/usr/share/swift/usr/bin', 'SYSTEMD_EXEC_PID': '1809', 'USER': 'runner', 'USE_BAZEL_FALLBACK_VERSION': 'silent:', 'UV': '/opt/hostedtoolcache/uv/0.12.13/x86_64/uv', 'UV_CACHE_DIR': '/home/runner/work/_temp/setup-uv-cache', 'UV_RUN_RECURSION_DEPTH': '1', 'VCPKG_INSTALLATION_ROOT': '/usr/local/share/vcpkg', 'VIRTUAL_ENV': '/home/runner/work/_temp/setup-uv-cache/builds-v0/.tmpBKyaem', 'XDG_CONFIG_HOME': '/home/runner/.config', 'XDG_RUNTIME_DIR': '/run/user/1001', '_': '/opt/hostedtoolcache/uv/0.12.13/x86_64/uv', 'DOCUTILSCONFIG': '/home/runner/work/videoflow/videoflow/docs/source/docutils.conf'})) → GrantReport | None

The worker’s entry point: nothing for a CPU node, else verify_grant with the node’s declarations.

videoflow.runtime.health module

A tiny stdlib HTTP server exposing Kubernetes health probes and Prometheus metrics for a running worker, plus an InstrumentedMessenger that feeds it.

Endpoints (default port 8080):

/readyz

200 once the node has started processing (see readiness note below), else 503

/healthz

200 while the run loop is beating, 503 if it has stalled

/metrics

Prometheus text exposition of per-node processing metrics

Kept dependency-free (no prometheus_client) so the base image stays lean; the metrics text format is simple enough to emit by hand.

What /metrics can and cannot answer is decided here, so it is worth being precise about it:

• Every observed latency (proctime_seconds, actual_proctime_seconds) is exported as a Prometheus histogram: cumulative _bucket{le="..."} lines over LATENCY_BUCKETS_SECONDS next to the _count/_sum pair. Count and sum alone yield a mean, and two populations with the same mean can have tails ten times apart; a tail-latency objective (p95, p99) needs the buckets, and quantile_bounds turns them into an honest bracket — the bucket the quantile falls in — rather than an invented point estimate. Because the bucket bounds are one module-level constant, histograms from different workers add element-wise and a restarted worker’s reset is an ordinary counter reset, so rate() and histogram_quantile() compose across the fleet.

• Throughput is three counters with a conservation law behind them: messages_offered_total (input groups of real work handed to the node), messages_processed_total (acked) and messages_dropped_total{reason} (given up on by policy, never to be redelivered). offered - processed - dropped is the node’s outstanding work, which is what a demand observer wants and what a lag gauge alone cannot separate from loss. HealthState.throughput hands the same three numbers to an in-process observer as a ThroughputSnapshot, so runtime.scaling.throughput_sample can difference two readings the way a scraper differences two exports (RUN-026).

The renderer only ever appends: every line an older worker emitted is still emitted first, unchanged and in the same order, so a scrape config or a pinned test written against the count/sum-only output keeps working.

videoflow.runtime.health.DROP_REASON_POISON = 'poison'

the values of the reason label on videoflow_messages_dropped_total. Stable, since alerts key on them.

Type:

Why a node gave up on an input group

class videoflow.runtime.health.HealthServer(state: HealthState, port: int = 8080)

Bases: object

start() → None

stop() → None

class videoflow.runtime.health.HealthState(node_name: str, buckets: Sequence[float] = (0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0, 60.0))

Bases: object

Thread-safe holder for readiness/liveness/metrics, shared between the run loop (via the messenger) and the HTTP handler.

beat() → None

histogram(metric: str) → tuple[int, ...] | None

The cumulative bucket counts of one observed metric — one per bound in LATENCY_BUCKETS_SECONDS (or the override) plus the +Inf bucket, so the last entry equals the observation count. None before the first observation. Feed it to quantile_bounds.

incr(counter: str, amount: int = 1) → None

is_live() → bool

is_ready() → bool

mark_ready() → None

observe(metric: str, value: float | None) → None

record_drop(reason: str, count: int = 1) → None

The node gave up on an input group: it will produce no output for it and the broker will not redeliver it. reason is one of the DROP_REASON_* values, or — for a drop decided below the Messenger seam, such as a join group evicted on timeout — a short stable word of the messenger’s own.

record_error(code: str, disposition: str) → None

Counts one failure under its stable code, so an alert can say what is failing.

record_offered() → None

One input group of real work handed to the node (end-of-stream and abort markers are not work).

render_metrics() → str

throughput(at: float | None = None) → ThroughputSnapshot

The throughput counters as one consistent reading (taken under the lock, so offered/processed/dropped describe the same instant). at defaults to time.time(); a test passes its own clock.

class videoflow.runtime.health.InstrumentedMessenger(inner: Messenger, state: HealthState, delivery_policy: DeliveryPolicy | None = None)

Bases: Messenger

Wraps a real Messenger and updates a HealthState as messages flow, without any change to the Task classes. Readiness note: the node is marked ready on its first messenger activity (first send or receive), which happens only after node.open() has returned inside NodeTask.run() — so a slow model-loading open() correctly keeps the pod un-ready until it finishes.

Drop accounting happens here too, from the delivery policy’s verdict on a failed input: a poison message is dead-lettered on its first failure in every mode, and best-effort delivery discards a transient failure instead of retrying it, so both count as dropped the moment fail_inputs sees them. What this seam cannot see is a message the messenger gives up on later — an at-least-once retry budget running out, a join group evicted on timeout, a collect buffer overflowing — because those verdicts are taken below Messenger with the delivery count and the group state in hand; they reach the counter only when the messenger reports them through HealthState.record_drop.

ack_inputs() → None

Acknowledge the input group last returned by receive_message — called by the task only after the node has processed it (and, for a processor, published its output). This ack-after-process ordering is what makes a crash mid-processing safe: the un-acked message is redelivered instead of lost.

check_for_termination() → bool

Returns true if a flow-wide termination signal has been received on the control channel. Used by videoflow.core.task.ProducerTask to stop pulling new input even before it naturally reaches StopIteration.

close() → None

Release any broker resources held by the messenger. Default: no-op.

fail_inputs(exc: BaseException) → None

Report that the node raised while processing the last input group. What that costs is decided by the node’s videoflow.core.policies.DeliveryPolicy together with how the error classified (see videoflow.core.errors): a poison message is dead-lettered immediately, a transient one is redelivered until its budget runs out, and a worker-fatal one is handed back for another replica and never blamed.

last_input_info() → dict | None

Per-parent envelope info (event_ts, metadata, trace_id, seq) for the input group last returned by receive_message (exposed to nodes as ctx.input_info). Default: None.

last_input_key() → str | None

A stable identity for the input group last returned by receive_message, used as a sink idempotency key. Default: None (no idempotency).

pending_count() → int

How many messages are waiting for this node across its parents. Used by videoflow.core.supervision.ProgressDeadline to tell a stalled node (work available, nothing acked) from an idle one. Default: 0, which reads as “idle” and so never trips the deadline.

pending_observation() → Any

pending_count as an observation: Known(count) when the broker answered, Unknown(reason) when it did not. The progress deadline reads this one, because “the query failed” and “nothing is pending” must lead to different decisions. Default: wraps pending_count as known.

publish_abort(error: Any) → None

Publishes an abnormal termination marker carrying why this node died. A clean end-of-stream and a crash are different facts, and only the first one used to exist on the wire — so a node that died mid-run left every child blocking forever on an EOS that was never coming. Downstream treats ABORT as end-of-stream-with-an-error: it stops, propagates the marker to its own children, and exits non-zero.

• Arguments:

  • error: a videoflow.core.errors.VideoflowError, a bare exception, or an already-normalized error dict being relayed from further upstream.

publish_message(message: Any, metadata: dict | None = None) → None

Publishes this node’s own output message. Depending on the flow’s configured retention policy (REALTIME vs BATCH), this may drop the message if downstream consumers are behind.

publish_stop_signal() → None

Publishes a termination marker on this node’s own subject. Unlike publish_message, this is never dropped regardless of retention policy — every downstream consumer must observe it exactly once.

quiesce() → None

Stop receiving new input; keep settling what is already held. Called on SIGTERM and by rollout/scale-down drains. Default: no-op.

receive_message() → dict

Blocks until this node has received a complete input: one message from every real parent, all derived from the same upstream originating event (see the trace_id propagation scheme in the concrete implementation) — or until every parent has signaled termination.

• Returns:

  • a dict {parent_name: entry} with exactly one entry per real parent of this node. Each entry carries message, metadata, event_ts, and two termination flags: is_stop_signal (that parent ended cleanly) and is_abort (it died). An aborted entry also carries abort_origin and abort_error — the originating node and its error record — so the failure can be reported and relayed downstream rather than degenerating into a hang.

set_output_event_timestamp(value: float) → None

Set the event time (epoch seconds) stamped on this node’s next published output (via RuntimeContext.set_event_timestamp) — when the underlying real-world event was captured. Producers of time-sensitive data (cameras, sensors) should set this; downstream nodes inherit their input group’s event time automatically. Default: no-op.

set_output_partition_key(value: Any) → None

Set the partition key attached to this node’s next published output (via RuntimeContext.set_partition_key), so a downstream partitioned node can route by a business key. Default: no-op.

stop_reason() → str | None

Which stop check_for_termination reports, one of the STOP_* constants, or None while the flow runs. Default: the control stop whenever a termination was signalled — the only kind a bare messenger has.

videoflow.runtime.health.LATENCY_BUCKETS_SECONDS: tuple[float, ...] = (0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0, 60.0)

Upper bounds, in seconds, of the latency histogram buckets. Prometheus buckets are cumulative: a bucket counts every observation <= le. The set is tuned for per-message processing time — dense from 5 ms to 1 s, where a video pipeline’s objectives live (a 30 fps budget is 33 ms), then sparse up to the minute-scale stages of a batch flow. A +Inf bucket is always appended, so the largest bound need not exceed every observation. Changing this set changes the le label values that dashboards and recording rules key on.

class videoflow.runtime.health.ThroughputSnapshot(offered: int, processed: int, dropped: int, at: float)

Bases: object

One reading of the throughput counters: what /metrics exports as messages_offered_total, messages_processed_total and messages_dropped_total summed over its reasons, with the time it was read. Two snapshots bracket a control window.

at: float

dropped: int

offered: int

processed: int

videoflow.runtime.health.parse_histogram(text: str, metric: str) → tuple[tuple[float, ...], tuple[int, ...]]

Read one histogram family back out of an exposition text, as the scrape side sees it: the finite bucket bounds (ascending) and the cumulative counts, one per bound plus the +Inf bucket last — the shape quantile_bounds takes. Both empty when the family has no _bucket lines: a count/sum-only exposition has no percentile to offer, and the empty answer is how a caller learns that instead of dividing a sum by a count.

• Arguments:

  • text: a /metrics body (one worker, so one node).

  • metric: the family name without the videoflow_ prefix or the _bucket suffix, e.g. proctime_seconds.

videoflow.runtime.health.quantile_bounds(cumulative: Sequence[int], q: float, bounds: Sequence[float] = (0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0, 60.0)) → tuple[float, float] | None

The bracket a nearest-rank quantile of a histogram lies in: the bucket that holds the ceil(q * count)-th observation, as (lower, upper) with the true quantile in (lower, upper]. upper is inf when the rank falls in the overflow bucket; lower is -inf for the first bucket.

This is the histogram’s declared error bound, and it is the most a bucketed export can truthfully say: an objective “p95 <= 50 ms” is met when upper <= 0.05, violated when lower >= 0.05, and undecidable when the threshold splits the bucket — which is a different answer from a guess.

• Arguments:

  • cumulative: the cumulative bucket counts, one per bound plus the final +Inf bucket, so cumulative[-1] is the observation count (HealthState.histogram and parse_histogram both produce this).

  • q: the quantile, in (0, 1].

  • bounds: the finite bucket bounds the counts were taken over.

• Returns:

  • (lower, upper), or None when there are no observations — a percentile of nothing is unavailable, not zero.

• Raises:

  • ValueError: q outside (0, 1], or counts and bounds of mismatched length.

videoflow.runtime.idempotency module

Optional sink-side idempotency: dedupe the effects of a consumer across message redelivery/restart, giving “exactly-once-ish” side effects on top of the broker’s at-least-once delivery. A consumer opts in with ConsumerNode(idempotent=True) and the flow must be given a Redis URL (reuses the blob-store Redis).

Consumers are single sinks (not replicated), so a plain check-then-mark against a shared store is race-free.

videoflow.runtime.idempotency.EFFECT_RETENTION_SECONDS = 86400.0

shorter than the dead-letter replay horizon, which is why a marker alone never certifies exactly-once (RUN-017) — the planner compares the two.

Type:

How long a sink’s effect marker is kept (the reviewed 86400 s default)

class videoflow.runtime.idempotency.IdempotencyStore

Bases: object

mark(key: str) → None

seen(key: str) → bool

class videoflow.runtime.idempotency.LedgerIdempotencyStore(runtime: Any, retention_seconds: float = 86400.0)

Bases: IdempotencyStore

Effect markers in the run ledger (RFC 0006 CTRL-4; RUN-017): the same seen/mark contract as the Redis store, kept where the node’s other durable facts are, with the retention the planner admitted (effect_retention_seconds). A marker is evidence that this runtime applied the effect; it never certifies exactly-once on its own — only a sink that keys its external effect (effect_guarantee = 'idempotent_key') does.

mark(key: str) → None

seen(key: str) → bool

class videoflow.runtime.idempotency.RedisIdempotencyStore(url: str, ttl_seconds: int = 86400)

Bases: IdempotencyStore

mark(key: str) → None

seen(key: str) → bool

videoflow.runtime.idempotency.idempotency_key(flow_id: str, node_name: str, message_id: str) → str

videoflow.runtime.logging_config module

Logging configuration for workers. Opt into JSON structured logs (one object per line, easy to ship to a log aggregator) by setting VF_STRUCTURED_LOGS=1; otherwise a plain human-readable format is used. Node-scoped fields (flow/run/node/ replica/trace ids) are included when a log record carries them as extra=....

class videoflow.runtime.logging_config.JsonFormatter(fmt=None, datefmt=None, style='%', validate=True, *, defaults=None)

Bases: Formatter

format(record: LogRecord) → str

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

videoflow.runtime.logging_config.configure_logging() → None

videoflow.runtime.provision module

Provisioning entrypoint (python -m videoflow.provision): creates every stream and durable consumer a flow needs, before its workers start. Run as a one-shot Kubernetes init Job so that BATCH interest-retention streams have their consumers registered before any message is published (otherwise early messages are dropped).

Driven by environment variables:

VF_NATS_URL         nats://host:port
VF_FLOW_ID          stable flow id
VF_RUN_ID           per-run id
VF_FLOW_TYPE        realtime | batch
VF_STREAM_REPLICAS  optional; stream copies to request (a replicated broker profile).
VF_MAX_RETRIES      optional; BATCH redelivery attempts (default 3)
VF_FLOW_SPECS_JSON  the compiled NodeSpecs as a JSON list, OR
VF_FLOW_SPECS_PATH  path to a file with that JSON (default /etc/videoflow/specs.json)
VF_BLOB_REDIS_URL   optional; the payload store the workers offload to — read
                    back for the admission below, never written here.
VF_PROFILE_REQUESTS_JSON  optional; the operator's explicit channel profiles
                    (deploy --require-profile) as JSON. Absent => the
                    flow-type presets, i.e. today's behaviour.
VF_ADMISSION_TIMEOUT_SECONDS optional; how long each read-back below may take,
                    connect included (default 60). A service that has not
                    answered by then is Unknown, never assumed.

Admission (RFC 0006 ENV-13). The composition is admitted before anything is created — the same admission videoflow deploy ran on the operator’s machine (deploy.admission), so a Job and a deploy never disagree — except that here the broker and the store are always read back live (jetstream_capabilities_observed / redis_payload_capabilities_observed): plan_composition admits them or rejects by name. A definite incompatibility is binding (IncompatibleProfile, exit 2); an unobservable capability binds only with explicit requests (UnobservableState, exit 3) and is a warning under the switch alone. After provisioning, explicit requests are checked once more against what the broker actually holds (topology.read_back_streams + verify_channel_profiles): a stream whose effective configuration contradicts its requested profile — reliable_work on a limits/discard-old stream, fewer copies than VF_STREAM_REPLICAS — is IncompatibleProfile whatever the switch says, because the request was explicit. Without requests and with the switch off nothing here changes: the streams are created exactly as they always were.

A typed rejection is rendered the way the CLI renders one (ERROR [code]: message and the remedy, on stderr — VideoflowError.render) and ends the process with the error’s exit code rather than a traceback: the Job’s termination message is what the deploy watchdog shows the operator. main raises SystemExit itself because the frozen videoflow.provision shim calls it bare and discards a return value.

videoflow.runtime.provision.admit_composition(nats_url: str, blob_redis_url: str | None, specs: Sequence[NodeSpec], flow_id: str, run_id: str, flow_type: str, explicit: Sequence[ProfileRequest], timeout: float) → None

The CLI’s admission, run again where the broker and store are actually reachable and always read back live. fail_fast = False: this Job may start while the broker it was deployed beside is still coming up, so the client’s own retry schedule applies within timeout.

• Raises:

  • IncompatibleProfile / UnobservableState: exactly as deploy.admission.admit.

videoflow.runtime.provision.ledger_durable_shared(url: str | None) → bool

Whether the runtime store behind url reads back durable and shared across processes.

videoflow.runtime.provision.main() → None

videoflow.runtime.provision.provision() → None

The whole entrypoint, raising typed errors for main to render: admission (when it applies), provisioning, verification (when requests were explicit).

videoflow.runtime.provision.verify_provisioned_profiles(nats_url: str, specs: Sequence[NodeSpec], flow_id: str, run_id: str, flow_type: str, explicit: Sequence[ProfileRequest], replicas: int, timeout: float) → None

Read the explicitly requested channels’ streams back after provisioning and bind the requests to what the broker holds — profile semantics and the requested fields (config_mismatches = True: this entrypoint knows the full request, VF_STREAM_REPLICAS included). Nothing to do without explicit requests.

• Raises:

  • IncompatibleProfile: a stream contradicts its requested profile or request.

  • BrokerUnavailable: a requested channel’s stream is not there after all.

  • UnobservableState: a requested channel could not be read back.

videoflow.runtime.redis_runtime_store module

The Redis RuntimeStore: the run ledger for workers spread across hosts (RFC 0006 CTRL-4, ENV-10). One Redis hash per key holds the value and its version; every mutation is an optimistic WATCH / MULTI / EXEC transaction on that one key, so a compare-and-swap is exactly one round of the same primitive the payload store uses — no Lua, no cross-slot commands, and a key layout a Redis Cluster can host (vfrt:{...} keys hash individually; a ledger never needs two keys in one transaction).

Durability is read back, never declared: the store is durable = Known(True) only when CONFIG GET shows persistence on (appendonly yes or an RDB save schedule) and maxmemory-policy noeviction, Known(False) when persistence is off, and Unknown('auth') when CONFIG is denied (a managed Redis). The planner admits restart_safe and durable_control only on Known(True); an unobservable ledger is not a durable one.

redis-py 8.0.1 facts relied on (.venv/lib/python3.12/site-packages/redis/): Pipeline.watch executes commands immediately until multi() (client.py:1763); a nil EXEC raises WatchError (client.py:1879); execute() always resets the pipeline (client.py:2039); hset(mapping=) (commands/core.py:9702); rpush returns the new length; scan_iter pages with SCAN (never KEYS).

class videoflow.runtime.redis_runtime_store.RedisRuntimeStore(url: str | None = None, client: Any = None, prefix: str = 'vfrt:')

Bases: RuntimeStore

• Arguments:

  • url: redis:// / rediss:// URL; ignored when client is given.

  • client: a ready redis.Redis (tests pass a fake with pipeline, hgetall, rpush, llen, scan_iter, config_get, info).

  • prefix: what every key is prefixed with on the server, so a ledger never collides with the payload store sharing the instance.

append(log: str, record: bytes) → int

capabilities() → RuntimeCapabilities

Read back: persistence on and noeviction ⇒ durable; denied CONFIG ⇒ Unknown.

cas(key: str, expected_version: str | None, value: bytes) → bool

delete(key: str, expected_version: str | None) → bool

get(key: str) → tuple[bytes | None, str | None]

log_entries(log: str) → list[bytes]

Every record appended to log, oldest first (empty for an unknown log).

scan(prefix: str) → list[tuple[str, bytes, str]]

videoflow.runtime.redis_runtime_store.TRANSACTION_RETRIES = 8

Bounded WATCH retries on a conflict that is not the caller’s (a lost reply, a concurrent touch of an unrelated field): a genuine version mismatch is reported after the first round, never retried.

videoflow.runtime.redis_runtime_store.runtime_store_from_url(url: str, client: Any | None = None) → RedisRuntimeStore

The registry factory for redis:// and rediss:// (runtime_stores.make_runtime_store).

videoflow.runtime.runtime_stores module

Runtime-store selection by URL scheme (RFC 0006 ENV-10, VF_RUNTIME_STORE_URL).

Three schemes are built in — memory:// (per-process, never durable: tests and the pre-RFC default), file://<dir> (one host, durable across restarts: the local engine’s default under the switch) and redis:// / rediss:// (cross-host; durable only when the server’s persistence is read back on). The registry follows the blob-store one: a module-level table seeded with the built-ins, an explicit register_runtime_store, and a lookup that names the known schemes and the fix. What each store is — durable or not, shared or not — is the store’s own capabilities(), which the planner reads; nothing here promises more than the store reports.

videoflow.runtime.runtime_stores.DEFAULT_RUNTIME_STORE_URL = 'memory://'

no durable state, nothing shared.

Type:

The default when VF_RUNTIME_STORE_URL is unset

videoflow.runtime.runtime_stores.known_runtime_store_schemes() → list[str]

videoflow.runtime.runtime_stores.make_runtime_store(url: str | None) → RuntimeStore

• Arguments:

  • url: VF_RUNTIME_STORE_URL; None or empty means memory://.

• Raises:

  • ValueError: an unknown scheme, naming the known ones and register_runtime_store.

videoflow.runtime.runtime_stores.register_runtime_store(scheme: str, factory: Callable[[str], RuntimeStore]) → None

Register a RuntimeStore factory for a URL scheme (case-insensitive); factory receives the full URL.

videoflow.runtime.scaling module

Autoscaling demand as a pure function of what the broker says about each of a node’s parents, and the rules that say when a node must not be autoscaled at all.

Why this exists. videoflow deploy --autoscaling renders a KEDA ScaledObject per processor that scales its Deployment on JetStream consumer lag. Two things were wrong with how that demand was derived, and both are the kind of bug no test fails on because the wrong answer is a plausible number:

• Demand was read from one parent. The scaler was keyed on the node’s first declared parent only, so a join whose first parent was drained while its second backed up reported no lag at all — and which parent was “first” was an accident of the process(a, b) declaration, so two equivalent graphs scaled differently. observe_demand evaluates every parent it is given, returns a canonical (name-sorted) decision that is identical for any permutation of the input, and takes the replica recommendation as the maximum over parents. That is also exactly what a ScaledObject with one trigger per parent does — the HPA “will calculate metrics for every scaler and use the highest desired replica count” (KEDA FAQ) — so the pure function and the rendered scaler agree by construction.

• A failed read was a zero. A consumer_info that timed out looked like an empty consumer. Here a parent whose observation is Unknown makes the whole demand unknown: DemandDecision.replicas is None and the parent is named. Never a zero that would scale a busy node down.

Demand is diagnosed as well as sized, because RUN-046’s second question is whether extra workers can help at all. A multi-parent node is a join: every group needs a message from each required parent. When every parent has work waiting, the backlog is a processing-capacity shortfall that replicas repair. When some parents are empty while others back up, the backed-up messages are waiting for counterparts that have not arrived — a missing-input stall that no replica count repairs, and that a quorum or timeout join policy, not a scaler, resolves. The decision carries both the recommendation and the diagnosis; the caller that knows the join policy decides what a starved join gets.

Eligibility is separate from demand. scaling_rejections names every reason a node stays at a fixed scale, mirroring the rules deploy.manifests applies when it renders: only processors; not partitioned nodes, whose key ownership would rehash; not a multi-parent join at a single replica, which a scaler would turn into competing multi-worker joins (RUN-018); not a node that declares partition_by at nb_tasks = 1, because the intent is not erased by the replica count — scaling it from one binds every new replica to the same competing durable and splits one key’s history among unfenced states (RUN-019); GPU nodes only on explicit opt-in. One rule is an error rather than a silent skip. A BATCH flow renders every node as a Kubernetes Job, and a Job’s parallelism is fixed when it is created — it is not what a scaler scales: KEDA’s ScaledObject and the HPA drive the /scale subresource of a Deployment or StatefulSet, and a Job has none. A ScaledObject whose scaleTargetRef names the Deployment a Job-rendered node never gets is a dangling scaler that does nothing while the operator believes it scales (RUN-027), so job_autoscaling_error is raised at render time instead.

Two more observers live here because they are the same kind of pure decision:

• observe_rate_demand sizes demand from throughput rather than lag, for the live-video path whose lossy retention conceals overload (RUN-026): a live_latest channel keeps one message per subject, so a node at a third of the offered rate shows a lag of one and a scaler keyed on lag never fires, while two frames in three are evicted before delivery. The health counters — offered, processed, dropped over a control window — see the loss the queue depth hides, and the declared objective (RateObjective) says how much of it is acceptable. A missing counter is DEMAND_UNKNOWN, never idle.

• reconcile_capacity keeps desired concurrency (what demand asks for) apart from granted (claims the allocator holds) and ready (workloads observed processing) (RUN-028): a Pending pod or an allocation request is not processing capacity, a shortfall is a named capacity_constrained decision rather than a silently smaller scale-out, and a claim that could not be observed is listed as unknown instead of counted either way.

Nothing here touches a broker or a cluster: the inputs are backends.messaging.SubscriptionObservation values wrapped in backends.outcomes.Known/Unknown, and every function is deterministic.

videoflow.runtime.scaling.CAPACITY_ADMITTED = 'admitted'

Diagnoses of a CapacityDecision (reconcile_capacity).

class videoflow.runtime.scaling.CapacityDecision(desired: int, granted: int, ready: int, admitted: int, diagnosis: str, reason: str = '', unknown_claims: tuple[str, ...] = ())

Bases: object

The outcome of reconcile_capacity.

• Arguments:

  • desired: the replicas demand asked for.

  • granted: claims observed holding a grant (allocated, prepared or ready).

  • ready: claims whose workload is observed ready — the only count that is processing capacity.

  • admitted: min(desired, granted): what may be scheduled now.

  • diagnosis: one of CAPACITY_DIAGNOSES.

  • reason: the sentence a report prints for a constrained or unknown decision.

  • unknown_claims: claims whose observation was Unknown, sorted; neither granted nor ready, and named.

admitted: int

desired: int

diagnosis: str

granted: int

ready: int

reason: str = ''

unknown_claims: tuple[str, ...] = ()

videoflow.runtime.scaling.DEFAULT_TARGET_LAG_PER_REPLICA = 10

lagThreshold: '10' in deploy.manifests.scaled_object and the CLI’s --max-replicas default.

Type:

The rule’s defaults, mirroring the rendered scaler

videoflow.runtime.scaling.DEMAND_UNKNOWN = 'unknown'

Diagnoses a DemandDecision carries alongside its replica recommendation.

videoflow.runtime.scaling.DEMAND_WITHIN_OBJECTIVE = 'within_objective'

Diagnoses of a RateDecision (observe_rate_demand), alongside DEMAND_UNKNOWN and DEMAND_IDLE.

class videoflow.runtime.scaling.DemandDecision(replicas: int | None, diagnosis: str, parents: tuple[ParentDemand, ...], unknown_parents: tuple[str, ...] = (), starved_parents: tuple[str, ...] = ())

Bases: object

The outcome of observe_demand. Canonical: two calls over the same observations handed over in any order compare (and hash) equal.

• Arguments:

  • replicas: the recommendation — the maximum over parents, clamped to [min_replicas, max_replicas] — or None when the diagnosis is DEMAND_UNKNOWN. Never 0 for “could not tell”.

  • diagnosis: one of DIAGNOSES.

  • parents: every evaluated parent, sorted by name.

  • unknown_parents: the parents that could not be observed, sorted.

  • starved_parents: the parents with nothing waiting while another parent has a backlog, sorted; empty unless the diagnosis is DEMAND_STARVED.

diagnosis: str

property known: bool

Whether every parent was observed, i.e. whether replicas is a number.

parents: tuple[ParentDemand, ...]

replicas: int | None

starved_parents: tuple[str, ...] = ()

unknown_parents: tuple[str, ...] = ()

videoflow.runtime.scaling.JOB_REJECTION = "the node renders as a Kubernetes Job, whose parallelism is fixed when it is created and is not what a scaler scales: KEDA's ScaledObject and the HPA drive the /scale subresource of a Deployment or StatefulSet, and a Job has none."

The one rejection that is an error rather than a fixed-scale skip (RUN-027).

videoflow.runtime.scaling.JOIN_REJECTION = 'the node joins several parents at a single replica: a scaler would turn that singleton join into competing multi-worker joins, delivering the halves of one group to different replicas where neither can assemble it, and no runtime store advertises the elastic join state that would let group ownership follow the replica count.'

A singleton join must not become competing multi-worker joins (RUN-018).

videoflow.runtime.scaling.MISSING_OBSERVATION = 'missing'

ParentDemand.unknown_reason for a required parent nobody observed at all.

videoflow.runtime.scaling.PARTITION_INTENT_REJECTION = "the node declares partition_by: a replica count of 1 does not erase that intent, and scaling it from 1 would bind every new replica to the same competing durable, splitting one key's history among independent, unfenced states — a partitioned node scales only by redeploying with the nb_tasks it should own its keys at."

Partition intent declared at nb_tasks = 1 is intent, not an accident of the replica count (RUN-019).

class videoflow.runtime.scaling.ParentDemand(parent: str, lag: int | None, replicas: int | None, unknown_reason: str = '')

Bases: object

One parent’s contribution to a decision.

• Arguments:

  • parent: the parent node’s name.

  • lag: the messages the node still owes this parent — not yet delivered plus delivered-but-unacked — or None when the observation was Unknown.

  • replicas: what this parent alone asks for under the rule, or None when unknown.

  • unknown_reason: the Unknown.reason ('timeout', MISSING_OBSERVATION …); empty when the parent was observed.

lag: int | None

parent: str

replicas: int | None

unknown_reason: str = ''

class videoflow.runtime.scaling.RateDecision(replicas: int | None, diagnosis: str, delivered_fraction: float | None = None, sample: ThroughputSample | None = None, unknown_reason: str = '')

Bases: object

The outcome of observe_rate_demand.

• Arguments:

  • replicas: the recommendation, or None when the diagnosis is DEMAND_UNKNOWN or DEMAND_STALLED. Never 0 for “could not tell”.

  • diagnosis: one of RATE_DIAGNOSES.

  • delivered_fraction: what the window measured, None when unknown or idle.

  • sample: the sample decided on, None when unknown.

  • unknown_reason: why the sample could not be used (Unknown.reason, MISSING_OBSERVATION, or 'window' for a sample shorter than the control window).

delivered_fraction: float | None = None

diagnosis: str

replicas: int | None

sample: ThroughputSample | None = None

unknown_reason: str = ''

class videoflow.runtime.scaling.RateObjective(min_delivered_fraction: float, control_window_seconds: float, stabilization_seconds: float = 0.0)

Bases: object

The declared delivery objective a live node must meet, per control window.

• Arguments:

  • min_delivered_fraction: the least processed / offered that is acceptable over one window (1.0 means every offered frame).

  • control_window_seconds: how long a window is; a sample shorter than this is not yet a decision.

  • stabilization_seconds: how long after a replica change a further change is held back, so a decision is not taken against a fleet that has not finished starting.

control_window_seconds: float

min_delivered_fraction: float

stabilization_seconds: float = 0.0

class videoflow.runtime.scaling.ThroughputSample(offered: int, processed: int, dropped: int, window_seconds: float)

Bases: object

One control window of a node’s throughput, from the health counters and the subscription observation.

• Arguments:

  • offered: inputs the source offered into the node’s channel during the window — what the node was handed (messages_offered_total) plus what the channel evicted before delivery (SubscriptionObservation.dropped). Under lossy retention the second term is the whole story.

  • processed: inputs the node acknowledged during the window (messages_processed_total).

  • dropped: inputs lost during the window — evicted before delivery, or given up on by policy (messages_dropped_total).

  • window_seconds: the window’s length.

property delivered_fraction: float | None

processed / offered, or None when nothing was offered.

dropped: int

offered: int

processed: int

window_seconds: float

videoflow.runtime.scaling.admit_autoscaling(node_name: str, flow_type: str, spec_kind: str, is_partitioned: bool, is_job: bool, has_parents: bool = True, device_type: str = 'cpu', gpu_autoscaling: bool = False, is_join: bool = False, declares_partition: bool = False) → list[str]

The render-time admission for one node when --autoscaling is on: the fixed-scale rejections to log (render no scaler, keep the declared scale), or the Job refusal raised.

• Returns:

  • scaling_rejections(...) for this node; empty means render a scaler.

• Raises:

  • CapabilityError: the node is a processor that renders as a Job — the flag cannot mean anything for it, and a scaler would dangle.

videoflow.runtime.scaling.job_autoscaling_error(node_name: str, flow_type: str) → CapabilityError

The RUN-027 refusal, built in one place so the renderer and an admission check raise the same words: autoscaling was requested for a node that renders as a Job. Raised before anything is applied — the alternative is a ScaledObject whose scaleTargetRef names a Deployment that does not exist, which the API server accepts and the HPA then cannot resolve, so the flow runs at nb_tasks forever while the operator believes it scales.

videoflow.runtime.scaling.observe_demand(observations: Mapping[str, Known[SubscriptionObservation] | Unknown], required_parents: Iterable[str] | None = None, target_lag_per_replica: int = 10, min_replicas: int = 1, max_replicas: int = 10) → DemandDecision

Size and diagnose a node’s demand from every parent’s subscription observation.

• Arguments:

  • observations: parent name -> what the broker said about the node’s subscription on that parent (Known or Unknown). Iteration order is irrelevant: the decision is canonical.

  • required_parents: the parents that must be evaluated. Defaults to the observed ones; a required parent with no observation at all counts as Unknown(MISSING_OBSERVATION), because “nobody asked” is not “no work”.

  • target_lag_per_replica, min_replicas, max_replicas: the rule; see replicas_for_lag.

• Returns:

  • a DemandDecision: replicas is the maximum over parents when every one was observed, else None with the unknown parents named.

• Raises:

  • ConfigError: an invalid rule (see validate_scaling_rule), or no parents at all to evaluate.

videoflow.runtime.scaling.observe_rate_demand(sample: Known[ThroughputSample] | Unknown | None, objective: RateObjective, current_replicas: int, min_replicas: int = 1, max_replicas: int = 10, last_change_at: float | None = None, now: float = 0.0) → RateDecision

Size and diagnose a live node’s demand from its throughput over one control window, against the declared objective.

• Arguments:

  • sample: the window’s ThroughputSample, as Known/Unknown; None when nobody read the counters (MISSING_OBSERVATION).

  • objective: the declared RateObjective.

  • current_replicas: how many replicas produced sample.processed; the per-replica capacity is derived from it.

  • min_replicas, max_replicas: the admissible range; the recommendation is clamped, never invented beyond it.

  • last_change_at, now: when the replica count last changed and the current time on the same clock; a breach within objective.stabilization_seconds of the change is DEMAND_STABILIZING and holds current_replicas.

• Returns:

  • a RateDecision. A breach asks for ceil(offered / (processed / current_replicas)) replicas — enough for the measured per-replica capacity to absorb the offered rate — clamped to the range.

• Raises:

  • ConfigError: an invalid objective or range.

videoflow.runtime.scaling.parent_lag(observation: SubscriptionObservation) → int

The messages the node still owes one parent: available (not yet delivered) plus leased (delivered, not yet acked) — the same sum the KEDA nats-jetstream scaler reads (num_pending + num_ack_pending), so a decision here matches the rendered trigger. unresolved is excluded on purpose: work the broker can no longer deliver is not work another replica could take.

videoflow.runtime.scaling.reconcile_capacity(desired: int, claims: Mapping[str, Known[ClaimObservation] | Unknown]) → CapacityDecision

Reconcile the replicas demand wants with what the allocator actually holds.

• Arguments:

  • desired: the demand decision’s replica count (>= 0).

  • claims: claim id -> the allocator’s observe(claim_id); one claim per replica the plan asked for.

• Returns:

  • a CapacityDecision. A claim counts as granted only when observed allocated/prepared/ready with a grant, as ready only when observed ready; a pending or failed claim is neither, and an Unknown observation is listed, never counted. desired > granted is capacity_constrained with the shortfall named; a shortfall that unknown claims could cover is unknown.

videoflow.runtime.scaling.replicas_for_lag(lag: int, target_lag_per_replica: int, min_replicas: int, max_replicas: int) → int

The per-parent rule: enough replicas for each to carry at most target_lag_per_replica messages — ceil(lag / target), the HPA’s AverageValue arithmetic — clamped to [min_replicas, max_replicas].

videoflow.runtime.scaling.scaling_rejections(spec_kind: str, is_partitioned: bool, is_job: bool, has_parents: bool = True, device_type: str = 'cpu', gpu_autoscaling: bool = False, is_join: bool = False, declares_partition: bool = False) → list[str]

Every reason this node stays at a fixed scale — empty means a scaler may be rendered. Mirrors the rules deploy.manifests applies, in one place, so an admission check and the renderer cannot drift.

• Arguments:

  • spec_kind: NodeSpec.kind.

  • is_partitioned: partition_by set with nb_tasks > 1 (manifests._is_partitioned).

  • is_job: whether the workload renders as a Kubernetes Job — every node of a BATCH flow, and a finite producer of a REALTIME one (manifests.workload).

  • has_parents: whether the node consumes anything, i.e. has a stream whose lag could be measured.

  • device_type, gpu_autoscaling: NodeSpec.device_type and the deploy’s --gpu-autoscaling opt-in; GPU nodes are excluded without it.

  • is_join: the node has more than one parent. A join at one replica is a singleton join; scaled, its groups’ halves land on different replicas (JOIN_REJECTION, RUN-018).

  • declares_partition: partition_by is set, whatever nb_tasks is. Distinct from is_partitioned on purpose: at nb_tasks = 1 the node binds a competing durable, and a scaler would add replicas to that durable (PARTITION_INTENT_REJECTION, RUN-019).

• Returns:

  • the applicable reasons, each a sentence naming the rule and its consequence. JOB_REJECTION is the one that must not be skipped silently: admit_autoscaling raises for it.

videoflow.runtime.scaling.throughput_sample(offered_before: int, offered_after: int, processed_before: int, processed_after: int, dropped_before: int, dropped_after: int, evicted_before: int, evicted_after: int, window_seconds: float) → ThroughputSample

A ThroughputSample from two readings of the counters: the node’s offered / processed / dropped totals (HealthState.throughput) and the channel’s evicted-before-delivery count (SubscriptionObservation.dropped), which is folded into both offered and dropped because the node never saw those inputs at all. Counters only grow, so a reading that went down is a restarted exporter: the sample is taken from zero for that counter.

videoflow.runtime.scaling.validate_rate_objective(objective: RateObjective) → None

• Raises:

  • ConfigError: a delivered fraction outside (0, 1], a control window that is not positive, or a negative stabilization window.

videoflow.runtime.scaling.validate_scaling_rule(target_lag_per_replica: int, min_replicas: int, max_replicas: int) → None

• Raises:

  • ConfigError: a target below 1 (a replica must absorb some lag), a minimum below 1 (a processor at zero replicas never observes its end-of-stream), or a maximum below the minimum.

videoflow.runtime.watchdog module

The progress watchdog: the thread that re-checks ProgressDeadline while the node is inside process()/consume().

Why a thread at all. The task loop consults the deadline between messages, which catches a node that is alive but no longer acking — and misses the one failure the deadline was written for: a callback that never returns. A process() wedged on a lock, an unbounded read or a device call never gets back to the loop, so the loop never checks, and the pod sits there with its broker lease heartbeats perfectly healthy (the messenger extends acks from its own I/O thread) while the message it holds goes nowhere. The watchdog owns nothing new: it calls the same ProgressDeadline the loop uses, on its own daemon thread, every interval_seconds. A node that is merely slow keeps calling record_progress() on every ack and is never touched; a node that is stuck with work pending trips the deadline within timeout + interval — the “declared tolerance” of RUN-011 — and the watchdog hands the resulting error to on_stall exactly once, then stops.

What on_stall does is the caller’s decision, which is what keeps this module pure enough to test with a fake clock: the worker’s callback writes the termination reason and calls os._exit (a thread cannot unwind the main thread’s wedged frame, so exiting the process is the only recovery there is); a test’s callback records the error and releases whatever latch stood in for the hang.

tick() is public for the same reason ProgressDeadline takes a clock: one check, no thread, so “does it fire, and does it not fire” is answerable without waiting on real time. start() merely runs tick() on a schedule.

videoflow.runtime.watchdog.DEFAULT_WATCHDOG_INTERVAL_SECONDS = 5.0

Seconds between deadline checks. Small next to the default 300s timeout, so the detection tolerance it adds is negligible, and large enough that the broker query behind pending_observation (consulted only once the silence threshold is crossed) is never a load of its own.

class videoflow.runtime.watchdog.ProgressWatchdog(deadline: ProgressDeadline, interval_seconds: float, on_stall: Callable[[BaseException], None], name: str = '')

Bases: object

Re-checks a ProgressDeadline on a daemon thread and reports the first stall to on_stall.

• Arguments:

  • deadline: the node’s deadline — the same instance the task loop calls record_progress()/check() on, or the watchdog would be measuring silence nobody resets.

  • interval_seconds: how often check() runs. Must be positive; the worker treats 0 as “no watchdog” and never constructs one.

  • on_stall: called exactly once, on the watchdog thread, with the ProgressStalled or BrokerUnavailable the deadline raised. May not return (os._exit); the watchdog is finished either way.

  • name: names the thread, so a stack dump says which node it belongs to.

• Raises:

  • ConfigError: if interval_seconds is not positive.

property fired: bool

Whether a stall was detected (and on_stall called).

property running: bool

property stall: BaseException | None

The error the watchdog fired with, or None while nothing has stalled.

start() → None

Starts the thread. A second call while running, or after firing, is a no-op.

stop() → None

Stops the thread and joins it. Idempotent, and safe from on_stall itself (a thread cannot join itself, so that call only signals).

tick() → bool

One deadline check. Returns whether the watchdog has fired — on this tick or an earlier one; on_stall is never called twice.

A probe that raises something the deadline does not classify is logged and ignored: the watchdog exists to notice a node that stopped, and a flaky observation is not evidence of that.

videoflow.runtime.worker module

The single process entrypoint that runs exactly one graph node — used identically whether the node is launched as a local subprocess (LocalProcessEngine) or as a Kubernetes pod. It’s fully driven by environment variables so it needs no access to the original graph-building script:

VF_NODE_CLASS       fully-qualified class, e.g. videoflow.processors.basic.IdentityProcessor
VF_NODE_PARAMS_JSON JSON dict of constructor kwargs (from NodeSpec.params)
VF_NODE_KIND        producer | processor | consumer. Must match the family of
                    VF_NODE_CLASS; the compiler writes both together, and the
                    worker rejects a disagreement up front (see require_node_kind).
VF_NODE_NAME        this node's stable name
VF_PARENT_NAMES     comma-separated parent node names ('' if none)
VF_HAS_CHILDREN     '1' or '0'
VF_NATS_URL         nats://host:port
VF_FLOW_ID          shared flow identifier (stable across runs)
VF_FLOW_TYPE        realtime | batch
VF_RUN_ID           per-run identifier that scopes this run's broker streams
VF_REPLICA_ID       index of this replica (0 for single-task nodes); absent and with no
                    pod ordinal either, a competing replica claims the lowest free slot
                    through the run ledger (ENV-5 step 3, up to VF_REPLICA_SLOTS — the
                    scaler's ceiling when one drives the node — else VF_NB_TASKS)
VF_ACK_WAIT_SECONDS optional; per-message ack deadline (default 60)
VF_MAX_RETRIES      optional; BATCH redelivery attempts before dead-letter (default 3)
VF_EOS_QUIESCENCE_MS optional; drain quiescence window before honoring EOS (default 500)
VF_NB_TASKS         optional; replica count of this node (for partition ownership)
VF_PARTITION_BY     optional; partition key ('trace_id' or a metadata field)
VF_JOIN_POLICY_JSON optional; JSON JoinPolicy for a multi-parent node
VF_BLOB_REDIS_URL   optional; enables the external blob store for large payloads.
                    The store is chosen by the URL's scheme (redis:// and
                    rediss:// built in; others via register_blob_store), so the
                    name is historical rather than a restriction to Redis.
VF_FAULT_SCHEDULE_JSON / VF_FAULT_MARKER_DIR
                    optional; a conformance test's fault schedule (RFC 0006 ENV-16/17),
                    installed before the node is built. Never set by a control plane.
VF_PREFETCH_BYTES   optional; the most envelope bytes a worker holds unsettled before it
                    stops fetching (RUN-025). Unset ⇒ bounded by the prefetch count only.
VF_RUNTIME_STORE_URL optional; the run ledger (RFC 0006 ENV-10): memory:// (default),
                    file://<dir> (one host) or redis:// (read back for persistence).
                    A durable, shared store turns on the EOS-7 completion
                    barrier, the partition lease and the ledger-budgeted
                    delivery cap (D11).
VF_PARENT_REPLICAS  optional; per-parent replica counts aligned with VF_PARENT_NAMES
                    (ENV-11); a count mismatch fails fast. Absent ⇒ the EOS-3 drain.
VF_BLOB_READER_IDS  optional; the reader obligations each payload this node
                    publishes is held for, comma-separated (RFC 0006 BLOB-13);
                    honoured with a redis:// blob store (the obligation store).
VF_BLOB_READERS     optional; how many downstream reads each message this node
                    publishes receives — enables refcounted blob reclamation
                    (PROTOCOL.md BLOB-5). Unset ⇒ blobs are TTL-only.
VF_BLOB_TTL_SECONDS optional; TTL for offloaded payloads (PROTOCOL.md BLOB-7).
                    Unset ⇒ flow-type default (3600 realtime / 86400 batch).
VF_GPU_COUNT        optional; whole GPUs granted to this worker (GPU nodes
                    only, RFC 0003). Visible devices are exactly 0..count-1.
                    Informational — a Python node's own gpu_count param is
                    authoritative; native components read this.
VF_GPU_RESOURCE_NAME optional; extended-resource name the GPUs were requested
                    as, e.g. a MIG profile (RFC 0003).
VF_GPU_GRANT_JSON   optional; the DeliveredGrant the launcher recorded (RFC 0006
                    ENV-14): device identities, the requested count, whether the
                    grant is exclusive and the policy that made it. Checked
                    against the node's gpu_count/gpu_fallback/requires_peer_access
                    before the node opens (runtime.gpucheck); absent ⇒ the
                    CUDA-visible namespace is enumerated instead.
VF_ENVELOPE_VERSION optional; wire envelope version to emit (only 4, protobuf)
VF_DELIVERY         optional; 'at-least-once' | 'best-effort', overriding the
                    flow type's preset for this node (PROTOCOL.md §7)
VF_ON_ERROR         optional; disposition for exceptions nothing classifies —
                    'poison' | 'transient' | 'worker_fatal' (default transient)
VF_BREAKER_THRESHOLD optional; consecutive failures before the worker declares
                    itself unhealthy and exits so its inputs go to another
                    replica (default 10; 0 disables)
VF_PROGRESS_TIMEOUT_SECONDS optional; seconds this node may ack nothing while
                    work is pending before it is declared stalled (default 300;
                    0 disables)
VF_PROFILE_REQUESTS_JSON  optional; the operator's explicit channel profiles
                    (deploy --require-profile), as JSON. Enforced at bind
                    (RFC 0006 ENV-13): before the node is built or opened,
                    the streams of this node's own channel and its parents'
                    are read back and a stream that does not carry its
                    requested profile ends the worker with
                    IncompatibleProfile (exit 2); one that could not be read
                    back, with UnobservableState (exit 3). Absent => the
                    flow-type presets, i.e. today's behaviour — nothing is
                    read back.
VF_ADMISSION_TIMEOUT_SECONDS optional; how long that read-back may take,
                    connect included (default 60)
VF_WATCHDOG_INTERVAL_SECONDS optional; how often a watchdog thread re-checks
                    that same progress deadline while the node is *inside*
                    process()/consume(), so a callback that never returns is
                    still caught — the run loop only checks between messages
                    (default 5; 0 disables the thread, leaving the loop's own
                    check). A stall found this way is written to the
                    termination log and ends the process with the error's
                    exit code (ProgressStalled: 5)
VF_TERMINATION_LOG  optional; path the structured termination reason is written
                    to (default /dev/termination-log, which Kubernetes surfaces
                    in the pod's containerStatuses)

SIGTERM — a pod being deleted, a rollout, a local kill — first calls messenger.quiesce() (stop admitting input; what is already held keeps being settled) and then takes the signal’s default action, so a worker still dies of SIGTERM exactly as it always has: nothing is acked that was not processed, and the un-acked inputs are redelivered to its replacement.

videoflow.runtime.worker.DEFAULT_TERMINATION_LOG = '/dev/termination-log'

Where the structured termination reason is written. Kubernetes reads this path by default and surfaces its contents in pod.status.containerStatuses[].state.terminated.message, which is how the deploy watchdog learns why a pod died without scraping its logs.

videoflow.runtime.worker.build_node_from_env() → Node

videoflow.runtime.worker.build_watchdog(deadline: ~videoflow.core.supervision.ProgressDeadline, interval_seconds: float, progress_timeout: float, node_name: str, on_stall: ~typing.Callable[[BaseException], None] = <function exit_on_stall>) → ProgressWatchdog | None

The watchdog for a non-producer node, or None when either knob disables it: an interval of 0 means no thread, and a progress timeout of 0 means the deadline itself never trips, so a thread would only be re-checking nothing.

• Arguments:

  • deadline: the node’s ProgressDeadline — the same instance the task loop is given, which is what makes the watchdog’s reading of “silence” the loop’s own.

  • interval_seconds: VF_WATCHDOG_INTERVAL_SECONDS.

  • progress_timeout: VF_PROGRESS_TIMEOUT_SECONDS.

  • node_name: names the thread.

  • on_stall: what to do with the stall; the process-ending default is replaced in tests.

videoflow.runtime.worker.exit_on_stall(error: BaseException, exit_process: ~typing.Callable[[int], ~typing.Any] = <built-in function _exit>) → None

What the progress watchdog does with a stall: record the reason, then end the process. os._exit rather than an exception, because the watchdog runs on its own thread and the frame that is wedged — a process() that never returned — belongs to the main thread, which nothing can unwind. The un-acked inputs go back to the broker for the replacement, exactly as they would after any other death.

• Arguments:

  • error: the ProgressStalled (or BrokerUnavailable) the deadline raised.

  • exit_process: the process-ending call; injected so a test can observe the exit status instead of losing the interpreter.

videoflow.runtime.worker.install_sigterm_quiesce(messenger: Messenger, then: Callable[[int], None] | None = None) → Callable[[], None]

SIGTERM → messenger.quiesce(), then the signal’s default action.

Quiescing first is what lets the messenger stop admitting input (and hand back what it prefetched but never delivered) before the process is gone, instead of leaving those messages to time out on the broker; the default action afterwards keeps every existing contract about how a SIGTERMed worker dies — exit status, no clean end-of-stream, un-acked inputs redelivered (as PID 1 of a container the process exits 128 + SIGTERM itself, since init is never killed by a default-action signal). Finishing the in-flight message before exiting is the rollout drain’s job (a later phase), not this hook’s.

Only the main thread may install a signal handler, so from any other thread (a test driving run_from_env in-process) this installs nothing.

• Arguments:

  • messenger: whose quiesce() runs on SIGTERM.

  • then: what follows the quiesce, given the signal number. Default: restore SIG_DFL and re-raise the signal to this process. Injected so a test can deliver a real SIGTERM without dying of it.

• Returns:

  • a callable that restores the previous SIGTERM disposition (a no-op when nothing was installed).

videoflow.runtime.worker.main() → int

Entrypoint. Returns the process exit status rather than raising, so a typed failure exits with the code its class carries (see videoflow.core.errors) and an operator can tell a bad node from a bad cluster without reading the log.

videoflow.runtime.worker.parent_replicas_from_env(parent_names: Sequence[str], raw: str | None) → dict[str, int]

VF_PARENT_REPLICAS (RFC 0006 ENV-11): comma-separated replica counts aligned with VF_PARENT_NAMES. Absent ⇒ empty (the EOS-7 barrier is not evaluated).

• Raises:

  • ConfigError: an entry count that differs from the parent count, or a value that is not a non-negative integer.

videoflow.runtime.worker.require_node_kind(node: Node, expected: Type[_N], kind: str) → _N

Check that the node built from VF_NODE_CLASS matches the node family declared by VF_NODE_KIND, and return it narrowed to that family.

The two env vars are written together by the compiler, so a mismatch means the ConfigMap was hand-edited or the image is stale. Without this check the wrong Task type is constructed and the disagreement only surfaces later, deep in the run loop, as an opaque AttributeError (e.g. calling next() on a consumer).

• Raises:

  • ValueError: if node is not an instance of expected.

videoflow.runtime.worker.run_from_env() → None

videoflow.runtime.worker.verify_explicit_profiles(nats_url: str, flow_id: str, run_id: str, flow_type: str, node_name: str, parent_names: Sequence[str], has_children: bool, requests: Sequence[ProfileRequest], timeout: float) → None

Bind the operator’s explicit channel profiles (VF_PROFILE_REQUESTS_JSON) to the streams this node touches — its own output channel and its parents’ — before it opens. The streams are read back over a connection of their own (topology.read_back_streams; nothing reaches into the messenger), and a stream whose effective configuration contradicts the profile it was requested to carry is refused: IncompatibleProfile, which main turns into exit 2 with the reason in the termination log. A stream that could not be read back is refused too (UnobservableState, exit 3): an explicit request does not pass on an unread guarantee. Requests for channels this node neither publishes nor consumes are other workers’ to judge; no requests at all means nothing is read and nothing changes.

videoflow.runtime.worker.watchdog_interval_from_env() → float

VF_WATCHDOG_INTERVAL_SECONDS as a number: the default when unset, 0 for “no watchdog thread”.

• Raises:

  • ConfigError: if the value is not a number, or is negative.

videoflow.runtime.worker.write_termination_reason(error: BaseException, path: str | None = None) → None

Records why this worker is exiting, in machine-readable form.

Deliberately best-effort and completely silent on failure: the file may not exist outside Kubernetes, and a worker that cannot explain its death must still die of the original cause rather than of a logging problem.

• Arguments:

  • error: the exception that ended the worker.

  • path: override for the termination-log path (VF_TERMINATION_LOG).