videoflow.deploy package

Submodules

videoflow.deploy.admission module

Composition admission: does the broker and payload store a flow is about to run on actually provide the delivery guarantees its channels ask for?

Every channel of a flow carries a messaging profile (live_latest for a REALTIME flow, reliable_work for a BATCH one, unless the operator asks for another with --require-profile), and every backend advertises what it can honour (videoflow.backends.capabilities). The planner either admits the composition or rejects it by name — it never downgrades a request to whatever the backend happens to offer, which is how a batch job used to end up on an evictable store with nobody told.

This module is the deploy-time half of that: it turns the broker and store a flow will run on into capability records, parses the operator’s explicit requests, and runs the planner. An auto-provisioned broker or store is judged by its declared profile (jetstream_capabilities / redis_payload_capabilities — the render is what the profile says it is); a bring-your-own --nats / --blob-redis-url is read back live before anything is created (jetstream_capabilities_observed / redis_payload_capabilities_observed: the connection’s payload limit, the JetStream account’s storage allowance, the run’s streams when they already exist; Redis’ persistence and eviction settings), and whatever the probe could not read stays Unknown with the reason — never assumed. A definite incompatibility is binding (RFC 0006, accepted); an unobservable capability of a bring-your-own broker or store is a warning unless the operator named a profile explicitly. Explicit requests also travel to the provision Job and the workers as VF_PROFILE_REQUESTS_JSON, emitted only when there are any (D8); there they bind again, against the streams as provisioned (runtime.provision) and as bound (runtime.worker, before open()), through messaging.topology.verify_channel_profiles.

videoflow.deploy.admission.PROBE_TIMEOUT_SECONDS = 5.0

How long a deploy-time probe of a bring-your-own broker or store waits, connect included, before reporting what it could not observe. Short on purpose: it runs on the operator’s machine before anything is created, and an unreachable service is reported as Unknown('unreachable') for the planner to rule on, not waited for.

videoflow.deploy.admission.admit(requirements: FlowRequirements, messaging: MessagingCapabilities, payload: PayloadCapabilities | None, *, payload_refs_in_use: bool, enforce: bool, unknown_is_fatal: bool, where: str, runtime: RuntimeCapabilities | None = None, execution: ExecutionCapabilities | None = None) → CompositionPlan | None

Run the planner. A rejection that is not binding is printed as a warning and None returned — today’s behaviour, with the reason on record; a binding one propagates the planner’s error (exit 2 or 3).

Two kinds of rejection, bound separately: a definite incompatibility (the declared store is evictable, the transport retains nothing) binds under enforce; an unobservable capability (a bring-your-own broker or store whose probe could not read the setting — unreachable, refused, timed out) binds only under unknown_is_fatal: with the switch on, a broker that happens to be slow must not turn every deploy into a rejection, so it is a warning there; an explicit request is the one case where “unobserved” must not pass.

• Arguments:

  • payload_refs_in_use: a payload store is configured, so envelopes over the inline threshold offload to it and its durability is part of the channel guarantee.

  • enforce: definite rejections are binding (explicit requests, or the RFC 0006 switch — enforce_admission).

  • unknown_is_fatal: unobservable capabilities are binding too (explicit requests only — unknown_admission).

  • where: deploy / run-local, for the message.

  • runtime: the runtime store’s read-back (VF_RUNTIME_STORE_URL), which restart_safe and durable_control are admitted against; None when no store is configured.

  • execution: what the engine advertises (fused groups, batching); None when the caller is not deploying through an engine.

videoflow.deploy.admission.enforce_admission(explicit: Sequence[ProfileRequest]) → bool

Whether a definite rejection is binding: always, since RFC 0006 was accepted (kept for its callers’ symmetry).

videoflow.deploy.admission.jetstream_capabilities(profile: BrokerProfile | None, unread: str = 'the broker configuration was not read back (bring-your-own --nats)') → MessagingCapabilities

What a JetStream broker offers, from its declared profile: retained, recoverable delivery always (INTEREST retention with redelivery is what the topology provisions); persistence and replication as the profile says. With no profile (a bring-your-own --nats, or a Service the namespace already had with no profile record — unread says which) those two are Unknown — the broker was not read back, and unknown is not “yes”; the CLI reads a bring-your-own broker back with jetstream_capabilities_observed instead, and the provision Job reads a reused one back in-cluster.

videoflow.deploy.admission.jetstream_capabilities_observed(nats_url: str, timeout: float = 5.0, stream_names: Sequence[str] = (), fail_fast: bool = True) → MessagingCapabilities

What a bring-your-own JetStream broker offers, read back from the live server rather than declared:

• max_payload_bytes: the connection’s max_payload (nats-py 2.15.0, nats/aio/client.py:1285, the server’s INFO max_payload).

• persistent_storage: from js.account_info() (nats/js/manager.py:69 -> api.AccountInfo, nats/js/api.py:761): limits.max_storage (AccountLimits, api.py:720) is the account’s file-store allowance, -1 unlimited and 0 none — a stream provisioned with storage unset is a file stream by server default, so the allowance says whether this run’s streams will be file-backed before any exists. Once a stream of the run exists, its applied config.storage (stream_info, manager.py:85 -> api.StreamInfo.config) is the answer instead. File storage is what the JetStream API can see; whether the directory behind it survives a pod loss is the deployment’s business (an auto-provisioned broker answers that through its declared profile).

• replication_factor: the smallest config.num_replicas over the run’s existing streams; Unknown('unread') while none exists, because the copies a stream keeps are decided when it is created (VF_STREAM_REPLICAS).

A server that answers without JetStream is reported as offering no retained, recoverable delivery (_core_nats_only). Anything that could not be read is Unknown with the reason — timeout, auth (credentials refused, or a permissions violation on the API subjects, which the server reports through the error callback while the request itself times out), unreachable, malformed — and never a guess.

• Arguments:

  • timeout: overall bound, connect included.

  • stream_names: the run’s stream names (run_stream_names), read when they exist.

  • fail_fast: no reconnects — the operator’s machine. False lets the client retry a broker that is still starting (the in-cluster provision Job).

videoflow.deploy.admission.local_dev_capabilities() → tuple[MessagingCapabilities, PayloadCapabilities]

What videoflow run-local’s docker dev containers offer (deploy.localinfra): a JetStream server without a volume and the Redis RedisProfile.dev() describes — an append-only file, noeviction — which localinfra starts with exactly those arguments.

videoflow.deploy.admission.parse_profile_requests(values: Sequence[str] | None, specs: Sequence[NodeSpec]) → List[ProfileRequest]

--require-profile CHANNEL=PROFILE entries as requests. A channel is the name of the node whose output it carries; the profile is one of MESSAGING_PROFILES.

• Raises:

  • ConfigError: a malformed entry, a channel no node publishes, an unknown profile, or the same channel named twice.

videoflow.deploy.admission.redis_payload_capabilities(profile: RedisProfile | None, unread: str = 'the store configuration was not read back (bring-your-own --blob-redis-url)') → PayloadCapabilities

What a Redis payload store offers, from its declared profile: durable only with append-only persistence and noeviction (either alone lets an accepted envelope outlive its bytes); persistent_storage only on a claim (the dev profile’s emptyDir is durable across a container restart, gone with the pod). No profile (bring-your-own --blob-redis-url, or a reused Service with no profile record — unread says which): Unknown until read back, which the CLI does with redis_payload_capabilities_observed for a bring-your-own store and the provision Job does in-cluster for a reused one.

videoflow.deploy.admission.redis_payload_capabilities_observed(url: str, timeout: float = 5.0) → PayloadCapabilities

What a bring-your-own Redis payload store offers, read back live (wire.redis_payload_store.redis_capabilities_observed: CONFIG GET appendonly / save / maxmemory-policy, INFO cluster, CLUSTER KEYSLOT of the obligation keys). Refused credentials are Unknown('auth'); the delegate reports every other failure with its reason.

• Arguments:

  • url: redis:// / rediss:// (redis.Redis.from_url, redis-py 8.0.1).

  • timeout: socket connect and read timeout per command.

videoflow.deploy.admission.requests_env(explicit: Sequence[ProfileRequest]) → dict[str, str]

The worker environment entry for explicit requests — empty when there are none.

videoflow.deploy.admission.requests_from_env(value: str | None) → list[ProfileRequest]

The inverse of requests_env on the worker side; [] when unset.

videoflow.deploy.admission.requirements_for(flow_type: str, specs: Sequence[NodeSpec], explicit: Sequence[ProfileRequest] = (), declared: FlowRequirements | None = None) → FlowRequirements

The flow-type presets for every channel, with the operator’s explicit requests replacing theirs, plus what the nodes themselves declared (deploy.compile.declared_requirements: sink guarantees, execution groups, batching contracts) when the caller has the compiled document.

videoflow.deploy.admission.run_stream_names(flow_id: str, run_id: str, specs: Sequence[NodeSpec]) → list[str]

The stream names a run’s nodes publish on, for a probe to read back when they already exist.

videoflow.deploy.admission.unknown_admission(explicit: Sequence[ProfileRequest]) → bool

Whether an unobservable capability is binding: only an explicit request asks for that.

videoflow.deploy.admission.verify_topology_shape(flow_type: str, flow_id: str, run_id: str, explicit: Sequence[ProfileRequest]) → None

Reject an explicit request the flow’s own topology cannot carry, before anything is built, provisioned or applied. Streams are shaped by the flow type (topology.stream_config_for: REALTIME ⇒ limits / discard-old, BATCH ⇒ interest / discard-new), so reliable_work on a REALTIME channel or live_latest on a BATCH one contradicts the stream that will exist whatever the broker can do — the same finding the provision Job and the workers report from the read-back, an hour earlier and with nothing to tear down.

• Raises:

  • IncompatibleProfile: naming every such channel.

videoflow.deploy.allocation_dra module

Dynamic Resource Allocation (resource.k8s.io/v1) as an AcceleratorAllocationBackend — render-only in this release (plan decision D9).

What it does: turn a WorkloadRequest into the manifests and pod fragments a DRA-scheduled worker needs — a ResourceClaimTemplate for independent replicas (one claim per pod) or a shared ResourceClaim for an explicit execution group, the pod’s spec.resourceClaims entry and the container’s resources.claims reference, with the DeviceClass named, never created (it is the driver’s and the cluster administrator’s object). And say, per feature, whether the cluster and driver at hand can serve a request: DraEnvironment is the version matrix — Kubernetes ≥ 1.34 for the v1 API, DRAPartitionableDevices for dynamic MIG (alpha and off by default on 1.34/1.35, beta and on from 1.36), DRAConsumableCapacity for shared capacity (same schedule), MPS as a driver capability with no Kubernetes gate, and the pairs no driver can honour at once (MPS with dynamic MIG, MPS with consumable shares) — rejected by name before anything is rendered (ALLOC-019/020/021/023/024).

What it does not do: read a ResourceSlice, create or watch a claim, or release one. Those need a DRA driver running in the cluster (the NVIDIA DRA driver), which the cluster this was built against does not have; every lifecycle call raises CapabilityError naming that follow-up rather than pretending. gpu.DraGpu (--gpu-mode dra) is the same rendering behind the CLI.

Field names follow the resource.k8s.io/v1 API reference (read at implementation time, Kubernetes 1.36 docs): a request is exactly: {deviceClassName, allocationMode: ExactCount | All, count, selectors[].cel, capacity.requests}; a pod lists resourceClaims[].resourceClaimTemplateName or resourceClaimName and a container resources.claims[].name.

videoflow.deploy.allocation_dra.DRA_GA_VERSION = (1, 34)

The first Kubernetes release serving resource.k8s.io/v1 (DRA GA).

class videoflow.deploy.allocation_dra.DraAllocationBackend(environment: DraEnvironment | None = None, device_class: str = 'gpu.nvidia.com')

Bases: AcceleratorAllocationBackend

• Arguments:

  • environment: the DraEnvironment capabilities are judged against; capabilities(mapping) may also carry one (DraEnvironment.from_mapping).

  • device_class: the DeviceClass every rendered claim references.

bindings(claim_id: str, workload_id: str) → WorkloadBindings

capabilities(environment: Mapping[str, Any]) → AllocationCapabilities

inventory(scope: Mapping[str, Any]) → Known[InventorySnapshot] | Unknown

observe(claim_id: str) → Known[ClaimObservation] | Unknown

plan(requests: Sequence[WorkloadRequest], snapshot: InventorySnapshot) → FeasiblePlan | Infeasible

reconcile(claim_id: str, desired: str, expected_generation: str) → ClaimObservation

release(claim_id: str, operation_id: str, expected_generation: str, keep_workloads: bool = False) → ReleaseObservation

render(request: WorkloadRequest, namespace: str, shared_claim: str | None = None, capacity: Mapping[str, str] | None = None) → WorkloadBindings

reserve(plan: FeasiblePlan, operation_id: str, expected_generation: str | None) → ClaimObservation

class videoflow.deploy.allocation_dra.DraEnvironment(kubernetes_version: str, feature_gates: Mapping[str, bool]=<factory>, device_classes: tuple[str, ...]=(), driver: str | None = None, driver_features: frozenset[str] = frozenset({}))

Bases: object

The cluster and driver a DRA request is judged against.

• Arguments:

  • kubernetes_version: the API server’s version string.

  • feature_gates: gate states known for certain (read from the API server flags, or declared by the operator); a gate not listed takes its version default.

  • device_classes: the DeviceClass names the cluster serves.

  • driver: the DRA driver name (gpu.nvidia.com), None when no driver publishes ResourceSlices.

  • driver_features: the gated features the driver itself implements (dynamic-mig, mps, consumable-capacity).

property api_served: bool

available_features() → tuple[str, ...]

device_classes: tuple[str, ...] = ()

driver: str | None = None

driver_features: frozenset[str] = frozenset({})

feature_gates: Mapping[str, bool]

static from_mapping(environment: Mapping[str, Any]) → DraEnvironment

gate_enabled(gate: str) → bool | None

True/False when known (declared, or the version default); None when the gate does not exist at this version.

kubernetes_version: str

to_mapping() → dict[str, Any]

unavailable_reasons() → dict[str, str]

Gated feature → why this environment cannot serve it; a feature absent here is available.

property version: tuple[int, int, int]

videoflow.deploy.allocation_dra.FEATURE_GATES = {'consumable-capacity': 'DRAConsumableCapacity', 'dynamic-mig': 'DRAPartitionableDevices'}

Which Kubernetes gate each gated feature needs (MPS is purely a driver matter).

videoflow.deploy.allocation_dra.GATE_SCHEDULE: dict[str, tuple[tuple[int, str, bool], ...]] = {'DRAAdminAccess': ((32, 'alpha', False), (34, 'beta', True), (36, 'stable', True)), 'DRAConsumableCapacity': ((34, 'alpha', False), (36, 'beta', True)), 'DRADeviceTaints': ((33, 'alpha', False), (36, 'beta', True), (37, 'stable', True)), 'DRAPartitionableDevices': ((33, 'alpha', False), (36, 'beta', True)), 'DRAPrioritizedList': ((33, 'alpha', False), (34, 'beta', True), (36, 'stable', True))}

Feature-gate schedule per the Kubernetes feature-gates reference (1.37 docs): (first minor, stage, on by default) rows in ascending order; a gate is absent before its first row. stable gates are locked on.

videoflow.deploy.allocation_dra.claim_name_for(workload_id: str) → str

The pod-local claim name (spec.resourceClaims[].name) for a workload.

videoflow.deploy.allocation_dra.device_request(request: WorkloadRequest, device_class: str, capacity: Mapping[str, str] | None = None) → dict

One spec.devices.requests[] entry: an exactly request for device_count devices of the class, with the request’s hard constraints as CEL selectors on device attributes and any capacity requests. Plain dicts throughout — the shape is the Kubernetes API’s, not ours.

videoflow.deploy.allocation_dra.gate_stage(gate: str, version: tuple[int, int, int]) → tuple[str, bool] | None

(stage, on by default) of a gate at a Kubernetes version, or None when the gate does not exist there.

videoflow.deploy.allocation_dra.observe_environment(kubectl: str = 'kubectl') → DraEnvironment

The environment as the API server reports it, read-only: server version, the DeviceClass names served (empty when the API is absent), and the driver behind the first GPU ResourceSlice (None when none exists). Gate states are not readable through the API and stay at version defaults; driver features are unknown to a read and stay empty — an operator who knows the driver declares them.

videoflow.deploy.allocation_dra.parse_version(version: str) → tuple[int, int, int]

v1.36.3+k3s1 → (1, 36, 3); anything unparseable is (0, 0, 0).

videoflow.deploy.allocation_dra.render_bindings(request: WorkloadRequest, device_class: str, namespace: str, shared_claim: str | None = None, capacity: Mapping[str, str] | None = None) → WorkloadBindings

The DRA manifests and fragments for one workload: a ResourceClaimTemplate (every pod gets its own claim — independent replicas) unless shared_claim names a ResourceClaim all pods of an execution group share. The DeviceClass is referenced by name and never rendered.

videoflow.deploy.allocation_kubernetes module

The Kubernetes AcceleratorAllocationBackend: the videoflow GPU pool as the contract sees it, over the two registered strategies.

• exclusive — the device plugin is the authority. Videoflow reads the pool (GFD labels, allocatable, running pods) and plans a per-host packing, but it writes nothing: the scheduler accounts whole-device claims, so reserve is a plan admitted for rendering and readiness is the pod’s, not the claim’s.

• mix — videoflow itself becomes an authority over geometry (RFC 0004): it claims nodes by a server-enforced compare-and-swap on their resourceVersion, re-reads the occupancy after claiming and before any geometry write, publishes the nvidia-mig-parted entries under CAS, and judges readiness by evidence correlated with this operation — the per-run entry name and owner epoch it wrote, the manager’s success and the requested slices in status.allocatable (ALLOC-003/011/033).

What the reads could not see stays unknown: a pod listing the API refused makes the inventory partial and no plan may admit on it; an unreadable node makes observe Unknown, never “ready”. Release is fenced on the claim’s generation (the owner epoch) and keep_workloads=True retains ownership and geometry for a later explicit release (ALLOC-013). The shared ConfigMap is never deleted: the last flow out restores the operator’s pointer and leaves a tombstone (decision D3).

The strategies keep their registry API for the CLI (resolve_specs → preflight_problems → prepare → cleanup); this module is the same machinery behind the contract, with the plan explicit (gpu.AllocationPlan).

videoflow.deploy.allocation_kubernetes.CONSTRAINT_KEYS = {'kubernetes.io/hostname': <function <lambda>>, 'nvidia.com/gpu.memory': <function <lambda>>, 'nvidia.com/gpu.product': <function <lambda>>}

GFD attributes a hard constraint may name, and how the inventory answers them.

class videoflow.deploy.allocation_kubernetes.KubernetesAllocationBackend(strategy_name: str = 'exclusive', kubectl: str = 'kubectl', clock: Callable[[], float]=<built-in function time>)

Bases: AcceleratorAllocationBackend

• Arguments:

  • strategy_name: a registered --gpu-mode (exclusive or mix).

  • kubectl: the kubectl binary every read and write goes through.

bindings(claim_id: str, workload_id: str) → WorkloadBindings

What a pod of workload_id carries: the extended-resource limit the strategy renders (a MIG profile name for a sharer), a hostname affinity to the planned node, and — under mix — the owner label the pool selector keys on.

capabilities(environment: Mapping[str, Any]) → AllocationCapabilities

inventory(scope: Mapping[str, Any]) → Known[InventorySnapshot] | Unknown

property managed_mig: bool

observe(claim_id: str) → Known[ClaimObservation] | Unknown

plan(requests: Sequence[WorkloadRequest], snapshot: InventorySnapshot) → FeasiblePlan | Infeasible

Per-host packing for whole-device requests (a replica’s devices sit on one node) and, under mix, the layout solver for MIG sharers. A partial snapshot plans nothing: occupancy that could not be read is not idle capacity.

reconcile(claim_id: str, desired: str, expected_generation: str) → ClaimObservation

release(claim_id: str, operation_id: str, expected_generation: str, keep_workloads: bool = False) → ReleaseObservation

mix: this flow’s geometry restored and its claims released through MixGpu.cleanup (idempotent; a node that did not revert keeps its restore record for a retry, reported as pending_recovery). With keep_workloads nothing is touched: ownership and geometry stay for a later explicit release. A stale generation releases nothing.

reserve(plan: FeasiblePlan, operation_id: str, expected_generation: str | None) → ClaimObservation

exclusive: nothing to write — the plan is admitted and the scheduler accounts the claims. mix: the node claims (CAS), the occupancy re-read, the ConfigMap/ClusterPolicy publication and the geometry labels, through MixGpu.apply_plan; the claim’s generation is the owner epoch stamped on its nodes.

• Raises:

  • OwnershipConflict: the plan was made on another snapshot, a node was taken, or occupancy changed after the claim.

  • UnobservableState: occupancy could not be re-read after the claim.

  • ClusterError: the manager refused or never applied the geometry.

videoflow.deploy.allocation_kubernetes.constraint_expressions(constraints: Sequence[Constraint]) → list[dict]

The hard constraints as node-affinity matchExpressions (soft ones are preferences, not terms).

videoflow.deploy.allocation_kubernetes.constraint_holds(constraint: Constraint, node: NodeInventory) → bool | None

Whether node satisfies a constraint on a verifiable attribute; None when the key is not one.

videoflow.deploy.allocation_kubernetes.requests_as_specs(requests: Sequence[WorkloadRequest]) → list[NodeSpec]

The layout solver speaks NodeSpec: one single-replica spec per request, a MIG sharer carrying its minimum usable memory as gpu_memory_gib.

videoflow.deploy.allocation_kubernetes.snapshot_from_inventory(inventory: Sequence[NodeInventory], observed_at: float, generation: str | None) → InventorySnapshot

The contract’s view of cluster.gpu_inventory_observed: one identity per card (host ordinal, no UUID — the API does not expose one), occupancy as GPU units held by running pods, the node’s sharing classification and owner stamp, and partial completeness whenever any node’s pod listing failed.

videoflow.deploy.allocation_local module

The local AcceleratorAllocationBackend: this host’s GPUs, read through nvidia-smi, partitioned across the workers of a run-local flow.

Two policies, one contract (plan §C step 4, decision D4):

• shared (the default until RFC 0006 is accepted) reproduces today’s wrap-around walk — when demand exceeds the visible devices, workers share them. What changes is the reporting: every worker receives a DeliveredGrant (VF_GPU_GRANT_JSON) that says how many devices it really got and that the grant is not exclusive, so a requested gpu_count is never presented as delivered capacity (ALLOC-014, RUN-044).

• strict refuses, before any worker is launched, a flow whose exclusive requests do not fit the host’s distinct devices, and admits cooperative sharers only within a declared peak-memory budget plus headroom (VF_GPU_HEADROOM_BYTES, ALLOC-016). A missing declaration is a rejection, never a guess.

What the host says, the backend repeats: a failed nvidia-smi read is an Unknown inventory (“could not observe”), never a confirmed zero-GPU machine; an inherited CUDA_VISIBLE_DEVICES narrows the pool by ordinal, card UUID or MIG UUID exactly as the CUDA runtime would (ALLOC-015). Grants are written to the workers as UUIDs, the identity that survives renumbering.

There is no server-side compare-and-swap on a single host, so reserve fences on the inventory generation instead: a plan made on one snapshot is refused when the host has changed since (a device disappeared, a foreign process appeared), and release refuses a stale generation the same way.

videoflow.deploy.allocation_local.GRANT_ENV = 'VF_GPU_GRANT_JSON'

the worker-side copy of its DeliveredGrant (RFC 0006 ENV-14).

Type:

Env

videoflow.deploy.allocation_local.HEADROOM_ENV = 'VF_GPU_HEADROOM_BYTES'

bytes kept free on every shared device beyond the declared peaks (allocator fragmentation, the CUDA context of each process). Strict policy only.

Type:

Env

class videoflow.deploy.allocation_local.LocalAllocationBackend(policy: str = 'shared', headroom_bytes: int | None = None, host_reader: Callable[[], ~videoflow.backends.outcomes.Known[list[~videoflow.backends.allocation.DeviceIdentity]] | ~videoflow.backends.outcomes.Unknown]=<function host_devices_observed>, used_reader: Callable[[], ~videoflow.backends.outcomes.Known[dict[str, int]] | ~videoflow.backends.outcomes.Unknown]=<function host_memory_used_observed>, mask: str | None = None, inherit_mask: bool = True, clock: Callable[[], float]=<built-in function time>)

Bases: AcceleratorAllocationBackend

• Arguments:

  • policy: shared (wrap-around, labelled) or strict (refuse short or over-budget grants before launch).

  • headroom_bytes: strict-policy memory headroom per shared device; defaults to VF_GPU_HEADROOM_BYTES or 1 GiB.

  • host_reader / used_reader: the nvidia-smi reads, injectable so a process-level test can present one, zero or unobservable GPUs.

  • mask: a CUDA_VISIBLE_DEVICES value applied to the host pool before planning; by default the one this process inherited from its environment (inherit_mask=False ignores the environment: the whole host).

bindings(claim_id: str, workload_id: str) → WorkloadBindings

capabilities(environment: Mapping[str, Any]) → AllocationCapabilities

grant(claim_id: str, workload_id: str) → DeliveredGrant

The grant a workload received under this claim (what bindings serialises).

inventory(scope: Mapping[str, Any]) → Known[InventorySnapshot] | Unknown

The devices this process may hand out: the host’s, narrowed by the inherited mask. Unknown when the host could not be read — the caller decides what an unobservable host means, never “no GPUs”.

observe(claim_id: str) → Known[ClaimObservation] | Unknown

Ready when every granted device is still enumerated by the host; Unknown when the host cannot be read.

plan(requests: Sequence[WorkloadRequest], snapshot: InventorySnapshot) → FeasiblePlan | Infeasible

property policy: str

reconcile(claim_id: str, desired: str, expected_generation: str) → ClaimObservation

release(claim_id: str, operation_id: str, expected_generation: str, keep_workloads: bool = False) → ReleaseObservation

reserve(plan: FeasiblePlan, operation_id: str, expected_generation: str | None) → ClaimObservation

Fences the plan on the host: re-reads the inventory and refuses when its generation differs from the one the plan was made on (a device or a foreign process came or went since), returning failed with the evidence rather than granting devices the plan never saw.

videoflow.deploy.allocation_local.device_key(device: DeviceIdentity) → str

The mask entry a worker receives for a device: its MIG UUID, else its card UUID, else the ordinal.

videoflow.deploy.allocation_local.grant_from_env(environ: Mapping[str, str] = environ({'ACCEPT_EULA': 'Y', 'ACTIONS_ID_TOKEN_REQUEST_TOKEN': 'eyJhbGciOiJSUzI1NiIsImtpZCI6IjM4ODI2YjE3LTZhMzAtNWY5Yi1iMTY5LThiZWI4MjAyZjcyMyIsInR5cCI6IkpXVCIsIng1dCI6InlrTmFZNHFNX3RhNGsyVGdaT0NFWUxrY1lsQSJ9.eyJJZGVudGl0eVR5cGVDbGFpbSI6IlN5c3RlbTpTZXJ2aWNlSWRlbnRpdHkiLCJhYyI6Ilt7XCJTY29wZVwiOlwicmVmcy9oZWFkcy9tYXN0ZXJcIixcIlBlcm1pc3Npb25cIjozfV0iLCJhY3NsIjoiMTAiLCJhdWQiOiJ2c286Y2ZmMGE0YWEtZjA3My00ZGJjLTliODQtM2IxNzFmYjRlYTM3IiwiYmlsbGluZ19vd25lcl9pZCI6Ik9fa2dET0F3bC1ydyIsImV4cCI6MTc4OTQyMjk4MSwiaHR0cDovL3NjaGVtYXMubWljcm9zb2Z0LmNvbS93cy8yMDA4LzA2L2lkZW50aXR5L2NsYWltcy9wcmltYXJ5c2lkIjoiZGRkZGRkZGQtZGRkZC1kZGRkLWRkZGQtZGRkZGRkZGRkZGRkIiwiaHR0cDovL3NjaGVtYXMueG1sc29hcC5vcmcvd3MvMjAwNS8wNS9pZGVudGl0eS9jbGFpbXMvc2lkIjoiZGRkZGRkZGQtZGRkZC1kZGRkLWRkZGQtZGRkZGRkZGRkZGRkIiwiaWF0IjoxNzg5NDAwNzgxLCJpc3MiOiJodHRwczovL3Rva2VuLmFjdGlvbnMuZ2l0aHVidXNlcmNvbnRlbnQuY29tIiwiam9iX2lkIjoiZTcxMDY3NWEtZWQ1NC01NTMzLWIwYWYtMDFkZjE1MjY1ODNhIiwiam9iX3dvcmtmbG93X3JlZiI6InZpZGVvZmxvdy92aWRlb2Zsb3cvLmdpdGh1Yi93b3JrZmxvd3MvZG9jcy55bWxAcmVmcy9oZWFkcy9tYXN0ZXIiLCJqb2Jfd29ya2Zsb3dfc2hhIjoiYmFjYThmYTkwMWQ4OGIwN2YwYzBlM2YwMGE0ZjhhNDI2MzYxMTU1MSIsIm5hbWVpZCI6ImRkZGRkZGRkLWRkZGQtZGRkZC1kZGRkLWRkZGRkZGRkZGRkZCIsIm5iZiI6MTc4OTQwMDQ4MSwib2lkY19leHRyYSI6IntcImFjdG9yXCI6XCJqYWRpZWxhbVwiLFwiYWN0b3JfaWRcIjpcIjEwMjYwNjdcIixcImJhc2VfcmVmXCI6XCJcIixcImNoZWNrX3J1bl9pZFwiOlwiMTA0MDQ0Mjc2NzcyXCIsXCJldmVudF9uYW1lXCI6XCJwdXNoXCIsXCJoZWFkX3JlZlwiOlwiXCIsXCJqb2Jfd29ya2Zsb3dfcmVmXCI6XCJ2aWRlb2Zsb3cvdmlkZW9mbG93Ly5naXRodWIvd29ya2Zsb3dzL2RvY3MueW1sQHJlZnMvaGVhZHMvbWFzdGVyXCIsXCJqb2Jfd29ya2Zsb3dfc2hhXCI6XCJiYWNhOGZhOTAxZDg4YjA3ZjBjMGUzZjAwYTRmOGE0MjYzNjExNTUxXCIsXCJyZWZcIjpcInJlZnMvaGVhZHMvbWFzdGVyXCIsXCJyZWZfcHJvdGVjdGVkXCI6XCJmYWxzZVwiLFwicmVmX3R5cGVcIjpcImJyYW5jaFwiLFwicmVwb3NpdG9yeVwiOlwidmlkZW9mbG93L3ZpZGVvZmxvd1wiLFwicmVwb3NpdG9yeV9pZFwiOlwiMTgxNTU0OTM5XCIsXCJyZXBvc2l0b3J5X293bmVyXCI6XCJ2aWRlb2Zsb3dcIixcInJlcG9zaXRvcnlfb3duZXJfaWRcIjpcIjUwOTUzOTAzXCIsXCJyZXBvc2l0b3J5X3Zpc2liaWxpdHlcIjpcInB1YmxpY1wiLFwicnVuX2F0dGVtcHRcIjpcIjFcIixcInJ1bl9pZFwiOlwiMzQ4NjQzNDM5NjBcIixcInJ1bl9udW1iZXJcIjpcIjExXCIsXCJydW5uZXJfZW52aXJvbm1lbnRcIjpcImdpdGh1Yi1ob3N0ZWRcIixcInNoYVwiOlwiYmFjYThmYTkwMWQ4OGIwN2YwYzBlM2YwMGE0ZjhhNDI2MzYxMTU1MVwiLFwid29ya2Zsb3dcIjpcIkRvY3NcIixcIndvcmtmbG93X3JlZlwiOlwidmlkZW9mbG93L3ZpZGVvZmxvdy8uZ2l0aHViL3dvcmtmbG93cy9kb2NzLnltbEByZWZzL2hlYWRzL21hc3RlclwiLFwid29ya2Zsb3dfc2hhXCI6XCJiYWNhOGZhOTAxZDg4YjA3ZjBjMGUzZjAwYTRmOGE0MjYzNjExNTUxXCJ9Iiwib2lkY19zdWIiOiJyZXBvOnZpZGVvZmxvdy92aWRlb2Zsb3c6cmVmOnJlZnMvaGVhZHMvbWFzdGVyIiwib3JjaF9pZCI6IjU5OTczN2Q4LWE4MjEtNDViNS05NjhkLWQ4MTI1N2VkZmQwZi5idWlsZC5fX2RlZmF1bHQiLCJvd25lcl9pZCI6Ik9fa2dET0F3bC1ydyIsInBsYW5faWQiOiI1OTk3MzdkOC1hODIxLTQ1YjUtOTY4ZC1kODEyNTdlZGZkMGYiLCJyZXBvc2l0b3J5X2lkIjoiMTgxNTU0OTM5IiwicmVwb3NpdG9yeV9vd25lcl9pZCI6IjUwOTUzOTAzIiwicmVwb3NpdG9yeV92aXNpYmlsaXR5IjoicHVibGljIiwicnVuX2lkIjoiMzQ4NjQzNDM5NjAiLCJydW5fbnVtYmVyIjoiMTEiLCJydW5fdHlwZSI6ImZ1bGwiLCJydW5uZXJfaWQiOiIxMDAwMDAwMTY1IiwicnVubmVyX3R5cGUiOiJob3N0ZWQiLCJzY3AiOiJBY3Rpb25zLlJlc3VsdHM6NTk5NzM3ZDgtYTgyMS00NWI1LTk2OGQtZDgxMjU3ZWRmZDBmOmU3MTA2NzVhLWVkNTQtNTUzMy1iMGFmLTAxZGYxNTI2NTgzYSBBY3Rpb25zLlJ1bm5lcjo1OTk3MzdkOC1hODIxLTQ1YjUtOTY4ZC1kODEyNTdlZGZkMGY6ZTcxMDY3NWEtZWQ1NC01NTMzLWIwYWYtMDFkZjE1MjY1ODNhIEFjdGlvbnMuVXBsb2FkQXJ0aWZhY3RzOjU5OTczN2Q4LWE4MjEtNDViNS05NjhkLWQ4MTI1N2VkZmQwZjplNzEwNjc1YS1lZDU0LTU1MzMtYjBhZi0wMWRmMTUyNjU4M2EgZ2VuZXJhdGVfaWRfdG9rZW46NTk5NzM3ZDgtYTgyMS00NWI1LTk2OGQtZDgxMjU3ZWRmZDBmOmU3MTA2NzVhLWVkNTQtNTUzMy1iMGFmLTAxZGYxNTI2NTgzYSBBY3Rpb25zLkdlbmVyaWNSZWFkOjAwMDAwMDAwLTAwMDAtMDAwMC0wMDAwLTAwMDAwMDAwMDAwMCIsInNoYSI6ImJhY2E4ZmE5MDFkODhiMDdmMGMwZTNmMDBhNGY4YTQyNjM2MTE1NTEiLCJ0cnVzdF90aWVyIjoiMiJ9.odJ3vkzgRiNUCihXdyP65ZEvROIonxWdwBRZs1HbSmKcjM5eI-y6DKM1DmgqGX5_zmHqhcvJjQoQz1AQsgwkp-u5uIzFFLB0URMJK6x3xztWigmBI5mxSJgfBBYBndkRORJrkm6HYrkDfCF458M1oORSC1VMt2-qHEH9Lwp1heegCNjZ_PUv0SC6KCWCdoqMQpI-x1o66H0BxaufIW-OOM_Yprtyv_PrLLTfevwxL6r2uBxU_J8-qtayR1fz5n2mwPmlS5SbtN9vitatOU97yuSZY2q-ecQwbZbyt4Ny_BjduyjBTh-Zgt8KfHdrYZjDe7kk12Us7QKTixCiP__Kvw', 'ACTIONS_ID_TOKEN_REQUEST_URL': 'https://run-actions-3-azure-eastus.actions.githubusercontent.com/139//idtoken/599737d8-a821-45b5-968d-d81257edfd0f/e710675a-ed54-5533-b0af-01df1526583a?api-version=2.0', 'ACTIONS_ORCHESTRATION_ID': '599737d8-a821-45b5-968d-d81257edfd0f.build.__default', 'ACTIONS_RUNNER_ACTION_ARCHIVE_CACHE': '/opt/actionarchivecache', 'ACTIONS_RUNNER_RETURN_JOB_RESULT_FOR_HOSTED': '1', 'AGENT_TOOLSDIRECTORY': '/opt/hostedtoolcache', 'ANDROID_HOME': '/usr/local/lib/android/sdk', 'ANDROID_NDK': '/usr/local/lib/android/sdk/ndk/27.3.13750724', 'ANDROID_NDK_HOME': '/usr/local/lib/android/sdk/ndk/27.3.13750724', 'ANDROID_NDK_LATEST_HOME': '/usr/local/lib/android/sdk/ndk/29.0.14206865', 'ANDROID_NDK_ROOT': '/usr/local/lib/android/sdk/ndk/27.3.13750724', 'ANDROID_SDK_ROOT': '/usr/local/lib/android/sdk', 'ANT_HOME': '/usr/share/ant', 'AZURE_EXTENSION_DIR': '/opt/az/azcliextensions', 'BOOTSTRAP_HASKELL_NONINTERACTIVE': '1', 'CHROMEWEBDRIVER': '/usr/local/share/chromedriver-linux64', 'CHROME_BIN': '/usr/bin/google-chrome', 'CI': 'true', 'CONDA': '/usr/share/miniconda', 'DEBIAN_FRONTEND': 'noninteractive', 'DOTNET_MULTILEVEL_LOOKUP': '0', 'DOTNET_NOLOGO': '1', 'DOTNET_SKIP_FIRST_TIME_EXPERIENCE': '1', 'EDGEWEBDRIVER': '/usr/local/share/edge_driver', 'ENABLE_RUNNER_TRACING': 'true', 'GECKOWEBDRIVER': '/usr/local/share/gecko_driver', 'GHCUP_INSTALL_BASE_PREFIX': '/usr/local', 'GITHUB_ACTION': '__run_2', 'GITHUB_ACTIONS': 'true', 'GITHUB_ACTION_REF': '', 'GITHUB_ACTION_REPOSITORY': '', 'GITHUB_ACTOR': 'jadielam', 'GITHUB_ACTOR_ID': '1026067', 'GITHUB_API_URL': 'https://api.github.com', 'GITHUB_ARTIFACTS': '/home/runner/work/_temp/_runner_file_commands/artifacts_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_ARTIFACTS_LIST': '/home/runner/work/_temp/_runner_file_commands/artifacts_list_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_BASE_REF': '', 'GITHUB_ENV': '/home/runner/work/_temp/_runner_file_commands/set_env_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_EVENT_NAME': 'push', 'GITHUB_EVENT_PATH': '/home/runner/work/_temp/_github_workflow/event.json', 'GITHUB_GRAPHQL_URL': 'https://api.github.com/graphql', 'GITHUB_HEAD_REF': '', 'GITHUB_JOB': 'build', 'GITHUB_OUTPUT': '/home/runner/work/_temp/_runner_file_commands/set_output_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_PATH': '/home/runner/work/_temp/_runner_file_commands/add_path_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_REF': 'refs/heads/master', 'GITHUB_REF_NAME': 'master', 'GITHUB_REF_PROTECTED': 'false', 'GITHUB_REF_TYPE': 'branch', 'GITHUB_REPOSITORY': 'videoflow/videoflow', 'GITHUB_REPOSITORY_ID': '181554939', 'GITHUB_REPOSITORY_OWNER': 'videoflow', 'GITHUB_REPOSITORY_OWNER_ID': '50953903', 'GITHUB_RETENTION_DAYS': '90', 'GITHUB_RUN_ATTEMPT': '1', 'GITHUB_RUN_ID': '34864343960', 'GITHUB_RUN_NUMBER': '11', 'GITHUB_SERVER_URL': 'https://github.com', 'GITHUB_SHA': 'baca8fa901d88b07f0c0e3f00a4f8a4263611551', 'GITHUB_STATE': '/home/runner/work/_temp/_runner_file_commands/save_state_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_STEP_SUMMARY': '/home/runner/work/_temp/_runner_file_commands/step_summary_39e01021-587e-44b7-9b9d-f6103a2911fc', 'GITHUB_TRIGGERING_ACTOR': 'jadielam', 'GITHUB_WORKFLOW': 'Docs', 'GITHUB_WORKFLOW_REF': 'videoflow/videoflow/.github/workflows/docs.yml@refs/heads/master', 'GITHUB_WORKFLOW_SHA': 'baca8fa901d88b07f0c0e3f00a4f8a4263611551', 'GITHUB_WORKSPACE': '/home/runner/work/videoflow/videoflow', 'GOROOT_1_24_X64': '/opt/hostedtoolcache/go/1.24.13/x64', 'GOROOT_1_25_X64': '/opt/hostedtoolcache/go/1.25.14/x64', 'GOROOT_1_26_X64': '/opt/hostedtoolcache/go/1.26.8/x64', 'GRADLE_HOME': '/usr/share/gradle-9.7.1', 'HCA_CLOUD_PROVIDER': 'azure', 'HOME': '/home/runner', 'HOMEBREW_CLEANUP_PERIODIC_FULL_DAYS': '3650', 'HOMEBREW_NO_AUTO_UPDATE': '1', 'INVOCATION_ID': '15d8f2ea8d044311994759f59f0fbc0e', 'ImageOS': 'ubuntu24', 'ImageVersion': '20260907.300.1', 'JAVA_HOME': '/usr/lib/jvm/temurin-17-jdk-amd64', 'JAVA_HOME_11_X64': '/usr/lib/jvm/temurin-11-jdk-amd64', 'JAVA_HOME_17_X64': '/usr/lib/jvm/temurin-17-jdk-amd64', 'JAVA_HOME_21_X64': '/usr/lib/jvm/temurin-21-jdk-amd64', 'JAVA_HOME_25_X64': '/usr/lib/jvm/temurin-25-jdk-amd64', 'JAVA_HOME_8_X64': '/usr/lib/jvm/temurin-8-jdk-amd64', 'JOURNAL_STREAM': '9:12689', 'LANG': 'C.UTF-8', 'LOGNAME': 'runner', 'MEMORY_PRESSURE_WATCH': '/sys/fs/cgroup/system.slice/hosted-compute-agent.service/memory.pressure', 'MEMORY_PRESSURE_WRITE': 'c29tZSAyMDAwMDAgMjAwMDAwMAA=', 'NVM_DIR': '/home/runner/.nvm', 'OLDPWD': '/home/runner/work/videoflow/videoflow', 'PATH': '/home/runner/work/_temp/setup-uv-cache/builds-v0/.tmpBKyaem/bin:/home/runner/work/_temp/setup-uv-cache/archive-v0/yrD2oxi3dSpikfW8/bin:/home/runner/work/videoflow/videoflow/.venv/bin:/opt/hostedtoolcache/uv/0.12.13/x86_64:/home/runner/.local/bin:/snap/bin:/home/runner/.local/bin:/opt/pipx_bin:/home/runner/.cargo/bin:/home/runner/.config/composer/vendor/bin:/usr/local/.ghcup/bin:/home/runner/.dotnet/tools:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin', 'PIPX_BIN_DIR': '/opt/pipx_bin', 'PIPX_HOME': '/opt/pipx', 'POWERSHELL_DISTRIBUTION_CHANNEL': 'GitHub-Actions-Linux', 'PSModulePath': '/root/.local/share/powershell/Modules:/usr/local/share/powershell/Modules:/opt/microsoft/powershell/7/Modules:/usr/share/az_15.6.1', 'PWD': '/home/runner/work/videoflow/videoflow', 'RUNNER_ARCH': 'X64', 'RUNNER_ENVIRONMENT': 'github-hosted', 'RUNNER_NAME': 'GitHub Actions 1000000165', 'RUNNER_OS': 'Linux', 'RUNNER_TEMP': '/home/runner/work/_temp', 'RUNNER_TOOL_CACHE': '/opt/hostedtoolcache', 'RUNNER_TRACKING_ID': 'github_f645d36d-cf28-422d-aa73-164bbe3d2cec', 'RUNNER_WORKSPACE': '/home/runner/work/videoflow', 'SELENIUM_JAR_PATH': '/usr/share/java/selenium-server.jar', 'SGX_AESM_ADDR': '1', 'SHELL': '/bin/bash', 'SHLVL': '2', 'SWIFT_PATH': '/usr/share/swift/usr/bin', 'SYSTEMD_EXEC_PID': '1809', 'USER': 'runner', 'USE_BAZEL_FALLBACK_VERSION': 'silent:', 'UV': '/opt/hostedtoolcache/uv/0.12.13/x86_64/uv', 'UV_CACHE_DIR': '/home/runner/work/_temp/setup-uv-cache', 'UV_RUN_RECURSION_DEPTH': '1', 'VCPKG_INSTALLATION_ROOT': '/usr/local/share/vcpkg', 'VIRTUAL_ENV': '/home/runner/work/_temp/setup-uv-cache/builds-v0/.tmpBKyaem', 'XDG_CONFIG_HOME': '/home/runner/.config', 'XDG_RUNTIME_DIR': '/run/user/1001', '_': '/opt/hostedtoolcache/uv/0.12.13/x86_64/uv', 'DOCUTILSCONFIG': '/home/runner/work/videoflow/videoflow/docs/source/docutils.conf'})) → DeliveredGrant | None

The DeliveredGrant a worker was launched with, or None outside a local grant.

videoflow.deploy.allocation_local.host_memory_used_observed() → Known[dict[str, int]] | Unknown

card uuid -> bytes in use from nvidia-smi, or Unknown when the read failed.

videoflow.deploy.allocation_local.snapshot_generation(devices: Sequence[DeviceIdentity], used: Mapping[str, int]) → str

A digest of what was observed: the fence reserve/release check the host against.

videoflow.deploy.broker_profiles module

Broker profiles: the sizing knobs videoflow deploy provisions its in-cluster NATS and Redis with when the user brings no broker of their own.

deploy.infra renders the dev-grade single-replica Deployments this project has always shipped, and those must stay byte-identical — the k8s integration tests and every cluster that already runs them depend on that shape. The durability work (multi-node JetStream, file-store persistence, an append-only Redis) needs a second shape, and threading six positional knobs through ensure_infra/nats_manifests/redis_manifests would have turned every call site into a lie waiting to happen. So the knobs travel as two small records this module owns, with one constructor per named profile:

• BrokerProfile.dev() — today’s render: one server, emptyDir, no

  persistence. profile = None everywhere in deploy.infra means exactly this.

• BrokerProfile.durable() — a NATS StatefulSet with cluster { routes }

  and a PVC per pod, so a stream with jetstream_replicas copies survives a pod and a node.

• RedisProfile.dev() — one server with an append-only file on an

  emptyDir and noeviction: the same standing as the dev NATS (its file store is on an emptyDir too), so a BATCH flow’s reliable_work channels are admitted on the dev pair and a pod loss costs both alike.

• RedisProfile.durable() — the same append-only file on a PVC, so the

  blobs survive a pod and a node.

The dev Redis used to be a transport-only cache (persistence off, volatile-lru). RFC 0006 made composition admission binding, and an evictable store cannot certify reliable_work (PAY-010: a blob whose obligations are outstanding must not be evicted, and a store that keeps nothing across a restart lets an accepted envelope outlive its bytes). Every key still carries a TTL (PROTOCOL.md BLOB-7) and the reconciler still reclaims orphans, so memory stays bounded; under pressure a full store now refuses a write — a typed TransientFailure the publisher sees — instead of silently dropping the oldest blob. An operator who wants the old cache shape brings their own Redis (--blob-redis-url), which admission reads back live and refuses for a BATCH flow.

The records are dataclasses rather than dicts because we own their shape (the Kubernetes objects they become stay dicts, per deploy.manifests). Validation happens at construction so a bad profile fails at the CLI, before any manifest is rendered — a JetStream cluster with an even replica count or a stream replicated more times than there are servers cannot elect a leader, and the only symptom in the cluster would be a provision Job that never completes.

Priority: priority_class lands on every pod a profile renders. The --priority-class flag sets it on the workers and on the infra alike, so a deploy told to yield to higher-priority work yields everything it created.

videoflow.deploy.broker_profiles.BROKER_PROFILE_NAMES = ('dev', 'durable')

The profile names --broker-profile accepts, in the order the help shows them.

class videoflow.deploy.broker_profiles.BrokerProfile(replicas: int = 1, jetstream_replicas: int = 1, storage_class: str | None = None, storage_size: str = '10Gi', persistence: bool = False, max_file_store: str = '10GB', priority_class: str | None = None)

Bases: object

How the auto-provisioned NATS is shaped.

• Arguments:

  • replicas: NATS server pods. 1 renders the Deployment of k8s/nats.yaml; more renders a StatefulSet whose pods route to each other over a headless Service (cluster { routes }).

  • jetstream_replicas: copies each stream should keep — what the provisioner asks JetStream for. At most replicas and at most 5.

  • storage_class: StorageClass of the per-pod PersistentVolumeClaim when persistence is on; None takes the cluster default.

  • storage_size: size of that claim (a Kubernetes quantity, 10Gi).

  • persistence: keep the JetStream file store on a PersistentVolumeClaim instead of an emptyDir that dies with the pod.

  • max_file_store: the jetstream { max_file_store } server limit; keep it under storage_size when persisting.

  • priority_class: priorityClassName for the NATS pods, or none.

classmethod dev(priority_class: str | None = None) → BrokerProfile

Today’s single-replica, emptyDir NATS — what profile = None means.

classmethod durable(replicas: int = 3, storage_class: str | None = 'local-path', priority_class: str | None = None) → BrokerProfile

A JetStream cluster whose streams keep up to three copies on persistent volumes. jetstream_replicas is min(replicas, 3): three copies is the standard JetStream deployment and five buys little at this scale.

jetstream_replicas: int = 1

max_file_store: str = '10GB'

property name: str

The --broker-profile name this shape answers to (deploy.infra records it on the Service).

persistence: bool = False

priority_class: str | None = None

replicas: int = 1

property stateful: bool

Whether the profile renders a StatefulSet. True as soon as the pods need a stable identity: a persistent claim per pod, or route peers that must find each other by a predictable DNS name.

storage_class: str | None = None

storage_size: str = '10Gi'

videoflow.deploy.broker_profiles.DEFAULT_STORAGE_CLASS = 'local-path'

The default StorageClass the durable profiles claim from. local-path is what k3s (and kind) ship; a cluster without it names its own through --broker-storage-class.

videoflow.deploy.broker_profiles.REDIS_EVICTION_POLICIES = ('noeviction', 'volatile-lru', 'allkeys-lru', 'volatile-lfu', 'allkeys-lfu', 'volatile-random', 'allkeys-random', 'volatile-ttl')

Redis maxmemory-policy values, as the server spells them.

videoflow.deploy.broker_profiles.REDIS_PERSISTENCE_MODES = ('none', 'appendonly')

What RedisProfile.persistence may be. none renders --appendonly no with RDB snapshots off (transport, not storage); appendonly renders an AOF under /data — on the volume RedisProfile.storage names.

videoflow.deploy.broker_profiles.REDIS_STORAGE_MODES = ('emptyDir', 'claim')

an emptyDir that lives as long as the pod (a container restart replays the file, a pod loss does not — the dev NATS file store has the same standing) or a PersistentVolumeClaim.

Type:

Where RedisProfile.persistence writes

class videoflow.deploy.broker_profiles.RedisProfile(persistence: str = 'none', eviction: str = 'noeviction', storage: str = 'emptyDir', storage_class: str | None = None, storage_size: str = '10Gi', priority_class: str | None = None)

Bases: object

How the auto-provisioned Redis (the large-payload blob store) is shaped.

• Arguments:

  • persistence: 'none' (RDB and AOF off; the store is transport) or 'appendonly' (an AOF under /data on the volume storage names).

  • eviction: the maxmemory-policy. Both shipped profiles refuse a write when the store is full (noeviction) rather than drop a blob whose readers have not released it; every videoflow key still carries a TTL (PROTOCOL.md BLOB-7), which is what bounds orphans, not eviction.

  • storage: 'emptyDir' (the pod’s lifetime; the dev profile) or 'claim' (a PersistentVolumeClaim; the durable profile). Meaningful only with persistence on — a store that writes nothing needs no volume.

  • storage_class: StorageClass of the claim; None takes the cluster default.

  • storage_size: size of that claim.

  • priority_class: priorityClassName for the Redis pod, or none.

classmethod dev(priority_class: str | None = None) → RedisProfile

One server, an append-only file on an emptyDir, noeviction: a blob outlives a container restart (like the dev NATS file store) and is never dropped while its readers hold it, which is what reliable_work asks of a payload store; a pod loss takes it, which is what tolerated_failures refuses the dev pair for.

classmethod durable(storage_class: str | None = 'local-path', priority_class: str | None = None) → RedisProfile

The same append-only, never-evicting Redis on a claim, so its blobs survive a pod and a node.

eviction: str = 'noeviction'

property name: str

durable (on a claim), dev (persistent on an emptyDir) or cache (nothing written — the shape an operator asks for explicitly; no --broker-profile name renders it).

Type:

The profile name recorded on the Service

persistence: str = 'none'

property persistent: bool

Whether the server writes its data set to disk at all (an AOF under /data).

priority_class: str | None = None

property stateful: bool

Whether the profile keeps its data on a PersistentVolumeClaim.

storage: str = 'emptyDir'

storage_class: str | None = None

storage_size: str = '10Gi'

videoflow.deploy.broker_profiles.broker_profiles(name: str, replicas: int | None = None, storage_class: str | None = None, priority_class: str | None = None) → tuple[BrokerProfile, RedisProfile]

The (BrokerProfile, RedisProfile) pair a profile name denotes — the one lookup --broker-profile goes through, so the CLI and any programmatic caller resolve a name the same way.

• Arguments:

  • name: one of BROKER_PROFILE_NAMES.

  • replicas: NATS replica override (--broker-replicas); durable only.

  • storage_class: claim StorageClass override (--broker-storage-class); durable only.

  • priority_class: priorityClassName for every infra pod.

• Raises:

  • ConfigError: an unknown name, or a durable-only override given with the dev profile (which has nothing to apply it to).

videoflow.deploy.build module

Image auto-build for videoflow deploy: when no --image is given, find the Dockerfile next to the graph module (gpu.Dockerfile for a flow with GPU nodes), build the videoflow-base image it is FROM if missing, and build the solution image from the enclosing git root (solution Dockerfiles COPY sibling packages, so the repo root is the context).

The base images can only be auto-built from a videoflow source checkout (docker/base/Dockerfile COPYs the source tree); a wheel-only install gets a precise error with the manual commands instead.

videoflow.deploy.build.autobuild(graph_dir: str, needs_gpu: bool, context_override: str | None = None) → str | None

The whole auto-build path: find the Dockerfile, ensure its base image, build the solution image. Returns the built tag, or None when the solution ships no Dockerfile (caller falls back to explicit-image resolution).

videoflow.deploy.build.base_image_for(dockerfile_path: str) → str | None

The default of the Dockerfile’s ARG BASE_IMAGE= line, or None if it has none.

videoflow.deploy.build.build_context_for(graph_dir: str, override: str | None = None) → str

The docker build context: an explicit --build-context, else the git root enclosing the graph (solution Dockerfiles COPY sibling packages from the repo root — e.g. the offside solution copies nine of them), else the graph dir.

videoflow.deploy.build.build_image(dockerfile: str, context: str, tag: str) → None

Builds the solution image, streaming docker output (layer cache makes unchanged rebuilds fast).

videoflow.deploy.build.default_tag(graph_dir: str) → str

Deterministic human-readable tag for the auto-built solution image, e.g. videoflow-offside:latest.

videoflow.deploy.build.docker_gpus_available() → bool

Whether the local docker daemon has the NVIDIA runtime (for –gpus all).

videoflow.deploy.build.ensure_base_image(base_ref: str) → None

Makes sure base_ref (a videoflow-base:* image) exists locally, building it from the videoflow source checkout if missing.

• Raises:

  • RuntimeError when the image is missing and videoflow is not an editable/source install (the base Dockerfile COPYs the source tree, so there is nothing to build from).

videoflow.deploy.build.find_dockerfile(graph_dir: str, needs_gpu: bool) → str | None

The Dockerfile deploy builds the node image from: gpu.Dockerfile when the flow has GPU nodes and one exists, else Dockerfile. None when the solution ships neither (the caller falls back to requiring --image).

videoflow.deploy.build.image_exists(ref: str) → bool

videoflow.deploy.build.run_in_image(image: str, command: List[str], mounts: List[Mount] | None = None, workdir: str | None = None, gpus: bool = False, capture: bool = False, interactive: bool = False) → str | None

Runs a command in the solution image with the given hostPath-style mounts (Mount records from manifests.parse_mounts) — how deploy executes the prepare hook and the graph compile without the graph’s deps on the host.

Claim mounts (manifests.parse_pvc_mounts, Mount.claim set) are skipped: a PersistentVolumeClaim exists only inside the cluster, and this container runs on the operator’s host, where the same directory is reached by the hostPath the claim shadows in the pods (see manifests.pod_mounts).

• Returns:

  • the command’s stdout when capture, else None.

• Raises:

  • RuntimeError on a non-zero exit (with stderr when captured).

videoflow.deploy.cli module

Command-line entrypoint for deploying a videoflow graph to Kubernetes.

videoflow deploy path/to/graph.py

deploy is one command that does everything: it generates the solution config (asking the template’s questions when none exists), runs the solution’s prepare hook, builds and loads the node image into the detected local cluster, provisions the broker (an in-cluster dev NATS + Redis when --nats is omitted), applies the flow, and — for a BATCH flow — waits for it to run to completion and then tears down every resource (Kubernetes workloads + broker streams + owned infra). A REALTIME flow is applied and left running; stop it later with videoflow teardown. Each automatic step has an explicit override (--config, --image, --nats, --mount, --no-prepare, --no-build, …). Nothing is written to disk (beyond a generated config.yaml) unless --render-only (write manifest files + kustomization) or --dry-run (print YAML to stdout) is given.

The graph module must expose a factory (default name build_flow) that returns a built videoflow.core.flow.Flow without calling .run() on it — the CLI needs the graph, not a running flow. When the graph’s dependencies are not importable on this machine, deploy compiles it inside the solution image instead.

videoflow.deploy.cli.build_parser() → ArgumentParser

videoflow.deploy.cli.main(argv: list[str] | None = None) → int

The one place a videoflow failure becomes an operator-facing message and a process exit status.

The exit status carries the class of failure — 2 your flow, 3 your environment, 4 the flow ran and lost nodes, 5 it stalled — so CI and wrapper scripts can triage without parsing stderr. Everything used to exit 1.

videoflow.deploy.cli.parse_resources(entries: list[str] | None) → dict[str, dict[str, str]]

--resources values as {node or '*': {cpu, memory, cpu_limit, memory_limit}}: each entry is NODE=key:quantity[,key:quantity...] (* for every node).

• Raises:

  • ConfigError: an entry is malformed or names an unknown key.

videoflow.deploy.cli.render_error(error: VideoflowError) → None

Prints a failure the way an operator needs to read it: what broke, then what to do about it, then the structured detail — never a traceback, which is a stack of framework internals the reader did not write and cannot act on. Set VF_DEBUG=1 when the traceback is the thing you want.

videoflow.deploy.cluster module

Best-effort detection of what kind of Kubernetes cluster kubectl points at, and the cluster-flavor-specific mechanics that depend on it: how to load a locally built image into it, whether hostPath mounts see the local filesystem, and whether GPU pods are schedulable. Everything here is advisory — deploy still works with explicit flags when detection gets it wrong.

Each flavor is one ClusterFlavorHandler registered at import. Supporting a new one (microk8s, colima, k0s) is a class plus a register_cluster_flavor call rather than an edit to three parallel if-ladders, which is what this used to be — and what made it easy to teach detection about a flavor while forgetting to teach image loading about it.

class videoflow.deploy.cluster.ClusterFlavorHandler

Bases: object

One local-cluster flavor: how to recognize it, how to get a locally built image into it, and whether its hostPath mounts see the host filesystem.

Subclasses set name and implement matches. load_images defaults to refusing, which is the right answer for anything remote.

hostpath_warning() → str | None

Message when hostPath will not resolve against the local filesystem, else None.

load_images(images: List[str], kubectl: str = 'kubectl') → None

Side-loads locally built images so pods can run them without a registry.

• Raises:

  • RuntimeError when the images cannot be loaded (remote cluster, missing tool, failed command). The message must name the fix.

matches(context_name: str, node_labels: Callable[[], str]) → bool

Whether the current cluster is this flavor. node_labels is a callable so a handler that can decide from the context name alone costs no kubectl call; the result is shared between handlers that do need it.

name: str = ''

Flavor identifier, returned by detect_cluster and passed back to load_images/hostpath_warning.

class videoflow.deploy.cluster.GpuAvailability(per_node_allocatable: Dict[str, int]=<factory>, per_node_in_use: Dict[str, int]=<factory>, occupancy_known: bool = True)

Bases: object

One GPU extended resource’s pool capacity with occupancy subtracted: what is allocatable per node, what running pods already claim, and the derived free numbers preflight compares demand against. Raw allocatable alone lies in a shared cluster — the scheduler will not grant units that other workloads hold.

property allocatable: int

property free: int

property in_use: int

property max_free_on_node: int

the most units any single node can still grant.

Type:

The per-pod bound

occupancy_known: bool = True

False when the pod listing behind per_node_in_use could not be read: free is then an upper bound, not a fact, and preflight says so.

per_node_allocatable: Dict[str, int]

per_node_in_use: Dict[str, int]

videoflow.deploy.cluster.active_runs_observed(kubectl: str, namespace: str, flow_id: str) → Known[set[str]] | Unknown

The run ids of every workload of flow_id in namespace (the videoflow.io/run-id label of its Deployments, StatefulSets and Jobs), or Unknown when the listing failed — a failed read is never “no other run”.

videoflow.deploy.cluster.allocatable_gpus(kubectl: str = 'kubectl', resource: str = 'nvidia.com/gpu') → int

Total allocatable units of one GPU extended resource across the pool’s nodes (0 when no pool node advertises it or the cluster is unreachable).

videoflow.deploy.cluster.classify_gfd_labels(labels: Mapping[str, str], advertised_units: object = None) → str

What one node’s GPU units are, from its GPU Feature Discovery labels: 'physical' (one unit = one whole device), 'mig' (hardware-isolated slices), 'time-sliced' or 'mps' (shares of a device — MPS is recognised from its explicit strategy label alone, with or without the replica/-SHARED auxiliaries), or 'unknown' (no GFD labels to judge by). The pure rule behind classify_gpu_resource; the reference allocator’s node fixtures use the same one so both agree on every fixture.

• Arguments:

  • advertised_units: the node’s allocatable count of the resource, when known. Under the single MIG strategy slices are advertised under the plain resource name, so more units than gpu.count cards is geometry evidence; a MIG-capable node with MIG disabled advertises whole cards and is physical.

videoflow.deploy.cluster.classify_gpu_resource(kubectl: str = 'kubectl', resource: str = 'nvidia.com/gpu', exclude_nodes: AbstractSet[str] = frozenset({})) → str

What one advertised GPU extended resource’s units actually are, from GPU Feature Discovery node labels: 'physical' (one unit = one whole device), 'mig' (units are hardware-isolated MIG slices), 'time-sliced' (units are shares of a device), or 'unknown' (no GFD labels to judge by — e.g. a non-NVIDIA resource, or a cluster without GFD).

This is what makes gpu_count > 1 checkable: the scheduler happily grants N units of any integer resource, but only whole physical devices can be spanned by one model. Classification looks only at pool nodes (the same videoflow.io/gpu-pool=true snapshot the capacity math reads, minus exclude_nodes) that advertise resource — a time-sliced node outside the pool advertising the same name is not somewhere a pool workload can land:

• a mig- final path segment, or nvidia.com/mig.strategy=single on a MIG-capable advertising node (slices renamed to nvidia.com/gpu) → mig;

• nvidia.com/gpu.sharing-strategy=time-slicing, a -SHARED product suffix, or nvidia.com/gpu.replicas > 1 → time-sliced;

• GFD labels present and none of the above → physical.

Worst case wins across nodes (any time-sliced advertiser taints the answer): the scheduler may place the pod on any advertising node, so the safe claim is the weakest one.

videoflow.deploy.cluster.combine_classifications(kinds: Iterable[str]) → str

One answer for a pool: worst case wins across advertisers. A share kind anywhere makes multi-unit claims impossible everywhere (the scheduler may place the pod on any advertising node), MIG likewise; and an unlabeled advertiser next to a physical one is not “physical” — nothing proves its units are whole devices — it is unresolved.

videoflow.deploy.cluster.current_context(kubectl: str = 'kubectl') → str

videoflow.deploy.cluster.detect_cluster(kubectl: str = 'kubectl') → str

Classifies the cluster kubectl currently points at, by asking each registered flavor in order. Context-name conventions identify kind/minikube/docker-desktop; k3s installs use a generic context name (‘default’), so they are confirmed via node facts instead — fetched at most once per call, and only if some handler asks for them.

videoflow.deploy.cluster.dra_owned_nodes_observed(kubectl: str = 'kubectl') → Known[set] | Unknown

The nodes a DRA driver publishes GPU ResourceSlices for — the other allocator’s territory (ALLOC-018). Known(set()) when the API serves no slices (or the resource.k8s.io group at all: a cluster without DRA has no DRA-owned nodes), Unknown only when a listing that exists could not be read.

videoflow.deploy.cluster.get_cluster_flavor(cluster: str) → ClusterFlavorHandler

The handler registered under cluster.

• Raises:

  • RuntimeError when no flavor is registered under that name.

videoflow.deploy.cluster.gpu_availability(kubectl: str = 'kubectl', resource: str = 'nvidia.com/gpu', in_use: Dict[str, Dict[str, int]] | None = None, exclude_nodes: AbstractSet[str] = frozenset({})) → GpuAvailability

Pool-scoped availability of one GPU extended resource. in_use accepts a pre-fetched gpu_units_in_use() result so a caller checking several resources fetches the pod list once; None fetches it here. exclude_nodes drops nodes the caller has already ruled out (e.g. mix planning excluded them as another flow’s), so capacity math matches what will be planned.

videoflow.deploy.cluster.gpu_inventory(kubectl: str = 'kubectl') → List[NodeInventory]

gpu_inventory_observed for display-only callers: [] when unknown.

videoflow.deploy.cluster.gpu_inventory_observed(kubectl: str = 'kubectl') → Known[List[NodeInventory]] | Unknown

The videoflow pool’s GPU inventory as NodeInventory records — only nodes labeled videoflow.io/gpu-pool=true, because that is the only place the rendered pods can schedule and the only nodes mix mode may repartition. Physical facts come off GPU Feature Discovery labels (nvidia.com/gpu.count, .product, .memory — the last in MiB); nodes without those labels contribute nothing: without GFD there is no inventory to lay out, and the mix strategy reports that as its own preflight problem rather than guessing.

Each record also carries the node’s sharing/ownership state (time-slicing signals, existing MIG geometry, the videoflow.io/gpu-owner stamp, units in use by running pods). These are facts, not decisions: the cluster is multi-tenant, and MixGpu decides which nodes are usable.

Unknown when the node listing itself failed; when only the pod listing failed, every record carries occupancy_known=False so the mix strategy refuses to plan on it rather than treat the pool as idle.

videoflow.deploy.cluster.gpu_preflight(kubectl: str = 'kubectl', gpu_runtime_class: str | None = None, demand: dict | None = None, gpu_mode: str = 'exclusive', max_per_pod: dict | None = None, pod_claims: dict | None = None) → List[str]

Checks what a GPU node workload needs (see manifests._pod_spec): a node labeled videoflow.io/gpu-pool=true; enough allocatable units of each requested extended resource to satisfy the flow’s whole demand (an under-provisioned flow schedules partially and stalls with the rest of its pods Pending); and — where the NVIDIA container runtime is an opt-in RuntimeClass rather than the node default — a --gpu-runtime-class, without which the pod schedules and then runs with no device. Returns problem strings with copy-pasteable fixes (empty list = OK). A problem prefixed with gpu.IMPOSSIBLE_GPU_REQUEST is fatal regardless of --strict-preflight (the request cannot work by construction).

• Arguments:

  • demand: dict of extended-resource name -> total units the flow requests (sum over GPU nodes of nb_tasks * gpu_count), or None to skip the capacity comparison and only check that the resource exists.

  • gpu_mode: the --gpu-mode strategy name whose preflight_problems runs the mode-specific checks.

  • max_per_pod: dict of extended-resource name -> largest single-pod claim (manifests.gpu_max_per_pod), or None to skip the per-node capacity and resource-classification checks (RFC 0003).

  • pod_claims: dict of extended-resource name -> one claim per pod replica (manifests.gpu_pod_claims), or None to skip the per-host packing check — the one that catches a pool whose free devices are fragmented across hosts although the total and the largest node both suffice.

videoflow.deploy.cluster.gpu_units_in_use(kubectl: str = 'kubectl') → Dict[str, Dict[str, int]]

gpu_units_in_use_observed for display-only callers: {} when unknown. Anything that plans or mutates on occupancy must use the observed form.

videoflow.deploy.cluster.gpu_units_in_use_observed(kubectl: str = 'kubectl') → Known[Dict[str, Dict[str, int]]] | Unknown

Extended-resource units currently claimed by pods, as node name -> resource -> units, summed over the resources.limits of every non-terminated pod in the cluster (all namespaces — a foreign workload’s claim occupies a device just as much as ours). Extended resources always carry a domain, so keys containing / are counted and native resources (cpu, memory, hugepages-*) are not.

Unknown when the pod listing could not be read or parsed. Callers that decide anything destructive on occupancy must branch on that: a failed read does not prove a device idle.

videoflow.deploy.cluster.hostpath_warning(cluster: str) → str | None

A message when hostPath mounts will NOT resolve against the local filesystem (the cluster “node” is a VM/container with its own filesystem), else None. An unregistered flavor warns nothing rather than raising: this is advisory, and deploy should not fail over a missing warning.

videoflow.deploy.cluster.is_registry_qualified(image: str) → bool

Whether an image reference names a registry — its first path component has a dot or a port, or is localhost (the Docker reference grammar’s rule for telling registry.example/ns/img from the implicit docker.io ns/img). Such an image was pushed somewhere the cluster pulls from; a bare local tag was only ever built here.

videoflow.deploy.cluster.load_images(cluster: str, images: List[str], kubectl: str = 'kubectl') → None

Loads locally built docker images into the detected cluster so pods can pull them without a registry.

• Raises:

  • RuntimeError when the cluster is remote (push to a registry instead), when a required tool is missing, or when a load command fails.

videoflow.deploy.cluster.max_allocatable_gpus_per_node(kubectl: str = 'kubectl', resource: str = 'nvidia.com/gpu') → int

The largest allocatable count of one GPU extended resource on any single pool node (0 when no pool node advertises it or the cluster is unreachable). The per-pod schedulability bound, complementing allocatable_gpus: total capacity answers “will the whole flow schedule”, this answers “can any single node host the biggest pod” — all of a pod’s gpu_count devices must come from one node, so a flow can pass the total-capacity check and still never schedule.

videoflow.deploy.cluster.nvidia_runtimeclass(kubectl: str = 'kubectl') → str | None

The NVIDIA RuntimeClass name when the cluster registers one, else None. Prefers the conventional nvidia but also recognizes variant names (e.g. a distro registering nvidia-container-runtime) so the shared-mode escalation cannot silently no-op on them.

videoflow.deploy.cluster.refuse_concurrent_run(kubectl: str, namespace: str, flow_id: str, run_id: str) → None

The --single-run policy (RFC 0006 §10, RUN-047): refuse to start run_id while another run of the flow holds workloads in the namespace. Decided before any resource is created.

• Raises:

  • ActiveRunConflict: another run is active.

  • UnobservableState: the listing failed; an unverifiable namespace is not a free one.

videoflow.deploy.cluster.register_cluster_flavor(handler: ClusterFlavorHandler, before: str | None = 'generic-remote') → None

Registers a cluster flavor. Detection tries handlers in registration order, so a new handler is inserted before before (by default the generic-remote fallback, which matches everything and must stay last).

• Arguments:

  • handler: the flavor handler instance.

  • before: name of the handler to insert ahead of, or None to append.

• Raises:

  • ValueError: before names no registered flavor.

videoflow.deploy.compile module

Compiles a graph module to a JSON specs document, for use where the operator machine cannot import the graph (its ML dependencies live only in the solution image). videoflow deploy runs this inside that image:

docker run --rm -v <graph_dir>:<graph_dir> -w <graph_dir> <image>         python -m videoflow.compile graph.py[:factory]

and rebuilds the NodeSpec list on the host with NodeSpec.from_dict — the same serialization the provision Job’s specs ConfigMap uses.

Output (stdout): {"flow_id": ..., "flow_type": ..., "specs": [...]}.

videoflow.deploy.compile.compile_to_dict(target: str, envelope_version: int | None = None) → dict

videoflow.deploy.compile.declared_requirements(flow: Any) → FlowRequirements

What a flow’s nodes declare beyond the graph (sink effect guarantees, fused execution groups, batching contracts): the document half of the composition admission. Empty for a flow that declares nothing.

videoflow.deploy.compile.load_flow(target: str) → Flow

• Arguments:

  • target: path/to/graph.py or path/to/graph.py:factory_name (factory defaults to build_flow).

• Returns:

  • a built Flow produced by calling the factory.

videoflow.deploy.compile.main(argv: List[str] | None = None) → None

videoflow.deploy.compile.requirements_from_document(document: dict | str) → FlowRequirements

The declared requirements of a compile-JSON document (empty when it carries none).

videoflow.deploy.compile.specs_from_document(document: dict | str) → tuple[str, str, List[NodeSpec]]

(flow_id, flow_type, specs) from a compile-JSON document (dict or JSON string).

videoflow.deploy.gpu module

GPU allocation strategies: how a GPU node’s pods claim devices, what to preflight before deploying them, and any cluster state a strategy needs to set up for a run and restore afterwards.

Three strategies ship, two of them matching the two-mode GPU design (RFC 0004):

• exclusive (default) claims whole physical devices through an integer extended resource, so the scheduler accounts for them and a flow that outgrows the cluster stays Pending rather than thrashing. Under it a GPU unit is one whole device — sharing a device between components is not a capability of the mode, and the node API offers no way to ask for it. (Dev clusters share devices by advertising time-sliced units and keeping gpu_count == 1; the preflight below understands that arrangement.)

• mix (opt-in) serves declared demands: nodes with gpu_memory_gib get exclusive MIG slices chosen by the layout solver (deploy/mig.py), nodes with gpu_count — declared or defaulted — get whole physical cards. Its prepare/cleanup hooks apply and restore MIG geometry for the run — over an explicit AllocationPlan (plan_layout → apply_plan → observe_geometry / cleanup), with ownership claimed by compare-and-swap, the occupancy re-read before any geometry write, and readiness judged by evidence of this operation rather than by the mig.config.state label a previous geometry left behind (plan Phase 4; deploy/allocation_kubernetes.py is the same machinery behind the AcceleratorAllocationBackend contract).

• dra renders Dynamic Resource Allocation claims (deploy/allocation_dra.py) and refuses at preflight unless a DRA driver publishes GPU ResourceSlices.

The mode used to be a bare string branched on in four places (pod resources, manifest validation, preflight, the CLI’s choices), which is why adding a third meant finding all four. A strategy is now one class registered here.

class videoflow.deploy.gpu.AllocationPlan(layout: GpuLayout, excluded: Mapping[str, str], flow_id: str | None, generation: str | None = None)

Bases: object

One deploy’s MIG decision, explicit and immutable (plan Phase 4, ALLOC-011): the solver’s layout, the pool nodes it was not allowed to touch (with the reason each), the flow it was planned for and the inventory generation it was planned on. Passed between resolve_specs/preflight/prepare instead of living in the registered strategy singleton, so two deploys in one process — or a retried prepare in a fresh one — cannot act on each other’s cached geometry.

excluded: Mapping[str, str]

flow_id: str | None

generation: str | None = None

layout: GpuLayout

mig_nodes() → List[str]

class videoflow.deploy.gpu.AppliedGeometry(nodes: tuple[str, ...], entries: Mapping[str, str], epoch: str, namespace: str, expected_allocatable: Mapping[str, Mapping[str, int]])

Bases: object

What MixGpu.apply_plan wrote, so readiness can be judged against this operation: the per-node entry names (nonce’d per run), the owner epoch the claim was stamped under, the namespace the manager runs in, and the allocatable resources each node must advertise once the geometry is live.

entries: Mapping[str, str]

epoch: str

expected_allocatable: Mapping[str, Mapping[str, int]]

namespace: str

nodes: tuple[str, ...]

videoflow.deploy.gpu.DEFAULT_GPU_RESOURCE = 'nvidia.com/gpu'

Extended resource used when neither the node nor the deploy names one.

class videoflow.deploy.gpu.DraGpu(device_class: str = 'gpu.nvidia.com')

Bases: GpuStrategy

--gpu-mode dra: GPU pods claim devices through a ResourceClaimTemplate instead of an extended resource. Renders only — prepare/cleanup are no-ops and preflight_problems reports the missing driver, so a deploy stops before applying claims nothing would allocate.

claim_manifests(spec: NodeSpec, namespace: str, flow_id: str, run_id: str) → list[dict]

The ResourceClaimTemplate this node’s pods instantiate, one per pod.

name: str = 'dra'

Mode name, as used by --gpu-mode.

pod_claims(spec: NodeSpec) → list[dict]

The pod’s spec.resourceClaims entries for this node.

pod_resources(spec: NodeSpec, gpu_resource_name: str | None = None) → dict

The container resources fragment for one GPU pod — {} for a strategy that requests nothing.

Stays a plain dict on purpose: it is spliced straight into a container spec, so its shape is Kubernetes’ ResourceRequirements schema and not a record we own. A strategy may legitimately emit requests, limits or neither, with arbitrary extended-resource keys.

• Arguments:

  • spec: the node’s NodeSpec (gpu_count, gpu_resource_name).

  • gpu_resource_name: deploy-level default extended-resource name.

preflight_problems(kubectl: str = 'kubectl', demand: dict[str, int] | None = None, gpu_runtime_class: str | None = None, max_per_pod: dict[str, int] | None = None, pod_claims: dict[str, list[int]] | None = None) → List[str]

Strategy-specific preflight problems, each a string naming its fix. The flavor-independent checks (cluster reachable, a labeled GPU node) are run by cluster.gpu_preflight before this is called. A problem prefixed with IMPOSSIBLE_GPU_REQUEST is fatal regardless of --strict-preflight.

Third-party strategies should tolerate future keyword inputs (accept **kwargs): new preflight inputs arrive as keywords with None defaults, as max_per_pod (RFC 0003) and pod_claims did.

• Arguments:

  • demand: extended-resource name -> units the flow requests, or None to skip capacity comparison.

  • gpu_runtime_class: the --gpu-runtime-class value, if given.

  • max_per_pod: extended-resource name -> the largest single-pod claim (manifests.gpu_max_per_pod), or None to skip per-node checks.

  • pod_claims: extended-resource name -> one claim per pod replica (manifests.gpu_pod_claims), or None to skip the per-host packing check (pack_pod_claims).

class videoflow.deploy.gpu.ExclusiveGpu

Bases: GpuStrategy

Whole-device claims through an integer extended resource (the default).

name: str = 'exclusive'

Mode name, as used by --gpu-mode.

pod_resources(spec: NodeSpec, gpu_resource_name: str | None = None) → dict

The container resources fragment for one GPU pod — {} for a strategy that requests nothing.

Stays a plain dict on purpose: it is spliced straight into a container spec, so its shape is Kubernetes’ ResourceRequirements schema and not a record we own. A strategy may legitimately emit requests, limits or neither, with arbitrary extended-resource keys.

• Arguments:

  • spec: the node’s NodeSpec (gpu_count, gpu_resource_name).

  • gpu_resource_name: deploy-level default extended-resource name.

preflight_problems(kubectl: str = 'kubectl', demand: dict[str, int] | None = None, gpu_runtime_class: str | None = None, max_per_pod: dict[str, int] | None = None, pod_claims: dict[str, list[int]] | None = None) → List[str]

Strategy-specific preflight problems, each a string naming its fix. The flavor-independent checks (cluster reachable, a labeled GPU node) are run by cluster.gpu_preflight before this is called. A problem prefixed with IMPOSSIBLE_GPU_REQUEST is fatal regardless of --strict-preflight.

Third-party strategies should tolerate future keyword inputs (accept **kwargs): new preflight inputs arrive as keywords with None defaults, as max_per_pod (RFC 0003) and pod_claims did.

• Arguments:

  • demand: extended-resource name -> units the flow requests, or None to skip capacity comparison.

  • gpu_runtime_class: the --gpu-runtime-class value, if given.

  • max_per_pod: extended-resource name -> the largest single-pod claim (manifests.gpu_max_per_pod), or None to skip per-node checks.

  • pod_claims: extended-resource name -> one claim per pod replica (manifests.gpu_pod_claims), or None to skip the per-host packing check (pack_pod_claims).

videoflow.deploy.gpu.GPU_OWNER_EPOCH_LABEL = 'videoflow.io/gpu-owner-epoch'

Companion of GPU_OWNER_LABEL, stamped in the same write with a fresh per-claim value: a release can then tell the claim it is undoing from a later re-claim by the same flow (a crashed deploy’s teardown racing a redeploy), and leave the newer claim standing.

videoflow.deploy.gpu.GPU_OWNER_LABEL = 'videoflow.io/gpu-owner'

Node label recording which flow owns a node’s MIG geometry. The cluster is multi-tenant: mix’s prepare() stamps it (compare-and-swap, no –overwrite) before partitioning, other flows exclude stamped nodes from planning and scheduling, and cleanup() restores only the nodes its flow stamped. The value is manifests.k8s_name(flow_id) — identical to the pods’ flow-id label.

videoflow.deploy.gpu.GPU_POOL_LABEL = 'videoflow.io/gpu-pool'

Node label a GPU pod selects on, and the taint key it tolerates.

class videoflow.deploy.gpu.GpuStrategy

Bases: object

One GPU allocation mode.

A strategy owns three decisions that must agree with each other: what a GPU pod asks the scheduler for (pod_resources), what makes that request satisfiable and is therefore worth checking first (preflight_problems), and whether the cluster needs temporary reconfiguration to honour it (prepare/cleanup). Splitting them across modules is what made the old string-mode version easy to extend incorrectly.

claim_manifests(spec: NodeSpec, namespace: str, flow_id: str, run_id: str) → list[dict]

Extra manifests a GPU node needs applied beside its workload (a ResourceClaimTemplate under DRA); [] by default. Plain dicts — Kubernetes API objects, rendered where the workload is.

cleanup(kubectl: str = 'kubectl', flow_id: str | None = None) → None

Undoes prepare. Must be idempotent and tolerant: it is called after a prepare that only partly succeeded, and — for a REALTIME flow, whose lifetime outlives the deploy command — from a later videoflow teardown that passes --gpu-mode but shares no state with the deploy that ran prepare. So it cannot assume prepare completed, or ran at all. With a flow_id it must restore only that flow’s state; without one it may sweep everything videoflow owns (single-operator escape hatch).

name: str = ''

Mode name, as used by --gpu-mode.

pod_claims(spec: NodeSpec) → list[dict]

The pod-level spec.resourceClaims entries a GPU pod of spec needs — [] for the extended-resource strategies, which claim through pod_resources alone. A DRA strategy names its claim here (plan Phase 4).

pod_resources(spec: NodeSpec, gpu_resource_name: str | None = None) → dict

The container resources fragment for one GPU pod — {} for a strategy that requests nothing.

Stays a plain dict on purpose: it is spliced straight into a container spec, so its shape is Kubernetes’ ResourceRequirements schema and not a record we own. A strategy may legitimately emit requests, limits or neither, with arbitrary extended-resource keys.

• Arguments:

  • spec: the node’s NodeSpec (gpu_count, gpu_resource_name).

  • gpu_resource_name: deploy-level default extended-resource name.

preflight_problems(kubectl: str = 'kubectl', demand: dict[str, int] | None = None, gpu_runtime_class: str | None = None, max_per_pod: dict[str, int] | None = None, pod_claims: dict[str, list[int]] | None = None) → List[str]

Strategy-specific preflight problems, each a string naming its fix. The flavor-independent checks (cluster reachable, a labeled GPU node) are run by cluster.gpu_preflight before this is called. A problem prefixed with IMPOSSIBLE_GPU_REQUEST is fatal regardless of --strict-preflight.

Third-party strategies should tolerate future keyword inputs (accept **kwargs): new preflight inputs arrive as keywords with None defaults, as max_per_pod (RFC 0003) and pod_claims did.

• Arguments:

  • demand: extended-resource name -> units the flow requests, or None to skip capacity comparison.

  • gpu_runtime_class: the --gpu-runtime-class value, if given.

  • max_per_pod: extended-resource name -> the largest single-pod claim (manifests.gpu_max_per_pod), or None to skip per-node checks.

  • pod_claims: extended-resource name -> one claim per pod replica (manifests.gpu_pod_claims), or None to skip the per-host packing check (pack_pod_claims).

prepare(demand: dict[str, int] | None = None, kubectl: str = 'kubectl', flow_id: str | None = None) → None

Cluster setup this strategy needs before a run’s manifests are applied. Default: nothing. A strategy that mutates cluster state here is responsible for restoring it in cleanup — and, in a multi-tenant cluster, for marking that state with flow_id so concurrent flows keep out of each other’s way.

resolve_specs(specs: List[NodeSpec], kubectl: str = 'kubectl', default_resource: str | None = None, flow_id: str | None = None) → List[NodeSpec]

The strategy’s chance to decide names and geometry before anything is rendered or preflighted: called once per deploy, with the compiled specs, before gpu_demand/gpu_max_per_pod/render_manifests consume them. Default: identity. The mix strategy returns specs whose gpu_resource_name carries each sharer’s solver-chosen MIG profile — after which the entire downstream pipeline runs unchanged.

Like preflight_problems, new lifecycle inputs arrive as keywords with None defaults (flow_id did): third-party strategies should accept **kwargs. flow_id identifies the deploying flow so a multi-tenant strategy can tell its own cluster state from another flow’s.

• Raises:

  • ValueError: the flow’s demands cannot be laid out (mix’s LayoutError is one) — the deploy should stop before rendering.

videoflow.deploy.gpu.IMPOSSIBLE_GPU_REQUEST = 'impossible GPU request'

Marker prefix for a preflight problem that is fatal regardless of --strict-preflight: the request is impossible by construction (a multi-unit claim against a MIG or time-sliced resource), so deploying anyway can only end in an admission error or a silently broken visibility contract. A constant so the CLI’s check never depends on message prose.

videoflow.deploy.gpu.MIG_APPLY_TIMEOUT_SECONDS = 600

How long prepare() waits for the MIG manager to report success per apply.

videoflow.deploy.gpu.MIG_CONFIGMAP_NAME = 'videoflow-mig-parted-config'

the operator’s current mig-parted config with the generated videoflow-<node> entries merged in. The MIG manager only reads the ConfigMap named in ClusterPolicy migManager.config.name, so prepare() points that field here for the run.

Type:

ConfigMap videoflow publishes in the operator namespace

videoflow.deploy.gpu.MIG_CONFIG_LABEL = 'nvidia.com/mig.config'

The node label the GPU Operator’s MIG manager watches, and its status twin.

videoflow.deploy.gpu.MIG_CONFIG_NAME_ABSENT = '__absent__'

Sentinel recorded in MIG_CONFIG_NAME_RESTORE_ANNOTATION when the field was absent — never ‘’, which a failed read could be mistaken for.

videoflow.deploy.gpu.MIG_CONFIG_NAME_RESTORE_ANNOTATION = 'videoflow.io/mig-config-name-restore'

ClusterPolicy annotation recording the pre-videoflow migManager.config.name, so a teardown in a fresh shell can restore it (the ClusterPolicy twin of MIG_RESTORE_ANNOTATION on nodes).

videoflow.deploy.gpu.MIG_DISABLED_CONFIG = 'videoflow-all-disabled'

Config entry always injected into the merged file so cleanup() can un-MIG a node whose pre-videoflow mig.config label was absent: removing the label triggers no reconfiguration, so such nodes are pointed here first.

videoflow.deploy.gpu.MIG_ENTRY_ANNOTATION = 'videoflow.io/mig-entry'

Node annotation recording the node’s current mig-parted entry name. Entry names carry a per-run nonce (see _mig_config_name), so unlike the old fixed videoflow-<node> scheme they cannot be reconstructed — cleanup() reads this annotation (and the mig.config label) to know which entries in the shared ConfigMap are this flow’s to strip. Stamped before the ConfigMap publish, so even a prepare that crashes between publishing and labeling leaves the record.

videoflow.deploy.gpu.MIG_LABEL_ABSENT = '__absent__'

The restore record’s value for “the label did not exist” (annotation values cannot be null, and ‘’ is a legitimate label value of its own).

videoflow.deploy.gpu.MIG_MANAGER_ROLLOUT_TIMEOUT_SECONDS = 300

How long prepare() waits for the mig-manager DaemonSet to remount the videoflow ConfigMap after the ClusterPolicy patch.

videoflow.deploy.gpu.MIG_OPERATOR_DEFAULT_CONFIGMAP = 'default-mig-parted-config'

The GPU Operator’s stock mig-parted ConfigMap — the merge base when ClusterPolicy names none, and what cleanup() restores when the migManager.config.name field was absent before videoflow touched it.

videoflow.deploy.gpu.MIG_RESTORE_ANNOTATION = 'videoflow.io/mig-config-restore'

Node annotation where mix’s prepare() records the node’s previous nvidia.com/mig.config label value, so a later cleanup() — possibly a teardown in a fresh shell — can restore it without sharing any state with the deploy that ran prepare(). Key-presence semantics (ALLOC-033): an absent label is recorded as MIG_LABEL_ABSENT and removed again on restore; an explicitly empty label is recorded as ‘’ and restored as ‘’. (Records written before the sentinel existed hold ‘’ for both; they restore as an empty label, which the MIG manager treats exactly like an absent one.)

videoflow.deploy.gpu.MIG_TOMBSTONE_ANNOTATION = 'videoflow.io/mig-config-tombstone'

ConfigMap annotation the last flow out stamps on MIG_CONFIGMAP_NAME instead of deleting it (value: UTC time of retirement). kubectl cannot express delete-with-precondition, and a map nothing references is harmless where a wrong delete pulls the file out from under a manager still mounting it — so retirement is strip + tombstone, and removal is the operator’s: kubectl delete configmap videoflow-mig-parted-config -n <gpu-operator namespace>

class videoflow.deploy.gpu.MixGpu

Bases: ExclusiveGpu

Declared-demand MIG partitioning (RFC 0004): sharers (gpu_memory_gib) get exclusive MIG slices chosen by the layout solver, spanners and undeclared-demand nodes get whole physical cards. Pod claims are exclusive-style integer limits — only the names differ, and those are decided in resolve_specs, which is why this subclasses ExclusiveGpu.

apply_plan(plan: AllocationPlan, kubectl: str = 'kubectl', flow_id: str | None = None) → AppliedGeometry | None

prepare for an explicit plan; returns what was written (None when the plan MIGs nothing). Between claiming the nodes and touching any geometry the occupancy is read again: a pod that landed on a planned card since the plan was made would be destroyed by repartitioning, so a changed or unreadable occupancy releases the claims and aborts (ALLOC-011).

static apply_plan_to_specs(plan: AllocationPlan, specs: List[NodeSpec]) → List[NodeSpec]

The specs with each sharer’s solver-chosen MIG profile as its gpu_resource_name.

cleanup(kubectl: str = 'kubectl', flow_id: str | None = None) → None

Restores every node carrying MIG_RESTORE_ANNOTATION to its recorded pre-videoflow nvidia.com/mig.config value — a node whose label was absent is first pointed at MIG_DISABLED_CONFIG so the manager actually un-partitions the cards (removing the label triggers nothing), then unlabeled — and waits for mig.config.state=success before declaring a node done. Only when every node reverted AND no other flow’s entries remain in the published map does it restore ClusterPolicy migManager.config.name from MIG_CONFIG_NAME_RESTORE_ANNOTATION and delete the published ConfigMap (last one out); otherwise it strips only this flow’s entries. Entry names carry a per-run nonce, so which entries are this flow’s is read off its nodes’ mig.config labels and MIG_ENTRY_ANNOTATION stamps, never reconstructed. A node that failed keeps its annotation, and the policy patch and ConfigMap stay wired, so a retried teardown can resume — and a node already sitting at its restore target with state=failed is first bounced through a nonce’d alias of the disabled entry, because rewriting the identical label value is a no-op for the manager and would deadlock every retry. State lives entirely in the cluster, so this works from a teardown that shares nothing with the deploy that ran prepare — and is a no-op when prepare never ran.

With a flow_id, only nodes stamped GPU_OWNER_LABEL=<this flow> are touched — other flows’ geometry stays up. Without one, every node videoflow owns is swept (the single-operator escape hatch, and the pre-ownership behaviour). A node stamped but never restore-annotated — prepare crashed between claiming and labeling — has no geometry to revert, so its claim is simply released.

name: str = 'mix'

Mode name, as used by --gpu-mode.

observe_geometry(kubectl: str, applied: AppliedGeometry) → Known[dict[str, str]] | Unknown

One correlated readiness read per node of an apply_plan result: ready when the node still carries this operation’s entry name and owner epoch, reports success and advertises the expected slices; failed on the manager’s verdict; pending otherwise; lost when the entry or the claim is no longer this operation’s. Unknown when a node could not be read.

plan_layout(specs: List[NodeSpec], kubectl: str = 'kubectl', flow_id: str | None = None) → AllocationPlan

Reads the pool and solves the layout for specs — an explicit, immutable AllocationPlan (ALLOC-011). Raises UnobservableState on an unreadable pool and LayoutError when the demands cannot be laid out; never caches.

preflight_for(plan: AllocationPlan, kubectl: str = 'kubectl', demand: dict[str, int] | None = None, gpu_runtime_class: str | None = None) → List[str]

preflight_problems for an explicit plan.

preflight_problems(kubectl: str = 'kubectl', demand: dict[str, int] | None = None, gpu_runtime_class: str | None = None, max_per_pod: dict[str, int] | None = None, pod_claims: dict[str, list[int]] | None = None) → List[str]

Strategy-specific preflight problems, each a string naming its fix. The flavor-independent checks (cluster reachable, a labeled GPU node) are run by cluster.gpu_preflight before this is called. A problem prefixed with IMPOSSIBLE_GPU_REQUEST is fatal regardless of --strict-preflight.

Third-party strategies should tolerate future keyword inputs (accept **kwargs): new preflight inputs arrive as keywords with None defaults, as max_per_pod (RFC 0003) and pod_claims did.

• Arguments:

  • demand: extended-resource name -> units the flow requests, or None to skip capacity comparison.

  • gpu_runtime_class: the --gpu-runtime-class value, if given.

  • max_per_pod: extended-resource name -> the largest single-pod claim (manifests.gpu_max_per_pod), or None to skip per-node checks.

  • pod_claims: extended-resource name -> one claim per pod replica (manifests.gpu_pod_claims), or None to skip the per-host packing check (pack_pod_claims).

prepare(demand: dict[str, int] | None = None, kubectl: str = 'kubectl', flow_id: str | None = None) → None

Applies the layout’s MIG geometry through the GPU Operator: claim the target nodes for this flow, merge the generated mig-parted config into the operator’s file (preserving other flows’ published entries), publish the result as MIG_CONFIGMAP_NAME, point ClusterPolicy migManager.config.name at it (recording the original name in MIG_CONFIG_NAME_RESTORE_ANNOTATION), wait for the mig-manager DaemonSet to remount, then label each MIG’d node with its per-run entry name — videoflow-<node>-<nonce>, see _mig_config_name — (recording its previous label in MIG_RESTORE_ANNOTATION and the entry name in MIG_ENTRY_ANNOTATION) and wait for nvidia.com/mig.config.state=success. The per-run nonce guarantees the label value changes: the manager reacts only to changes, so a leftover state=failed from a previous attempt cannot deadlock the retry. The manager only reads the ConfigMap that ClusterPolicy names — a side ConfigMap it never mounts cannot carry the config. Without a MIG manager or a ClusterPolicy it fails actionably, with the config to apply by hand.

resolve_specs(specs: List[NodeSpec], kubectl: str = 'kubectl', default_resource: str | None = None, flow_id: str | None = None) → List[NodeSpec]

The strategy’s chance to decide names and geometry before anything is rendered or preflighted: called once per deploy, with the compiled specs, before gpu_demand/gpu_max_per_pod/render_manifests consume them. Default: identity. The mix strategy returns specs whose gpu_resource_name carries each sharer’s solver-chosen MIG profile — after which the entire downstream pipeline runs unchanged.

Like preflight_problems, new lifecycle inputs arrive as keywords with None defaults (flow_id did): third-party strategies should accept **kwargs. flow_id identifies the deploying flow so a multi-tenant strategy can tell its own cluster state from another flow’s.

• Raises:

  • ValueError: the flow’s demands cannot be laid out (mix’s LayoutError is one) — the deploy should stop before rendering.

videoflow.deploy.gpu.PACKING_EXACT_MAX_PODS = 12

within them a negative answer is a proof; beyond them it is first-fit-decreasing’s opinion, and the problem string says so.

Type:

Bounds of the exhaustive placement search in pack_pod_claims

class videoflow.deploy.gpu.PodPacking(feasible: bool, proven: bool, placement: Dict[int, str], unplaced: tuple[int, ...])

Bases: object

Whether a list of per-pod whole-unit claims fits a pool’s per-node free units with every pod’s whole claim on one node.

• Attributes:

  • feasible: a placement was found.

  • proven: the answer is exact — a placement is always a proof, and an infeasible answer is one when an exhaustive search found nothing. False only for an infeasible answer on an instance too large to search, where the scheduler might still succeed.

  • placement: claim index -> node, when feasible.

  • unplaced: the claim indices first-fit-decreasing could not place — the evidence the report quotes when nothing fits.

feasible: bool

placement: Dict[int, str]

proven: bool

unplaced: tuple[int, ...]

videoflow.deploy.gpu.UNOBSERVABLE_GPU_STATE = 'unobservable GPU state'

Prefix of a preflight problem saying that an occupancy or inventory read the capacity math depends on could not be made. The numbers that follow assume an idle pool — an upper bound, not a fact. Fatal for a mode that mutates the cluster on the strength of it (mix repartitions cards); for exclusive claims it is a warning --strict-preflight promotes.

videoflow.deploy.gpu.expected_allocatable(layout: GpuLayout) → dict[str, dict[str, int]]

Per MIG’d node, the extended resources the device plugin must advertise once the layout is applied (mixed strategy naming, nvidia.com/mig-<profile> per slice, nvidia.com/gpu per untouched card). The readiness evidence _wait_for_geometry demands beyond a success label: a label left by an earlier geometry says nothing about this one, the advertised resources do.

videoflow.deploy.gpu.flow_owner_value(flow_id: str) → str

The GPU_OWNER_LABEL value for a flow: manifests.k8s_name(flow_id), so an arbitrary --flow-id charset becomes a legal <= 63-char label value — the same value the flow’s pods already carry in their flow-id label.

videoflow.deploy.gpu.geometry_advertised(allocatable: Mapping[str, int], expected: Mapping[str, int]) → bool

Whether a node advertises at least the resources a layout needs on it.

videoflow.deploy.gpu.get_gpu_mode(name: str) → GpuStrategy

The strategy registered under name.

• Raises:

  • ValueError: no strategy is registered under that name; the message names the known modes and register_gpu_mode.

videoflow.deploy.gpu.gpu_resource_provenance(spec: NodeSpec, default: str | None = None) → tuple[str, str]

resolve_gpu_resource with its answer’s source: (name, source) where the source is 'strategy' (the spec’s resolved name), 'cli-default' (--gpu-resource-name) or 'default' (nvidia.com/gpu). The one merge point for the resource name, expressed as declarations to the shared resolver in videoflow.core.provenance so precedence and provenance are stated once: a strategy’s resolution is a hard decision, the two defaults are soft.

videoflow.deploy.gpu.pack_pod_claims(claims: Sequence[int], free: Mapping[str, int]) → PodPacking

Decide whether every pod’s whole-unit claim can be placed on one node of a pool with free units per node — the per-host check the aggregate arithmetic cannot make: per-node free [3, 3] against pods [2, 2, 2] passes both the total (6 <= 6) and the largest-pod bound (2 <= 3), yet only two pods fit.

First-fit-decreasing answers first, and a placement it finds is a proof. When it finds none and the instance is small (at most PACKING_EXACT_MAX_PODS pods over PACKING_EXACT_MAX_NODES nodes) an exhaustive search settles the question either way; a larger instance is reported infeasible but unproven.

• Arguments:

  • claims: one entry per pod replica (manifests.gpu_pod_claims).

  • free: node -> free units of the resource being claimed.

• Returns:

  • a PodPacking.

videoflow.deploy.gpu.register_gpu_mode(strategy: GpuStrategy) → None

Registers a GPU allocation strategy under its name. Registering makes the mode selectable via --gpu-mode — the CLI builds its choices from here — so a new strategy needs no CLI edit.

• Arguments:

  • strategy: the strategy instance.

• Raises:

  • ValueError: the strategy has no name.

videoflow.deploy.gpu.registered_gpu_modes() → list[str]

Registered GPU mode names, sorted. The CLI’s --gpu-mode choices.

videoflow.deploy.gpu.resolve_gpu_resource(spec: NodeSpec, default: str | None = None) → str

The extended-resource name a GPU spec requests: the spec’s resolved name (internal — only a GPU strategy sets it, e.g. mix assigning a MIG profile), else the deploy default (--gpu-resource-name), else nvidia.com/gpu.

videoflow.deploy.images module

Resolve the container image a node’s worker runs in on Kubernetes.

There is no module-path “family” inference: a user defines their processors in their own package and builds their own image (their code + deps on top of videoflow-base), so the image must be stated explicitly. Resolution order, first match wins:

1. a deploy-time override for the node (--image-override <name>=<ref>)

2. the node’s own image= kwarg (declared in graph code)

3. the deploy-time default (--image <ref>)

If none apply, resolution raises with an actionable message instead of guessing.

This module also owns the pull policy — how the cluster obtains that image once it is named. It lives here rather than in manifests because the CLI must read the default while building its parser, and manifests imports the optional yaml extra at module scope; this module imports nothing outside the stdlib.

videoflow.deploy.images.parse_override(spec: str) → tuple

Parses a name=ref CLI override into a (name, ref) tuple.

videoflow.deploy.images.resolve_image(node_name: str, node_image: str | None, default_image: str | None = None, overrides: dict | None = None) → str

• Arguments:

  • node_name: the node’s stable name (matched against overrides).

  • node_image: the image declared on the node (Node.image), or None.

  • default_image: the deploy-time flow default (--image), or None.

  • overrides: mapping of node name to image ref (--image-override).

• Returns:

  • the resolved image ref (str).

• Raises:

  • ValueError if no image can be determined for the node.

videoflow.deploy.images.validate_image_pull_policy(policy: str) → str

• Returns:

  • policy unchanged when it is a valid Kubernetes imagePullPolicy.

• Raises:

  • ValueError naming the accepted values, so a typo fails on the operator’s machine rather than as an unschedulable pod in the cluster.

videoflow.deploy.infra module

Auto-provisioning of the dev broker infrastructure videoflow deploy needs when the user doesn’t bring their own: an in-cluster NATS JetStream server and, when the blob store is wanted, a Redis. Built as plain dicts (same convention as manifests) so they ship inside the package, parametrize by namespace, and carry an ownership label for selective teardown.

Two shapes, chosen by a deploy.broker_profiles profile:

• the dev profile (profile = None): single replica, emptyDir — a faithful port of k8s/nats.yaml for NATS, and a Redis whose append-only file lives on an emptyDir with noeviction (RedisProfile.dev()), so the pair has one standing: both survive a container restart, neither a pod loss, and a BATCH flow’s reliable_work channels are admitted on it. For production, bring your own broker (the official NATS Helm chart, a managed Redis) and pass --nats/--blob-redis-url.

• the durable profile: a NATS StatefulSet whose pods route to each other through a headless Service (cluster { routes }) and keep the JetStream file store on a PersistentVolumeClaim each, plus an append-only Redis on a claim of its own. Enough for a stream to survive a pod, a node, or a rollout; still not a tuned production broker.

Ownership rule: a pre-existing nats/redis Service in the namespace is reused as-is and never owned; only components this module applied are returned as “created” and later torn down. Each Service records the profile it was rendered from (videoflow.io/profile, plus videoflow.io/replicas for NATS) so a later deploy that finds it can judge what it is reusing — admission runs against the recorded profile, not the one the deploy would have rendered — and refuse an explicit --broker-profile that contradicts it (reused_infra / adopt_profiles). Persistent claims a durable profile created are deliberately not torn down with the workloads — the data is the point — so a redeploy finds it; reclaim them by hand with kubectl delete pvc -n <namespace> -l videoflow.io/infra.

videoflow.deploy.infra.LABEL_PROFILE = 'videoflow.io/profile'

the profile it was rendered from (BrokerProfile.name / RedisProfile.name) and, for NATS, its replica count — what a later deploy reads to judge the infrastructure it reuses.

Type:

Recorded on each client Service

videoflow.deploy.infra.NATS_SERVICE = 'nats'

The client Service every profile exposes (what infra_urls names) and the headless one a StatefulSet’s pods address each other through.

videoflow.deploy.infra.REDIS_CLAIM = 'redis-data'

The claim a durable Redis keeps its append-only file on.

class videoflow.deploy.infra.ReusedInfra(nats: dict | None, redis: dict | None)

Bases: object

What ensure_infra will find and reuse: the labels of the nats and redis Services already in the namespace (None = absent, so this deploy creates it). A component that is present but carries no profile record was created by hand or by an older videoflow.

components() → List[str]

nats: dict | None

redis: dict | None

videoflow.deploy.infra.adopt_profiles(reuse: ReusedInfra, requested: str | None, broker: BrokerProfile, redis: RedisProfile, namespace: str) → tuple[BrokerProfile | None, RedisProfile | None]

The profiles admission judges a deploy by, given what the namespace already runs: a component this deploy creates is judged by the profile it renders; a reused one by the profile its creator recorded on the Service, or by nothing (None — unread, for the planner to rule on) when it carries no record. An operator who named a profile that contradicts a record is refused: ensure_infra would reuse the other shape silently otherwise.

• Arguments:

  • reuse: what reused_infra found.

  • requested: the --broker-profile name the operator passed, or None when they left the choice to the deploy.

  • broker / redis: the profiles the deploy renders for what is missing.

• Returns: (BrokerProfile | None, RedisProfile | None).

• Raises:

  • ConfigError: an explicit profile contradicts a reused component’s record.

videoflow.deploy.infra.ensure_infra(kubectl: str, namespace: str, need_redis: bool, profile: BrokerProfile | None = None, redis_profile: RedisProfile | None = None) → tuple

Applies the NATS (and, when need_redis, Redis) of the given profiles unless a Service of the same name already exists in the namespace (bring-your-own is reused, not owned).

• Arguments:

  • profile: how to shape NATS; None is the dev profile.

  • redis_profile: how to shape Redis; None is the dev profile.

• Returns:

  • (urls, created) where urls maps nats/redis to in-cluster URLs (redis is None when not needed) and created lists only the components THIS call applied (what teardown may later delete).

videoflow.deploy.infra.ensure_namespace(kubectl: str, namespace: str) → None

videoflow.deploy.infra.infra_urls(namespace: str) → dict

The in-cluster URLs workers use once the dev infra is up.

videoflow.deploy.infra.nats_conf(namespace: str, profile: BrokerProfile | None = None) → str

The nats.conf the server pods run with.

The dev profile’s text is the one k8s/nats.yaml carries. A stateful profile adds server_name (JetStream clustering requires a unique name per server; $POD_NAME is substituted by the server from the pod’s downward-API env) and, for more than one replica, a cluster block whose routes name every peer through the headless Service — the seed list JetStream’s Raft groups form over.

videoflow.deploy.infra.nats_manifests(namespace: str, profile: BrokerProfile | None = None) → list

The NATS JetStream server for namespace + infra labels: the single-replica Deployment of k8s/nats.yaml for the dev profile (None), a StatefulSet with a headless Service, route peers and one claim per pod for a durable one. The client Service is named nats in every profile, which is what infra_urls and the reuse rule key on.

videoflow.deploy.infra.redis_manifests(namespace: str, profile: RedisProfile | None = None) → list

Single-replica Redis for the large-payload blob store.

Both shipped profiles run noeviction with an append-only file under /data: a blob is never dropped while a reader still holds it (every key carries a TTL, PROTOCOL.md BLOB-7, and the reconciler reclaims orphans, so that is what bounds memory), and a container restart replays the file. maxmemory stays capped at 4 GB so a stuck pipeline hits a refused write — a typed failure the publisher sees — before the node OOMs (the redis:7 default is unlimited memory); the container limit sits above it to leave headroom for allocator fragmentation.

Dev profile (None): the file lives on an emptyDir, the pod’s lifetime — the same standing as the dev NATS file store.

Durable profile: the file lives on a PersistentVolumeClaim, and the Deployment uses a Recreate strategy, since a ReadWriteOnce claim cannot be held by the old and the new pod at once during a rollout.

RedisProfile(persistence = 'none') still renders the old transport-only cache (no volume, nothing written) for an operator who asks for it.

videoflow.deploy.infra.reused_infra(kubectl: str, namespace: str, need_redis: bool) → ReusedInfra

The Services a deploy into namespace would reuse rather than create.

videoflow.deploy.infra.service_exists(kubectl: str, namespace: str, name: str) → bool

videoflow.deploy.infra.service_labels(kubectl: str, namespace: str, name: str) → dict | None

The labels of a Service in the namespace, or None when there is no such Service.

videoflow.deploy.infra.teardown_infra(kubectl: str, namespace: str, components: List[str], profile: BrokerProfile | None = None) → None

Deletes the given auto-provisioned components by ownership label. Best-effort (never raises). A stateful profile adds the StatefulSet to the kinds deleted; the PersistentVolumeClaims either profile created are left in place (see the module docstring).

videoflow.deploy.infra.wait_infra_ready(kubectl: str, namespace: str, created: List[str], timeout_secs: int = 120, profile: BrokerProfile | None = None) → None

Blocks until each freshly created infra workload rolls out; raises on timeout. Pass the same profile as ensure_infra so a stateful NATS is awaited as the StatefulSet it is.

videoflow.deploy.localinfra module

Auto-provisioning of the dev broker videoflow run-local needs when the user doesn’t bring their own: a NATS JetStream container and, for the large-payload blob store, a Redis container, both published on localhost. The local analogue of videoflow.infra, which does the same job in-cluster for videoflow deploy.

Ownership rule, identical to infra: if something is already listening on the port — docker compose up -d, a bare nats-server -js, a previous --keep-infra run — it is reused as-is and never torn down. Only containers this module started are returned as “created”, and only those are stopped later.

Dev-grade only: no persistence, no auth, host-published ports. Point --nats / --blob-redis-url at a real broker for anything else.

videoflow.deploy.localinfra.docker_available() → bool

True when a docker daemon is reachable.

videoflow.deploy.localinfra.ensure_local_infra(need_redis: bool = True, nats_url: str | None = None, redis_url: str | None = None) → Tuple[dict, List[str]]

Starts a dev NATS (and, when need_redis, Redis) container unless something is already listening on the port.

• Returns:

  • (urls, created) where urls maps nats/redis to localhost URLs (redis is None when not needed) and created lists only the components THIS call started — the only ones teardown may stop.

• Raises:

  • RuntimeError when a container must be started but docker is unavailable, or when docker run fails.

videoflow.deploy.localinfra.local_infra_urls() → dict

The localhost URLs workers use once the dev containers are up.

videoflow.deploy.localinfra.nats_running(url: str = 'nats://localhost:4222') → bool

videoflow.deploy.localinfra.port_open(host: str, port: int, timeout: float = 1.0) → bool

True when something accepts a TCP connection — the “is it already up?” probe.

videoflow.deploy.localinfra.redis_running(url: str = 'redis://localhost:6379/0') → bool

videoflow.deploy.localinfra.teardown_local_infra(components: List[str] | None) → None

Removes the given auto-started containers. Best-effort; never raises.

videoflow.deploy.localinfra.wait_local_infra_ready(created: List[str] | None, urls: dict, timeout_secs: int = 45) → None

Polls each freshly started component’s port until it accepts connections.

• Raises:

  • RuntimeError (after dumping the container’s logs) on timeout.

videoflow.deploy.manifests module

Renders Kubernetes manifests for a compiled flow (a list of NodeSpec). One workload per node, chosen by flow type:

• BATCH flow -> every node is a Job (each worker exits 0 when its upstream

  end-of-stream drains, so the whole flow runs to completion).

• REALTIME flow -> a finite producer is a Job; every other node is a Deployment

  (or a StatefulSet if partitioned) that stays up until the control-channel stop.

plus a per-node ConfigMap holding the node’s env, a shared ConfigMap for the NATS URL, and a default-deny-except-broker NetworkPolicy.

Manifests are built as plain dicts and serialized with yaml.dump rather than text-templated, so the output is always structurally valid YAML.

The dict/dataclass split here is deliberate: anything that mirrors the Kubernetes API schema (workloads, ConfigMaps, Services, pod specs, …) stays a dict, because we do not own that schema and a typed mirror of it would be a second source of truth that silently drifts from the real API — and would still have to become a dict at the YAML boundary. Records this module owns outright (Mount, ProvisionSplit) are dataclasses, since nothing external constrains their shape.

class videoflow.deploy.manifests.Mount(name: str, host_path: str, container_path: str, read_only: bool, claim: str | None = None)

Bases: object

One volume threaded into every node workload — a host path (and then also into the prepare / compile containers by deploy.build.run_in_image) or a PersistentVolumeClaim. Produced by parse_mounts / parse_pvc_mounts; rendered into a volume + volumeMount by _pod_spec.

• Arguments:

  • name: the volume name, unique within the pod (vf-mount-<i> for a host path, vf-pvc-<i> for a claim).

  • host_path: absolute path on the cluster node. Empty for a claim mount, which has no host side — run_in_image skips those.

  • container_path: absolute path it appears at inside the container.

  • read_only: whether the container gets it read-only.

  • claim: the PersistentVolumeClaim name for a claim mount; None for a host path. The claim must exist in the flow’s namespace — it is what lets a multi-node cluster share a work directory that no node’s own filesystem holds (an RWX claim over NFS), where a hostPath would silently mount an empty directory on every node but one.

claim: str | None = None

container_path: str

host_path: str

name: str

read_only: bool

class videoflow.deploy.manifests.ProvisionSplit(provision: List[dict], worker: List[dict])

Bases: NamedTuple

The two apply phases of a flow’s manifests, in the order they must be applied.

A NamedTuple rather than a plain @dataclass: the phases are genuinely ordered (phase 1 must be applied and awaited before phase 2), and existing callers unpack the result positionally, which keeps working unchanged.

• Arguments:

  • provision: manifests the provision Job needs before any worker publishes.

  • worker: every node’s own ConfigMap/workload/service/PDB.

provision: List[dict]

Alias for field number 0

worker: List[dict]

Alias for field number 1

videoflow.deploy.manifests.delete_resources(kubectl: str, namespace: str, flow_id: str, run_id: str | None = None) → None

Single source of truth for tearing down a flow’s Kubernetes resources: deletes every kind in DELETABLE_KINDS matching the flow-id label, scoped to one run when run_id is given (else every run of the flow). Best-effort (never raises).

videoflow.deploy.manifests.dump_manifests(manifests: List[dict]) → str

Serializes a list of manifest dicts to a single multi-document YAML string.

videoflow.deploy.manifests.flow_name(flow_id: str, *parts: object) → str

The name of a flow-wide resource shared by every run of the flow (today: the NetworkPolicy).

videoflow.deploy.manifests.flow_spec_configmap(specs: List[NodeSpec], flow_id: str, run_id: str) → dict

ConfigMap holding the compiled specs as JSON, mounted by the provisioning init Job.

videoflow.deploy.manifests.gpu_demand(specs: List[NodeSpec], default_resource: str | None = None) → dict[str, int]

Whole-flow GPU demand per extended-resource name: every replica of a GPU node claims its own gpu_count devices exclusively, so this is the allocatable capacity the cluster must have for the flow to fully schedule. The single source of truth for deploy’s preflight and videoflow explain — the two must never disagree.

Deliberately a dict[str, int] and not a dataclass: the keys are cluster-defined extended-resource names (nvidia.com/gpu, amd.com/gpu, a MIG profile, a vendor’s own), discovered from the specs at runtime. There is no fixed field set to name, and every caller iterates it against the cluster’s equally open-ended allocatable map, so this is a mapping in substance, not a record wearing a dict’s clothes.

• Arguments:

  • specs: the compiled flow. Non-GPU nodes contribute nothing.

  • default_resource: extended-resource name for GPU nodes that don’t name their own (--gpu-resource-name). Defaults to nvidia.com/gpu.

• Returns:

  • resource name -> total devices the flow needs at full scale. Empty when no node requests a GPU.

videoflow.deploy.manifests.gpu_max_per_pod(specs: List[NodeSpec], default_resource: str | None = None) → dict[str, int]

The largest single-pod claim per extended-resource name — gpu_demand’s sibling and the other half of preflight’s input. Total demand answers “will the whole flow schedule”; this answers “can any single node host the biggest pod”: all of one replica’s gpu_count devices must sit on one Kubernetes host, so a flow can satisfy the total and still never schedule (RFC 0003).

Same dict[str, int]-not-dataclass shape as gpu_demand, for the same reason: the keys are open-ended cluster-defined extended-resource names.

• Arguments:

  • specs: the compiled flow. Non-GPU nodes contribute nothing.

  • default_resource: as on gpu_demand.

• Returns:

  • resource name -> the largest gpu_count any one replica requests. Empty when no node requests a GPU.

videoflow.deploy.manifests.gpu_pod_claims(specs: List[NodeSpec], default_resource: str | None = None) → dict[str, list[int]]

Every pod’s claim per extended-resource name — the third of gpu_demand’s siblings and the input to preflight’s per-host packing check. gpu_demand sums the claims and gpu_max_per_pod takes the largest; both are bounds a fragmented pool can satisfy while still placing only some of the pods (free [3, 3] per node holds two of three gpu_count = 2 replicas), so the packing check needs the claims themselves: one entry per replica — nb_tasks copies of the node’s gpu_count — in spec order.

Same dict-not-dataclass shape as its siblings, for the same reason.

• Arguments:

  • specs: the compiled flow. Non-GPU nodes contribute nothing.

  • default_resource: as on gpu_demand.

• Returns:

  • resource name -> one gpu_count per replica. Empty when no node requests a GPU.

videoflow.deploy.manifests.headless_service(spec: NodeSpec, flow_id: str, run_id: str) → dict

Headless Service backing a partitioned node’s StatefulSet (required for stable pod network identity/ordinals).

videoflow.deploy.manifests.host_resources_for(spec: NodeSpec, resources: Mapping[str, Mapping[str, str]] | None) → dict[str, str]

The host requests/limits one node’s container renders: the descriptor’s spec.resources.cpu/memory defaults, under the operator’s * entry, under the entry for this node (--resources). Empty when nobody asked.

videoflow.deploy.manifests.k8s_name(*parts: object) → str

Joins parts into a DNS-1123-safe Kubernetes resource name.

videoflow.deploy.manifests.nats_configmap(flow_id: str, run_id: str, nats_url: str, blob_redis_url: str | None = None, blob_ttl_seconds: int | None = None) → dict

videoflow.deploy.manifests.network_policy(flow_id: str) → dict

Allows worker pods to talk to each other and out to the broker/DNS, and denies everything else ingress by default. Kept intentionally permissive on egress since the NATS/Redis services may live in another namespace.

videoflow.deploy.manifests.node_configmap(spec: NodeSpec, flow_id: str, flow_type: str, run_id: str, envelope_version: int, profile_requests: dict | None = None, blob_reader_ids: List[str] | None = None, parent_replicas: List[int] | None = None, replica_slots: int | None = None) → dict

videoflow.deploy.manifests.parse_mounts(values: List[str] | None) → List[Mount]

Parses repeatable --mount specs into Mount records consumed by _pod_spec.

• Arguments:

  • values: list of /host/path:/container/path[:ro] strings. The single-path shorthand /path[:ro] mounts the same absolute path on both sides (what a flow compiled against local files needs, since the paths baked into node params must resolve identically in the pods).

• Returns:

  • a list of Mount, in argument order. name is only unique within one call, so a concatenation of two parse_mounts results repeats names — fine for run_in_image (docker -v ignores them), which is the only caller that concatenates.

• Raises:

  • ValueError on a relative path or a malformed suffix.

videoflow.deploy.manifests.parse_pvc_mounts(values: List[str] | None) → List[Mount]

Parses repeatable --mount-pvc specs into claim Mount records consumed by _pod_spec (a persistentVolumeClaim volume + volumeMount on every node workload; the provision Job never touches node data and gets none).

• Arguments:

  • values: list of claim:/container/path[:ro] strings. The claim must already exist in the namespace the flow deploys to.

• Returns:

  • a list of Mount with claim set and host_path empty, in argument order, named vf-pvc-<i> — a namespace disjoint from parse_mounts’s vf-mount-<i>, so the two lists concatenate into one pod without a name collision.

• Raises:

  • ValueError on a missing or invalid claim name, a relative path, or a malformed suffix.

videoflow.deploy.manifests.pod_disruption_budget(spec: NodeSpec, flow_id: str, run_id: str) → dict

Keeps at least one replica of a multi-replica node available during voluntary disruptions (node drains, upgrades).

videoflow.deploy.manifests.pod_mounts(mounts: List[Mount]) → List[Mount]

The mounts a pod actually renders, from the full list the deploy collected.

The rule, deterministic and applied nowhere else: every claim mount is kept, and a hostPath mount whose container path lies at or under a claim mount’s container path is dropped from the pod. Such a path is served by the claim — a solution’s work_dir staged inside an RWX share, say — and a hostPath over it would shadow the claim on every node whose own filesystem does not hold that directory, mounting an empty root-owned directory where the artifacts should land. The dropped mount is still honoured by deploy.build.run_in_image, which runs the prepare/compile containers on the host, where the path does resolve. Input order is preserved.

• Arguments:

  • mounts: one parse_mounts result plus one parse_pvc_mounts result.

• Returns:

  • mounts itself (same order, every element) when it holds no claim; otherwise the filtered list.

videoflow.deploy.manifests.provision_init_job(flow_id: str, run_id: str, flow_type: str, image: str, nats_cm_name: str, image_pull_policy: str = 'IfNotPresent', priority_class: str | None = None, stream_replicas: int = 1, profile_requests: dict | None = None) → dict

A one-shot Job that runs videoflow.provision to create all streams/durables before workers start (required so BATCH interest-retention streams don’t drop early messages). Runs on image — any worker image has the framework + broker client installed. profile_requests (the VF_PROFILE_REQUESTS_JSON entry, RFC 0006 ENV-13) reaches it too, so the entrypoint admits the composition against the live broker before creating anything and verifies the streams it created carry the requested profiles; absent when none were made.

Its pull policy must match the workers’: this Job runs first, so when it is the one that cannot pull, the flow fails before any worker starts and the only symptom is the provision-wait timeout. The same goes for priority_class: a flow that yields to other tenants’ work must yield here too, or its provision pod is the one that cannot schedule.

videoflow.deploy.manifests.render_manifests(specs: List[NodeSpec], flow_id: str, flow_type: str, nats_url: str, run_id: str, namespace: str = 'default', default_image: str | None = None, image_overrides: dict | None = None, blob_redis_url: str | None = None, blob_ttl_seconds: int | None = None, autoscaling: bool = False, max_replicas: int = 10, nats_monitoring_endpoint: str | None = None, envelope_version: int | None = None, provision_image: str | None = None, mounts: List[Mount] | None = None, gpu_runtime_class: str | None = None, gpu_mode: str = 'exclusive', gpu_resource_name: str | None = None, gpu_autoscaling: bool = False, image_pull_policy: str = 'IfNotPresent', supervision: SupervisionPolicy | None = None, priority_class: str | None = None, profile_requests: dict | None = None, stream_replicas: int = 1, rollout_policy: str | None = None, gpu_nodes: List[str] | None = None, resources: Mapping[str, Mapping[str, str]] | None = None) → list

Returns a list of manifest dicts for the whole flow. The caller decides whether to yaml.dump them to files (CLI) or apply them via the API (engine).

• rollout_policy: drain or surge for every Deployment (--rollout-policy, see rollout_strategy); None keeps the API default.

• gpu_nodes: hostnames every GPU pod is pinned to (--gpu-nodes); None leaves placement to the pool label.

• resources: host requests/limits per node name (* for all), merged over the descriptors’ spec.resources.cpu/memory (--resources, see host_resources_for).

• run_id: per-run identifier stamped into each node’s env (scopes broker streams).

• blob_ttl_seconds: TTL override for offloaded payloads (--blob-ttl-seconds, PROTOCOL.md BLOB-7); None lets workers pick the flow-type default.

• supervision: restart policy for every node workload, rendered into the Job backoffLimit. The local engine honours the same object, which is what makes the two engines’ failure behaviour identical rather than merely similar.

• default_image: image ref used for any node that didn’t declare its own (--image).

• image_overrides: mapping of node name to image ref (--image-override).

• autoscaling: if True, emit a KEDA ScaledObject per processor node.

• max_replicas: upper bound for autoscaled processors.

• nats_monitoring_endpoint: NATS monitoring host:port for KEDA (defaults to nats.<namespace>.svc:8222).

• envelope_version: wire version for the whole run. The only supported version is 4 (the language-neutral protobuf wire); defaults to the ambient DEFAULT_ENVELOPE_VERSION, and an explicit incompatible pin is rejected.

• provision_image: image the one-shot provision Job runs on — must have videoflow + the broker client (i.e. a Python image), which a vendor default_image may not be. Defaults to default_image; set it explicitly (--provision-image) for flows whose default image is a non-Python vendor image.

• mounts: Mount records from parse_mounts (hostPath volumes) and parse_pvc_mounts (persistentVolumeClaim volumes), concatenated — each becomes a volume + volumeMount on every node workload (not the provision Job, which never touches node data). A hostPath whose path lies under a claim’s is dropped from the pods: see pod_mounts.

• priority_class: priorityClassName for every pod this flow creates, the provision Job included (--priority-class). None leaves the cluster’s default priority in place.

• gpu_runtime_class: runtimeClassName to put on GPU pods (--gpu-runtime-class). Needed where the NVIDIA container runtime is registered as an opt-in RuntimeClass rather than the node default — on k3s, nvidia. Without it such a pod schedules onto a GPU node and then runs with no device visible.

• gpu_mode: GPU strategy name ('exclusive', the default — each GPU replica claims whole physical devices via the extended resource; see deploy.gpu).

• gpu_resource_name: deploy-level extended-resource name for GPU claims (--gpu-resource-name), for clusters advertising whole devices under a non-default name. Defaults to nvidia.com/gpu. Must denote whole physical devices — a MIG profile here is rejected (see validate_gpu_specs).

• gpu_autoscaling: emit KEDA ScaledObjects for GPU nodes too. Off by default: each autoscaled replica claims its own GPUs, so scaling to max_replicas can demand more devices than the cluster has and strand pods Pending.

• image_pull_policy: imagePullPolicy for every rendered container, workers and the provision Job alike (--image-pull-policy). Defaults to IfNotPresent, which is what makes a locally built-and-loaded image actually run — see DEFAULT_IMAGE_PULL_POLICY.

• Raises:

  • ValueError if a node has no resolvable image (see videoflow.deploy.images), if image_pull_policy is not a valid k8s policy, or if the wire settings are incompatible with the flow’s components.

videoflow.deploy.manifests.rollout_strategy(rollout_policy: str) → dict

The Deployment strategy for a rollout policy (RUN-029, ALLOC-029): drain → Recreate (every old replica is gone before a new one starts — what a GPU node needs when its devices cannot be held by two generations at once); surge → RollingUpdate with one extra replica and none unavailable, which needs spare capacity for that extra replica.

videoflow.deploy.manifests.run_name(flow_id: str, run_id: str, *parts: object) → str

The name of one run’s resource: vf-<flow>-<run>[-<node>][-<suffix>] (RFC 0006 §10, run-scoped names). Two runs of one flow therefore never kubectl apply over each other’s ConfigMaps or workloads; only the flow-wide NetworkPolicy (flow_name) is shared between them.

videoflow.deploy.manifests.scaled_object(spec: NodeSpec, flow_id: str, run_id: str, nats_monitoring_endpoint: str, max_replicas: int) → dict | None

A KEDA ScaledObject that scales a processor Deployment on NATS JetStream consumer lag. nb_tasks becomes the floor (minReplicaCount); KEDA scales up toward max_replicas when the node’s input stream backs up. Returns None for node kinds that aren’t autoscaled (producers, consumers).

Requires KEDA installed in the cluster and the NATS monitoring endpoint (port 8222) reachable at nats_monitoring_endpoint.

videoflow.deploy.manifests.split_provision_manifests(manifests: List[dict], flow_id: str, run_id: str) → ProvisionSplit

Partitions rendered manifests into a ProvisionSplit so the caller can apply provisioning first and wait for it before starting workers.

The provision phase is everything the provision Job needs to create the broker streams/durables (including its EOS interest anchors) before any worker publishes: the broker + specs ConfigMaps, the NetworkPolicy, and the provision Job itself. The worker phase is every node’s ConfigMap/workload/service/PDB.

• Arguments:

  • manifests: the full rendered manifest list from render_manifests.

  • flow_id, run_id: what the provision-phase resource names are derived from.

• Returns:

  • a ProvisionSplit(provision, worker), each preserving input order.

videoflow.deploy.manifests.validate_gpu_specs(specs: List[NodeSpec], default_resource: str | None = None) → None

Hard build-time GPU validation — the checks that are provable from the specs alone, with no cluster in sight. Distinct from preflight (which compares against live cluster state and is skippable): what fails here can never work on any cluster, so it raises instead of warning.

Today that is one rule: a multi-device grant (gpu_count > 1) against a MIG resource. MIG slices are hardware-isolated partitions — one CUDA process addresses one MIG instance and there is no P2P between instances — so the pod would receive devices its model cannot span (RFC 0003).

• Raises:

  • ValueError: naming the node and the fix.

videoflow.deploy.manifests.workload(spec: NodeSpec, flow_id: str, run_id: str, flow_type: str, image: str, nats_cm_name: str, mounts: List[Mount] | None = None, gpu_runtime_class: str | None = None, gpu_mode: str = 'exclusive', gpu_resource_name: str | None = None, image_pull_policy: str = 'IfNotPresent', supervision: SupervisionPolicy | None = None, priority_class: str | None = None, rollout_policy: str | None = None, gpu_nodes: List[str] | None = None, host_resources: Mapping[str, str] | None = None) → dict

• Arguments:

  • supervision: restart policy for this node’s workload. Rendered into the Job backoffLimit; the local engine honours the same object in its supervisor, which is what keeps the two engines’ failure behaviour identical instead of merely similar.

  • priority_class: priorityClassName for the pod, or none (the cluster’s default priority).

  • rollout_policy: how a Deployment replaces its pods on an update — drain (Recreate: the old replicas stop before the new start; a GPU node whose devices cannot be held twice) or surge (one extra replica at a time, none unavailable). None keeps the API default. Jobs and StatefulSets are not rolled this way.

  • gpu_nodes: hostnames the node’s GPU pods may schedule on (--gpu-nodes).

  • host_resources: cpu/memory requests and cpu_limit/memory_limit for the worker container (--resources, descriptor defaults).

videoflow.deploy.mig module

MIG geometry knowledge and the mix-mode layout solver (RFC 0004).

The mix GPU strategy turns declared demands — spanners needing whole physical devices (gpu_count) and sharers needing an isolated fraction of one device (gpu_memory_gib) — into a concrete per-card plan: which cards stay whole, which get MIG-partitioned, and into which profiles. Everything in this module is pure and deterministic: it computes against an inventory handed to it and never talks to a cluster, which is what makes the solver exhaustively unit-testable. The strategy (deploy/gpu.py) owns fetching the inventory and applying the result; deploy/cluster.py owns reading the inventory off GPU Feature Discovery labels.

A card holds a multiset of profiles only when it can place it: each profile occupies a fixed width on the card’s memory-slice grid and may start only at the positions NVIDIA’s placement table allows (nvidia-smi mig -lgipp; the MIG user guide’s “GPU instance profile placements”), so compute slices, memory and legal start positions are three separate budgets. Two 3g.20gb on an A100 fill the whole grid (widths 4 at starts 0 and 4) and leave no room for a 1g.5gb although the compute slices (3 + 3 + 1 = 7) would fit — a layout the totals accept and nvidia-mig-parted would refuse after the operator has already been told it is feasible (ALLOC-002). A profile without placement data (a third-party table) is checked by totals only. nvidia-mig-parted remains the executor: it chooses the instance positions; the solver only promises that a legal assignment exists.

class videoflow.deploy.mig.CardPlan(node: str, card_index: int, profiles: Dict[str, int] | None = None)

Bases: object

What one physical card becomes under the layout: whole (profiles is None — spanner or undeclared-demand territory) or MIG’d into profiles (profile name -> instance count).

card_index: int

property is_mig: bool

node: str

profiles: Dict[str, int] | None = None

class videoflow.deploy.mig.GpuLayout(cards: List[CardPlan] = <factory>, spec_resources: Dict[str, str]=<factory>, slice_demand: Dict[str, int]=<factory>)

Bases: object

The solver’s answer: the per-card plan, each sharer spec’s resolved extended resource, and the total slice demand per resource (what the device plugin must advertise once the geometry is applied — preflight compares against it).

cards: List[CardPlan]

mig_nodes() → List[str]

Nodes with at least one MIG’d card, sorted — the nodes whose geometry prepare() must apply.

slice_demand: Dict[str, int]

spec_resources: Dict[str, str]

exception videoflow.deploy.mig.LayoutError

Bases: ValueError

No feasible MIG layout exists for the flow’s demands on this inventory. The message names the demand that failed and the fix.

class videoflow.deploy.mig.MigProfile(name: str, memory_gib: float, slices: int, max_per_gpu: int, width: int = 0, placements: tuple[int, ...] = ())

Bases: object

One MIG profile a card family supports: 1g.10gb needs 1 of the card’s compute slices and yields a 10 GiB instance, at most max_per_gpu per card.

• Arguments:

  • width: how many positions of the card’s memory-slice grid an instance occupies (nvidia-smi mig -lgipp prints {starts}:width); 0 when the table carries no placement data and the profile is checked by totals only.

  • placements: the grid positions an instance may start at, from the same query — a 3g.20gb on an A100 starts at 0 or 4, never at 2.

max_per_gpu: int

memory_gib: float

name: str

placements: tuple[int, ...] = ()

property resource: str

The Kubernetes extended resource the device plugin advertises for it.

slices: int

width: int = 0

class videoflow.deploy.mig.MigTable(family: str, match_substrings: List[str], total_slices: int, profiles: List[MigProfile], grid: int = 0)

Bases: object

The MIG geometry of one GPU family: how to recognize it from the GFD product label, how many compute slices a card has, the profiles it offers, and the size of the memory-slice grid the profiles’ placements refer to (grid; 0 when the table carries no placement data).

family: str

grid: int = 0

match_substrings: List[str]

matches(product: str) → bool

placement_checked() → bool

Whether every profile carries placement data, so place is a real check.

profiles: List[MigProfile]

smallest_profile_for(memory_gib: float) → MigProfile | None

total_slices: int

class videoflow.deploy.mig.NodeInventory(name: str, product: str, card_count: int, memory_gib_per_card: float, time_sliced: bool = False, mig_config: str | None = None, mig_partitioned: bool = False, owner: str | None = None, used_units: Dict[str, int]=<factory>, mig_allowed: bool = True, occupancy_known: bool = True, allocatable: Dict[str, int]=<factory>, dra_owned: bool = False)

Bases: object

One GPU node’s inventory as read off node labels and allocatable resources (cluster.gpu_inventory): the physical facts (product, card count, memory) plus the sharing/ownership state the mix strategy filters on before solving. The fields past memory_gib_per_card are facts, not decisions — this module never excludes a node; gpu.MixGpu partitions the inventory into usable and excluded nodes and hands the solver only the usable ones.

allocatable: Dict[str, int]

The node’s advertised extended resources (status.allocatable, integer quantities only): what a static MIG slice request is matched against.

card_count: int

dra_owned: bool = False

no device-plugin planning and never videoflow’s managed-MIG hooks (ALLOC-018).

Type:

The node is owned by a DRA driver (it publishes ResourceSlices)

memory_gib_per_card: float

mig_allowed: bool = True

Whether the solver may plan MIG geometry here. The mix strategy clears it on busy nodes: repartitioning destroys running workloads, but whole-card spanner claims are scheduler-accounted and remain safe.

mig_config: str | None = None

The node’s nvidia.com/mig.config label value, None when absent.

mig_partitioned: bool = False

The node already carries carved MIG geometry (advertises nvidia.com/mig-* or its product names a MIG profile) — GFD’s gpu.count then counts only non-MIG cards, so card_index no longer maps to physical positions.

name: str

occupancy_known: bool = True

False when the pod listing behind used_units could not be read. Unknown occupancy is not zero occupancy: the mix strategy refuses to plan on such a node rather than repartition cards another tenant may hold.

owner: str | None = None

videoflow.io/gpu-owner label value — the flow that MIG’d this node.

product: str

time_sliced: bool = False

Units are shares of a device (-SHARED product, time-slicing strategy, or replicas > 1) — physical card positions are not addressable.

used_units: Dict[str, int]

Extended-resource units currently requested by running pods, per resource.

videoflow.deploy.mig.layout_to_mig_parted_config(layout: GpuLayout, config_prefix: str = 'videoflow') → str

The nvidia-mig-parted config implementing layout, as YAML text — one named config per MIG’d node (<prefix>-<node>), covering every card on that node so untouched cards are explicitly mig-enabled: false. This is both what prepare() applies through the GPU Operator’s MIG manager and what preflight prints for manual application when no MIG manager is present.

Emitted by string assembly rather than a YAML library on purpose: the shape is fixed and tiny, and this module must import cleanly without the optional yaml extra.

videoflow.deploy.mig.mig_table_for_product(product: str) → MigTable | None

The registered MIG table matching a GFD nvidia.com/gpu.product label, or None for a card with no MIG support (consumer GPUs, unknown families).

videoflow.deploy.mig.place(profiles: Sequence[MigProfile], grid: int) → List[int] | None

A legal, non-overlapping start position per profile on a grid-position memory-slice grid, or None when the multiset cannot be placed. Exhaustive (a card has at most eight positions and seven instances) and order-free: the caller’s profile order never decides feasibility, only which of the equivalent assignments is returned. Widest profiles are tried first so the search prunes early; the result is reported in the caller’s order.

videoflow.deploy.mig.register_mig_table(table: MigTable) → None

Registers a GPU family’s MIG geometry. Same extension shape as the GPU strategy and cluster-flavor registries: a new card family is one call, not an edit to the solver.

videoflow.deploy.mig.solve_layout(inventory: List[NodeInventory], specs: List[NodeSpec]) → GpuLayout

Computes a feasible card layout for the flow’s GPU demands, or raises LayoutError naming the demand that cannot be placed and the fix.

The inventory handed in is assumed to be already filtered to usable nodes — the mix strategy excludes nodes owned by other flows, time-sliced or pre-MIG’d nodes, and clears mig_allowed on busy ones before calling this.

Deterministic (sorted walks throughout) so re-running against the same inventory reproduces the same geometry — which is what makes prepare() idempotent and explain truthful about what deploy will do. Placement order:

1. Spanners first — every replica of a GPU spec with no declared gpu_memory_gib takes gpu_count whole cards on one node (gpu_count defaults to 1: the undeclared-demand node is the degenerate spanner). Rigid demand, so it goes first, largest claims first.

2. Sharers packed second — every replica of a spec with gpu_memory_gib takes one MIG instance of the smallest fitting profile, first-fit onto already-open MIG cards, opening new cards as needed. All of one spec’s replicas use one profile on one card family, so the spec resolves to a single extended-resource name.

videoflow.deploy.solution module

Solution conventions for one-command deploys. A “solution” is a graph module shipped with optional sibling files:

• config.template.yaml — a valid config plus two extension blocks stripped on write: x-questions (inputs deploy asks for interactively when no config exists) and x-mounts (paths from the resolved config that must be hostPath-mounted into prep containers and worker pods).

• prepare.py — an idempotent prep hook deploy runs inside the solution image before compiling (its outputs get baked into the compiled specs).

x-questions entries: {key, prompt, type, default, choices, item_key, item_value}. key is a dotted path into the config (digit segments index int-keyed maps). Types: str (default), int, float, choice (with choices), path (one filesystem path, validated and absolutized), and paths (comma-separated paths expanded into a mapping via item_key, e.g. 'cam{i}', and item_value, e.g. {video: '{path}'}). That set is extensible: see register_question_type.

x-mounts entries are path templates: '{cameras.*.video}:ro' (dotted lookup into the resolved config, * fans out), '{work_dir}', '~/.videoflow:/root/.videoflow'. A single path resolves to a same-path hostPath mount (host and container see the same absolute path — required because the paths baked into node params at compile time must resolve identically in the pods); a host:container pair maps them explicitly (e.g. the operator’s home caches onto the container root’s). A pvc:<claim>:<path>[:ro] entry mounts an existing PersistentVolumeClaim at <path> in the pods instead — the form for data that lives in the cluster rather than on the operator’s machine (a shared model cache, an RWX work directory on a multi-node cluster); <path> takes the same {dotted} lookup and resolves relative to the solution directory like the others.

videoflow.deploy.solution.PVC_MOUNT_PREFIX = 'pvc:'

Marks an x-mounts entry (and a resolved mount spec) as a claim mount.

videoflow.deploy.solution.ask_questions(questions: ~typing.List[dict], base_dir: str, input_fn: ~typing.Callable[[str], str] = <built-in function input>) → dict

Prompts for each x-question on the terminal; returns {dotted_key: typed_value}.

• Raises:

  • ValueError: a question names an unregistered type (checked before prompting — see validate_question_types).

videoflow.deploy.solution.ensure_config(graph_dir: str, config_arg: str | None = None, interactive: bool = True, input_fn: ~typing.Callable[[str], str] = <built-in function input>) → str | None

The config file deploy should use: an explicit --config, an existing config.yaml next to the graph, or — when the solution ships a template — one generated by asking the template’s questions. None when the solution has no config convention at all.

• Raises:

  • SystemExit listing every question when a config must be generated but the session is non-interactive.

videoflow.deploy.solution.find_prepare(graph_dir: str) → str | None

The solution’s prepare.py hook, or None when it ships none.

videoflow.deploy.solution.find_template(graph_dir: str) → str | None

videoflow.deploy.solution.load_template(template_path: str) → dict

videoflow.deploy.solution.prepare_command(config_path: str | None = None, python_exe: str = 'python') → List[str]

The argv for the prepare hook, run with the solution directory as cwd.

videoflow.deploy.solution.register_question_type(qtype: str, coercer: Callable[[dict, str, str], Any]) → None

Registers a coercer for an x-questions type, called as coercer(question, answer, base_dir) where question is the raw x-questions entry (so a coercer can read its own extra keys, as choice reads choices), answer is the operator’s raw string, and base_dir is the solution directory that relative paths resolve against.

A coercer signals a bad answer by raising ValueError; the prompt loop catches it, shows the message, and re-asks.

• Arguments:

  • qtype: the type string used in config.template.yaml.

  • coercer: callable returning the coerced config value.

videoflow.deploy.solution.registered_question_types() → list

The x-questions type names currently registered, sorted.

videoflow.deploy.solution.render_config(template: dict, answers: dict) → str

The final config YAML: template body with answers applied and x-* blocks stripped.

videoflow.deploy.solution.resolve_mounts(template: dict | None, config: dict, graph_dir: str) → List[str]

Expands the template’s x-mounts against the resolved config into mount specs: /abs/path[:ro] and /host:/container[:ro] for manifests.parse_mounts, and pvc:<claim>:/abs/path[:ro] for manifests.parse_pvc_mounts (split_mount_specs separates the two). Relative config values resolve against graph_dir (matching a prep/compile container whose workdir is the graph dir); ~ expands.

• Raises:

  • ValueError: a pvc: entry names no claim or no path.

videoflow.deploy.solution.run_prepare_local(graph_dir: str, config_path: str | None = None) → bool

Runs the solution’s prepare.py on this host, with the solution directory as the working directory so its import common resolves. Used by run-local (whose workers are local anyway) and as deploy’s fallback when there is no image to run it in.

• Returns:

  • False when the solution ships no hook (nothing was run).

• Raises:

  • subprocess.CalledProcessError when the hook exits non-zero.

videoflow.deploy.solution.split_mount_specs(specs: List[str]) → tuple[List[str], List[str]]

Partitions resolve_mounts output into (host_specs, claim_specs): the host-path specs as manifests.parse_mounts takes them, and the claim specs with their pvc: prefix stripped, as manifests.parse_pvc_mounts takes them. Order is preserved within each list.

videoflow.deploy.solution.validate_question_types(questions: list | None) → None

Checks every question’s type is registered, before any prompting starts.

Done up front deliberately: the prompt loop treats ValueError as a bad answer and re-asks, so an unregistered type discovered mid-loop would re-prompt forever over something the operator cannot fix by typing.

• Raises:

  • ValueError: a question names an unregistered type; the message names the offending key, the known types, and how to add one.