GPU sharing: running more GPU nodes than you have GPUs

Why local runs work and Kubernetes runs stall

Locally (LocalProcessEngine), every node is an OS subprocess on one machine. Nothing arbitrates the GPU: all N processes open it concurrently and share it, bounded only by VRAM. A 9-GPU-node graph runs fine on a single card if the models fit.

On Kubernetes, each GPU replica requests nvidia.com/gpu — an integer extended resource that cannot be overcommitted. The scheduler allocates whole devices exclusively: a graph with N GPU replicas needs N allocatable devices. On a cluster with fewer, some pods bind and the rest sit Pending (Insufficient nvidia.com/gpu) forever. The failure is silent in both flow modes, in different ways:

  • BATCH — the missing node’s input stream fills, backpressure blocks the producer, and the flow hangs making no progress. (Deploy’s wait loop now detects the unschedulable pod and aborts with an actionable error instead of hanging.)

  • REALTIME — producers never block; frames headed for the dead node are silently evicted and everything downstream of it produces nothing, while every running pod looks healthy. (Deploy now runs a bounded post-apply schedulability check and warns.)

videoflow explain my_flow.py prints a flow’s total GPU demand, and deploy’s preflight compares demand against the cluster’s allocatable capacity before applying anything (--strict-preflight makes a shortfall a hard error).

Three ways to close the gap

1. Reduce demand. Not every “GPU” stage needs one: trackers and light pose models often run fine on CPU. Prefer solution-level device knobs (e.g. per-stage device.tracker: cpu in a solution config) — the floor is the number of nodes doing genuinely GPU-bound inference.

2. Share the physical GPU, keep scheduler accounting: device-plugin time-slicing. The NVIDIA device plugin can advertise each physical GPU as N schedulable units:

# nvidia-plugin-config.yaml (namespace kube-system)
apiVersion: v1
kind: ConfigMap
metadata:
  name: nvidia-plugin-configs
  namespace: kube-system
data:
  config.yaml: |
    version: v1
    sharing:
      timeSlicing:
        renameByDefault: false
        failRequestsGreaterThanOne: true
        resources:
          - name: nvidia.com/gpu
            replicas: 4

Mount it into the device-plugin DaemonSet and point the plugin at it (CONFIG_FILE env var), then kubectl -n kube-system rollout restart ds/<device-plugin>. Allocatable nvidia.com/gpu flips from the physical count to replicas x physical, and — because renameByDefault: false keeps the resource name — videoflow manifests need no change at all. Caveats: time-slicing is round-robin temporal multiplexing with no memory isolation and one shared fault domain; the sum of all co-tenant models must fit in VRAM, and gpu_count > 1 is rejected (failRequestsGreaterThanOne).

3. Skip scheduler accounting entirely: ``–gpu-mode shared``.

videoflow deploy my_flow.py --gpu-mode shared --gpu-runtime-class nvidia

Shared mode emits no GPU resource limit: every GPU pod schedules onto the GPU pool (the nodeSelector/toleration stay) and shares the physical devices through the NVIDIA container runtime — exactly the semantics of a local run, with VRAM as the only limit. This is a dev-cluster tool:

  • There is no scheduler accounting and no memory isolation; an OOM takes out whichever pod the driver picks.

  • On a multi-GPU node every shared pod sees all devices (the CUDA base images set NVIDIA_VISIBLE_DEVICES=all) and frameworks default to device 0 — expect pile-up unless your node code picks a device.

  • On clusters where the NVIDIA runtime is an opt-in RuntimeClass (k3s), --gpu-runtime-class is required: with no resource limit, the runtime class is the only mechanism that injects the device, so deploy refuses shared mode without it.

Production clusters

Keep the default exclusive mode in production and size node pools to demand — the preflight numbers are exactly the capacity you need. Where sharing with real isolation is required:

  • MIG (A100/H100-class GPUs): hardware partitions exposed as their own resources. Point a node at a profile with gpu_resource_name='nvidia.com/mig-1g.10gb' (or --gpu-resource-name); no other videoflow change.

  • MPS (any Volta+ GPU, via the device plugin’s Helm chart): concurrent kernels with hard per-client memory caps of total/replicas. Stronger isolation than time-slicing; the heaviest model bounds the replica count. Pod specs are unchanged.

  • KEDA autoscaling excludes GPU nodes by default (each extra replica claims whole devices); opt in deliberately with --gpu-autoscaling once capacity math says it is safe.