DBOS pins workflows to their starting code version, meaning standard Kubernetes deployments risk stranding active work. To solve this, we built a lightweight controller that keeps legacy versions until all pending tasks finish, deleting them only after we verify completion.
Here at Trustle, we’re big fans of DBOS and how easy it makes building reliable and durable workflows. We use DBOS extensively to orchestrate and run ETL pipelines that pull data from a wide range of cloud and SaaS platforms, so we can offer better insights and observability to our customers. Much of our work is ensuring our data is up to date, complete, and available.
Building our pipelines revealed a core challenge: many extraction jobs run for hours. DBOS provides durability, allowing replacing workers to recover crashes. However, workers only handle workflows matching their specific version stamp. Immediate deployments replace all pods, making active workflows unrecoverable and stranding queued jobs. In short:
- You deploy version B. The new pods take new work.
- Version A still owns in-flight and enqueued workflows.
- If the deploy deletes the version A pods, that work has no executor. It stops until somebody brings version A back.
The cluster must keep legacy versions active until all work finishes. We update versions only when code changes make executing old workflows unsafe, such as modified extractions, since replaying incomplete workflows on updated steps causes errors.
We had no clean way to dynamically manage the list of versions that needed to run on our cluster at any given time (we manage our deployments with GitOps).
- A version should leave the list once all work associated with it is finished, which could be hours after the commit replaced it.
- With auto-prune enabled, Argo CD deletes whatever Git stops listing.
- The list of live versions is a runtime state like pod count, and runtime state belongs to a controller.
DBOS says to run one Deployment per active version and delete it when its workflows are done, but that's the problem: how to reliably detect when all work on a previous version is complete and safely tear down the associated Deployment.
Git defines intent, while our controller reflects actual state. The controller tracks active versions, their status, and when to delete them. Crucially, it works alongside KEDA rather than replacing it. The controller determines which Deployments exist, while KEDA controls how many pods run per Deployment.
Each application gets one WorkerVersionSet in Git. A deploy changes only the version and image; the controller does the rest.
apiVersion: dbos.trustle.com/v1alpha1
kind: WorkerVersionSet
metadata:
name: etl-worker
spec:
version: "v20261020143000" # the version new work should run on
image: "registry.example.com/etl:8332cd5"
graceSeconds: 600 # how long a drained version must stay idle before deletion
template: # stamped out once per live version
deployment: |
metadata:
name: etl-worker-__VERSION__
spec:
template:
spec:
containers:
- name: worker
image: IMAGE
env:
- name: DBOS_APPLICATION_VERSION
value: VERSION
scaledObject: |
... # KEDA: scale on this version's ENQUEUED + PENDING rows
podDisruptionBudget: |
...The controller replaces VERSION and IMAGE for each version it runs, and reports what it sees on the resource's status:
status:
versions:
- { version: v20261020143000, phase: Current, replicas: 2, outstanding: 4 }
- { version: v20261001090000, phase: Draining, replicas: 1, outstanding: 1 }There are three phases to the lifecycle of each Deployment that the controller manages, plus deletion.
stateDiagram-v2
[*] --> Current: Git names this version
Current --> Draining: Git names a newer version
Draining --> Drained: no work, no worker pods, no other pods on this version
Drained --> Draining: work or a pod comes back
Drained --> [*]: idle for the full grace period, then deleted
Draining --> Current: rollback
Drained --> Current: rollbackPhase | What it means | What the controller does |
|---|---|---|
Current | Git names this version. New work runs here. | Creates or updates its Deployment, KEDA ScaledObject, and PodDisruptionBudget. Never deletes it. |
Draining | A newer version exists, but this one still has work or pods. | Points its ScaledObject at this version's own ENQUEUED and PENDING workflows, with a minimum of zero pods. KEDA keeps pods only while that work exists. |
Drained | Zero workflows, zero worker pods, and zero other pods that carry this version. | Records the time it reached zero and starts the grace period (10 minutes by default). Any returning work or pod sends it back to Draining, and the clock starts over the next time it drains. |
Deleted | It stayed drained for the whole grace period. | Deletes the ScaledObject first, so KEDA cannot scale it back up, then the PodDisruptionBudget and the Deployment. |
Our pipeline service combines API and async worker Deployments. API pods enqueue workflows, but only worker pods execute them. Early on, our controller ignored terminating old API pods and deleted workers while API pods were still enqueuing unfulfillable workflows. As a result, any pod with a version label must keep that version active.
Thus, the controller deletes a version only after verifying no related resources remain. If database or pod queries fail, it safely retries on the next pass without interrupting active work. While it currently queries the DBOS database directly, we may transition to the DBOS API via Conductor in the future.
Since we started, DBOS has released an autoscaling API in Conductor that recommends a worker count for each application version, and it overlaps with the controller we built. If you run DBOS on Kubernetes and have hit similar problems, we'd love to hear how you solved them. You can review our implementation here: dbos-version-controller.