Skip to content

Gate template promotion with GitOps

This guide shows how to make Argo CD or Flux wait for a CoderTemplateTest result, and how to promote a template version only after its test passed. The health rules live in config/gitops/ in this repository. This page copies them, and a unit test keeps the copies equal to the files. The page also holds the health rules for CoderAIProvider, and for CoderWorkspacePool and CoderWorkspaceClaim.

Why custom health rules are needed

A GitOps tool decides from health rules when a resource is ready.

  • Argo CD has no built-in health check for CoderTemplateTest. Without one, it cannot use the test result to hold back a later sync wave.
  • Flux falls back to kstatus. kstatus reports a resource without conditions as current. A new test has no status until the controller writes the first one, and that lasts as long as the controller is down. In that window, Flux treats the test as passed.

Both rules below wait until status.observedGeneration equals metadata.generation. The controller sets status.observedGeneration on every status write. The rules then map status.phase:

status.phase Argo CD Flux
no status, or a stale observedGeneration Progressing InProgress
Pending, Running Progressing InProgress
Succeeded with condition Ready=True Healthy Current
Failed Degraded Failed

The controller sets Succeeded only after the test workspace is deleted, and always together with Ready=True. While a test is being deleted, Argo CD shows Progressing with the message Deleting the test workspace, which the script returns as deletionMessage. Argo CD versions without deletionMessage support show their default deletion message instead. Otherwise its message is <status.reason>: <status.message>.

Install the Argo CD health check

Add the script as the argocd-cm key resource.customizations.health.coder.com_CoderTemplateTest. If Git manages your argocd-cm, add the key there. Otherwise, patch the ConfigMap from the file:

kubectl -n argocd patch configmap argocd-cm --type merge -p "$(
  jq -n --rawfile lua config/gitops/argocd-health-codertemplatetest.lua \
    '{data: {"resource.customizations.health.coder.com_CoderTemplateTest": $lua}}')"

The script (config/gitops/argocd-health-codertemplatetest.lua):

-- Argo CD health check for CoderTemplateTest (coder.com/v1alpha1).
-- Install it as the argocd-cm key
-- resource.customizations.health.coder.com_CoderTemplateTest.
-- docs/how-to/gitops.md copies this file. A test keeps the copy equal.
local hs = { status = "Progressing", message = "Waiting for the controller" }
if obj.metadata.deletionTimestamp ~= nil then
  -- Argo CD shows deletionMessage, not message, for a resource being deleted.
  hs.deletionMessage = "Deleting the test workspace"
  hs.message = hs.deletionMessage
  return hs
end
local st = obj.status
if st == nil or st.observedGeneration == nil or st.observedGeneration ~= obj.metadata.generation then
  return hs
end
if st.reason ~= nil and st.message ~= nil then
  hs.message = st.reason .. ": " .. st.message
end
if st.phase == "Succeeded" then
  for _, c in ipairs(st.conditions or {}) do
    if c.type == "Ready" and c.status == "True" then
      hs.status = "Healthy"
    end
  end
elseif st.phase == "Failed" then
  hs.status = "Degraded"
end
return hs

Install the Flux health check

Flux v2.5 and later evaluate healthCheckExprs of a Kustomization with spec.wait: true. Add this entry (config/gitops/flux-healthcheck-codertemplatetest.yaml) to spec.healthCheckExprs:

# Flux health check expressions for CoderTemplateTest (coder.com/v1alpha1).
# Add this entry to spec.healthCheckExprs of a Flux v2.5+ Kustomization
# with spec.wait: true.
# docs/how-to/gitops.md copies this file. A test keeps the copy equal.
- apiVersion: coder.com/v1alpha1
  kind: CoderTemplateTest
  inProgress: "!has(status.observedGeneration) || status.observedGeneration != metadata.generation || !has(status.phase) || status.phase in ['Pending', 'Running']"
  failed: "status.phase == 'Failed'"
  current: "status.phase == 'Succeeded' && status.conditions.exists(c, c.type == 'Ready' && c.status == 'True')"

Flux evaluates inProgress, then failed, then current, and the first true expression wins. Before the controller writes the first status, the expressions cannot read status. Flux then reports the test as Unknown and keeps waiting, like InProgress.

Set spec.timeout of the Kustomization above the test's timeoutSeconds (default 900 s) plus the time that the controller needs to start or recover, or Flux stops waiting before the test ends. The Flux timeout starts with the reconciliation, but the test's clock starts only when the controller starts the test and sets status.startTime.

Patterns

One test per version

Create one test per template version, and name it after the version. The spec is immutable, so a new version needs a new object anyway, and the name shows which version a result belongs to. Object names allow only lowercase letters, digits, -, and ., but a version name can also hold uppercase letters and _. promote.sh name of the promotion example maps any valid version name to a safe name. It lowercases the name, replaces _ and . with -, and appends 8 characters of the SHA-256 hash of the exact version name, so V2 and v2 get different names:

$ examples/gitops-promotion/promote.sh name docker V2.0_rc1
docker-v2-0-rc1-1c72012b

Do not set ttlSecondsAfterFinished on tests that GitOps manages. The controller deletes the finished test, GitOps creates it again, and a new workspace starts, forever.

Gate before promotion

  1. CI pushes the new version under a fixed name, without activating it and without prompts: coder templates push docker --directory ./docker --name v2 --activate=false --yes.
  2. Git holds a test of that version, with spec.version.name.
  3. A promotion step waits until the test is final. Only after Succeeded, it reads status.templateVersionID of the test and calls the codertemplates/promote subresource of the aggregated API. Promoting the active version again answers AlreadyActive. After a 409 or a 503, the step must read the active version before it tries again, because another writer can have activated a third version, and the last writer wins. See the confirmation rules of the promote subresource.

With Argo CD, put the test in sync wave 1, the promotion step in wave 2, and at least one resource in wave 3. Argo CD applies a wave only after every resource of the previous wave is healthy. After it applies the last wave, it ends the sync without a wait, so a promotion step in the last wave can fail after a successful sync. With Flux, a plain dependsOn does not wait until the dependency has applied the same Git revision, so the promotion step can start while the previous test still shows its result. The promotion step must therefore wait for its own test.

Any promotion step must check these points. The promotion example checks all of them:

  • Full spec. The test's whole spec, including spec.parameters, equals the spec in Git. Read spec and status with one get, so both come from the same object. Anyone who can create tests can create the test name first with another spec.
  • Timeout. The wait and the deadline of the step cover the test's timeoutSeconds (900 s by default, at most 7200 s) and the workspace delete.
  • Names. Test names and the names of promotion objects, such as a Job per version, are Kubernetes-safe, as described in One test per version.
  • Result. The pipeline waits for the promotion result and shows a failed promotion as a failure. With Flux, set wait: true and a timeout above the step's deadline on the Kustomization that runs the step.

Run the promotion example

The directory examples/gitops-promotion/ in this repository is a Kustomize directory for one template, default.docker in namespace coder. Argo CD and Flux both render it with Kustomize. It holds:

File Content
codertemplatetest.json The test of one version, in sync wave 1. It is JSON, so the Job can read it with jq.
promote.sh The promotion script. It needs bash, kubectl, and jq. promote.sh name also needs sha256sum or shasum.
promote-job.yaml A Job that runs promote.sh promote, in sync wave 2, and the ConfigMap docker-promote-done in sync wave 3. The ConfigMap holds only the test name.
rbac.yaml A ServiceAccount that can get only this test and create only codertemplates/promote of the test's template.
kustomization.yaml Puts promote.sh and codertemplatetest.json into the ConfigMap docker-promote for the Job. Copies the test name to the Job, the Role, and the ConfigMap docker-promote-done, the template to the Role, and the namespace to every object.
argocd-application.yaml, flux-kustomization.yaml An Argo CD Application and a Flux Kustomization for the directory. Apply one of them, not both.

The Job reads the test in Git from its ConfigMap. It reads the test in the cluster with one get every 30 s, and then does one of these:

The test in the cluster Result
Its spec differs from the spec in Git. The API server default timeoutSeconds: 900, the order of spec.parameters, and an omitted versus an empty parameter value do not count. The Job fails at once and does not promote.
It does not exist, is being deleted, has a stale status.observedGeneration, or is Pending or Running The Job waits.
Failed The Job fails and does not promote. Its log shows status.reason and status.message.
Succeeded with Ready=True The Job promotes status.templateVersionID of that object and completes when the answer is Promoted or AlreadyActive.

The Job waits at most the test's timeoutSeconds plus 900 s for the controller to start the test. The controller writes Succeeded only after the workspace delete, and Failed at the latest at the test deadline, so this wait covers the delete. The Job's activeDeadlineSeconds is 8400: 7200 s, the longest timeoutSeconds, plus the 900 s, plus 300 s for the promotion. The Flux Kustomization has wait: true and timeout: 2h30m, so Flux is ready only after the Job completed, and it fails when the Job fails. Argo CD creates the Job in wave 2 after the test is healthy. Because the ConfigMap docker-promote-done is in wave 3, Argo CD waits until the Job is healthy and fails the sync when the Job fails. The ConfigMap holds the test name, so each version changes it, and the sync option ApplyOutOfSyncOnly=true cannot skip wave 3. The Job is not an Argo CD sync hook, because a hook runs on every sync and promotes the version again, even after another writer activated a newer one.

Each Job promotes once. The Job has backoffLimit: 0, and promote.sh sends the promotion only once. After a failed promotion, read the active version, and then delete the Job to run it again. A change of promote.sh changes the pod template of the Job, which Kubernetes cannot update, so the sync fails until the next version gets a new Job, or until you delete the Job. Do not set Flux force: true or the Argo CD sync option Replace=true for the Job: a replaced Job promotes again.

To use the example:

  1. Install the Argo CD health check if you use Argo CD. With Flux, the Job is the gate.
  2. Copy the directory into your Git repository. Set the control plane, the template, the namespace, and the parameters of your test in codertemplatetest.json. spec.template must be the exact name that kubectl get codertemplates shows, because the promote subresource refuses other casing with 400. Set repoURL or sourceRef and path in the Argo CD or Flux manifest, and apply it. For a second template, also replace the name docker-template of the Argo CD Application or the Flux Kustomization, and, in the same namespace, the name docker-promote in rbac.yaml, promote-job.yaml, and kustomization.yaml. Otherwise the second copy changes the objects of the first.
  3. For each new version, CI pushes the version without activating it, sets the version and the test name in codertemplatetest.json, and commits the file. Git then holds a new test and a new Job, both with the new name.

The CI commands for step 3, in the copied directory:

coder templates push docker --directory ./docker --name V2.0_rc1 --activate=false --yes
name=$(./promote.sh name docker V2.0_rc1)
jq --arg name "$name" --arg version V2.0_rc1 \
  '.metadata.name = $name | .spec.version.name = $version' \
  codertemplatetest.json >codertemplatetest.json.new
mv codertemplatetest.json.new codertemplatetest.json

CAUTION: Install the example only in a namespace where only the GitOps tool creates Pods, Jobs, and ConfigMaps. Anyone who can create a Pod there can run it as the ServiceAccount docker-promote and promote any version of the template without a test. A test creator who has only the test rights from Test templates cannot. To isolate the promoter, move the ServiceAccount, the ConfigMaps docker-promote and docker-promote-done, and the Job to a namespace that only the GitOps tool writes, and keep the Role and the RoleBinding in the namespace of the test. Set the new namespace in rbac.yaml, in promote-job.yaml, in the configMapGenerator of kustomization.yaml, and in subjects[0].namespace of the RoleBinding. Then remove the targets ServiceAccount, ConfigMap, and Job, and the field subjects.0.namespace, from the namespace replacement in kustomization.yaml. Otherwise Kustomize copies the test namespace back to the moved objects and to the RoleBinding subject.

Check after promotion

A CoderTemplate update of spec.files activates the new version at once, before any test. A test with spec.version.active: true in a later wave can still detect breakage, but only after users got the version.

If the import takes longer than the 34-second write budget of the aggregated API (#117), the update fails and the previous version stays active, so the test resolves the previous version. status.templateVersionName shows which version ran.

Health rules for CoderAIProvider

Argo CD and Flux also need health rules to wait for a CoderAIProvider. Without them, Argo CD has no health check for the kind, and Flux reports a new provider as current before the controller writes its first status. The controller writes the conditions Ready, Reconciling and Stalled together, with one reason and message. It sets status.observedGeneration to metadata.generation only when it writes the conditions for that generation. Both rules wait until status.observedGeneration equals metadata.generation. They also ignore a condition whose own observedGeneration is older, so a status that an older controller wrote in the middle of an update does not count as healthy. Then they read the conditions:

Conditions Argo CD Flux
no status, or a stale observedGeneration on the status or the Ready condition Progressing InProgress (Unknown before the first status)
Reconciling=True Progressing InProgress
Ready=True Healthy Current
Stalled=True Degraded Failed

A kill switch stalls the provider, so it shows Degraded or Failed until the spec or the key Secret changes. The Argo CD message is <reason>: <message> of the Ready condition. While the object is being deleted, Argo CD shows Deleting the Coder AI provider.

Add the script as the argocd-cm key resource.customizations.health.coder.com_CoderAIProvider (config/gitops/argocd-health-coderaiprovider.lua):

-- Argo CD health check for CoderAIProvider (coder.com/v1alpha1).
-- Install it as the argocd-cm key
-- resource.customizations.health.coder.com_CoderAIProvider.
-- docs/how-to/gitops.md copies this file. A test keeps the copy equal.
local hs = { status = "Progressing", message = "Waiting for the controller" }
if obj.metadata.deletionTimestamp ~= nil then
  -- Argo CD shows deletionMessage, not message, for a resource being deleted.
  hs.deletionMessage = "Deleting the Coder AI provider"
  hs.message = hs.deletionMessage
  return hs
end
local st = obj.status
if st == nil or st.observedGeneration == nil or st.observedGeneration ~= obj.metadata.generation then
  return hs
end
-- A condition from an older generation does not describe the current spec.
local conditions = {}
for _, c in ipairs(st.conditions or {}) do
  if c.observedGeneration == obj.metadata.generation then
    table.insert(conditions, c)
  end
end
for _, c in ipairs(conditions) do
  if c.type == "Ready" and c.reason ~= nil and c.message ~= nil then
    hs.message = c.reason .. ": " .. c.message
  end
end
for _, c in ipairs(conditions) do
  if c.type == "Ready" and c.status == "True" then
    hs.status = "Healthy"
  elseif c.type == "Stalled" and c.status == "True" then
    hs.status = "Degraded"
  end
end
return hs

Add this entry (config/gitops/flux-healthcheck-coderaiprovider.yaml) to spec.healthCheckExprs of a Flux v2.5+ Kustomization with spec.wait: true:

# Flux health check expressions for CoderAIProvider (coder.com/v1alpha1).
# Add this entry to spec.healthCheckExprs of a Flux v2.5+ Kustomization
# with spec.wait: true.
# docs/how-to/gitops.md copies this file. A test keeps the copy equal.
- apiVersion: coder.com/v1alpha1
  kind: CoderAIProvider
  inProgress: "!has(status.observedGeneration) || status.observedGeneration != metadata.generation || !has(status.conditions) || !status.conditions.exists(c, c.type == 'Ready' && has(c.observedGeneration) && c.observedGeneration == metadata.generation) || status.conditions.exists(c, c.type == 'Reconciling' && c.status == 'True')"
  failed: "status.conditions.exists(c, c.type == 'Stalled' && c.status == 'True')"
  current: "status.conditions.exists(c, c.type == 'Ready' && c.status == 'True')"

Health rules for CoderWorkspacePool and CoderWorkspaceClaim

Argo CD and Flux also need health rules to wait for a workspace pool and its claims. Without them, Argo CD has no health check for the kinds, and Flux reports a new pool or claim as current before the controller writes its first status. The controller writes the conditions Ready, Reconciling and Stalled of a pool or a claim together, with one reason and message, and with the object's metadata.generation as the condition's observedGeneration. The Argo CD rules ignore each condition whose observedGeneration is older. The Flux rules wait for a Ready condition of the current generation, and then read the conditions without that filter. The controller gives all three conditions one generation, so both rules agree on what it writes.

A pool also has status.observedGeneration, and both pool rules wait until it equals metadata.generation. A change of spec.replicas through /scale raises the generation, so after kubectl scale, or a resize by an HPA or KEDA, the pool shows Progressing until the controller has counted its members for the new spec.replicas.

Pool conditions Argo CD Flux
no status, or a stale observedGeneration on the status or the Ready condition Progressing InProgress (Unknown before the first status)
Reconciling=True, for example WaitingForMembers or OwnerWorkspaceLimit Progressing InProgress
Ready=True: at least spec.replicas current unclaimed workspaces are ready, and no outdated one is left Healthy Current
Stalled=True, for example MemberFailureLimit Degraded Failed

A claim has no status.observedGeneration, and its spec cannot change, so the claim rules read only the conditions and status.phase:

Claim Argo CD Flux
no status, or no conditions of the current generation Progressing InProgress (Unknown before the first status)
Pending, or Bound to a workspace that is not ready (Reconciling=True) Progressing InProgress
Bound to a ready workspace (Ready=True) Healthy Current
Lost, or Stalled=True Degraded Failed

Lost is final: the claim never binds again. To get a workspace again, delete the claim, and let the GitOps tool create it again. The new object binds a new workspace. In Share mode, a claim that is not Lost can also be Stalled: ClaimModeMismatch never binds, ShareRevoking means the operator removes the share, and ShareRefused, ShareRevoked, and ShareRemoved leave a bound workspace without its share. ShareRefused clears once the policy and Coder allow the share. After ShareRevoked or ShareRemoved, the operator does not share the workspace again.

A bound claim whose member failed with OwnershipUnknown or ChangedExternally stays Progressing and never becomes Degraded, so a GitOps wait for it ends only at its timeout (#454). The claim shows the member's reason. To recover, delete the claim, and let the GitOps tool create it again. See A pool member does not finish deleting. For every reason, see Troubleshoot workspace pools. The Argo CD message is <reason>: <message> of the Ready condition. While the object is being deleted, Argo CD shows Deleting the workspace pool or Deleting the workspace claim.

Add the scripts as the argocd-cm keys resource.customizations.health.coder.com_CoderWorkspacePool (config/gitops/argocd-health-coderworkspacepool.lua):

-- Argo CD health check for CoderWorkspacePool (coder.com/v1alpha1).
-- Install it as the argocd-cm key
-- resource.customizations.health.coder.com_CoderWorkspacePool.
-- docs/how-to/gitops.md copies this file. A test keeps the copy equal.
local hs = { status = "Progressing", message = "Waiting for the controller" }
if obj.metadata.deletionTimestamp ~= nil then
  -- Argo CD shows deletionMessage, not message, for a resource being deleted.
  hs.deletionMessage = "Deleting the workspace pool"
  hs.message = hs.deletionMessage
  return hs
end
local st = obj.status
if st == nil or st.observedGeneration == nil or st.observedGeneration ~= obj.metadata.generation then
  return hs
end
-- A condition from an older generation does not describe the current spec.
local conditions = {}
for _, c in ipairs(st.conditions or {}) do
  if c.observedGeneration == obj.metadata.generation then
    table.insert(conditions, c)
  end
end
for _, c in ipairs(conditions) do
  if c.type == "Ready" and c.reason ~= nil and c.message ~= nil then
    hs.message = c.reason .. ": " .. c.message
  end
end
for _, c in ipairs(conditions) do
  if c.type == "Ready" and c.status == "True" then
    hs.status = "Healthy"
  elseif c.type == "Stalled" and c.status == "True" then
    hs.status = "Degraded"
  end
end
return hs

and resource.customizations.health.coder.com_CoderWorkspaceClaim (config/gitops/argocd-health-coderworkspaceclaim.lua):

-- Argo CD health check for CoderWorkspaceClaim (coder.com/v1alpha1).
-- Install it as the argocd-cm key
-- resource.customizations.health.coder.com_CoderWorkspaceClaim.
-- docs/how-to/gitops.md copies this file. A test keeps the copy equal.
local hs = { status = "Progressing", message = "Waiting for the controller" }
if obj.metadata.deletionTimestamp ~= nil then
  -- Argo CD shows deletionMessage, not message, for a resource being deleted.
  hs.deletionMessage = "Deleting the workspace claim"
  hs.message = hs.deletionMessage
  return hs
end
local st = obj.status
if st == nil then
  return hs
end
-- A claim has no status.observedGeneration, so only the conditions tell
-- whether the controller saw the current spec.
local conditions = {}
for _, c in ipairs(st.conditions or {}) do
  if c.observedGeneration == obj.metadata.generation then
    table.insert(conditions, c)
  end
end
for _, c in ipairs(conditions) do
  if c.type == "Ready" and c.reason ~= nil and c.message ~= nil then
    hs.message = c.reason .. ": " .. c.message
  end
end
for _, c in ipairs(conditions) do
  if c.type == "Ready" and c.status == "True" then
    hs.status = "Healthy"
  elseif c.type == "Stalled" and c.status == "True" then
    hs.status = "Degraded"
  end
end
-- Lost is final: the claim never binds again.
if st.phase == "Lost" then
  hs.status = "Degraded"
end
return hs

Add these entries (config/gitops/flux-healthcheck-coderworkspacepool.yaml and config/gitops/flux-healthcheck-coderworkspaceclaim.yaml) to spec.healthCheckExprs of a Flux v2.5+ Kustomization with spec.wait: true:

# Flux health check expressions for CoderWorkspacePool (coder.com/v1alpha1).
# Add this entry to spec.healthCheckExprs of a Flux v2.5+ Kustomization
# with spec.wait: true.
# docs/how-to/gitops.md copies this file. A test keeps the copy equal.
- apiVersion: coder.com/v1alpha1
  kind: CoderWorkspacePool
  inProgress: "!has(status.observedGeneration) || status.observedGeneration != metadata.generation || !has(status.conditions) || !status.conditions.exists(c, c.type == 'Ready' && has(c.observedGeneration) && c.observedGeneration == metadata.generation) || status.conditions.exists(c, c.type == 'Reconciling' && c.status == 'True')"
  failed: "status.conditions.exists(c, c.type == 'Stalled' && c.status == 'True')"
  current: "status.conditions.exists(c, c.type == 'Ready' && c.status == 'True')"
# Flux health check expressions for CoderWorkspaceClaim (coder.com/v1alpha1).
# Add this entry to spec.healthCheckExprs of a Flux v2.5+ Kustomization
# with spec.wait: true.
# docs/how-to/gitops.md copies this file. A test keeps the copy equal.
- apiVersion: coder.com/v1alpha1
  kind: CoderWorkspaceClaim
  inProgress: "(!has(status.phase) || status.phase != 'Lost') && (!has(status.conditions) || !status.conditions.exists(c, c.type == 'Ready' && has(c.observedGeneration) && c.observedGeneration == metadata.generation) || status.conditions.exists(c, c.type == 'Reconciling' && c.status == 'True'))"
  failed: "(has(status.phase) && status.phase == 'Lost') || status.conditions.exists(c, c.type == 'Stalled' && c.status == 'True')"
  current: "status.conditions.exists(c, c.type == 'Ready' && c.status == 'True')"

What CI covers

The unit test in config/gitops/ runs the Lua scripts with gopher-lua, the Lua engine of Argo CD, and the Flux expressions with cel-go, both over fixed test objects. The unit test in examples/gitops-promotion/ renders the example with Kustomize and runs promote.sh against a fake kubectl. Its cases include a test that someone created first with other parameters, which the script must refuse, and a test that runs for the longest timeoutSeconds. CI does not run Argo CD or Flux.