Skip to main content

AutoClusters

Select your cluster's CMK version. To check a cluster's version, see What's new in CMK v2.

What AutoClusters does​

GPU hardware can fail in ways that stop a workload without taking the node down. A GPU can fall off the bus, report an uncorrectable memory error, or lose its NVLink connections. Kubernetes can still report the node as Ready, but the work scheduled on it stalls or crashes. In a large GPU cluster these faults are routine, and recovering from one usually means resetting the node's VM or replacing the node.

AutoClusters handles this for you. It watches every GPU node in the cluster for GPU and NVSwitch faults and reports each one on the node. Then it repairs the node by resetting its VM or replacing it with a healthy node. By default, Crusoe repairs nodes on its own. You can configure a node pool to have more control over remediation actions, or turn it off entirely. AutoClusters is installed on every CMK v2 cluster.

Choose a remediation policy​

The remediation policy is a setting on each node pool that decides how AutoClusters handles faults on the pool's nodes.

A repair resets or replaces a node, which interrupts every workload running on it. Whether that's acceptable at a given moment depends on the workload. A replica of an inference service can move to another node with little cost. A multi-day training run might need to save a checkpoint first, or its owners might want to move it off the node on their own schedule. The policy lets you decide who chooses when a repair starts. The CLI, Terraform, and the console call this setting the consent mode.

auto: Crusoe repairs on its own​

auto is the default. With auto, Crusoe repairs a node with an active fault without waiting for your approval. Crusoe schedules the repair and records a MaintenanceScheduled Kubernetes event on the node with the start time. When the repair starts, Crusoe drains the node and resets or replaces it.

Choose auto for workloads that tolerate losing a node, such as inference replicas, batch jobs that retry, or anything your scheduler already moves to healthy nodes. It's also the right choice when you want Crusoe to handle remediation end to end.

propose: you approve each repair​

With propose, when Crusoe finds a fault, it marks the node with the label crusoe.ai/maintenance-required=true and waits. To see why the node needs a repair, read the node's conditions whose type starts with Crusoe. You approve the repair by adding a second label, crusoe.ai/maintenance-approved=true, and Crusoe starts the repair.

Because approval is a label, you can approve by hand with kubectl or automate it. A common pattern is a small controller in your cluster that watches for nodes that need a repair, prepares the affected workloads, and then approves. Preparing might mean saving a checkpoint, waiting for a training step to finish, or telling your batch scheduler to stop placing work on the node. See Automate approvals.

Choose propose when the timing of losing a node matters. Until you approve, the faulty node stays in the cluster and keeps running its workloads, and its PreferNoSchedule taint makes it the last place the scheduler puts new pods. To keep new work off it entirely, run kubectl cordon <node-name>.

off: AutoClusters is turned off​

With off, AutoClusters takes no action on the node pool's nodes. Choose off if you run your own remediation. If you want to control when a repair happens but still have Crusoe perform it, use propose.

Set the policy​

The policy applies to all nodes in a node pool. A cluster can have node pools with different policies.

Set the policy when you create or update a node pool:

  • CLI (Crusoe CLI 1.0.10 or later): crusoe kubernetes nodepools update my-node-pool --consent-mode propose
  • Terraform (provider 1.5.0 or later): Set consent_mode = "propose" on crusoe_kubernetes_node_pool.
  • Console: Set Remediation Consent Mode when you create or edit a node pool.

How a repair works​

Under propose and auto, every fault goes through these steps. The policy only changes who decides when the repair starts.

  1. Detection: AutoClusters watches GPU nodes for fatal GPU and NVSwitch faults that the NVIDIA driver and NVIDIA Data Center GPU Manager (DCGM) report. A fault has to stay active for 60 seconds before Crusoe acts on it, so short, transient faults are ignored.
  2. Report: Crusoe sets the node's CrusoeGPUFault condition to True, with a reason and a message that describe the fault. It adds the label crusoe.ai/maintenance-required=true and a PreferNoSchedule taint, so the scheduler places new pods on the node only when no other node fits.
  3. Decision: Under propose, the repair waits until the node also has crusoe.ai/maintenance-approved=true. Under auto, Crusoe schedules the repair immediately.
  4. Repair: Crusoe cordons the node and evicts its pods, honoring PodDisruptionBudgets and termination grace periods. Pods still running after 5 minutes for a VM reset, or 10 minutes for a replacement, are deleted. Crusoe then resets the node's VM or replaces the node, depending on the fault.
  5. Cleanup: When the node is healthy, Crusoe sets CrusoeGPUFault to False and removes both labels and the taint.

If the fault clears before the repair starts, Crusoe sets the condition back to False and removes the labels and the taint, and the node goes back to normal service.

When a VM reset doesn't clear the fault​

For many faults, a VM reset is the appropriate fix, but the reset can sometimes fail to clear the problem. When that happens, Crusoe escalates to a replacement. It replaces the node when the reset fails or when the fault comes back within 15 minutes. If a node that had a reset in the last 24 hours gets another fault of the same kind, the next repair is a replacement. One approval covers the reset and the replacement that follows it.

When a repair fails​

If a node replacement fails, Crusoe records a MaintenanceActionFailed Kubernetes event, and Crusoe support is notified. Support reaches out to you through a support ticket to resolve the issue. The labels and the condition stay on the node until the problem is resolved.

See what's wrong with a node​

Crusoe reports faults as node conditions whose type starts with Crusoe. GPU faults are reported with the CrusoeGPUFault condition. The condition Status tells you whether the node has a GPU fault, and the condition Reason classifies the fault. The condition Message gives you details, such as which GPU reported which error:

StatusReasonMeaningExample message
TrueNvidiaXIDErrorThe NVIDIA driver reported a fatal Xid error.GPU 3 reported Xid 79 (GPU has fallen off the bus).
TrueNvidiaSXIDErrorAn NVSwitch reported a critical SXID error.NVSwitch reported SXID 24007.
TrueNvidiaDCGMErrorNVIDIA DCGM diagnostics found a critical failure.GPU 5 reported DCGM_FR_NVLINK_DOWN.
TrueGpuErrorAnother GPU fault that doesn't fit the reasons above.The message from the component that reported it.
FalseNoGpuFaultDetectedNo GPU fault.No GPU fault.

To read the condition on a node, run:

kubectl get node <node-name> -o jsonpath='{.status.conditions[?(@.type=="CrusoeGPUFault")]}'

kubectl describe node <node-name> shows the condition and the node's recent Kubernetes events together.

Approve repairs​

Under propose, a node that needs a repair carries the label crusoe.ai/maintenance-required=true. To list these nodes, run:

kubectl get nodes -l crusoe.ai/maintenance-required=true

To approve the repair, add the approval label:

kubectl label node <node-name> crusoe.ai/maintenance-approved=true

The repair starts after you approve. Until then, the node keeps the maintenance-required label for as long as the fault stays active.

The cluster accepts changes to these labels that follow these rules:

  • Crusoe manages crusoe.ai/maintenance-required.
  • You add crusoe.ai/maintenance-approved with the value true while the node has crusoe.ai/maintenance-required.
  • The approval stays on the node until the repair succeeds or the fault clears. Then Crusoe removes both labels.

Automate approvals​

A controller that automates approvals usually:

  1. Watches the nodes in your propose node pools for maintenance-required without maintenance-approved.
  2. Reads the node's Crusoe conditions to log the fault or decide what to do.
  3. Prepares the workloads on the node, for example by saving a checkpoint or pausing a job. The drain deletes pods that are still running after 5 minutes for a VM reset or 10 minutes for a replacement, so finish preparing before you approve.
  4. Adds maintenance-approved=true once preparation finishes.

This script shows the same loop with kubectl for one node pool. Replace prepare_node with your own logic:

#!/usr/bin/env bash
set -euo pipefail

POOL="my-training-pool"

prepare_node() {
  # Save checkpoints, pause jobs, or wait for a safe point here.
  echo "Preparing $1"
}

for node in $(kubectl get nodes \
    -l "crusoe.ai/nodepool.name=${POOL},crusoe.ai/maintenance-required=true,!crusoe.ai/maintenance-approved" \
    -o jsonpath='{.items[*].metadata.name}'); do
  prepare_node "$node"
  kubectl label node "$node" crusoe.ai/maintenance-approved=true \
    || echo "Skipped $node: the fault cleared"
done

If the fault clears while you prepare, the cluster rejects the approval because the node no longer needs a repair. Treat that as done. A repair is complete when Crusoe removes maintenance-required. A replacement adds a new node to the pool, so track work by node pool rather than by node name. A production controller watches Node objects instead of polling, and records what it did.

Follow repairs with Kubernetes events​

Crusoe records each step of a repair as a Kubernetes event on the Node object. You read these events through the Kubernetes API, for example with kubectl. Kubernetes keeps events for one hour, so they're the place to watch a repair as it happens. To see them, run:

kubectl get events -A --field-selector involvedObject.kind=Node,involvedObject.name=<node-name>
ReasonTypeWhen
MaintenanceRequiredWarningCrusoe adds the maintenance-required label. The message names the fault.
MaintenanceAwaitingApprovalNormalUnder propose, the repair is waiting for your approval.
MaintenanceApprovedNormalUnder propose, Crusoe sees your approval.
MaintenanceScheduledNormalUnder auto, Crusoe schedules the repair. The message gives the start time.
MaintenanceStartedNormalA VM reset or a node replacement starts.
MaintenanceEscalatedNormalThe VM reset didn't clear the fault, so Crusoe replaces the node.
MaintenanceActionFailedWarningA VM reset or a node replacement failed.
MaintenanceSucceededNormalThe node is healthy, and Crusoe removed both labels.

For example, in the Events section of kubectl describe node <node-name>, a repair that you approve and that a VM reset fixes looks like this:

Type     Reason                       Age   From              Message
Warning  MaintenanceRequired          18m   nodepool-service  Node needs maintenance: GPU 3 reported Xid 79 (GPU has fallen off the bus).
Normal   MaintenanceAwaitingApproval  18m   nodepool-service  Waiting for approval. Add the label crusoe.ai/maintenance-approved=true to start.
Normal   MaintenanceApproved          9m    nodepool-service  Maintenance approved.
Normal   MaintenanceStarted           9m    nodepool-service  VM reset started.
Normal   MaintenanceSucceeded         4m    nodepool-service  VM reset succeeded. Node is healthy.

View remediation history​

Kubernetes events cover only the last hour. To see past repairs, view the history of remediations in the Console UI.