Skip to main content

Advanced: Kubernetes Operations

Managed Slurm clusters run on Crusoe Managed Kubernetes (CMK). While the CLI and API handle most operations, you can also interact with the cluster directly via kubectl for advanced configuration, troubleshooting, and user management.

Accessing Your Cluster via kubectl​

Configure kubectl to connect to your cluster:

crusoe kubernetes clusters get-credentials <cluster-name>

Verify the connection:

kubectl cluster-info

Viewing Cluster State​

Slurm Custom Resources​

The Crusoe Slurm Operator (CSO) uses four Custom Resource types to manage your cluster. When you create a Slurm cluster through the Slurm UI or crusoe slurm CLI, these CRDs will be created, controlled, and reconciled by Crusoe in the slurm namespace.

kubectl get slurmclusters -n slurm # Clusters
kubectl get nodesets -n slurm # Node sets
kubectl get slurmusers -n slurm # All users
kubectl get slurmusergroups -n slurm # All groups

For detailed status with conditions:

kubectl describe slurmclusters <name> -n slurm
tip

Use kubectl explain for field-level documentation on any Custom Resource. The CRDs have built-in descriptions for every field:

kubectl explain slurmclusters.spec
kubectl explain slurmusers.spec
kubectl explain slurmusergroups.spec
kubectl explain nodesets.spec

Cluster Phase and Conditions​

The SlurmCluster status includes a phase field (possible values: Provisioning, Installing, Ready, NotReady, CreateFailed, Deleting, DeleteFailed) and detailed conditions for each component. Use kubectl describe to see what's happening if your cluster isn't healthy:

kubectl describe slurmclusters <name> -n slurm

Look for conditions like:

  • ControllerReady — Slurm controller pod is running
  • LoginReady — Login nodes are running
  • SlinkyReady — Slinky Helm chart is installed
  • CertManagerReady — cert-manager is installed
  • LoadBalancerReady — Load balancer controller is installed
  • TopographReady — Topology discovery is running

Node Set Readiness​

The NodeSet status shows a readyReplicas field in "ready/total" format:

kubectl get nodesets -n slurm
NAME READY AGE
slurm-worker-node-set 2/2 1h

Storage Configuration​

Managed Slurm uses a shared filesystem for the /home directory, mounted across all login and worker nodes. This is backed by a PersistentVolumeClaim (PVC) using the Crusoe CSI driver.

StorageClass​

The operator automatically creates a StorageClass named <cluster-name>-crusoe-csi-driver-fs-sc with:

  • Provisioner: fs.csi.crusoe.ai
  • Volume binding mode: WaitForFirstConsumer
  • Volume expansion: Enabled

Viewing Storage​

kubectl get pvc -n slurm # View persistent volume claims
kubectl get sc # View storage classes

What the Operator Manages​

The Crusoe Slurm Operator (CSO) continuously reconciles certain resources to keep your cluster in a healthy state. Understanding what CSO manages helps you know what's safe to modify and what will be reverted.

Reconciliation Reference​

ResourceManaged by CSO?Safe to Modify?Notes
SlurmUser CRsNo — customer-ownedYesThis is the intended way to manage users
SlurmUserGroup CRsNo — customer-ownedYesThis is the intended way to manage groups
SlurmClusterHealthCheck (-custom)Created once — never overwrittenYesAdd your own health/prolog/epilog checks here (see Node Health Checks)
SlurmClusterHealthCheck (-defaults)Yes — seeded and owned by CSONoCrusoe-provided node health checks
Slinky CRDs (Controller, LoginSet, NodeSet)Yes — full overwrite on every reconcileNoChanges will be reverted automatically
gres-config ConfigMapSeeded once — your edits are keptYesSets the GPU detection backend. Defaults to AutoDetect=nvml
cgroup-config ConfigMapSeeded once — your edits are keptYesHolds cgroup.conf. An admission policy rejects edits that disable the cgroup plugin
plugstack-config ConfigMapYes — reconciled every cycleNoLoads the Pyxis SPANK plugin via plugstack.conf
Auth Secrets (slurm-auth, jwt-auth)Create-once — never regeneratedDo not modifyBreaking these breaks cluster authentication
nsscache ConfigMapYes — regenerated on user changesNoAuto-managed from SlurmUser CRs
ssh-keys SecretYes — regenerated on user changesNoAuto-managed from SlurmUser CRs
topology.conf ConfigMapCreated by CSO, then managed by TopographNoTopograph updates this automatically based on network topology
Home PVCValidated but not overwritten after creationStorage size only (increase)Spec is immutable after initial creation
StorageClassesYes — reconciled every cycleNoProvisioner and settings are fixed
Your own resourcesNeverYesCSO ignores any resources it doesn't own

A ConfigMap marked seeded once keeps your edits. To return one to the Crusoe default, delete its key. CSO restores the default file on the next reconcile.

info

Key takeaway: You edit the SlurmUser and SlurmUserGroup CRs (see User Management), the SlurmClusterHealthCheck -custom object (see Node Health Checks), and the ConfigMaps marked seeded once. Everything else is either managed by CSO or managed via the Crusoe CLI/API.

Configuration Defaults​

CSO writes two configuration files with Crusoe defaults: the managed sections of slurm.conf and the cgroup.conf file. This section lists what they contain and how your own settings interact with them.

slurm.conf​

Your slurm.conf additions live in spec.extraConf of the Controller CR, above two managed sections that CSO rewrites on every reconcile (see Managing Partitions for an editing example). On a cluster with L40S nodes the managed sections look like this:

# ===========================HC-HOOKS-START===========================
HealthCheckProgram=/opt/crusoe/healthcheck/slurm-healthcheck.sh
HealthCheckInterval=300
HealthCheckNodeState=IDLE,CYCLE
Prolog=/opt/crusoe/healthcheck/slurm-prolog.sh
PrologFlags=Contain,Alloc
Epilog=/opt/crusoe/healthcheck/slurm-epilog.sh
# ============================HC-HOOKS-END============================

# THE FOLLOWING SETTINGS ARE AUTOMATICALLY INJECTED BY CRUSOE SLURM OPERATOR
# ===============================START======================================
SlurmctldDebug=debug5
SlurmdDebug=debug5

MetricsType=metrics/openmetrics
SlurmctldParameters=enable_configless,enable_stepmgr,requeue_setup_env_fail
SlurmdParameters=numa_node_as_socket
JobRequeue=1
PreemptType=preempt/partition_prio
PreemptMode=REQUEUE
PropagateResourceLimits=NONE
DefCpuPerGpu=8
TopologyPlugin=topology/tree
TopologyParam=dragonfly

PartitionName=all Nodes=ALL Default=YES MaxTime=UNLIMITED State=UP
# ================================END=======================================
SettingWhy it is set
HealthCheckProgram, Prolog, Epilog and their parametersWire the Crusoe node health checks into Slurm (see Node Health Checks).
SlurmctldParametersenable_configless lets the worker and login pods fetch their configuration from the controller; enable_stepmgr is required with the cgroup plugin; requeue_setup_env_fail requeues a job whose node failed to set up its environment. If you set SlurmctldParameters yourself, keep all three tokens.
SlurmdParameters=numa_node_as_socketEmitted only on clusters with L40S nodes. Required for the L40S slurmd to register. If you set SlurmdParameters yourself on such a cluster, include numa_node_as_socket.
JobRequeue, PreemptType, PreemptModeJobs are requeued when a node fails or when a higher-priority partition preempts them.
PropagateResourceLimits=NONELogin-node ulimit values are not propagated into jobs.
DefCpuPerGpuCPUs a job gets per GPU when it requests none explicitly. Derived from the GPU instance types in the cluster (CPUs per GPU of the instance type); a cluster that mixes GPU types gets the smallest ratio.
TopologyPlugin, TopologyParamMatch the topology that Topograph writes into topology.conf.
PartitionName=allA default partition over every node. If you define a partition with Default=YES, CSO drops Default=YES from this one.

Your settings win. Every line in the managed sections is a default. If your part of extraConf assigns the same key, CSO omits its own line and leaves a comment in its place:

# CSO default (overridden above): DefCpuPerGpu=8

Slurm then sees only your assignment. The same comments appear in the rendered file on the controller (/etc/slurm/slurm.conf) and on every node (/run/slurm/conf/slurm.conf), CSO logs Customer extraConf overrides CSO slurm.conf defaults, and the SlurmCluster records an ExtraConfOverrides event whenever the set of overridden keys changes. scontrol show config shows only the effective values.

cgroup.conf​

The cgroup-config ConfigMap is seeded once with:

CgroupPlugin=cgroup/v2
IgnoreSystemd=yes
ConstrainRAMSpace=yes
ConstrainSwapSpace=yes
ConstrainDevices=yes
ConstrainCores=yes
SettingEffect on jobs
ConstrainRAMSpace, ConstrainSwapSpaceA job that exceeds its memory allocation is killed instead of affecting the node. Jobs that request no memory are allocated the whole node, so request --mem or --mem-per-gpu when several jobs should share a node.
ConstrainDevicesA job can open only the GPUs it was allocated. nvidia-smi inside the job lists only those GPUs, and CUDA_VISIBLE_DEVICES numbers them from 0. Applies to srun steps and Pyxis containers, not to containers started with docker run.
ConstrainCoresA job runs only on the CPU cores it was allocated.

IgnoreSystemd=yes is required for slurmd to run as a pod and must stay. The admission policy rejects edits that disable the cgroup plugin.

Because the file is seeded once, any changes made to the cgroup-config ConfigMap will persist, if there are additions you would like to add. To revert to the defaults, delete the cgroup.conf key from the cgroup-config ConfigMap; CSO rewrites it within seconds and the nodes pick it up without a restart.

Slurm Commands Reference​

Once connected to a login node via SSH, use standard Slurm commands:

CommandDescription
sinfoView cluster status and node information
sinfo -RView nodes in drain state with reasons
squeueView the job queue
sbatchSubmit a batch job
srunRun a job interactively
scancelCancel a job
scontrol show nodeView detailed node information
scontrol show configView Slurm configuration

Running NCCL Tests​

To validate multi-node GPU communication, run an NCCL all-reduce test. SSH into a login node and create the following script named nccl_test.batch. This example runs NCCL tests on H200 nodes. Different hardware types will have their own topo files. Note that you can find more test examples on your login node at /opt/examples:

#!/bin/bash

#SBATCH --job-name=nccl_tests
#SBATCH --nodes=<number-of-nodes>
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --time=20:00
#SBATCH --output="%x_%j.out"
#SBATCH --exclusive

export NCCL_TOPO_FILE=/etc/crusoe/nccl_topo/h200-141gb-sxm-ib-cloud-hypervisor.xml
export NCCL_SOCKET_IFNAME=eth0
export NCCL_IB_HCA="mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1"
export NCCL_IB_MERGE_VFS=0
export NCCL_DEBUG=WARN

export OMPI_MCA_coll_hcoll_enable=0
export PMIX_MCA_gds='^ds12'

export UCX_NET_DEVICES="mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1"

srun --mpi=pmix /opt/nccl-tests/build/all_reduce_perf -b 2G -e 32G -f 2
note

Update the NCCL_TOPO_FILE path to match your GPU type. The example above is for H200 nodes. UCX_NET_DEVICES is required on every GPU type: the InfiniBand device list on SXM nodes, eth0 on L40S nodes.

L40S nodes

L40S nodes have no InfiniBand and no topology file, so drop the NCCL_TOPO_FILE, NCCL_IB_HCA and NCCL_IB_MERGE_VFS lines and set UCX_NET_DEVICES=eth0. On l40s-48gb.10x nodes also set NCCL_CUMEM_ENABLE=0; without it, any NCCL job that uses all ten GPUs fails to start (the variable is harmless on smaller instance types). Size --ntasks-per-node and --gpus-per-node to the instance type's GPU count, and keep -e at or below 16G on the 48 GB GPUs. Ready-made L40S scripts are on the login node at /opt/examples/l40s-pcie/.

Submit the test:

sbatch nccl_test.batch

Monitor with squeue, and check the output file nccl_tests_<job_id>.out once complete.

To connect interactively to a worker node:

srun --pty bash # Any available worker
srun --nodelist=<worker-pod-name> --pty bash # A specific worker

Health Checks, Prolog, and Epilog​

Managed Slurm ships a built-in node health-check suite (periodic, prolog, and epilog) and lets you add your own health, prolog, and epilog checks through the SlurmClusterHealthCheck custom resource. See Node Health Checks.

Automatic Hardware Remediation​

Managed Slurm clusters run on Crusoe Managed Kubernetes with AutoClusters enabled. AutoClusters automatically detects critical hardware failures — such as a GPU or HCA falling off the bus — and remediates them without manual intervention.

What Happens During Remediation​

  1. Crusoe's monitoring pipeline detects a hardware issue on a worker node
  2. The affected Slurm node is set to DOWN, which immediately cancels any running job on that node — the job process receives a SIGTERM signal before being terminated
  3. You have up to 2 minutes to handle the SIGTERM (save checkpoints, flush logs, etc.) before the node is replaced
  4. The cancelled job is automatically requeued (JobRequeue=1 is enabled by default) and runs on a healthy node

Handling SIGTERM in Your Jobs​

When a node goes down, Slurm sends SIGTERM to your job process. You can use trap to catch this signal and perform cleanup before the job is cancelled. For example:

#!/bin/bash
#SBATCH --job-name=my-training-job
#SBATCH --nodes=1
#SBATCH --output=%x-%j.out

trap 'echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] SIGTERM received — saving checkpoint"; save_checkpoint' SIGTERM

run_training &
wait $!

Next Steps​

Support​

If you encounter issues or need assistance, contact Crusoe Cloud Support.