Advanced: Kubernetes Operations
Managed Slurm clusters run on Crusoe Managed Kubernetes (CMK). While the CLI and API handle most operations, you can also interact with the cluster directly via kubectl for advanced configuration, troubleshooting, and user management.
Accessing Your Cluster via kubectl
Configure kubectl to connect to your cluster:
crusoe kubernetes clusters get-credentials <cluster-name>
Verify the connection:
kubectl cluster-info
Viewing Cluster State
Slurm Custom Resources
The Crusoe Slurm Operator (CSO) uses four Custom Resource types to manage your cluster. When you create a Slurm cluster through the Slurm UI or crusoe slurm CLI, these CRDs will be created, controlled, and reconciled by Crusoe in the slurm namespace.
kubectl get slurmclusters -n slurm # Clusters
kubectl get nodesets -n slurm # Node sets
kubectl get slurmusers -n slurm # All users
kubectl get slurmusergroups -n slurm # All groups
For detailed status with conditions:
kubectl describe slurmclusters <name> -n slurm
Use kubectl explain for field-level documentation on any Custom Resource. The CRDs have built-in descriptions for every field:
kubectl explain slurmclusters.spec
kubectl explain slurmusers.spec
kubectl explain slurmusergroups.spec
kubectl explain nodesets.spec
Cluster Phase and Conditions
The SlurmCluster status includes a phase field (possible values: Provisioning, Installing, Ready, NotReady, CreateFailed, Deleting, DeleteFailed) and detailed conditions for each component. Use kubectl describe to see what's happening if your cluster isn't healthy:
kubectl describe slurmclusters <name> -n slurm
Look for conditions like:
ControllerReady— Slurm controller pod is runningLoginReady— Login nodes are runningSlinkyReady— Slinky Helm chart is installedCertManagerReady— cert-manager is installedLoadBalancerReady— Load balancer controller is installedTopographReady— Topology discovery is running
Node Set Readiness
The NodeSet status shows a readyReplicas field in "ready/total" format:
kubectl get nodesets -n slurm
NAME READY AGE
slurm-worker-node-set 2/2 1h
Storage Configuration
Managed Slurm uses a shared filesystem for the /home directory, mounted across all login and worker nodes. This is backed by a PersistentVolumeClaim (PVC) using the Crusoe CSI driver.
StorageClass
The operator automatically creates a StorageClass named <cluster-name>-crusoe-csi-driver-fs-sc with:
- Provisioner:
fs.csi.crusoe.ai - Volume binding mode: WaitForFirstConsumer
- Volume expansion: Enabled
Viewing Storage
kubectl get pvc -n slurm # View persistent volume claims
kubectl get sc # View storage classes
What the Operator Manages
The Crusoe Slurm Operator (CSO) continuously reconciles certain resources to keep your cluster in a healthy state. Understanding what CSO manages helps you know what's safe to modify and what will be reverted.
Reconciliation Reference
| Resource | Managed by CSO? | Safe to Modify? | Notes |
|---|---|---|---|
| SlurmUser CRs | No — customer-owned | Yes | This is the intended way to manage users |
| SlurmUserGroup CRs | No — customer-owned | Yes | This is the intended way to manage groups |
SlurmClusterHealthCheck (-custom) | Created once — never overwritten | Yes | Add your own health/prolog/epilog checks here (see Node Health Checks) |
SlurmClusterHealthCheck (-defaults) | Yes — seeded and owned by CSO | No | Crusoe-provided node health checks |
| Slinky CRDs (Controller, LoginSet, NodeSet) | Yes — full overwrite on every reconcile | No | Changes will be reverted automatically |
| gres-config ConfigMap | Seeded once — your edits are kept | Yes | Sets the GPU detection backend. Defaults to AutoDetect=nvml |
| cgroup-config ConfigMap | Seeded once — your edits are kept | Yes | Holds cgroup.conf. An admission policy rejects edits that disable the cgroup plugin |
| plugstack-config ConfigMap | Yes — reconciled every cycle | No | Loads the Pyxis SPANK plugin via plugstack.conf |
| Auth Secrets (slurm-auth, jwt-auth) | Create-once — never regenerated | Do not modify | Breaking these breaks cluster authentication |
| nsscache ConfigMap | Yes — regenerated on user changes | No | Auto-managed from SlurmUser CRs |
| ssh-keys Secret | Yes — regenerated on user changes | No | Auto-managed from SlurmUser CRs |
| topology.conf ConfigMap | Created by CSO, then managed by Topograph | No | Topograph updates this automatically based on network topology |
| Home PVC | Validated but not overwritten after creation | Storage size only (increase) | Spec is immutable after initial creation |
| StorageClasses | Yes — reconciled every cycle | No | Provisioner and settings are fixed |
| Your own resources | Never | Yes | CSO ignores any resources it doesn't own |
A ConfigMap marked seeded once keeps your edits. To return one to the Crusoe default, delete its key. CSO restores the default file on the next reconcile.
Key takeaway: You edit the SlurmUser and SlurmUserGroup CRs (see User Management), the SlurmClusterHealthCheck -custom object (see Node Health Checks), and the ConfigMaps marked seeded once. Everything else is either managed by CSO or managed via the Crusoe CLI/API.
Configuration Defaults
CSO writes two configuration files with Crusoe defaults: the managed sections of slurm.conf and the cgroup.conf file. This section lists what they contain and how your own settings interact with them.
slurm.conf
Your slurm.conf additions live in spec.extraConf of the Controller CR, above two managed sections that CSO rewrites on every reconcile (see Managing Partitions for an editing example). On a cluster with L40S nodes the managed sections look like this:
# ===========================HC-HOOKS-START===========================
HealthCheckProgram=/opt/crusoe/healthcheck/slurm-healthcheck.sh
HealthCheckInterval=300
HealthCheckNodeState=IDLE,CYCLE
Prolog=/opt/crusoe/healthcheck/slurm-prolog.sh
PrologFlags=Contain,Alloc
Epilog=/opt/crusoe/healthcheck/slurm-epilog.sh
# ============================HC-HOOKS-END============================
# THE FOLLOWING SETTINGS ARE AUTOMATICALLY INJECTED BY CRUSOE SLURM OPERATOR
# ===============================START======================================
SlurmctldDebug=debug5
SlurmdDebug=debug5
MetricsType=metrics/openmetrics
SlurmctldParameters=enable_configless,enable_stepmgr,requeue_setup_env_fail
SlurmdParameters=numa_node_as_socket
JobRequeue=1
PreemptType=preempt/partition_prio
PreemptMode=REQUEUE
PropagateResourceLimits=NONE
DefCpuPerGpu=8
TopologyPlugin=topology/tree
TopologyParam=dragonfly
PartitionName=all Nodes=ALL Default=YES MaxTime=UNLIMITED State=UP
# ================================END=======================================
| Setting | Why it is set |
|---|---|
HealthCheckProgram, Prolog, Epilog and their parameters | Wire the Crusoe node health checks into Slurm (see Node Health Checks). |
SlurmctldParameters | enable_configless lets the worker and login pods fetch their configuration from the controller; enable_stepmgr is required with the cgroup plugin; requeue_setup_env_fail requeues a job whose node failed to set up its environment. If you set SlurmctldParameters yourself, keep all three tokens. |
SlurmdParameters=numa_node_as_socket | Emitted only on clusters with L40S nodes. Required for the L40S slurmd to register. If you set SlurmdParameters yourself on such a cluster, include numa_node_as_socket. |
JobRequeue, PreemptType, PreemptMode | Jobs are requeued when a node fails or when a higher-priority partition preempts them. |
PropagateResourceLimits=NONE | Login-node ulimit values are not propagated into jobs. |
DefCpuPerGpu | CPUs a job gets per GPU when it requests none explicitly. Derived from the GPU instance types in the cluster (CPUs per GPU of the instance type); a cluster that mixes GPU types gets the smallest ratio. |
TopologyPlugin, TopologyParam | Match the topology that Topograph writes into topology.conf. |
PartitionName=all | A default partition over every node. If you define a partition with Default=YES, CSO drops Default=YES from this one. |
Your settings win. Every line in the managed sections is a default. If your part of extraConf assigns the same key, CSO omits its own line and leaves a comment in its place:
# CSO default (overridden above): DefCpuPerGpu=8
Slurm then sees only your assignment. The same comments appear in the rendered file on the controller (/etc/slurm/slurm.conf) and on every node (/run/slurm/conf/slurm.conf), CSO logs Customer extraConf overrides CSO slurm.conf defaults, and the SlurmCluster records an ExtraConfOverrides event whenever the set of overridden keys changes. scontrol show config shows only the effective values.
cgroup.conf
The cgroup-config ConfigMap is seeded once with:
CgroupPlugin=cgroup/v2
IgnoreSystemd=yes
ConstrainRAMSpace=yes
ConstrainSwapSpace=yes
ConstrainDevices=yes
ConstrainCores=yes
| Setting | Effect on jobs |
|---|---|
ConstrainRAMSpace, ConstrainSwapSpace | A job that exceeds its memory allocation is killed instead of affecting the node. Jobs that request no memory are allocated the whole node, so request --mem or --mem-per-gpu when several jobs should share a node. |
ConstrainDevices | A job can open only the GPUs it was allocated. nvidia-smi inside the job lists only those GPUs, and CUDA_VISIBLE_DEVICES numbers them from 0. Applies to srun steps and Pyxis containers, not to containers started with docker run. |
ConstrainCores | A job runs only on the CPU cores it was allocated. |
IgnoreSystemd=yes is required for slurmd to run as a pod and must stay. The admission policy rejects edits that disable the cgroup plugin.
Because the file is seeded once, any changes made to the cgroup-config ConfigMap will persist, if there are additions you would like to add. To revert to the defaults, delete the cgroup.conf key from the cgroup-config ConfigMap; CSO rewrites it within seconds and the nodes pick it up without a restart.
Slurm Commands Reference
Once connected to a login node via SSH, use standard Slurm commands:
| Command | Description |
|---|---|
sinfo | View cluster status and node information |
sinfo -R | View nodes in drain state with reasons |
squeue | View the job queue |
sbatch | Submit a batch job |
srun | Run a job interactively |
scancel | Cancel a job |
scontrol show node | View detailed node information |
scontrol show config | View Slurm configuration |
Running NCCL Tests
To validate multi-node GPU communication, run an NCCL all-reduce test. SSH into a login node and create the following script named nccl_test.batch. This example runs NCCL tests on H200 nodes. Different hardware types will have their own topo files. Note that you can find more test examples on your login node at /opt/examples:
#!/bin/bash
#SBATCH --job-name=nccl_tests
#SBATCH --nodes=<number-of-nodes>
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --time=20:00
#SBATCH --output="%x_%j.out"
#SBATCH --exclusive
export NCCL_TOPO_FILE=/etc/crusoe/nccl_topo/h200-141gb-sxm-ib-cloud-hypervisor.xml
export NCCL_SOCKET_IFNAME=eth0
export NCCL_IB_HCA="mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1"
export NCCL_IB_MERGE_VFS=0
export NCCL_DEBUG=WARN
export OMPI_MCA_coll_hcoll_enable=0
export PMIX_MCA_gds='^ds12'
export UCX_NET_DEVICES="mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1,mlx5_8:1"
srun --mpi=pmix /opt/nccl-tests/build/all_reduce_perf -b 2G -e 32G -f 2
Update the NCCL_TOPO_FILE path to match your GPU type. The example above is for H200 nodes. UCX_NET_DEVICES is required on every GPU type: the InfiniBand device list on SXM nodes, eth0 on L40S nodes.
L40S nodes have no InfiniBand and no topology file, so drop the NCCL_TOPO_FILE, NCCL_IB_HCA and NCCL_IB_MERGE_VFS lines and set UCX_NET_DEVICES=eth0. On l40s-48gb.10x nodes also set NCCL_CUMEM_ENABLE=0; without it, any NCCL job that uses all ten GPUs fails to start (the variable is harmless on smaller instance types). Size --ntasks-per-node and --gpus-per-node to the instance type's GPU count, and keep -e at or below 16G on the 48 GB GPUs. Ready-made L40S scripts are on the login node at /opt/examples/l40s-pcie/.
Submit the test:
sbatch nccl_test.batch
Monitor with squeue, and check the output file nccl_tests_<job_id>.out once complete.
To connect interactively to a worker node:
srun --pty bash # Any available worker
srun --nodelist=<worker-pod-name> --pty bash # A specific worker
Health Checks, Prolog, and Epilog
Managed Slurm ships a built-in node health-check suite (periodic, prolog, and epilog) and lets you add your own health, prolog, and epilog checks through the SlurmClusterHealthCheck custom resource. See Node Health Checks.
Automatic Hardware Remediation
Managed Slurm clusters run on Crusoe Managed Kubernetes with AutoClusters enabled. AutoClusters automatically detects critical hardware failures — such as a GPU or HCA falling off the bus — and remediates them without manual intervention.
What Happens During Remediation
- Crusoe's monitoring pipeline detects a hardware issue on a worker node
- The affected Slurm node is set to DOWN, which immediately cancels any running job on that node — the job process receives a SIGTERM signal before being terminated
- You have up to 2 minutes to handle the SIGTERM (save checkpoints, flush logs, etc.) before the node is replaced
- The cancelled job is automatically requeued (
JobRequeue=1is enabled by default) and runs on a healthy node
Handling SIGTERM in Your Jobs
When a node goes down, Slurm sends SIGTERM to your job process. You can use trap to catch this signal and perform cleanup before the job is cancelled. For example:
#!/bin/bash
#SBATCH --job-name=my-training-job
#SBATCH --nodes=1
#SBATCH --output=%x-%j.out
trap 'echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] SIGTERM received — saving checkpoint"; save_checkpoint' SIGTERM
run_training &
wait $!
Next Steps
- Quickstart — Create your first Slurm cluster
- User Management — Add users and groups, manage partitions
- Managing Partitions — Create and manage partitions in your Slurm cluster
- Node Health Checks — Built-in health checks and adding your own health/prolog/epilog checks
- Slurm Metrics — Monitor cluster health and performance
- For Slurm command reference, see the official Slurm documentation
Support
If you encounter issues or need assistance, contact Crusoe Cloud Support.