Support Access
Support access allows Crusoe support engineers to access your CMK cluster for troubleshooting and maintenance purposes. When enabled, authorized Crusoe support personnel can perform diagnostic tasks, investigate issues, and provide technical assistance directly within your cluster.
This feature gives you control over when and how Crusoe support can access your infrastructure, ensuring you maintain visibility and control over cluster access.
Enabling Support Access
Support access is disabled by default on all clusters. You can enable it at any time using the CLI command. For existing clusters, you will also need to install a Helm chart.
Step 1: Enable Support Access via CLI
Enable support access on any cluster using the CLI:
crusoe kubernetes clusters support-access enable <cluster-name> \
--enabled-roles readonly,operator
You can choose which level of access to grant:
- readonly - View-only access to cluster resources for diagnostics (default)
- operator - Additional permissions for troubleshooting and configuration assistance
- readonly,operator - Both roles (recommended for comprehensive support)
Note that operator access will automatically enable readonly; so the last two are equivalent.
To enable only readonly access:
crusoe kubernetes clusters support-access enable <cluster-name>
Step 2: Install the Helm Chart (For Existing Clusters)
If you have an existing cluster, you will also need to install a Helm chart that provides the necessary roles and permissions:
- Add the Crusoe support roles Helm repository:
helm repo add crusoe-support https://crusoecloud.github.io/crusoe-support-roles-helm-charts/charts
helm repo update
- Install the support access chart:
helm install crusoe-support-roles crusoe-support/crusoe-support-roles \
--namespace crusoe-system \
--create-namespace
The Helm chart is available at: https://github.com/crusoecloud/crusoe-support-roles-helm-charts
The Helm chart is required for existing clusters to enable the necessary cluster roles and role bindings in the crusoe-system namespace. New clusters created after this feature was released will have this chart pre-installed and only require the CLI command (Step 1).
By default, CMK clusters include a firewall rule (cmk-cp-api-access-cp-<abcd1234>) in the associated project that allows all inbound traffic to port 443 on the control plane nodes. If you have restricted this access, you must also allow inbound connections from Crusoe's internal VPN exit node (4.7.95.218) so that support engineers can reach the Kubernetes API server.
Disabling Support Access
If you need to revoke support access to your cluster, you can do so at any time.
Disable via CLI
Disable support access using the CLI:
crusoe kubernetes clusters support-access disable <cluster-name>
Disabling via CLI prevents new or renewed support access sessions but does not immediately revoke active access. For immediate revocation, see the Helm chart removal option below.
Uninstall the Helm Chart (For Immediate Revocation)
You can uninstall the Helm chart to immediately disable all support access:
helm uninstall crusoe-support-roles --namespace crusoe-system
Removing the Helm chart will immediately revoke support access by removing the necessary cluster roles and role bindings from your cluster. Use this method when you need to ensure support access is terminated right away.
When to use each method:
- CLI disable - Prevents future access while allowing current support sessions to complete
- Helm chart removal - Immediately terminates all support access (for existing clusters with the Helm chart installed)
- Both - Use both methods for comprehensive access revocation on existing clusters
- Remove all access - Also delete leftover diagnostic pods. See Removing All Support Access
Verifying Support Access Status
You can verify whether support access is currently enabled on your cluster:
crusoe kubernetes clusters support-access get <cluster-name>
This command displays the current support access configuration, including:
- Whether access is enabled
- Which roles are granted
- Currently active support access sessions with details about each session
Understanding the Output
Example output when support access is enabled with an active session:
Support Access Status: Enabled
Enabled Roles: SUPPORT_ACCESS_ROLE_READONLY
Active Requests: 1
Request 1:
Role: SUPPORT_ACCESS_ROLE_READONLY
Requestor ID: 00uix9g0n77KrcdMq5d7
Reason: This is a test!
Requested At: 2026-03-30T23:12:53Z
Request Expires At: 2026-04-01T03:12:53Z
Credential Expires At: 2026-04-01T01:46:18Z
Active Requests shows currently live support credentials, including:
- Role - The access level granted (READONLY or OPERATOR)
- Requestor ID - The identifier of the support engineer
- Reason - The stated purpose for the access request
- Requested At - When the access was granted
- Request Expires At - When the access will automatically expire
- Credential Expires At - When the latest certificate will expire
This transparency allows you to monitor exactly who has access to your cluster and why. Note that if support access is disabled to the corresponding role, the requests are no longer valid but already minted certificates will not be invalidated (because Kubernetes does not support certificate revocation). For immediate revocation, see instructions above.
Checking Helm Chart Installation
If you need to check whether the Helm chart is already installed on your cluster:
helm list --namespace crusoe-system
What Access Does Support Have?
The level of access granted depends on which roles you enable:
Readonly Role
With the readonly role, Crusoe support engineers can:
- View cluster resources and configurations
- Access logs and metrics for troubleshooting
- Inspect resource status and health
- Diagnose issues without making changes
Operator Role
With the operator role, Crusoe support engineers have additional capabilities to:
- Execute diagnostic commands
- Assist with cluster configuration
- Perform troubleshooting actions
- Help resolve operational issues
For full details of this access, you can reference the helm chart README at https://github.com/crusoecloud/crusoe-support-roles-helm-charts/blob/main/charts/crusoe-support-roles/README.md
What Support Cannot Access
Regardless of which roles are enabled, support access does not grant permission to:
- Modify or delete your workloads without coordination
- Access data within your application containers
- Make infrastructure changes outside of coordinated support engagements
- Access secrets or sensitive application data
Removing All Support Access
Disabling support access prevents new sessions, and uninstalling the Helm chart removes the cluster roles and role bindings that support relies on. However, a support engagement may have left behind workloads that retain access to nodes even after the RBAC roles are gone. Use this procedure when you want to ensure that no Crusoe personnel retain any path into the cluster: it revokes RBAC and cleans up any such workloads.
Step 1: Revoke RBAC Access
Use both methods described in Disabling Support Access:
-
Disable support access via the CLI to prevent new or renewed sessions:
crusoe kubernetes clusters support-access disable <cluster-name> -
Uninstall the Helm chart to immediately remove the support cluster roles and role bindings:
helm uninstall crusoe-support-roles --namespace crusoe-system
Already-minted certificates remain valid until they expire, because Kubernetes does not support certificate revocation. Removing the cluster roles and role bindings (Step 1, item 2) ensures those credentials can no longer authorize any action.
Step 2: Remove Node-Shell and Similar Pods
During troubleshooting, support engineers may run helper pods, such as node-shell pods, that mount host paths or open a shell onto a node. These pods can provide node-level access independent of the RBAC roles you removed in Step 1, so they must be deleted explicitly.
Crusoe support roles are scoped to the following Crusoe-managed namespaces. Sweep each of them for leftover diagnostic pods:
crusoe-systemkube-systemnvidia-gpu-operatornvidia-network-operatorkube-amd-gpukube-amd-networkslinkyslurm
-
Check each Crusoe-managed namespace for any such pods. For example:
kubectl get pods --namespace crusoe-systemTo check all of the namespaces at once:
for ns in crusoe-system kube-system nvidia-gpu-operator nvidia-network-operator kube-amd-gpu kube-amd-network slinky slurm; doecho "== $ns =="kubectl get pods --namespace "$ns"doneLook for pods with names like
node-shell-*, debug/shell pods, or any pod you do not recognize as part of normal cluster operation. -
Delete any node-shell or similar diagnostic pods you find:
kubectl delete pod <pod-name> --namespace <namespace>
Only delete pods that you have confirmed are diagnostic or support-related. Deleting pods that are part of normal cluster operation may disrupt your workloads.
Step 3: Verify
Confirm that support access is fully disabled and no active sessions remain:
crusoe kubernetes clusters support-access get <cluster-name>
Confirm the Helm chart is no longer installed:
helm list --namespace crusoe-system
Confirm no node-shell or diagnostic pods remain in any of the Crusoe-managed namespaces:
for ns in crusoe-system kube-system nvidia-gpu-operator nvidia-network-operator kube-amd-gpu kube-amd-network slinky slurm; do
echo "== $ns =="
kubectl get pods --namespace "$ns"
done
Getting Help
If you encounter issues enabling or disabling support access, have questions about what level of access is granted, or need help identifying which pods are safe to remove, contact support.