Get started
This page covers the setup steps shared across Command Center features: installing the Crusoe Watch Agent, generating a monitoring token, and the access methods available.
Crusoe Watch Agent
The Crusoe Watch Agent collects telemetry from your infrastructure, enabling metrics, logs, health status, and alerting features. It is enabled by default when you create a VM or CMK cluster.
- CMK Clusters
- VMs
CMK version 1.33.4-cmk.31 and later automatically install the agent at cluster creation. For earlier versions, or to upgrade an existing installation, follow these steps.
Step 1: Switch kubectl context
Make sure the NVIDIA GPU Operator add-on is enabled on your cluster (required for NVIDIA GPU accelerated instances) or the AMD GPU Operator add-on is enabled (required for AMD GPU accelerated instances), then switch your kubectl context to the target cluster:
crusoe kubernetes clusters get-credentials <cluster-name> --project-id <project-id>
Step 2: Install or upgrade the agent
- Install
- Upgrade
helm repo add crusoe-watch-agent https://crusoecloud.github.io/crusoe-watch-agent/k8s/helm-charts
helm repo update
helm install crusoe-watch-agent crusoe-watch-agent/crusoe-watch-agent --namespace crusoe-system
helm repo update
helm upgrade crusoe-watch-agent crusoe-watch-agent/crusoe-watch-agent --namespace crusoe-system
Step 3: Verify the installation
kubectl get pods -n crusoe-system
Supported Kubernetes versions: 1.32, 1.33, 1.34, and 1.35. Run crusoe kubernetes versions list to see the latest available patch versions.
When you create a VM, the Crusoe Watch Agent is installed by default and a monitoring token is provisioned automatically. To turn the agent off at creation time, see Opt out during VM creation.
Install mode: By default, the agent runs as Docker containers. On NVIDIA GPU and non-GPU VMs, you can choose to run it natively as systemd services—see Choose an install mode.
To add the agent to an existing VM: Generate a monitoring token first (see Generate a monitoring token), then follow the installation instructions in the Crusoe Watch Agent repository.
Supported images:
- NVIDIA GPU:
ubuntu22.04-nvidia-sxm-docker,ubuntu22.04-nvidia-pcie-docker - AMD GPU:
ubuntu22.04with ROCm 6.2.0 or later - Non-GPU:
ubuntu22.04,ubuntu24.04
CMK Capability by Helm Chart Version
| Capability | Minimum Helm Chart Version |
|---|---|
| NVIDIA GPU metrics | 0.2.6 |
| AMD GPU metrics | 0.2.7 |
| Managed logs (JournalD, kubelet, container) | 0.3.2 |
| Slurm metrics | 0.3.11 |
| Custom pod metrics | 0.3.11 |
| NVIDIA bug reports | 0.3.12 |
| AMD GPU logs | 0.3.19 |
| AMD bug reports | 0.3.19 |
| Disk, NVMe, and object store metrics | 0.3.29 |
VM Capability by Agent Version
| Capability | Minimum VM Agent Version |
|---|---|
| NVIDIA GPU metrics | 1.0.0 |
| Managed logs (JournalD) | 1.0.1 |
| NVIDIA bug reports | 1.0.3 |
| AMD GPU metrics | 1.0.3 |
| AMD GPU monitoring and logs | 1.0.5 |
| AMD bug reports | 1.0.5 |
| Disk, NVMe, and object store metrics | 1.0.13 |
Choose an install mode
You can choose how the Crusoe Watch Agent runs on a VM:
| Install mode | How the agent runs |
|---|---|
| Docker (default) | The agent and its exporters run as Docker containers. |
| Native | The agent and its exporters run as systemd services directly on the host, without Docker. |
Choose native mode if you don't want the agent to depend on Docker, or if you prefer to manage host services with systemd.
Native mode supports NVIDIA GPU VMs and non-GPU VMs. AMD GPU VMs support Docker mode only.
Install the agent in native mode at VM creation
- CLI
- UI
- Terraform
Pass --crusoe-watch-agent-install-mode=native when you create the VM:
crusoe compute vms create \
--name my-vm \
--type a100-80gb.1x \
--location us-east1-a \
--image ubuntu22.04-nvidia-sxm-docker \
--crusoe-watch-agent-install-mode=native
- In the Console, go to Compute → Instances → Create Instance.
- Under Additional Options, keep Enable Observability turned on.
- Set Install Mode to Native.
Set crusoe_watch_agent_install_mode = "native" on the crusoe_compute_instance or crusoe_compute_instance_by_template resource. This setting applies only when install_crusoe_watch_agent is true, and changing it replaces the VM.
resource "crusoe_compute_instance" "my_vm" {
# ...
install_crusoe_watch_agent = true
crusoe_watch_agent_install_mode = "native"
}
If you don't set an install mode, the agent is installed in Docker mode.
Install the agent in native mode on an existing VM
To install the agent in native mode on an existing VM, pass --no-docker to the installer:
sudo ./crusoe_watch_agent.sh install --no-docker
The installer saves the install mode, so later upgrade and uninstall runs use the same mode without the flag.
Verify the install mode
Check which install mode a VM uses:
sudo cat /etc/crusoe/secrets/.install-mode
In native mode, check the agent's systemd service:
sudo systemctl status crusoe-watch-agent.service
On NVIDIA GPU VMs, also check the DCGM exporter service:
sudo systemctl status crusoe-dcgm-exporter.service
Opt out during VM creation
The Crusoe Watch Agent is installed by default at VM creation. If you don't want telemetry collected, opt out using any of the following:
- CLI
- UI
- Terraform
Pass --install-watch-agent=false when you create the VM:
crusoe compute vms create \
--name my-vm \
--type a100-80gb.1x \
--location us-east1-a \
--image ubuntu22.04-nvidia-sxm-docker \
--install-watch-agent=false
- In the Console, go to Compute → Instances → Create Instance.
- Under Additional Options, turn off Enable Observability.
Set install_crusoe_watch_agent = false on the crusoe_compute_instance or crusoe_compute_instance_by_template resource.
resource "crusoe_compute_instance" "my_vm" {
# ...
install_crusoe_watch_agent = false
}
Opting out means Crusoe support will not have visibility into your infrastructure health, utilization, or performance data, which may limit our ability to proactively identify issues or provide the in-depth assistance you need.
Disabling the Agent
You can uninstall the Crusoe Watch Agent or disable some of its capabilities if needed. Keep in mind that turning it off means Crusoe support will not have visibility into your infrastructure health, utilization, or performance data, which may limit our ability to proactively identify issues or provide the in-depth assistance you need.
CMK (full uninstall):
helm uninstall crusoe-watch-agent -n crusoe-system
VM (full uninstall):
- Docker mode
- Native mode
docker stop crusoe-watch-agent
docker rm crusoe-watch-agent
Run the installer's uninstall command. It detects the native install and removes the agent's services and packages:
sudo ./crusoe_watch_agent.sh uninstall
CMK and VM (collect only metrics or only logs): Create a values.yaml:
# Collect only metrics (disable logs)
metrics:
enabled: true
logs:
enabled: false
Apply for CMK with:
helm upgrade crusoe-watch-agent crusoe-watch-agent/crusoe-watch-agent --namespace crusoe-system -f values.yaml
Apply for VMs — download and re-run the installer with the same flags, passing the values file:
bash crusoe_watch_agent.sh --values values.yaml
Generate a monitoring token
A monitoring token is required to query metrics or logs via API, import data into Grafana, or use Telemetry Conduit. When the agent is installed at VM creation, a token is automatically provisioned for the agent; you still need to generate one separately to query the API manually.
crusoe monitoring tokens create
Store the token securely. You cannot retrieve it later.
If the token contains special characters such as $, reference it from a Kubernetes Secret using secretKeyRef to avoid parsing errors in Helm deployments.
Access Methods
Console
To access metrics, logs, and health status directly in the Crusoe Cloud Console:
- Metrics and health status: Navigate to Command Center in the left navigation, and then select Infrastructure Overview.
- Project-level logs: Navigate to Command Center in the left navigation, and then select Managed Logs.
- Resource-specific metrics and logs: Use a specific VM's Metrics or Logs tab under Compute, or a CMK cluster's tab under Orchestration.
API
Metrics (PromQL): Query metrics via the Prometheus-compatible endpoint:
https://api.cloud.crusoe.ai/v1/projects/<project-id>/metrics/timeseries
Logs (LogsQL): Query logs via:
https://api.crusoecloud.com/v1/projects/<project-id>/logs
Authenticate with your monitoring token as a Bearer token. See Logs for the full endpoint reference.
Grafana
Add a Prometheus data source to your Grafana instance pointing to the metrics API endpoint, with an Authorization: Bearer <token> HTTP header.
Pre-built Grafana dashboard templates for CMK and Managed Slurm clusters are available in the Crusoe solutions library. Templates cover GPU utilization, InfiniBand fabric health, power draw, XID error tracking, Slurm job performance, storage, and network.
Telemetry Conduit
Export metrics continuously to Grafana, Datadog, or Splunk via a Prometheus-compatible scraping endpoint. See Telemetry Conduit for setup.
Crusoe MCP
You can access metrics and logs data collected by the Command Center via Crusoe MCP. See the Crusoe MCP page for setup instructions.