Control plane logs

Control plane logs let you see who did what in your cluster and diagnose problems in the components that run it. Crusoe collects Kubernetes API server audit logs and control plane component logs for every CMK v2 cluster and makes them available in Managed Logs.

Control plane logs complement the node-level metrics and JournalD logs described in CMK telemetry. Unlike node telemetry, they're on by default: you don't need to install an agent, deploy a log collector, or change any cluster setting.

Availability

Control plane logs are available on every CMK v2 cluster, in every location where CMK v2 is available. To check a cluster's generation, see Check whether a cluster is CMK v2.

CMK v1 clusters

Control plane logs aren't available on CMK v1 clusters. CMK v1 clusters continue to send node-level JournalD logs, including kubelet and container runtime logs, through the Crusoe Watch Agent.

Log sources

Each control plane log entry carries a log_source field that identifies where it came from:

`log_source`	Contents
`k8s_audit`	Kubernetes API server audit events. Each event records one request to the API server: who made it, the verb, the target resource, and the result.
`k8s_apiserver`	Logs from `kube-apiserver`.
`k8s_controller_manager`	Logs from `kube-controller-manager`.
`k8s_etcd`	Logs from `etcd`.
`k8s_konnectivity`	Logs from `konnectivity-server`, which carries traffic between the control plane and your nodes.
`k8s_scheduler`	Logs from `kube-scheduler`.

Every entry also carries crusoe_cluster_id, the ID of the cluster it came from, so you can scope searches to a single cluster.

Log fields

Control plane log entries are structured JSON. The following fields are common to all control plane log sources:

Field	Description
`_msg`	Log message. For `k8s_audit`, a one-line summary in the form `User: <username>; <verb> <resource>; decision=<allow or forbid>`.
`_time`	Time of the event. For `k8s_audit`, the event's `stageTimestamp`.
`level`	Severity. Component logs use the component's own severity. Every `k8s_audit` entry has the severity `info`.
`log_source`	Log source (see Log sources).
`crusoe_cluster_id`	ID of the CMK cluster.
`crusoe_project_id`	ID of the project.
`payload.message`	The original log line, unmodified. For `k8s_audit`, the full audit event as a JSON string.
`payload.parsed`	The original log line parsed into fields (see below).

Audit event fields

For k8s_audit entries, payload.parsed holds the complete Kubernetes audit event. Commonly used fields include:

Field	Description
`payload.parsed.auditID`	Unique ID of the request
`payload.parsed.stage`	Request stage, such as `RequestReceived` or `ResponseComplete`
`payload.parsed.verb`	Kubernetes verb, such as `get`, `list`, `create`, `update`, `patch`, or `delete`
`payload.parsed.requestURI`	Request path, such as `/api/v1/namespaces/default/pods`
`payload.parsed.user.username`	User or service account that made the request
`payload.parsed.user.groups`	Groups the user belongs to
`payload.parsed.sourceIPs`	IP addresses the request came from
`payload.parsed.userAgent`	Client that made the request, such as `kubectl`
`payload.parsed.objectRef.resource`	Resource type, such as `pods`, `secrets`, or `deployments`
`payload.parsed.objectRef.namespace`	Namespace of the target resource
`payload.parsed.objectRef.name`	Name of the target resource
`payload.parsed.responseStatus.code`	HTTP status code the API server returned
`payload.parsed.annotations`	Annotations, including the authorization decision in `authorization.k8s.io/decision`

Component log fields

For component log sources, payload.parsed holds the component's log line parsed into fields:

k8s_apiserver, k8s_controller_manager, k8s_konnectivity, and k8s_scheduler: klog fields timestamp, level, file, line, id, and message
k8s_etcd: etcd fields ts, level, caller, and msg, plus fields specific to each message

View control plane logs in the console

Control plane logs are available on the Managed Logs page. They don't appear on a cluster's Logs tab under Orchestration, which shows node logs only.

To view control plane logs:

In the console, navigate to Command Center > Managed Logs.
Set Log source to a control plane log source, such as k8s_audit.
(Optional) Narrow the results with the time window and text search filters. For example, search for secrets to see audit events for Secret objects.

For the full list of filters, see Searching and filtering.

Query control plane logs with the API

The Managed Logs API accepts LogsQL queries against control plane logs, the same as node logs.

Prerequisites

To query the API, you need:

A monitoring token. Create one with crusoe monitoring tokens create. See Generate a monitoring token.
Your project ID and cluster ID. Find them with crusoe projects list and crusoe kubernetes clusters list.

Example queries

The following examples use the GET /logs/query and GET /logs/stats endpoints. For the full endpoint and parameter reference, see Querying logs via API.

List audit events for a cluster from the last hour:

curl -G "https://api.crusoecloud.com/v1/projects/$project_id/logs/query" \
  -H "Authorization: Bearer $monitoring_token" \
  --data-urlencode "query=crusoe_cluster_id:$cluster_id AND log_source:k8s_audit" \
  --data-urlencode "start=now-1h"

Find who deleted resources in a namespace:

curl -G "https://api.crusoecloud.com/v1/projects/$project_id/logs/query" \
  -H "Authorization: Bearer $monitoring_token" \
  --data-urlencode "query=crusoe_cluster_id:$cluster_id AND log_source:k8s_audit AND payload.parsed.verb:delete AND payload.parsed.objectRef.namespace:production" \
  --data-urlencode "start=now-24h"

Find requests the API server rejected as forbidden:

curl -G "https://api.crusoecloud.com/v1/projects/$project_id/logs/query" \
  -H "Authorization: Bearer $monitoring_token" \
  --data-urlencode "query=crusoe_cluster_id:$cluster_id AND log_source:k8s_audit AND payload.parsed.responseStatus.code:403" \
  --data-urlencode "start=now-24h"

Count audit events by user over the last 24 hours:

curl -G "https://api.crusoecloud.com/v1/projects/$project_id/logs/stats" \
  -H "Authorization: Bearer $monitoring_token" \
  --data-urlencode "query=crusoe_cluster_id:$cluster_id AND log_source:k8s_audit AND _time:24h | stats by (payload.parsed.user.username) count() AS requests"

Search control plane component logs for errors:

curl -G "https://api.crusoecloud.com/v1/projects/$project_id/logs/query" \
  -H "Authorization: Bearer $monitoring_token" \
  --data-urlencode "query=crusoe_cluster_id:$cluster_id AND log_source:in(k8s_apiserver, k8s_controller_manager, k8s_etcd, k8s_konnectivity, k8s_scheduler) AND error" \
  --data-urlencode "start=now-1h"

Export control plane logs

To send control plane logs to your own observability platform or security information and event management (SIEM) system, use OpenTelemetry streaming in Telemetry Conduit. Telemetry Conduit pushes logs and metrics to any endpoint that accepts OpenTelemetry Protocol (OTLP) over HTTP, with the same fields described on this page.

Retention and limits

Control plane logs follow the same retention and rate limits as other managed logs. Logs are retained for 7 days. Audit events are typically searchable within a few seconds of the API server writing them.

What's next

Logs: Search and filter all managed logs, and see the full API reference
Telemetry Conduit: Stream logs and metrics to external platforms
Support access: Control when Crusoe support can access your cluster
What's new in CMK v2: See what else changes in CMK v2 clusters

Availability​

Log sources​

Log fields​

Audit event fields​

Component log fields​

View control plane logs in the console​

Query control plane logs with the API​

Prerequisites​

Example queries​

Export control plane logs​

Retention and limits​

What's next​