Observability¶
This page describes what the tsuru API server (tsurud) emits about itself and about the applications it manages: its own logs, distributed traces, Prometheus metrics, application logs, the event audit trail, and the Kubernetes-side timeouts that most often explain a slow or failed deploy. Configuration keys link to the tsuru.conf reference; keys not documented there yet are explained inline. For the endpoints themselves (/healthcheck, /metrics, /debug/*) see Operating the API server.
API logs¶
tsurud writes two different kinds of log lines.
Internal log (errors, warnings and, with debug: true, debug messages). Its destinations are additive:
| Key | Default | Effect |
|---|---|---|
log:disable-syslog |
false |
Syslog is on by default. In a container without a syslog socket, tsurud fails at startup with unable to initialize logging; set this to true (the Helm chart does). |
log:syslog-tag |
tsurud |
Tag used for syslog messages. |
log:file |
unset | Also append to this file. |
log:use-stderr |
false |
Also write to stderr. This is the usual choice on Kubernetes. |
debug |
false |
Emit debug-level messages (startup steps, auth scheme, routers loaded, ...). |
Every failed request is logged as failure running HTTP request <METHOD> <path> (<status>): <error>. Errors that carry a stack trace are followed by a stack for error: line. Clients can ask for the same detail in the response by sending the header X-Tsuru-Verbosity: 1.
Access log, one line per request, always written to stdout regardless of the keys above. /healthcheck and /metrics are not logged. The default format is plain text:
2026-11-20T10:15:42.123456789Z https POST /apps/myapp/deploy 200 "tsuru-client/1.34.0" in 8123.456000ms [Request-ID: 7f3c...]
With log:format: json each line is a JSON object, which is easier to ship to Loki, Elasticsearch or similar:
{"time":"2026-11-20T10:15:42.123456789Z",
"request":{"scheme":"https","method":"POST","path":"/apps/myapp/deploy","userAgent":"tsuru-client/1.34.0","requestID":"7f3c...","sourceIP":"10.0.3.17"},
"response":{"statusCode":200,"durationMS":"8123.456000"},
"auth":{"username":"alice@example.com","engine":"oidc"}}
request.requestID is only present when request-id-header is configured (see Request IDs). request.sourceIP (JSON format only, since tsuru 1.18) is the TCP peer address, not X-Forwarded-For, so behind a load balancer it shows the balancer's address. auth is present when the request carried a valid token; engine tells which scheme validated it (native, oidc, oauth, team, apikey, peer).
Tracing with OpenTelemetry¶
Since tsuru 1.29 tsurud exports traces with the OpenTelemetry SDK over OTLP/gRPC (before that it used the Jaeger client). The exporter is configured with environment variables only; there are no tsuru.conf keys.
| Variable | Default | Meaning |
|---|---|---|
OTEL_EXPORTER_OTLP_ENDPOINT |
localhost:4317 |
host:port of the OTLP gRPC collector. The connection is insecure (plaintext), so point it at a collector sidecar or an in-cluster collector. |
OTEL_TRACES_SAMPLER_ARG |
0.001 |
Fraction of traces to sample (1 samples everything). |
JAEGER_ENDPOINT |
Legacy fallback for the endpoint, used only when OTEL_EXPORTER_OTLP_ENDPOINT is empty. It must now be an OTLP gRPC address. |
|
JAEGER_SAMPLER_PARAM |
Legacy fallback for the sampling fraction; OTEL_TRACES_SAMPLER_ARG wins when both are set. |
Other Jaeger variables such as JAEGER_AGENT_HOST and JAEGER_SAMPLER_TYPE are no longer read. The Helm chart's tracing.* values still set those, so with tsuru 1.29 or later pass OTEL_EXPORTER_OTLP_ENDPOINT through the pod environment instead.
Sampling is not purely probabilistic: every write request (POST, PUT, DELETE) is always sampled, except POST /node/status, so deploys and other mutations are always traceable; GET requests follow the ratio. Spans are reported with service name tsurud, named <METHOD> <route template> (for example POST /apps/{appname}/deploy), tagged with the authenticated user.name, and marked as errors on HTTP 5xx. Incoming trace context is accepted and outgoing context propagated in both W3C traceparent/baggage and B3 (single b3 header and X-B3-* headers), so a trace started at your ingress continues into tsurud and on to the Kubernetes API, routers and service brokers it calls.
Prometheus metrics¶
GET /metrics serves the Prometheus text format and requires no authentication; restrict it at the network level and scrape it from inside the cluster (the Helm chart can create a ServiceMonitor with serviceMonitor.enabled). Besides the standard Go runtime and process metrics, tsuru 1.34 exposes the following families.
HTTP and authentication
| Metric | Labels | Meaning |
|---|---|---|
tsuru_http_requests_total |
status (1xx..5xx), method, path (route template) |
Requests served. Pre-populated with zero for every route so rates work from the first scrape. |
tsuru_http_request_duration_seconds |
method, path |
Latency histogram (buckets from 1 ms to 30 s). |
tsuru_api_token_valid_total |
engine |
Tokens accepted, by the scheme that validated them (since tsuru 1.17.1). |
tsuru_api_token_invalid_total |
Tokens rejected by every scheme. A rising rate usually means an expired token in a CI job or a scanner. | |
tsuru_oauth_request_duration_seconds, tsuru_oauth_request_errors_total |
Calls to the OAuth provider, when auth:scheme is oauth. |
Deploys, builds and jobs
| Metric | Labels | Meaning |
|---|---|---|
tsuru_app_deploys_total, tsuru_app_deploy_duration_seconds |
app, status, kind, platform |
App deploys and their duration, by outcome and deploy kind. |
tsuru_app_builds_total, tsuru_app_build_duration_seconds |
app, status, kind, platform |
tsuru app build runs. |
tsuru_job_started_total, tsuru_job_completed_total |
job_name |
Job executions started and completed, taken from Kubernetes Job events. |
tsuru_job_failed_total |
job_name, reason |
Executions that hit BackoffLimitExceeded. |
Events and webhooks
| Metric | Labels | Meaning |
|---|---|---|
tsuru_events_current |
kind |
Events running right now, e.g. app.deploy. |
tsuru_events_duration_seconds |
kind |
Event duration histogram. |
tsuru_events_rejected_total |
kind, reason |
Events refused by locks, throttling or blocks. |
tsuru_webhooks_calls_total, tsuru_webhooks_calls_error, tsuru_webhooks_latency_seconds |
Webhook deliveries. | |
tsuru_webhooks_event_queue_current |
Events waiting for webhook processing. |
Application logs (see below)
| Metric | Labels | Meaning |
|---|---|---|
tsuru_logs_memory_received_total, tsuru_logs_memory_evicted_total |
app |
Entries added to, and dropped from, the in-memory buffer. Evictions mean the buffer is too small for that app. |
tsuru_logs_memory_size, tsuru_logs_memory_length |
app |
Current buffer size in bytes and in entries. |
tsuru_logs_memory_watch_dropped_total |
app |
Messages dropped because a tsuru app log -f client was too slow. |
tsuru_logs_app_tail_current, tsuru_logs_app_tail_entries_total |
app |
Active follow streams and entries sent to them. |
Integrations and housekeeping
| Metric | Labels | Meaning |
|---|---|---|
tsuru_router_request_duration_seconds, tsuru_router_request_errors_total |
router |
Calls to router APIs. |
tsuru_service_request_duration_seconds, tsuru_service_request_errors_total |
service |
Calls to service brokers (bind, unbind, create instance, ...). |
tsuru_cluster_info |
provisioner, name |
One series per registered cluster, value 1. |
tsuru_cluster_pool |
name, pool |
Pool-to-cluster mapping. |
tsuru_cluster_fetch_fail |
1 when the cluster list could not be read from MongoDB during the scrape. |
|
tsuru_gc_executions_total (phase), tsuru_gc_execution_duration_seconds, tsuru_gc_versions_marked_to_removal_total, tsuru_gc_registry_prune_*, tsuru_gc_storage_prune_* |
Old-image garbage collector activity and failures. | |
tsuru_node_status_not_found |
Legacy node-status reports for unknown nodes. |
A useful first alert is rate(tsuru_http_requests_total{status="5xx"}[5m]) per path, together with tsuru_events_current{kind="app.deploy"} staying high for longer than your deployment timeout.
Application logs¶
tsuru app log reads from two places and merges them by timestamp:
- Kubernetes pod logs. For every request tsurud asks the kubelet for the last N lines of each unit's container (
kubectl logs-style), filtered by--unitand--source(process name). This is where the application's own stdout and stderr come from. Retention is whatever your cluster keeps: logs disappear with the pod, and the kubelet rotates large files. tsuru does not store them. - The in-memory buffer. Messages pushed to the API through
POST /apps/{app}/logand the/logswebsocket go to a per-app ring buffer inside each tsurud process, capped atlog:app-log-memory-buffer-bytesbytes (default 1 MiB, oldest entries evicted first). It is lost whenever tsurud restarts. Job logs (tsuru job log) never use it; they come only from pod logs.
log:app-log-service chooses how the buffer behaves across replicas:
| Value | Behaviour |
|---|---|
memory-standalone (default) |
Each replica serves only its own buffer, so with several replicas behind a load balancer two consecutive tsuru app log calls can return different buffered entries. |
memory |
The replica that receives the request fans out to every live replica (discovered through the instance tracker), calls GET /apps/{app}/log-instance on each one authenticated with auth:peer:token, and merges the results. Use this whenever you run more than one replica. |
The request behind the client is GET /apps/{app}/log?lines=N with optional source, unit, invert-source and follow=1. In follow mode (tsuru app log -f) tsurud flushes the response at most every 500 ms and counts slow consumers in tsuru_logs_memory_watch_dropped_total. For durable application logs, ship pod logs from the cluster with your usual agent (Fluent Bit, Vector, Promtail); tsuru's buffer is a convenience for the last few minutes, not a log store.
Auditing with events¶
Every mutation in tsuru (deploys, env changes, unit scaling, role assignments, cluster edits, ...) is recorded as an event with the user or token that triggered it, its source IP, the tsurud instance that ran it, start and end time, the parameters and the full output. Events are stored in MongoDB and queried with tsuru event list / tsuru event info; they can be exported as they happen through webhooks and blocked by rule. They are the audit trail of the platform. See Events, blocks and webhooks.
Kubernetes-side signals¶
Most "my deploy hangs" reports end in one of these provisioner timeouts, all in seconds under kubernetes: in tsuru.conf. When one fires, the deploy event fails with the timeout in its error, and the Kubernetes events of the pod (kubectl -n <pool namespace> describe pod) tell you why.
| Key | Default | What it bounds |
|---|---|---|
kubernetes:deployment-progress-timeout |
600 |
Time for the new Deployment rollout to complete. Exceeding it rolls the app back to the previous version. The usual causes are an image that crashes on start or insufficient cluster capacity. |
kubernetes:pod-running-timeout |
600 |
Time for a pod to reach Running, including the image pull. Slow registries and large images show up here. |
kubernetes:pod-ready-timeout |
60 |
Time for a running pod to pass its readiness probe (the healthcheck section of tsuru.yaml). |
kubernetes:attach-after-finish-timeout |
60 |
Time tsurud keeps an attach (tsuru app run, build pods) open after the container exits, waiting for the stream to drain. |
kubernetes:api-timeout |
60 |
Client-side timeout for each call to the Kubernetes API server. |
On the cluster itself, watch the pool namespaces for ImagePullBackOff, CrashLoopBackOff and OOMKilled pods, and the tsuru.io/app-name / tsuru.io/app-process labels let you group metrics from kube-state-metrics or your node agent per tsuru app and process.