Multi-cluster services¶
A regular service has one API endpoint, and tsuru calls it for every instance regardless of where the apps that use it run. A multi-cluster service is different: its instances live inside the same Kubernetes cluster as the apps that bind to them, so each instance belongs to a pool, and tsuru can route calls to a different provider endpoint per cluster. The typical example is a service that creates its resources (a proxy, a database, a queue) inside the app's own cluster. Multi-cluster services are available since tsuru 1.9.
This page is for service providers. Users create and bind instances as described in Using services; the request and response formats are in the Service API workflow.
Declaring a multi-cluster service¶
Set multi-cluster: true in the service manifest you pass to tsuru service create (the manifest tsuru service template generates already has the key, set to false):
id: mydb
username: mydb
password: 1CWpoX2Zr46Jhc7u
team: platform
endpoint:
production: https://mydb-api.example.com
multi-cluster: true
The flag is immutable: tsuru service update and PUT /services/{name} ignore it, so a service created as regular cannot become multi-cluster later (or the other way around). Create a new service instead.
Instances need a pool¶
Every instance of a multi-cluster service must be created with --pool:
$ tsuru service instance add mydb orders-db small -t orders --pool prod-us
Without it the API rejects the request with multi-cluster service instance requires a pool. Instances of a regular service are the opposite: passing --pool fails with regular (non-multi-cluster) service instance cannot belong to a pool. The pool cannot be changed once the instance exists.
tsuru also checks that the pool allows the service through a pool constraint of field service. If it does not, the API answers 400 Service "mydb" is not available in pool "prod-us". An admin opens the pool to your service with:
$ tsuru pool constraint set prod-us service mydb --append
The same constraint is enforced when an app is bound to any service instance (multi-cluster or not): an app in a pool whose service constraint does not list the service gets service "mydb" is not available for pool "prod-us".
One endpoint per cluster¶
Every pool is bound to a cluster (see Multi-cluster in the cluster guide). When tsuru needs to call the provider for an instance, it resolves the instance's pool to its cluster and picks the endpoint in this order:
endpoints[<cluster-name>], the endpoint keyed by the cluster name (not the pool name);endpoints.production, the default.
A pool with no cluster bound uses production directly. Instances of regular services always use production.
Endpoints are stored as a map on the service. The CLI only sends endpoint.production from the YAML, so register per-cluster endpoints through the API, as form fields endpoints.<cluster-name>:
$ curl -X POST -H "Authorization: bearer $TOKEN" https://tsuru.example.com/1.0/services \
-d id=mydb -d username=mydb -d password=1CWpoX2Zr46Jhc7u -d team=platform \
-d multi-cluster=true \
-d endpoints.production=https://mydb-api.example.com \
-d endpoints.gke-us=https://mydb-api.us.example.com \
-d endpoints.gke-eu=https://mydb-api.eu.example.com
Warning
PUT /services/{name} replaces the whole endpoint map with the one in the request, and tsuru service update sends only production. Running tsuru service update on a service with per-cluster endpoints silently drops them. Update multi-endpoint services through the API with the full map.
You do not need one endpoint per cluster. A single production endpoint that reads the headers below and talks to the right cluster itself is a valid design.
What the provider receives¶
Every call about an instance that has a pool (create, update, bind, unbind, status, info, remove, proxied requests) carries these headers in addition to the ones listed under every request in the API workflow:
| Header | Value |
|---|---|
X-Tsuru-Pool-Name, X-Tsuru-Pool-Provisioner |
The instance's pool and its provisioner (kubernetes) |
X-Tsuru-Cluster-Name, X-Tsuru-Cluster-Provisioner |
The cluster bound to that pool; absent when the pool has no cluster |
X-Tsuru-Cluster-Addresses |
One header per cluster API server address |
X-Tsuru-Cluster-Kube-Config |
Base64 of the cluster's kubeConfig as JSON. Only when the cluster has custom data propagate-kubeconfig=true and a kubeConfig registered |
propagate-kubeconfig¶
A provider that creates Kubernetes objects inside the app's cluster needs credentials for it. Instead of configuring them in the provider, the admin can set tsuru cluster update <cluster> kubernetes --add-custom propagate-kubeconfig=true and tsuru ships the cluster's kubeConfig in X-Tsuru-Cluster-Kube-Config on every call. Decode it with base64, parse the JSON (the same structure as a kubeconfig file) and build your client from it. Keep in mind the header carries cluster credentials: use HTTPS endpoints and do not log headers.
Plans per pool¶
For a multi-cluster service tsuru service plan list <service> and tsuru service info <service> accept -p/--pool. tsuru then calls GET /resources/plans on the endpoint for that pool's cluster, with the pool and cluster headers above. Without -p the call goes to production with no pool headers.
If your plans depend on the cluster, answer a request without cluster headers with 400 and this body:
{"missing_params": ["cluster"]}
tsuru turns it into You must provide the pool name, available pools: prod-us, prod-eu for the user, who retries with -p. The same lookup happens inside tsuru service instance info, which uses the instance's pool.
Instance info¶
GET /services/{service}/instances/{instance} returns Pool and Cluster (the cluster the pool resolves to) for instances of a multi-cluster service. tsuru service instance info prints Pool but not Cluster.
Binding¶
When an app is bound to an instance, POST /resources/<instance>/bind-app carries the app's location as well, so the provider can compare it with the instance's: app-pool-name, app-pool-provisioner, app-cluster-name, app-cluster-provisioner and app-cluster-addresses (repeated) in form encoding, or appPoolName, appPoolProvisioner, appClusterName, appClusterProvisioner, appClusterAddresses with encoding: json. bind-job sends job-pool-name and job-cluster-name (jobPoolName, jobClusterName). tsuru does not refuse a bind between an app and an instance in different pools; if your service only works inside one cluster, reject it yourself.
Worked example¶
Two clusters, gke-us (pools prod-us, staging-us) and gke-eu (pool prod-eu). The provider mydb runs one API deployment per cluster, each creating databases in the cluster it runs in.
-
The admin binds pools to clusters and allows the service in them:
$ tsuru cluster update gke-us kubernetes --add-pool prod-us --add-pool staging-us --add-custom propagate-kubeconfig=true $ tsuru cluster update gke-eu kubernetes --add-pool prod-eu --add-custom propagate-kubeconfig=true $ tsuru pool constraint set prod-us service mydb --append $ tsuru pool constraint set staging-us service mydb --append $ tsuru pool constraint set prod-eu service mydb --append -
The provider registers the service with one endpoint per cluster name (see the
curlabove), keepingproductionas a fallback for pools in clusters it has no dedicated endpoint for. -
A user lists plans and creates an instance:
$ tsuru service plan list mydb Error: You must provide the pool name, available pools: prod-us, staging-us, prod-eu $ tsuru service plan list mydb -p prod-eu $ tsuru service instance add mydb orders-db small -t orders --pool prod-eutsuru resolves
prod-eutogke-eu, callshttps://mydb-api.eu.example.com/resourceswithX-Tsuru-Pool-Name: prod-eu,X-Tsuru-Cluster-Name: gke-euand the kubeconfig header. An instance instaging-uswould be created throughhttps://mydb-api.us.example.com. -
Binding an app in
prod-eusendsbind-appto the same EU endpoint withapp-pool-name=prod-euandapp-cluster-name=gke-eu.
What this gives you: data stays in the region of the app, each provider deployment only needs credentials for its own cluster, and users never see the cluster name; they only pick a pool.