Travels Demo - Multicluster

• 1: Introduction
• 2: Prerequisites
• 3: Deploy East cluster
• 4: Install Istio on East cluster
• 5: Install Kiali
• 6: Install Travels on East cluster
• 7: Deploy West cluster
• 8: Install Istio on West cluster
• 9: Configure Kiali for multicluster
• 10: Install Travels on West cluster

1 - Introduction

So far, we know how good Kiali can be to understand applications, their relationships with each other and with external applications.

In the previous tutorial, Kiali was setup to observe just a single cluster. Now, we will expand its capabilities to observe more than one cluster. The extra clusters are remotes, meaning that there is not a control plane on them, they only have user applications.

This topology is called primary-remote and it is very useful to spread applications into different clusters having just one primary cluster, which is where Istio and Kiali are installed.

This scenario is a good choice when as an application administrator or architect, you want to give a different set of clusters to different sets of developers and you also want all these applications to belong to the same mesh. This scenario is also very helpful to give applications high availability capabilities while keeping the observability together (we are referring to just applications in terms of high availability, for Istio, we might want to install a multi-primary deployment model, which is on the roadmap for the multicluster journey for Kiali).

In this tutorial we will be deploying Istio in a primary-remote deployment. At first, we will install the “east” cluster with Istio, then we will add the “west” remote cluster and join it to the mesh. Then we will see how Kiali allows us to observe and manage both clusters and their applications. Metrics will be aggregated into the “east” cluster using Prometheus federation and a single Kiali will be deployed on the “east” cluster.

If you already have a primary-remote deployment, you can skip to instaliing Kiali.

2 - Prerequisites

This tutorial is a walkthrough guide to install everything. For this reason, we will need:

• minikube
• istioctl
• helm

This tutorial was tested on:

• Minikube v1.30.1
• Istio v1.18.1
• Kiali v1.70

Clusters are provided by minikube instances, but this tutorial should work on on any Kubernetes environment.

We will set up some environment variables for the following commands:

CLUSTER_EAST="east"
CLUSTER_WEST="west"
ISTIO_DIR="absolute-path-to-istio-folder"

As Istio will be installed on more than one cluster and needs to communicate between clusters, we need to create certificates for the Istio installation. We will follow the Istio documentation related to certificates to achieve this:

mkdir -p certs
pushd certs

make -f $ISTIO_DIR/tools/certs/Makefile.selfsigned.mk root-ca

make -f $ISTIO_DIR/tools/certs/Makefile.selfsigned.mk $CLUSTER_EAST-cacerts
make -f $ISTIO_DIR/tools/certs/Makefile.selfsigned.mk $CLUSTER_WEST-cacerts

popd

The result is two certificates for then use when installing Istio in the future.

3 - Deploy East cluster

Run the following commands to deploy the first cluster:

minikube start -p $CLUSTER_EAST --network istio --memory 8g --cpus 4

For both clusters, we need to configure MetalLB, which is a load balancer. This is because we need to assign an external IP to the required ingress gateways to enable cross cluster communication between Istio and the applications installed.

minikube addons enable metallb -p $CLUSTER_EAST

We set up some environment variables with IP ranges that MetalLB will then assign to the services:

MINIKUBE_IP=$(minikube ip -p $CLUSTER_EAST)
MINIKUBE_IP_NETWORK=$(echo $MINIKUBE_IP | sed -E 's/([0-9]+\.[0-9]+\.[0-9]+)\.[0-9]+/\1/')
MINIKUBE_LB_RANGE="${MINIKUBE_IP_NETWORK}.20-${MINIKUBE_IP_NETWORK}.29"

cat <<EOF | kubectl --context $CLUSTER_EAST apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses: [${MINIKUBE_LB_RANGE}]
EOF

We should have the first cluster deployed and ready to use.

4 - Install Istio on East cluster

The east cluster is the primary one, consequently is where the istiod process will be installed alongside other applications like Kiali.

Run the following commands to install Istio:

kubectl create namespace istio-system --context $CLUSTER_EAST

kubectl create secret generic cacerts -n istio-system --context $CLUSTER_EAST \
      --from-file=certs/$CLUSTER_EAST/ca-cert.pem \
      --from-file=certs/$CLUSTER_EAST/ca-key.pem \
      --from-file=certs/$CLUSTER_EAST/root-cert.pem \
      --from-file=certs/$CLUSTER_EAST/cert-chain.pem

kubectl --context=$CLUSTER_EAST label namespace istio-system topology.istio.io/network=network1

cat <<EOF > $CLUSTER_EAST.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: $CLUSTER_EAST
      network: network1
EOF

istioctl install -y --set values.pilot.env.EXTERNAL_ISTIOD=true --context=$CLUSTER_EAST -f $CLUSTER_EAST.yaml

After the installation, we need to create what we called an “east-west” gateway. It’s an ingress gateway just for the cross cluster configuration as we are opting to use the installation for different networks (this will be the case in the majority of the production scenarios).

$ISTIO_DIR/samples/multicluster/gen-eastwest-gateway.sh \
    --mesh mesh1 --cluster $CLUSTER_EAST --network network1 | \
    istioctl --context=$CLUSTER_EAST install -y -f -

Then, we need to expose the istiod service as well as the applications for the cross cluster communication:

kubectl apply --context=$CLUSTER_EAST -n istio-system -f \
    $ISTIO_DIR/samples/multicluster/expose-istiod.yaml

kubectl --context=$CLUSTER_EAST apply -n istio-system -f \
    $ISTIO_DIR/samples/multicluster/expose-services.yaml

export DISCOVERY_ADDRESS=$(kubectl \
    --context=$CLUSTER_EAST \
    -n istio-system get svc istio-eastwestgateway \
    -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

Finally, we need to install Prometheus, which is important and required for Kiali to operate:

kubectl --context $CLUSTER_EAST -n istio-system apply -f $ISTIO_DIR/samples/addons/prometheus.yaml

5 - Install Kiali

Run the following command to install Kiali using the Kiali operator:

kubectl config use-context $CLUSTER_EAST

helm upgrade --install --namespace istio-system --set auth.strategy=anonymous --set deployment.logger.log_level=debug --set deployment.ingress.enabled=true --repo https://kiali.org/helm-charts kiali-server kiali-server 

Verify that Kiali is running with the following command:

istioctl dashboard kiali

There are other alternatives to expose Kiali or other Addons in Istio. Check Remotely Accessing Telemetry Addons for more information.

6 - Install Travels on East cluster

Run the following commands to install Travels application on east cluster:

kubectl create namespace travel-agency --context $CLUSTER_EAST
kubectl create namespace travel-portal --context $CLUSTER_EAST
kubectl create namespace travel-control --context $CLUSTER_EAST

kubectl label namespace travel-agency istio-injection=enabled --context $CLUSTER_EAST
kubectl label namespace travel-portal istio-injection=enabled --context $CLUSTER_EAST
kubectl label namespace travel-control istio-injection=enabled --context $CLUSTER_EAST

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_agency.yaml) -n travel-agency --context $CLUSTER_EAST
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_portal.yaml) -n travel-portal --context $CLUSTER_EAST
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_control.yaml) -n travel-control --context $CLUSTER_EAST

After the installation, we can see that the Travels application is running on the east cluster:

It is important to note that Kiali only observes one istio-system namespace as we did not configure it for multicluster yet.

Go to the Graph page and select the three namespaces related to the Travels demo in the namespace dropdown menu. This shows you the in-cluster traffic:

So far, we installed everything on one cluster, similarly to the Travels tutorial for a single cluster.

Now we will expand this topology to include a remote cluster. As we commented this situation can be very common in a production scenario, either because we might want to split some applications into different clusters, generally because they are maintained by different developers or for high availability or just making applications available in other zones to reduce latencies.

7 - Deploy West cluster

Run the following commands to deploy the second cluster:

minikube start -p $CLUSTER_WEST --network istio --memory 8g --cpus 4

Similar to the east cluster, we configure MetalLB:

minikube addons enable metallb -p $CLUSTER_WEST

MINIKUBE_IP=$(minikube ip -p $CLUSTER_WEST)
MINIKUBE_IP_NETWORK=$(echo $MINIKUBE_IP | sed -E 's/([0-9]+\.[0-9]+\.[0-9]+)\.[0-9]+/\1/')
MINIKUBE_LB_RANGE="${MINIKUBE_IP_NETWORK}.30-${MINIKUBE_IP_NETWORK}.39"

cat <<EOF | kubectl --context $CLUSTER_WEST apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses: [${MINIKUBE_LB_RANGE}]
EOF

8 - Install Istio on West cluster

This installation will be different as this cluster will be a remote. In a remote cluster, it won’t be an Istio control plane. Istio will install some resources that allows the primary control plane to configure the workloads in the remote cluster like injecting the sidecars and configuring the low level routing.

kubectl create namespace istio-system --context $CLUSTER_WEST

kubectl create secret generic cacerts -n istio-system --context $CLUSTER_WEST \
      --from-file=certs/$CLUSTER_WEST/ca-cert.pem \
      --from-file=certs/$CLUSTER_WEST/ca-key.pem \
      --from-file=certs/$CLUSTER_WEST/root-cert.pem \
      --from-file=certs/$CLUSTER_WEST/cert-chain.pem

kubectl --context=$CLUSTER_WEST annotate namespace istio-system topology.istio.io/controlPlaneClusters=$CLUSTER_EAST
kubectl --context=$CLUSTER_WEST label namespace istio-system topology.istio.io/network=network2

cat <<EOF > $CLUSTER_WEST.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: remote
  values:
    istiodRemote:
      injectionPath: /inject/cluster/$CLUSTER_WEST/net/network2
    global:
      remotePilotAddress: ${DISCOVERY_ADDRESS}
EOF

istioctl install -y --context=$CLUSTER_WEST -f $CLUSTER_WEST.yaml

We will also install a Prometheus instance on the remote. We will federate both Prometheus, with the east’s one being the place where all metrics will be gathered together:

kubectl apply -f $ISTIO_DIR/samples/addons/prometheus.yaml --context $CLUSTER_WEST

An important step is to create a secret on east cluster allowing it to fetch information of the remote cluster:

istioctl x create-remote-secret \
    --context=$CLUSTER_WEST \
    --name=$CLUSTER_WEST | \
    kubectl apply -f - --context=$CLUSTER_EAST

Finally, we create the east-west gateway

$ISTIO_DIR/samples/multicluster/gen-eastwest-gateway.sh \
    --mesh mesh1 --cluster $CLUSTER_WEST --network network2 | \
    istioctl --context=$CLUSTER_WEST install -y -f -

Prometheus federation

Kiali requires unified metrics from a single Prometheus endpoint for all clusters, even in a multi-cluster environment. In this tutorial, we will federate the two Prometheus instances, meaning that all the remote’s metrics should be fetched by the main Prometheus.

We will configure east’s Prometheus to fetch west’s metrics:

kubectl patch svc prometheus -n istio-system --context $CLUSTER_WEST -p "{\"spec\": {\"type\": \"LoadBalancer\"}}"

WEST_PROMETHEUS_ADDRESS=$(kubectl --context=$CLUSTER_WEST -n istio-system get svc prometheus -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

curl -L -o prometheus.yaml https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/multicluster/prometheus.yaml

sed -i "s/WEST_PROMETHEUS_ADDRESS/$WEST_PROMETHEUS_ADDRESS/g" prometheus.yaml

kubectl --context=$CLUSTER_EAST apply -f prometheus.yaml -n istio-system

9 - Configure Kiali for multicluster

We will configure Kiali to access the remote cluster. This will require a secret (similar to the Istio secret) containing the credentials for Kiali to fetch information from the remote cluster:

curl -L -o kiali-prepare-remote-cluster.sh https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/multicluster/kiali-prepare-remote-cluster.sh

chmod +x kiali-prepare-remote-cluster.sh

./kiali-prepare-remote-cluster.sh --kiali-cluster-context $CLUSTER_EAST --remote-cluster-context $CLUSTER_WEST

Finally, upgrade the installation for Kiali to pick up the secret:

kubectl config use-context $CLUSTER_EAST

helm upgrade --install --namespace istio-system --set auth.strategy=anonymous --set deployment.logger.log_level=debug --set deployment.ingress.enabled=true --repo https://kiali.org/helm-charts kiali-server kiali-server

As result, we can quickly see that a new namespace appear in the Overview, the istio-system namespace from west cluster:

10 - Install Travels on West cluster

We are going to deploy two new services just to distribute traffic on the new cluster. These services are travels v2 and v3:

kubectl create ns travel-agency --context $CLUSTER_WEST

kubectl label namespace travel-agency istio-injection=enabled --context $CLUSTER_WEST

kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travels-v2.yaml) -n travel-agency --context $CLUSTER_WEST
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travels-v3.yaml) -n travel-agency --context $CLUSTER_WEST

cat <<EOF | kubectl -n travel-agency --context $CLUSTER_WEST apply -f -
apiVersion: v1
kind: Service
metadata:
  name: travels
  labels:
    app: travels
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: travels
---
apiVersion: v1
kind: Service
metadata:
  name: insurances
  labels:
    app: insurances
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: insurances
---
apiVersion: v1
kind: Service
metadata:
  name: hotels
  labels:
    app: hotels
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: hotels
---
apiVersion: v1
kind: Service
metadata:
  name: flights
  labels:
    app: flights
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: flights
---
apiVersion: v1
kind: Service
metadata:
  name: discounts
  labels:
    app: discounts
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: discounts
---
apiVersion: v1
kind: Service
metadata:
  name: cars
  labels:
    app: cars
spec:
  ports:
    - name: http
      port: 8000
  selector:
    app: cars
EOF

After the installation, we can see that traffic is flowing to the remote cluster too:

This is happening automatically, Istio balances the traffic to both services. The key thing to notice here is that there is a concept called namespace sameness in Istio that is very important when planning our multicluster setup.

In both clusters, we can see that we have the same namespaces. They are called the same in both. Also, we can see that the services in both clusters need to exist and be called the same.

When we created the west’s namespaces, they are called the same, and also notice that even if we do not have instances of insurances or cars, we created the services. This is because travel services from the cluster will try to communicate with these services, not caring at all if the applications are in the west or east cluster. Istio will handle the routing in the back.

From this moment, we can start playing with Kiali to introduce some scenarios previously seen in the Travels tutorial.