Observe Your Service Mesh with Kiali

中文版 – Observability in a service mesh gives you details about the topology of your microservices-based application. It tells what microservices are part of the service mesh, how are they connected, how much traffic is routed to them, and other details. This post from Heiko Rupp shows how Kiali can be used to visualize an application using the Istio service mesh on Amazon EKS.

– Arun

A microservices architecture breaks up the software monolith into many smaller pieces that are composed together. Developers then add libraries and code to secure communications between services, provide fault tolerance to cater for network failures, and provide mechanisms to distribute traffic between different versions of a service.

A Service Mesh such as Istio, together with Amazon Elastic Container Service for Kubernetes (EKS), can now provide these services on a platform level, freeing application writers from those tasks. Routing decisions are made at the mesh level and abstracted away from the underlying ‘physical’ setup.

This power of the service mesh also brings a new complexity into the setup, including simply understanding the workings of the mesh.

Kiali to the Rescue

Kiali, an open source UI project under ASLv2 license, aims to make the mesh understandable by visualizing the effective routing in place with information about topology, throughput, and health.

First, follow the steps from this blog post to install Istio on Amazon EKS, with two slight variations as noted below to also install Kiali.

A condensed overview of the basic steps are (they have changed a bit from the blog post in Istio 1.1):

• Install EKS with the help of eksctl.
• Create the service account and initialize

  helm
  kubectl apply -f \
    install/kubernetes/helm/helm-service-account.yaml
  helm init --service-account tiller

• Initialize the Istio Custom Resource Definitions (needed in Istio 1.1 ):

  helm install --wait \
       --name istio-init \
       --namespace istio-system \
       install/kubernetes/helm/istio-init

• Install Istio with the “demo” profile which includes Kiali. (This way of installing creates the secret with the default credentials for Kiali, which are admin/admin.)

  helm install install/kubernetes/helm/istio \
       --name istio \
       --namespace istio-system \
       --values \
       install/kubernetes/helm/istio/values-istio-demo.yaml

• Expose Kiali either via Kubernetes port forwarding or via a gateway. The following forwarding command exposes Kiali on localhost, port 20001:

  kubectl -n istio-system port-forward svc/kiali 20001:20001 &

• Follow the instructions in blog post linked above to install the bookinfo demo application and send traffic to it.

When you enter Kiali by pointing your browser to http://localhost:2001/kiali/ and providing the credentials of admin/admin, you are greeted with an overview of the projects you have set up, along with their health:

Clicking on Graph in the left column leads you to a graph like the one below (you may need to select one or more namespaces in the ‘Namespace’ dropdown). Alternatively you can click on the icon in the overview card of a namespace to reach the graph for that namespace):

This visualizes the topology of the individual services and workloads that make up your application (the above shows the ‘bookinfo’ application, the Hello World of Istio).

This visualization again contains health information, but also has information about the traffic that the individual services receive. The sidebar on the right shows more information on elements in the graph once you click on them.

It is also possible to see the traffic rates in absolute or relative numbers by selecting the respective entry in the ‘Edge Labels’ drop-down. The next screenshot shows traffic rates as percentages. Because we have not installed any routing rules, the traffic is evenly distributed.

Installing a routing rule in Istio can be done by applying yaml files with definitions via kubectl. Kiali will pick up the effective rules and show that the rule exists directly on the graph, without the need to dig through yaml files in Kubernetes.

We are now using Kiali to install a rule to distribute the traffic differently to the reviews workloads. Click on the triangle icon and then on the Reviews link in the sidebar, which leads to the service detail page. You can also reach the page by going to the Services tab in the left menu and then choosing the service from the list.

Then in the top right, use the Actions menu and select Create traffic routing. Use the sliders or input fields to send 20% of the traffic to reviews v1, 50% to v2, and 30% to v3.

Click on Create when done.

After exiting the wizard, you will see that the rules have been created. You can click on the subtabs to take a peek at the rules:

Click on Reviews to get a graphical view of the rule; click on View YAML to view the source.

Now head back to the graph. The purple fork icon with “reviews” indicates the existence of such a rule in the graph, and the percentages on the edges between the reviews service triangle and the individual workload show the selected traffic flow:

When you choose View YAML above, it is also possible to edit the Istio rules.

If you made an error there (say, the weights of the versions of the reviews deployment above don’t add up to 100%), a warning will pop up and Kiali will refuse to save the change.

Modern versions of Istio should already reject such inconsistencies within one rule; the capabilities of Kiali go a lot deeper here, and allow you to cross-check settings in VirtualServices and their respective DestinationRules. In the following example, the edit changed the subset to one that cannot be found in DestinationRules (yet) – Kiali will warn about it:

Getting and Displaying Telemetry for Your Workloads

Kiali can also show telemetry data for your workload, using the Prometheus database that comes with Istio. To accomplish this, you put annotations on your workloads like the following:

apiVersion: extensions/v1beta1
kind: Deployment
[...]
spec:
  template:
    metadata:
      annotations:
        kiali.io/runtimes: quarkus,my-app
        prometheus.io/port: "9080"
        prometheus.io/scheme: http