AWS Open Source Blog

Observe Your Service Mesh with Kiali

中文版 – Observability in a service mesh gives you details about the topology of your microservices-based application. It tells what microservices are part of the service mesh, how are they connected, how much traffic is routed to them, and other details. This post from Heiko Rupp shows how Kiali can be used to visualize an application using the Istio service mesh on Amazon EKS.

Arun


A microservices architecture breaks up the software monolith into many smaller pieces that are composed together. Developers then add libraries and code to secure communications between services, provide fault tolerance to cater for network failures, and provide mechanisms to distribute traffic between different versions of a service.

A Service Mesh such as Istio, together with Amazon Elastic Container Service for Kubernetes (EKS), can now provide these services on a platform level, freeing application writers from those tasks. Routing decisions are made at the mesh level and abstracted away from the underlying ‘physical’ setup.

This power of the service mesh also brings a new complexity into the setup, including simply understanding the workings of the mesh.

Kiali to the Rescue

Kiali, an open source UI project under ASLv2 license, aims to make the mesh understandable by visualizing the effective routing in place with information about topology, throughput, and health.

First, follow the steps from this blog post to install Istio on Amazon EKS, with two slight variations as noted below to also install Kiali.

A condensed overview of the basic steps are (they have changed a bit from the blog post in Istio 1.1):

  • Install EKS with the help of eksctl.
  • Create the service account and initialize
    helm
    kubectl apply -f \
      install/kubernetes/helm/helm-service-account.yaml
    helm init --service-account tiller
  • Initialize the Istio Custom Resource Definitions (needed in Istio 1.1 ):
    helm install --wait \
         --name istio-init \
         --namespace istio-system \
         install/kubernetes/helm/istio-init
  • Install Istio with the “demo” profile which includes Kiali. (This way of installing creates the secret with the default credentials for Kiali, which are admin/admin.)
    helm install install/kubernetes/helm/istio \
         --name istio \
         --namespace istio-system \
         --values \
         install/kubernetes/helm/istio/values-istio-demo.yaml
  • Expose Kiali either via Kubernetes port forwarding or via a gateway. The following forwarding command exposes Kiali on localhost, port 20001:
    kubectl -n istio-system port-forward svc/kiali 20001:20001 &
  • Follow the instructions in blog post linked above to install the bookinfo demo application and send traffic to it.

When you enter Kiali by pointing your browser to http://localhost:2001/kiali/ and providing the credentials of admin/admin, you are greeted with an overview of the projects you have set up, along with their health:

Kiali console overview.

Clicking on Graph in the left column leads you to a graph like the one below (you may need to select one or more namespaces in the ‘Namespace’ dropdown). Alternatively you can click on the Kiali graph icon. icon in the overview card of a namespace to reach the graph for that namespace):

Kiali graph.

This visualizes the topology of the individual services and workloads that make up your application (the above shows the ‘bookinfo’ application, the Hello World of Istio).

This visualization again contains health information, but also has information about the traffic that the individual services receive. The sidebar on the right shows more information on elements in the graph once you click on them.

It is also possible to see the traffic rates in absolute or relative numbers by selecting the respective entry in the ‘Edge Labels’ drop-down. The next screenshot shows traffic rates as percentages. Because we have not installed any routing rules, the traffic is evenly distributed.

Kiali, graph showing even traffic distribution.

Installing a routing rule in Istio can be done by applying yaml files with definitions via kubectl. Kiali will pick up the effective rules and show that the rule exists directly on the graph, without the need to dig through yaml files in Kubernetes.

We are now using Kiali to install a rule to distribute the traffic differently to the reviews workloads. Click on the triangle icon and then on the Reviews link in the sidebar, which leads to the service detail page. You can also reach the page by going to the Services tab in the left menu and then choosing the service from the list.

Kiali reviews view.

Then in the top right, use the Actions menu and select Create traffic routing. Use the sliders or input fields to send 20% of the traffic to reviews v1, 50% to v2, and 30% to v3.

Click on Create when done.

Kiali create traffic routing.

After exiting the wizard, you will see that the rules have been created. You can click on the subtabs to take a peek at the rules:

Kiali rules.

Click on Reviews to get a graphical view of the rule; click on View YAML to view the source.

Kiali graphical view of rule.

Now head back to the graph. The purple fork icon with “reviews” indicates the existence of such a rule in the graph, and the percentages on the edges between the reviews service triangle and the individual workload show the selected traffic flow:

Kiali: rules displayed in graph view.

When you choose View YAML above, it is also possible to edit the Istio rules.

If you made an error there (say, the weights of the versions of the reviews deployment above don’t add up to 100%), a warning will pop up and Kiali will refuse to save the change.

Modern versions of Istio should already reject such inconsistencies within one rule; the capabilities of Kiali go a lot deeper here, and allow you to cross-check settings in VirtualServices and their respective DestinationRules. In the following example, the edit changed the subset to one that cannot be found in DestinationRules (yet) – Kiali will warn about it:

 

Kiali subset not found error.

Getting and Displaying Telemetry for Your Workloads

Kiali can also show telemetry data for your workload, using the Prometheus database that comes with Istio. To accomplish this, you put annotations on your workloads like the following:

apiVersion: extensions/v1beta1
kind: Deployment
[...]
spec:
  template:
    metadata:
      annotations:
        kiali.io/runtimes: quarkus,my-app
        prometheus.io/port: "9080"
        prometheus.io/scheme: http

The prometheus.io annotations tell Prometheus from where to scrape the data, and the kiali.io one lists the dashboards that should be shown. Dashboard definitions are yaml files which define the layout on screen and the names of metrics to display.

Once this is set up, there will be new tabs in the application and workload detail pages, in this case “Quarkus Metrics” and “App Rating.”
Kiali workload telemetry.As the data obtained is kept in Prometheus, it is also possible to display it in Grafana or other similar tools. You can find more information about this feature in my post How to view application-metrics in Kiali.

Link to Distributed Tracing

In a classical monolith, it is easy to add a profiler to the running process to get statistics on where time is spent. For distributed applications, this does not work.

Out of the box, Istio provides not only provide telemetry but also distributed tracing data, according to the OpenTracing standard. Kiali integrates with Jaeger, one of the tracing systems that supports OpenTracing.
Kiali integrates with Jaeger screenshot.This screenshot shows a standalone version of the Jaeger UI, tracing calls through the bookinfo application. The horizontal bars show the time spent in each of the components, so it is easy to see that the reviews service spends a lot of time in its inner workings, and the total latency of the bookinfo application is dominated by it.

Future Work

While Kiali is already pretty good at visualizing the state of the service mesh, there are still too many complex tasks left for the Istio user involving yaml files, a text editor, and command line tools. While folks like power of Istio a lot, its complexity is still high.

Kiali will in the future better support creating and updating of Istio resources without needing to fall back on the command line (as you saw in the Create Weighted Routing wizard above).

Kiali is a standard add-on to Istio since version 1.1, replacing the static service graph.

Kiali is currently tied to Istio, but a colleague has already had some success getting data from a LinkerD-based mesh. It would certainly be cool if someone would be interested in abstracting some of the Istio-specific code to allow Kiali to use other service meshes on top of Kubernetes, e.g., Amazon App Mesh with Kiali.

More

As I wrote above, Kiali is open source under ASLv2 license. We started it as an open source effort because this is in our DNA. You can find Kiali on GitHub, and we welcome contributions. While code contributions are almost always welcome, you can also help us a lot by reporting errors, telling us about your use cases, or creating translations of the documentation.

To keep up with Kiali news, you can follow Kiali on Twitter and Medium, and myself on Twitter and Medium.

Heiko Rupp.

Heiko Rupp

Heiko is an open source enthusiast working for more than a decade at Red Hat in the area of middleware monitoring and management. He helps leading and shaping the Kiali effort.
He also helps defining the next way of Java Microservices with his work on Eclipse MicroProfile. As such he is the lead of the Eclipse MicroProfile Metrics effort. He lives with his family in Stuttgart, Germany.

The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.

Arun Gupta

Arun Gupta

Arun Gupta is a Principal Open Source Technologist at Amazon Web Services. He focuses on everything containers and open source at AWS. He is responsible for CNCF strategy within AWS, and participates at CNCF Board and technical meetings actively. He has built and led developer communities for 12+ years at Sun, Oracle, Red Hat and Couchbase. He has extensive speaking experience in more than 40 countries on myriad topics and is a JavaOne Rock Star for four years in a row. Gupta also founded the Devoxx4Kids chapter in the US and continues to promote technology education among children. A prolific blogger, author of several books, an avid runner, a globe trotter, a Docker Captain, a Java Champion, a JUG leader, NetBeans Dream Team member, he is easily accessible at @arungupta.