Life of a Packet in ISTIO — Control Plane— Part 3

In Part 2 of the “Life of a Packet in ISTIO” series, we explored the Envoy proxy configuration and its significance in managing inbound and outbound traffic within the Istio service mesh. Envoy, a CNCF project developed by Lyft, operates as a sidecar proxy alongside each service instance, intercepting and directing traffic between services. We discussed Envoy’s essential features, such as load balancing, traffic routing, and security, which are crucial for Istio’s functionality.

The article provided a detailed overview of the processing steps that a packet undergoes when it reaches an Envoy proxy, including ingress, network filtering, HTTP stream processing, load balancing, and egress. A hands-on demo was presented to demonstrate how to configure Envoy as a front proxy using YAML configuration, enabling round-robin load balancing and securing communication with TLS authentication.

Furthermore, we explored tracing in Envoy, which helps developers understand communication between services in a distributed application. A hands-on demo illustrated how to enable Zipkin tracing in Envoy and visualize traces using Jaeger. The article also introduced Envoy XDS, a core component of Istio, which dynamically configures Envoy sidecars with service mesh configuration. A hands-on demo showcased how to use Envoy XDS by starting the proxy container and upstream HTTP echo servers.

Overall, Part 2 provided a comprehensive understanding of Envoy’s role in Istio’s traffic management, tracing capabilities, and dynamic configuration updates. In Part 3, we will delve further into other critical aspects of Istio’s architecture.

In this article, we will delve deeper into another critical component of Istio’s architecture: Pilot. We will explore Pilot’s role in Istio’s control plane, including service discovery, traffic management, and configuration distribution. We will also discuss how Pilot generates a detailed service topology map and ensures that the Istio mesh can handle service-to-service communication efficiently, reliably, and responsively.

As you read this article, you will come across the term “Pilot” frequently when discussing service discovery, traffic management, and configuration distribution within the Istio service mesh. It’s worth noting that if you’re familiar with Istio’s architecture in versions prior to 1.5, you might know about Pilot, Galley, Citadel, and Mixer as separate control plane components, each handling different aspects of the service mesh. However, starting with Istio 1.5, these components have been combined into a single binary and deployment called istiod.

Pilot’s Role in ISTIO’s Control Plane

Pilot is a crucial component of the Istio service mesh. It is responsible for converting high-level configuration and service discovery information into specific configuration objects that Envoy can understand, such as listeners, clusters, and endpoints. Whenever a new service is deployed or updated within the Kubernetes cluster, Pilot automatically updates the Envoy configuration to reflect these changes. This involves creating or modifying listeners to intercept traffic, updating clusters to define how traffic is managed, and maintaining an up-to-date list of healthy endpoints for traffic routing. This dynamic configuration ensures that the Istio mesh can handle service-to-service communication efficiently, reliably, and responsively.

Service Registration

• Pilot stays up-to-date with the Kubernetes API server by subscribing to updates. Whenever there are changes to services or endpoints (such as when a new service is added or an existing service is modified), Pilot is notified immediately.
• These notifications are received through the Kubernetes event stream, which Pilot listens to as part of its regular operation. This guarantees that Pilot always has the most recent information about the status of services within the Kubernetes cluster.

Discovery

With the information received from the Kubernetes API server, Pilot performs several critical functions:

• When Pilot receives updates from the Kubernetes API server, it creates an internal service registry that contains information about all services, endpoints, and their health statuses.
• Pilot obtains this data from Kubernetes Custom Resource Definitions (CRDs), such as ServiceEntries and Endpoints.
• Additionally, Pilot collects metadata and health check information for each service instance, which is also stored in the registry.

Topology Generation

Using the data gathered from Kubernetes CRDs, Pilot creates a detailed service topology map. This map is essential for several reasons.

• Dynamic Representation: The topology map dynamically illustrates how services are related and interact with each other within the mesh. This includes their endpoints, dependencies, and the policies that affect their communication.
• Routing and Policies: The topology map includes routing rules, load balancing policies, and failover strategies that govern how traffic flows through the mesh and how services should interact with one another.

Endpoint Selection

• When a request arrives at a sidecar proxy, the proxy uses pre-configured rules and policies to determine where to forward the request. This decision is based on factors such as routing rules (e.g., URI paths, headers), load balancing policies (e.g., round-robin, least connections), and the health status of potential destination endpoints.
• The proxy selects a healthy endpoint from its local cache of the service registry, which is regularly synchronized with Pilot. The selection criteria include the load balancing policy and the health checks of the endpoints.
• Once the destination endpoint is chosen, the sidecar proxy sends the request directly to that endpoint, without routing the request back through Pilot or any other part of Istio’s control plane.

Health Monitoring

• Istio’s Pilot (or Istiod in newer versions, which combines Pilot with other Istio control plane components) doesn’t perform health checks directly. Instead, it relies on the health status information managed by Kubernetes. Pilot receives periodic updates from Kubernetes about the status of service instances (such as new instances becoming ready or existing instances becoming unavailable).
• For Kubernetes-managed services, Pilot’s service registry is updated based on Kubernetes endpoints, which reflect the results of Kubernetes health checks.
• For Envoy’s active health checks, the health checks are performed between the Envoy proxies themselves, and unhealthy endpoints can be dynamically excluded from load balancing without directly updating Pilot’s service registry. This is more about real-time traffic management than service registry management.

Dynamic Updates

• As services and endpoints are added, modified, or removed in Kubernetes, Pilot updates the service registry and the service topology map dynamically. This ensures that if a service instance is considered unhealthy and is either removed from Kubernetes endpoints or marked as unhealthy by Envoy health checks, the routing configurations distributed by Pilot will reflect these changes, preventing traffic from being directed to unhealthy instances.
• Pilot distributes these updates to all sidecar proxies within the service mesh, ensuring that they have the most up-to-date information about service endpoints and configurations.

Dynamic Service Discovery and Configuration in Istio

Envoy as SideCar

Envoy Proxy, a critical component of the Istio service mesh, uses several abstractions to manage traffic: listeners, clusters, and endpoints. These elements play essential roles in how Envoy processes incoming and outgoing requests within the mesh, ensuring efficient and flexible traffic management. Here’s how they relate to the Istio mesh:

• Listeners: Listeners are configuration objects that tell Envoy where to listen for incoming connections and enable sidecar proxies to intercept and manage traffic entering and leaving the service pods.
• Clusters: A cluster is a collection of endpoints (service instances) to which Envoy can route traffic, representing services within the mesh. Istio configures Envoy with the necessary clusters based on the services defined in the mesh, along with policies for load balancing, retries, and other traffic management features.
• Endpoints: Endpoints are the individual instances of a service (e.g., pods in Kubernetes) to which traffic can be routed. They are the actual destinations for network connections managed by Envoy. In an Istio mesh, endpoints correspond to the pods running the components of the microservices. Pilot dynamically updates the Envoy proxies with the latest endpoint information, ensuring traffic is always routed to the appropriate service instance.

Step-by-Step Guide to Istio Service Discovery

Let’s use a simple example to understand how Istio manages service discovery and configuration distribution. We will use a productpage application and two versions of a reviews application. After deploying these services and exposing them via Kubernetes services, we will use Istio to manage traffic between them. This setup will demonstrate how Istio communicates service discovery information to Envoy sidecars and dynamically updates the configuration based on changes within the cluster.

Prerequisites

• Istio Installed: Ensure Istio is installed in your Kubernetes cluster. If not, follow Istio’s installation instructions.
  - kind — Quick Start (k8s.io)
  - Istio / Install with Istioctl
• Namespace Prepared: Confirm automatic sidecar injection is enabled in your working namespace (e.g., default). If necessary, enable it by running:

kubectl label namespace default istio-injection=enabled

Step 1: Deploy the Services

Create the following deployments and services in Kubernetes. We are using simplified versions of the BookInfo application’s productpage and reviews services for clarity.

ProductPage Deployment and Service

• Save the following YAML as productpage.yaml and apply it with kubectl apply -f productpage.yaml:

Reviews v1 and v2 Deployments and Service

• Save the following YAML as reviews.yaml and apply it with kubectl apply -f reviews.yaml

Once the services are deployed, and sidecars are injected, let’s explore the initial Envoy configuration.

Check Clusters:

A cluster is a group of logically similar upstream hosts that Envoy proxy connects to.

$ POD_NAME=$(kubectl get pod -l app=productpage -n default -o jsonpath='{.items[0].metadata.name}')
$ istioctl proxy-config clusters $POD_NAME --namespace default

SERVICE FQDN                                            PORT      SUBSET     DIRECTION     TYPE             DESTINATION RULE
                                                        9080      -          inbound       ORIGINAL_DST     
BlackHoleCluster                                        -         -          -             STATIC           
InboundPassthroughClusterIpv4                           -         -          -             ORIGINAL_DST     
PassthroughCluster                                      -         -          -             ORIGINAL_DST     
agent                                                   -         -          -             STATIC           
istio-egressgateway.istio-system.svc.cluster.local      80        -          outbound      EDS              
istio-egressgateway.istio-system.svc.cluster.local      443       -          outbound      EDS              
istio-ingressgateway.istio-system.svc.cluster.local     80        -          outbound      EDS              
istio-ingressgateway.istio-system.svc.cluster.local     443       -          outbound      EDS              
istio-ingressgateway.istio-system.svc.cluster.local     15021     -          outbound      EDS              
istio-ingressgateway.istio-system.svc.cluster.local     15443     -          outbound      EDS              
istio-ingressgateway.istio-system.svc.cluster.local     31400     -          outbound      EDS              
istiod.istio-system.svc.cluster.local                   443       -          outbound      EDS              
istiod.istio-system.svc.cluster.local                   15010     -          outbound      EDS              
istiod.istio-system.svc.cluster.local                   15012     -          outbound      EDS              
istiod.istio-system.svc.cluster.local                   15014     -          outbound      EDS              
kube-dns.kube-system.svc.cluster.local                  53        -          outbound      EDS              
kube-dns.kube-system.svc.cluster.local                  9153      -          outbound      EDS              
kubernetes.default.svc.cluster.local                    443       -          outbound      EDS              
productpage.default.svc.cluster.local                   9080      -          outbound      EDS              
prometheus_stats                                        -         -          -             STATIC           
reviews.default.svc.cluster.local                       9080      -          outbound      EDS              
sds-grpc                                                -         -          -             STATIC           
xds-grpc                                                -         -          -             STATIC           
zipkin                                                  -         -          -             STRICT_DNS  

Check Listeners:

A listener is a named network location (e.g., port, unix domain socket, etc.) that can be connected to by downstream clients. Envoy exposes one or more listeners that downstream hosts connect to.

$ istioctl proxy-config listeners $POD_NAME --namespace default

ADDRESSES     PORT  MATCH                                                                                           DESTINATION
10.96.0.10    53    ALL                                                                                             Cluster: outbound|53||kube-dns.kube-system.svc.cluster.local
0.0.0.0       80    Trans: raw_buffer; App: http/1.1,h2c                                                            Route: 80
0.0.0.0       80    ALL                                                                                             PassthroughCluster
10.96.0.1     443   ALL                                                                                             Cluster: outbound|443||kubernetes.default.svc.cluster.local
10.96.160.151 443   ALL                                                                                             Cluster: outbound|443||istio-ingressgateway.istio-system.svc.cluster.local
10.96.63.230  443   ALL                                                                                             Cluster: outbound|443||istio-egressgateway.istio-system.svc.cluster.local
10.96.85.208  443   ALL                                                                                             Cluster: outbound|443||istiod.istio-system.svc.cluster.local
0.0.0.0       9080  Trans: raw_buffer; App: http/1.1,h2c                                                            Route: 9080
0.0.0.0       9080  ALL                                                                                             PassthroughCluster
10.96.0.10    9153  Trans: raw_buffer; App: http/1.1,h2c                                                            Route: kube-dns.kube-system.svc.cluster.local:9153
10.96.0.10    9153  ALL                                                                                             Cluster: outbound|9153||kube-dns.kube-system.svc.cluster.local
0.0.0.0       15001 ALL                                                                                             PassthroughCluster
0.0.0.0       15001 Addr: *:15001                                                                                   Non-HTTP/Non-TCP
0.0.0.0       15006 Addr: *:15006                                                                                   Non-HTTP/Non-TCP
0.0.0.0       15006 Trans: tls; App: istio-http/1.0,istio-http/1.1,istio-h2; Addr: 0.0.0.0/0                        InboundPassthroughClusterIpv4
0.0.0.0       15006 Trans: raw_buffer; App: http/1.1,h2c; Addr: 0.0.0.0/0                                           InboundPassthroughClusterIpv4
0.0.0.0       15006 Trans: tls; App: TCP TLS; Addr: 0.0.0.0/0                                                       InboundPassthroughClusterIpv4
0.0.0.0       15006 Trans: raw_buffer; Addr: 0.0.0.0/0                                                              InboundPassthroughClusterIpv4
0.0.0.0       15006 Trans: tls; Addr: 0.0.0.0/0                                                                     InboundPassthroughClusterIpv4
0.0.0.0       15006 Trans: tls; App: istio,istio-peer-exchange,istio-http/1.0,istio-http/1.1,istio-h2; Addr: *:9080 Cluster: inbound|9080||
0.0.0.0       15006 Trans: raw_buffer; Addr: *:9080                                                                 Cluster: inbound|9080||
0.0.0.0       15010 Trans: raw_buffer; App: http/1.1,h2c                                                            Route: 15010
0.0.0.0       15010 ALL                                                                                             PassthroughCluster
10.96.85.208  15012 ALL                                                                                             Cluster: outbound|15012||istiod.istio-system.svc.cluster.local
0.0.0.0       15014 Trans: raw_buffer; App: http/1.1,h2c                                                            Route: 15014
0.0.0.0       15014 ALL                                                                                             PassthroughCluster
0.0.0.0       15021 ALL                                                                                             Inline Route: /healthz/ready*
10.96.160.151 15021 Trans: raw_buffer; App: http/1.1,h2c                                                            Route: istio-ingressgateway.istio-system.svc.cluster.local:15021
10.96.160.151 15021 ALL                                                                                             Cluster: outbound|15021||istio-ingressgateway.istio-system.svc.cluster.local
0.0.0.0       15090 ALL                                                                                             Inline Route: /stats/prometheus*
10.96.160.151 15443 ALL                                                                                             Cluster: outbound|15443||istio-ingressgateway.istio-system.svc.cluster.local
10.96.160.151 31400 ALL                                                                                             Cluster: outbound|31400||istio-ingressgateway.istio-system.svc.cluster.local

Check Routes:

$ istioctl proxy-config routes $POD_NAME --namespace default

NAME                                                          VHOST NAME                                                    DOMAINS                                              MATCH                  VIRTUAL SERVICE
15010                                                         istiod.istio-system.svc.cluster.local:15010                   istiod.istio-system, 10.96.85.208                    /*                     
15014                                                         istiod.istio-system.svc.cluster.local:15014                   istiod.istio-system, 10.96.85.208                    /*                     
istio-ingressgateway.istio-system.svc.cluster.local:15021     istio-ingressgateway.istio-system.svc.cluster.local:15021     *                                                    /*                     
80                                                            istio-egressgateway.istio-system.svc.cluster.local:80         istio-egressgateway.istio-system, 10.96.63.230       /*                     
80                                                            istio-ingressgateway.istio-system.svc.cluster.local:80        istio-ingressgateway.istio-system, 10.96.160.151     /*                     
kube-dns.kube-system.svc.cluster.local:9153                   kube-dns.kube-system.svc.cluster.local:9153                   *                                                    /*                     
9080                                                          productpage.default.svc.cluster.local:9080                    productpage, productpage.default + 1 more...         /*                     
9080                                                          reviews.default.svc.cluster.local:9080                        reviews, reviews.default + 1 more...                 /*                     
InboundPassthroughClusterIpv4                                 inbound|http|0                                                *                                                    /*                     
inbound|9080||                                                inbound|http|9080                                             *                                                    /*                     
inbound|9080||                                                inbound|http|9080                                             *                                                    /*                     
                                                              backend                                                       *                                                    /healthz/ready*        
InboundPassthroughClusterIpv4                                 inbound|http|0                                                *                                                    /*                     
                                                              backend                                                       *                                                    /stats/prometheus*   

Initially, you might not see custom routes until we define VirtualServices. These commands give you a snapshot of how traffic is configured to flow between the productpage and the reviews services, as seen by the productpage's sidecar proxy.

To understand the flow of egress traffic from the productpage service to the reviews service within an Istio service mesh, it's crucial to grasp how Istio's control plane (specifically istiod in versions 1.5 and later, which subsumes the functionality previously attributed to Pilot) uses routes and endpoints to direct traffic. This process is central to enabling the dynamic, resilient, and secure service-to-service communication that Istio is known for. Here's an end-to-end explanation, simplifying the concepts of routes and endpoints as per your request:

Overview of the Flow

When a request is made from the productpage service to the reviews service within an Istio-enabled Kubernetes cluster, several steps are involved in routing the request correctly and ensuring that it reaches a healthy instance of the reviews service. This process leverages the configurations defined in VirtualServices, DestinationRules, and the dynamic service discovery provided by istiod.

Routes

Routes in Istio are defined through VirtualServices. They tell the Envoy sidecar proxies how to route traffic to different services based on rules like URI paths, HTTP headers, or even weights for A/B testing or canary rollouts. In the istioctl proxy-config routes output:

• You see entries like 9080 for both productpage.default.svc.cluster.local and reviews.default.svc.cluster.local. These entries represent routing rules that apply to traffic directed to these services on port 9080.
• The “VHOST NAME” and “DOMAINS” columns indicate the virtual host within the proxy that handles this traffic, essentially grouping routing rules.
• The “MATCH” and “VIRTUAL SERVICE” columns would show specific conditions and the associated VirtualService resource that defines the routing behavior (if custom routing rules were applied).

Endpoints

Endpoints are specific instances (pods) of a service that are healthy and available to handle requests. istiod communicates with Envoy sidecars to provide them with up-to-date lists of endpoints for each service, using the Endpoint Discovery Service (EDS). In the istioctl proxy-config endpoints output:

• The “ENDPOINT” column shows the IP addresses and ports of service instances that Envoy can route traffic to. For instance, entries under outbound|9080||reviews.default.svc.cluster.local represent the IPs and ports of the reviews service pods.
• The “STATUS” indicates the health status of each endpoint, ensuring that traffic is only routed to healthy instances.
• The “CLUSTER” refers to the logical grouping of these endpoints as defined by Istio configurations (e.g., reviews.default.svc.cluster.local).

The Egress Traffic Flow from productpage to reviews

1. Request Initiation: A user or an application makes a request to the productpage service, which in turn needs to retrieve data from the reviews service.
2. Envoy Sidecar Interception: The request from productpage is intercepted by its Envoy sidecar proxy, based on the predefined routing rules (as seen in the routes output).
3. Route Evaluation: The Envoy proxy evaluates the request against the routing rules defined in the associated VirtualService (e.g., headers, paths) and selects the appropriate route for the reviews service.
4. Endpoint Selection: Based on the selected route and the load balancing policies defined in DestinationRule, the Envoy proxy consults the list of healthy endpoints for the reviews service (as updated by istiod).
5. Forwarding: The Envoy proxy forwards the request to one of the healthy reviews service instances, based on the load balancing policy and the health statuses of the endpoints.
6. Response Path: The response from the reviews service travels back through its Envoy sidecar, to the productpage's Envoy sidecar, and finally to the original requester.

This flow demonstrates Istio’s capability to dynamically manage service-to-service communication within the mesh, leveraging real-time updates on service topology and health statuses to ensure efficient and reliable request routing and load balancing.

Step 3: Define VirtualService for Reviews

Create a VirtualService to control traffic flow to different versions of the reviews service based on HTTP headers.

This configuration tells Istio to check the incoming request for a header named end-user. If the header is present and its value is exactly tester, the request is routed to reviews-v2. All other traffic is routed to reviews-v1. This allows you to selectively route traffic based on client identity or any other criteria you choose to implement, making it highly useful for testing new features or releases with a subset of your users before rolling them out more broadly.

• Hosts: Specifies the hostnames that this rule applies to. Matching the reviews service means it affects traffic intended for the reviews service.
• HTTP Route: Defines routing rules:
  – The first rule routes traffic to the v2 subset when the end-user header exactly matches tester. This targets traffic to pods labeled with version: v2, as defined in the DestinationRule.
  – The second route rule, which doesn’t specify a match condition, acts as the default path, routing all other traffic to the v1 subset.

Save the following as reviews-virtualservice.yaml and apply it.

Create DestinationRule for Reviews:

Define a DestinationRule to specify subsets corresponding to the two versions of the reviews service. This will allow us to route traffic selectively to each version.

• Host: Specifies the service this rule applies to. In this case, it’s reviews, which matches the Kubernetes service name.
• Subsets: Define logical divisions of the reviews service based on labels. Here, two subsets are defined:
  –v1: Represents pods of the reviews service with the label version: v1.
  –v2: Represents pods of the reviews service with the label version: v2.

The purpose of these subsets is to provide identifiable groups of service instances that can be referenced in routing rules.

How VS and DS works together?

1. Traffic Routing: When a request is made to the reviews service, Istio evaluates the VirtualService rules to determine how the traffic should be routed.
   – If the request includes an end-user header with the value tester, the request is routed to instances of the reviews service labeled with version: v2 because of the match condition in the VirtualService that references the v2 subset defined in the DestinationRule.
   – All other requests, which don’t meet the specified match condition, fall through to the default route, which directs traffic to instances labeled with version: v1.
2. Dynamic Configuration: This setup allows for dynamic configuration of traffic routing based on request attributes (like headers) without needing to update the application or the service’s deployment configuration. Changes can be made by adjusting the VirtualService or DestinationRule resources.
3. Use Cases: This model is particularly useful for canary rollouts, A/B testing, or any scenario where you want to control traffic distribution to different versions of a service based on specific criteria.

In summary, DestinationRules define policies and subsets for traffic targets within the mesh, while VirtualServices specify how traffic is routed to these targets. Working together, they enable fine-grained control over traffic routing, allowing for sophisticated traffic management strategies.

Demonstrating Dynamic Configuration with Pilot

Now that we’ve applied a more complex routing rule with our VirtualService, Pilot will distribute this configuration to the Envoy sidecars of the pods that match the hosts field, in this case, the reviews service.

You can verify the updated Envoy configurations to reflect this new routing logic by repeating the inspection commands:

1. Review Cluster Configuration: Check the clusters again to ensure they include both subsets (v1 and v2).

$ POD_NAME=$(kubectl get pod -l app=productpage -n default -o jsonpath='{.items[0].metadata.name}')
$ istioctl proxy-config clusters $POD_NAME --namespace default

SERVICE FQDN                                            PORT      SUBSET     DIRECTION     TYPE             DESTINATION RULE
                                                        9080      -          inbound       ORIGINAL_DST     
BlackHoleCluster                                        -         -          -             STATIC           
InboundPassthroughClusterIpv4                           -         -          -             ORIGINAL_DST     
PassthroughCluster                                      -         -          -             ORIGINAL_DST     
agent                                                   -         -          -             STATIC           
istio-egressgateway.istio-system.svc.cluster.local      80        -          outbound      EDS              
istio-egressgateway.istio-system.svc.cluster.local      443       -          outbound      EDS              
istio-ingressgateway.istio-system.svc.cluster.local     80        -          outbound      EDS              
istio-ingressgateway.istio-system.svc.cluster.local     443       -          outbound      EDS              
istio-ingressgateway.istio-system.svc.cluster.local     15021     -          outbound      EDS              
istio-ingressgateway.istio-system.svc.cluster.local     15443     -          outbound      EDS              
istio-ingressgateway.istio-system.svc.cluster.local     31400     -          outbound      EDS              
istiod.istio-system.svc.cluster.local                   443       -          outbound      EDS              
istiod.istio-system.svc.cluster.local                   15010     -          outbound      EDS              
istiod.istio-system.svc.cluster.local                   15012     -          outbound      EDS              
istiod.istio-system.svc.cluster.local                   15014     -          outbound      EDS              
kube-dns.kube-system.svc.cluster.local                  53        -          outbound      EDS              
kube-dns.kube-system.svc.cluster.local                  9153      -          outbound      EDS              
kubernetes.default.svc.cluster.local                    443       -          outbound      EDS              
productpage.default.svc.cluster.local                   9080      -          outbound      EDS              
prometheus_stats                                        -         -          -             STATIC     
reviews.default.svc.cluster.local                       9080      -          outbound      EDS              reviews-destination.default
reviews.default.svc.cluster.local                       9080      v1         outbound      EDS              reviews-destination.default
reviews.default.svc.cluster.local                       9080      v2         outbound      EDS              reviews-destination.default 
sds-grpc                                                -         -          -             STATIC           
xds-grpc                                                -         -          -             STATIC           
zipkin                                                  -         -          -             STRICT_DNS 

Difference:

Before and After VirtualService and DestinationRules

2. Review Route Configuration: This will show the detailed routing rules based on the end-user header.

$ istioctl proxy-config routes $POD_NAME --namespace default   
NAME                                                          VHOST NAME                                                    DOMAINS                                              MATCH                  VIRTUAL SERVICE
15010                                                         istiod.istio-system.svc.cluster.local:15010                   istiod.istio-system, 10.96.85.208                    /*                     
15014                                                         istiod.istio-system.svc.cluster.local:15014                   istiod.istio-system, 10.96.85.208                    /*                     
istio-ingressgateway.istio-system.svc.cluster.local:15021     istio-ingressgateway.istio-system.svc.cluster.local:15021     *                                                    /*                     
80                                                            istio-egressgateway.istio-system.svc.cluster.local:80         istio-egressgateway.istio-system, 10.96.63.230       /*                     
80                                                            istio-ingressgateway.istio-system.svc.cluster.local:80        istio-ingressgateway.istio-system, 10.96.160.151     /*                     
80                                                            reviews.default.svc.cluster.local:80                          reviews.default.svc.cluster.local                    /*                     reviews.default
80                                                            reviews.default.svc.cluster.local:80                          reviews.default.svc.cluster.local                    /*                     reviews.default
kube-dns.kube-system.svc.cluster.local:9153                   kube-dns.kube-system.svc.cluster.local:9153                   *                                                    /*                     
9080                                                          productpage.default.svc.cluster.local:9080                    productpage, productpage.default + 1 more...         /*                     
9080                                                          reviews.default.svc.cluster.local:9080                        reviews, reviews.default + 1 more...                 /*                     reviews.default
9080                                                          reviews.default.svc.cluster.local:9080                        reviews, reviews.default + 1 more...                 /*                     reviews.default
InboundPassthroughClusterIpv4                                 inbound|http|0                                                *                                                    /*                     
inbound|9080||                                                inbound|http|9080                                             *                                                    /*                     
inbound|9080||                                                inbound|http|9080                                             *                                                    /*                     
                                                              backend                                                       *                                                    /healthz/ready*        
InboundPassthroughClusterIpv4                                 inbound|http|0                                                *                                                    /*                     
                                                              backend                                                       *                                                    /stats/prometheus*

After adding VirtualServices and DestinationRules in Istio, the configuration and behavior of the Envoy proxies within the service mesh are significantly altered to accommodate the defined routing policies and service subsets. Let’s break down the impact of these changes and understand the end-to-end flow when egress traffic goes from the productpage service to the reviews service.

Difference:

Before and After VirtualService and DestinationRules

Before VirtualServices and DestinationRules

Initially, without specific VirtualServices and DestinationRules, traffic routing within the Istio mesh is based on standard Kubernetes service discovery and load balancing. Envoy proxies would route requests to the reviews service without any fine-grained control over which version of the service (e.g., v1 or v2) the traffic should be directed to.

After Adding VirtualServices and DestinationRules

1. DestinationRule Impact: By introducing a DestinationRule for the reviews service, we define subsets based on service versions (v1 and v2). This allows for more granular control over traffic routing to different versions of the reviews service.
   — In the clusters output, you now see clusters for reviews differentiated by subsets (v1 and v2). This indicates that Envoy is aware of these subsets and can route traffic accordingly.
2. VirtualService Impact: A VirtualService specifies how traffic to a service (or from one service to another) is routed. By applying a VirtualService to the reviews service, we can create rules that determine how traffic is directed between the subsets defined in the DestinationRule.
   — In the routes output, you now see routes for the reviews service indicating how traffic is matched (e.g., based on headers or URIs) and which subset it is routed to. The existence of these routes is directly tied to the rules defined in the VirtualService.

If you’re seeing port 80 (If you noticed this, well done!) in the istioctl proxy-config routes output for the reviews service despite not specifying it in your configuration, and after applying a VirtualService that does not mention port 80, it’s to Istio's handling of unspecified ports in the VirtualService routing definitions. Here's a deeper look into why this might happen and how Istio processes such configurations:

Default Port Handling in Istio

When a VirtualService does not explicitly specify a port in the destination of a routing rule, Istio might use default port assignments based on the protocol or service configuration. HTTP services, for instance, commonly use port 80 as a default when no other port is specified. This behavior helps ensure that the traffic routing rules can still apply even if specific port details are omitted, but it can lead to confusion if the actual service uses a different port.

How Istio Interprets the Configuration

Your VirtualService configuration for the reviews service routes traffic based on the end-user header but does not specify ports in the routing destinations. Given this:

• Implicit Default Port: In the absence of a specific port in the VirtualService's route.destination field, Istio might implicitly use a default port (commonly 80 for HTTP) for routing decisions.
• Protocol Inference: Istio infers the protocol (HTTP in this case) from the VirtualService definition. If the service definition in Kubernetes for reviews lacks a specified port or uses a name implying HTTP (e.g., http), Istio could default to using port 80 for HTTP traffic routing.

Addressing the Issue

1. Specify Ports Explicitly: To ensure that traffic is routed to the correct port, you can explicitly specify the port in your VirtualService routing rules. For instance, if the reviews service is intended to be accessed on port 9080, modify the VirtualService to include the port number:
2. Review Service Definitions: Ensure the Kubernetes Service definition for reviews specifies the correct target port (9080), and it's consistently used across your configurations.
3. Protocol Naming Conventions: Verify the port naming conventions in your Kubernetes Service definition align with the expected protocol, which can influence Istio's default behavior.
4. Check for Other Influencing Resources: Look for other Istio resources, such as Gateways or DestinationRules, that might influence traffic routing and port selection, ensuring they are consistent with your intended configuration.

vs-with-port.yaml (github.com)

By explicitly defining all aspects of the traffic routing, including ports, in your VirtualService and ensuring consistency across your service mesh configuration, you can control and predict Istio's traffic routing behavior more accurately, avoiding unexpected defaults and ensuring traffic flows as intended.

End-to-End Flow of Egress Traffic from productpage to reviews

1. Initiation: A request is made to the productpage service, which then needs to fetch data from the reviews service.
2. Interception by Envoy: The request from productpage to reviews is intercepted by the Envoy sidecar proxy in the productpage pod.
3. Route Matching: The Envoy proxy uses the VirtualService configuration to match the request based on specified criteria (e.g., URI, headers). The VirtualService dictates whether the request should go to reviews-v1 or reviews-v2 based on these rules.
4. Subset Selection: Once the route is determined, the Envoy proxy consults the DestinationRule to understand the subsets (versions) available for the reviews service. The DestinationRule helps Envoy make a decision on which subset (e.g., v1 or v2) the request should be routed to.
5. Endpoint Discovery: The Envoy proxy then looks up the healthy endpoints within the chosen subset by querying the service discovery information provided by istiod. This is where the EDS (Endpoint Discovery Service) comes into play, dynamically providing the list of healthy endpoints for reviews-v1 or reviews-v2.
6. Request Forwarding: The Envoy proxy forwards the request to one of the healthy endpoints in the selected subset. This decision also considers load balancing policies defined in the DestinationRule.
7. Response Routing: The response from the reviews service is routed back through the Envoy proxies to the client that initiated the request to the productpage.

This demonstrates the dynamic nature of traffic routing within an Istio service mesh, enabled by the configuration of VirtualServices and DestinationRules. These Istio resources empower operators to define sophisticated traffic routing and load balancing policies that react to real-time changes in the service landscape.

This detailed setup, incorporating a DestinationRule for defining subsets and a VirtualService for header-based routing, illustrates how Istio enables sophisticated traffic management within the service mesh. By leveraging these features, you can perform canary releases, A/B testing, and ensure that new features are gradually and safely exposed to your users. Additionally, it demonstrates how Istio’s Pilot component dynamically manages and distributes service discovery and configuration information to Envoy proxies, ensuring the mesh adapts in real-time to configured policies.

In the Next Part

In the next article of our Istio deep dive series, we will explore Istio’s extensive traffic management capabilities. Building upon our foundational understanding of how Istio’s Pilot component orchestrates service discovery and configuration distribution, we’ll take a closer look at how to harness Istio for advanced traffic routing, resilience patterns like circuit breaking, fault injection, traffic mirroring, and rate limiting.

Each of these features enables us to create more robust, reliable, and observable microservice architectures. We’ll go through practical examples to see these capabilities in action, helping us to fine-tune our mesh’s behavior to meet the needs of complex, real-world applications. Stay tuned for an in-depth exploration of how to leverage Istio to its full potential for managing traffic within your service mesh.

References

To deepen your understanding and further explore the concepts discussed in this series, consider the following resources:

1. Istio Official Documentation: The primary source of comprehensive documentation covering all aspects of Istio, from getting started to advanced configuration.
2. Envoy Proxy Documentation: Since Istio uses Envoy as its sidecar proxy, understanding Envoy can be very beneficial.
3. Kubernetes Documentation: A deep understanding of Kubernetes is essential when working with Istio, as it operates within the Kubernetes ecosystem.

These resources provide a solid foundation for both beginners and experienced users looking to leverage Istio within their service mesh for efficient service-to-service communication, traffic management, security, and observability.

Disclaimer

This article is for informational purposes only and does not provide any technical advice or recommendations. The views expressed in this article are solely those of the author and do not represent the views of any company or organization. Any action you take based on the information in this article is at your own risk.