Life of a Packet in ISTIO — Part 1

The term “service mesh” has probably insinuated your consciousness sometimes if you are working with backend systems. Istio has been criticized as notoriously complex, but today’s Istio is much more straightforward. In this series, I will try to follow an HTTP request in the ISTIO service mesh to deep dive to find how it handles the inbound and outbound traffic by intercepting the traffic.

Topics — Part 1 (Traffic Interception)

1. Sidecar injection
2. Init-container iptable configuration
3. Traffic Interception in SideCar Proxy

Topics — Part 2 (SideCar proxy config)

1. Envoy proxy configuration introduction

Link: Envoy Proxy

Topics — Part 3 (Architecture)

1. Service Discovery and Configuration

Topics — Part 4 (Traffic Management)

1. Ingress Gateway
2. Egress Gateway
3. Gateway, VirtualService, and Destination rules

Topics — Part 5 (Observability)

1. Metrics
2. Tracing

Topics — Part 6 (Security)

1. TLS

Life of a Packet in Kubernetes

Life of a Packet in Kubernetes — Part 1

What is a Service Mesh?

Modern applications are typically architected as distributed collections of microservices, with each collection of microservices performing some discrete business function. A service mesh is a dedicated infrastructure layer that you can add to your applications. It allows you to transparently add capabilities like observability, traffic management, and security, without adding them to your own code. The term “service mesh” describes both the type of software you use to implement this pattern and the security or network domain that is created when you use that software.

As the deployment of distributed services, such as in a Kubernetes-based system, grows in size and complexity, it can become harder to understand and manage. Its requirements can include discovery, load balancing, failure recovery, metrics, and monitoring. A service mesh also often addresses more complex operational requirements, like A/B testing, canary deployments, rate limiting, access control, encryption, and end-to-end authentication.

What is Istio?

Istio is an open-source service mesh that layers transparently onto existing distributed applications. Istio’s powerful features provide a uniform and more efficient way to secure, connect, and monitor services. Istio is the path to load balancing, service-to-service authentication, and monitoring — with few or no service code changes. Istio provides a complete solution to satisfy the diverse requirements of microservice applications by providing behavioral insights and operational control over the service mesh as a whole. It provides a number of key capabilities uniformly across a network of services:

Traffic Management: Control the flow of traffic and API calls between services, make calls more reliable, and make the network more robust in the face of adverse conditions.

Observability: Gain an understanding of the dependencies between services and the nature and flow of traffic between them, providing the ability to quickly identify issues.

Policy Enforcement: Apply the organizational policy to the interaction between services, ensure access policies are enforced and resources are fairly distributed among consumers. Policy changes are made by configuring the mesh, not by changing the application code.

Service Identity and Security: Provide services in the mesh with a verifiable identity and provide the ability to protect service traffic as it flows over networks of varying degrees of trustability.

How it Works

Istio has two main components: the data plane and the control plane.

Data plane: The data plane, or data layer, is composed of a collection of proxy services represented as sidecar containers in each Kubernetes pod, using an extended Envoy proxy. Those sidecars mediate and control all network communication between the microservices while also collecting and reporting useful telemetry data.

Control plane: The control plane, or control layer, consists of a single binary called istiod that is responsible for converting high-level routing rules and traffic control behavior into Envoy-specific configurations, then propagating them to sidecars at runtime. Additionally, the control plane provides security measures enabling strong service-to-service and end-user authentication with built-in identity and credential management while enforcing security policies based on service identity.

The control plane takes your desired configuration, and its view of the services, and dynamically programs the proxy servers, updating them as the rules or the environment changes.

I’ll be using the book info application from the ISTIO documentation as an example to explore and explain ISTIO.

Bookinfo Application

Traffic routing without ISTIO

A Kubernetes service can be used to easily expose an application deployed on a set of pods using a single endpoint. Services keep track of the changes in IP addresses and DNS names of the pods and expose them to the end-user as a single IP or DNS.

Here is the link to the ‘Life of a Packet in Kubernetes — Part 3’ to know more about Kubernetes services.

Life of a Packet in Kubernetes — Part 3

Traffic routing with ISTIO

Envoy Proxy is deployed as a sidecar to the relevant service in the same Kubernetes pod. A sidecar is just a container that runs on the same Pod as the application container, because it shares the same volume and network as the main container, it can “help” or enhance how the application operates.

SideCar Injection

In simple terms, sidecar injection is adding the configuration of additional containers to the pod template. The added containers needed for the Istio service mesh are,

istio-proxy (sideCar)

istio-proxy This is the actual sidecar proxy (based on Envoy).

This allows Istio to extract a wealth of signals about traffic behavior as attributes, which in turn it can use in Mixer to enforce policy decisions and be sent to monitoring systems to provide information about the behavior of the entire mesh. The sidecar proxy model also allows you to add Istio capabilities to an existing deployment with no need to rearchitect or rewrite code.

istio-proxy container runs with restricted privileges as a user 1337. As this is reserved, the UID (User ID) for an application workload must be different and must not conflict with 1337. The 1337 UID has been chosen arbitrarily by the Istio team to bypass traffic redirection to istio-proxy container. You can also see 1337 being used as an argument to istio-iptables when initializing iptables. As this container is actively running along with the application workload, Istio also ensures that if it’s compromised, it only has read-only access to the root filesystem.

istio-init

istio-init This init container is used to set up the iptables rules so that inbound/outbound traffic will go through the sidecar proxy. An init container is different than an app container in the following ways:

• It runs before an app container is started and it always runs to completion.
• If there are many init containers, each should be complete with success before the next container is started.

So, you can see how this type of container is perfect for a set-up or initialization job which does not need to be a part of the actual application container. In this case, istio-init does just that and sets up the iptables rules.

To minimize the attack surface, securityContext stanza in the istio-init container signifies that the container runs with root privileges (runAsUser: 0), however all Linux capabilities are dropped with the exception of the NET_ADMIN and NET_RAW capabilities. These capabilities provide the istio-init init container with runtime privileges to rewrite the application pod’s iptables.

istio-init iptable rules

How does the sidecar proxy grab the inbound and outbound traffic to and from the container? It is done by the iptable rules created by theistio-init container command.

SideCar injection methods

In the manual injection method, you can use istioctl to modify the pod template and add the configuration of the two containers previously mentioned. For both manual as well as automatic injection, Istio takes the configuration from the istio-sidecar-injector configuration map (configmap) and the mesh’s istio configmap.

istioctl kube-inject -f application.yaml | kubectl apply -f -

Most of the time, you don’t want to manually inject a sidecar every time you deploy an application, using the istioctl command, but would prefer that Istio automatically inject the sidecar to your pod. This is the recommended approach and for it to work, all you need to do is to label the namespace where you are deploying the app with istio-injection=enabled. Istio relies on Mutating Admission Webhook for injecting the sidecar.

kubectl label namespace <namespaceName> istio-injection=enabled

Here’s the process that Kubernetes mutating admission controller handles in the sidecar injection:

1. First, the istio-sidecar-injector mutating configuration injected during the Istio installation process (shown below) sends a webhook request with all pod information to the istiod controller.
2. Next, the controller modifies the pod specification in runtime introducing an init and sidecar container agents to the actual pod specification.
3. Then, the controller returns the modified object back to the admission webhook for object validation.
4. Finally after validation, the modified pod specification is deployed with all the sidecar containers.

For the full configuration, take a look at kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml.

Traffic Interception (Inbound traffic)

TCP requests that are sent or received from the Pod will be hijacked by iptables. After the inbound traffic is hijacked, it is processed by the Inbound Handler and then forwarded to the application container for processing.

In Istio prior to release 1.10, the Envoy proxy, running in the same pod as the application, binds to the eth0 interface and redirects all inbound traffic to the lo interface.

This has two important side effects that cause the behavior to differ from standard Kubernetes:

• Applications binding only to lo will receive traffic from other pods, when otherwise this is not allowed.
• Applications binding only to eth0 will not receive traffic.

Applications that bind to both interfaces (which is typical) will not be impacted.

Starting with Istio 1.10, the networking behavior is changed to align with the standard behavior present in Kubernetes.

Here we can see that the proxy no longer redirects the traffic to the lo interface, but instead forwards it to the application on eth0. As a result, the standard behavior of Kubernetes is retained, but we still get all the benefits of Istio. This change allows Istio to get closer to its goal of being a drop-in transparent proxy that works with existing workloads with zero configuration. Additionally, it avoids unintended exposure of applications binding only to lo.

Traffic Interception (Outbound traffic)

The outbound traffic is hijacked by iptables and then forwarded to the Outbound Handler for processing.

iptable analysis

Following are the NAT table rules that handle inbound and outbound traffic routing.

Outbound Traffic Routing

1. Product Page service sends requests to the reviews service which is intercepted by netfilter again and forwarded to the export traffic OUTPUT chain
2. OUTPUT chain forwarding traffic to ISTIO_OUTPUT chain
3. Traffic that sends non-localhost requests and is istio proxy user space is forwarded to ISTIO_REDIRECT chain
4. ISTIO_REDIRECT chain is directly redirected to the 15001 outlet traffic port monitored by envoy
5. After a series of export traffic governance actions, envoy continues to send response data, which will be intercepted by netfilter and forwarded to the export traffic OUTPUT chain
6. OUTPUT chain forwarding traffic to ISTIO_OUTPUT chain
7. The traffic is directly returned to the POSTROUTING and then reaches the Reviews service.

Inbound Traffic Routing

1. The product page service sends a TCP connection request to the reviews service
2. The request enters the Pod kernel space where the reviews service is located, is intercepted by netfilter, passes through the PREROUTING chain, and then forwarded to the ISTIO_INBOUND chain
3. ISTIO_INBOUND chain is defined by this rule — A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT intercepts and forwards to ISTIO_IN_REDIRECT
4. ISTIO_IN_REDIRECT chain is directly redirected to the 15006 inlet traffic port monitored by the envoy.
5. After a series of inbound traffic governance actions, the envoy sends a TCP connection request to review the service. This step belongs to outbound traffic for the envoy and will be intercepted by netfilter and forwarded to the outbound traffic OUTPUT chain
6. OUTPUT chain forwarding traffic to ISTIO_OUTPUT chain
7. The destination is localhost, which can’t be matched to the forwarding rule chain. RETURN directly to the next chain.
8. The request from sidecar arrives at the review service 9080 port

References:

https://istio.io/
https://www.solo.io/blog/
https://istio.io/latest/blog/
https://github.com/istio
https://github.com/istio/istio/wiki/Design-Doc-Links
https://github.com/istio/old_pilot_repo/blob/master/doc/design.md

Disclaimer

This article does not provide any technical advice or recommendation; if you feel so, it is my personal view, not the company I work for.