Life of a Packet in ISTIO — Part 1

The term “service mesh” has probably insinuated your consciousness sometimes if you are working with backend systems. Istio has been criticized as notoriously complex, but today’s Istio is much more straightforward. In this series, I will try to follow an HTTP request in the ISTIO service mesh to deep dive to find how it handles the inbound and outbound traffic by intercepting the traffic.

Topics — Part 1 (Traffic Interception)

  1. Init-container iptable configuration
  2. Traffic Interception in SideCar Proxy

Topics — Part 2 (SideCar proxy config)

Link: Envoy Proxy

Topics — Part 3 (Architecture)

  1. Citadel
  2. Galley

Topics — Part 4 (Traffic Management)

  1. Egress Gateway
  2. Gateway, VirtualService, and Destination rules

Topics — Part 5 (Observability)

  1. Tracing

Topics — Part 6 (Security)

Life of a Packet in Kubernetes

What is a Service Mesh?

Modern applications are typically architected as distributed collections of microservices, with each collection of microservices performing some discrete business function. A service mesh is a dedicated infrastructure layer that you can add to your applications. It allows you to transparently add capabilities like observability, traffic management, and security, without adding them to your own code. The term “service mesh” describes both the type of software you use to implement this pattern and the security or network domain that is created when you use that software.

As the deployment of distributed services, such as in a Kubernetes-based system, grows in size and complexity, it can become harder to understand and manage. Its requirements can include discovery, load balancing, failure recovery, metrics, and monitoring. A service mesh also often addresses more complex operational requirements, like A/B testing, canary deployments, rate limiting, access control, encryption, and end-to-end authentication.

What is Istio?

Istio is an open-source service mesh that layers transparently onto existing distributed applications. Istio’s powerful features provide a uniform and more efficient way to secure, connect, and monitor services. Istio is the path to load balancing, service-to-service authentication, and monitoring — with few or no service code changes. Istio provides a complete solution to satisfy the diverse requirements of microservice applications by providing behavioral insights and operational control over the service mesh as a whole. It provides a number of key capabilities uniformly across a network of services:

Traffic Management: Control the flow of traffic and API calls between services, make calls more reliable, and make the network more robust in the face of adverse conditions.

Observability: Gain an understanding of the dependencies between services and the nature and flow of traffic between them, providing the ability to quickly identify issues.

Policy Enforcement: Apply the organizational policy to the interaction between services, ensure access policies are enforced and resources are fairly distributed among consumers. Policy changes are made by configuring the mesh, not by changing the application code.

Service Identity and Security: Provide services in the mesh with a verifiable identity and provide the ability to protect service traffic as it flows over networks of varying degrees of trustability.

How it Works

Istio has two main components: the data plane and the control plane.

Data plane: The data plane, or data layer, is composed of a collection of proxy services represented as sidecar containers in each Kubernetes pod, using an extended Envoy proxy. Those sidecars mediate and control all network communication between the microservices while also collecting and reporting useful telemetry data.

Control plane: The control plane, or control layer, consists of a single binary called istiod that is responsible for converting high-level routing rules and traffic control behavior into Envoy-specific configurations, then propagating them to sidecars at runtime. Additionally, the control plane provides security measures enabling strong service-to-service and end-user authentication with built-in identity and credential management while enforcing security policies based on service identity.

The control plane takes your desired configuration, and its view of the services, and dynamically programs the proxy servers, updating them as the rules or the environment changes.

I’ll be using the book info application from the ISTIO documentation as an example to explore and explain ISTIO.

Traffic routing without ISTIO

Here is the link to the ‘Life of a Packet in Kubernetes — Part 3’ to know more about Kubernetes services.

Traffic routing with ISTIO

Book info app with ISTIO

SideCar Injection

istio-proxy (sideCar)

This allows Istio to extract a wealth of signals about traffic behavior as attributes, which in turn it can use in Mixer to enforce policy decisions and be sent to monitoring systems to provide information about the behavior of the entire mesh. The sidecar proxy model also allows you to add Istio capabilities to an existing deployment with no need to rearchitect or rewrite code.

istio-proxy container runs with restricted privileges as a user 1337. As this is reserved, the UID (User ID) for an application workload must be different and must not conflict with 1337. The 1337 UID has been chosen arbitrarily by the Istio team to bypass traffic redirection to istio-proxy container. You can also see 1337 being used as an argument to istio-iptables when initializing iptables. As this container is actively running along with the application workload, Istio also ensures that if it’s compromised, it only has read-only access to the root filesystem.

istio-init

  • It runs before an app container is started and it always runs to completion.
  • If there are many init containers, each should be complete with success before the next container is started.

So, you can see how this type of container is perfect for a set-up or initialization job which does not need to be a part of the actual application container. In this case, istio-init does just that and sets up the iptables rules.

To minimize the attack surface, securityContext stanza in the istio-init container signifies that the container runs with root privileges (runAsUser: 0), however all Linux capabilities are dropped with the exception of the NET_ADMIN and NET_RAW capabilities. These capabilities provide the istio-init init container with runtime privileges to rewrite the application pod’s iptables.

istio-init iptable rules

SideCar injection methods

istioctl kube-inject -f application.yaml | kubectl apply -f -

Most of the time, you don’t want to manually inject a sidecar every time you deploy an application, using the istioctl command, but would prefer that Istio automatically inject the sidecar to your pod. This is the recommended approach and for it to work, all you need to do is to label the namespace where you are deploying the app with istio-injection=enabled. Istio relies on Mutating Admission Webhook for injecting the sidecar.

kubectl label namespace <namespaceName> istio-injection=enabled

Here’s the process that Kubernetes mutating admission controller handles in the sidecar injection:

  1. First, the istio-sidecar-injector mutating configuration injected during the Istio installation process (shown below) sends a webhook request with all pod information to the istiod controller.
  2. Next, the controller modifies the pod specification in runtime introducing an init and sidecar container agents to the actual pod specification.
  3. Then, the controller returns the modified object back to the admission webhook for object validation.
  4. Finally after validation, the modified pod specification is deployed with all the sidecar containers.

For the full configuration, take a look at kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml.

Traffic Interception (Inbound traffic)

In Istio prior to release 1.10, the Envoy proxy, running in the same pod as the application, binds to the eth0 interface and redirects all inbound traffic to the lo interface.

This has two important side effects that cause the behavior to differ from standard Kubernetes:

  • Applications binding only to lo will receive traffic from other pods, when otherwise this is not allowed.
  • Applications binding only to eth0 will not receive traffic.

Applications that bind to both interfaces (which is typical) will not be impacted.

Starting with Istio 1.10, the networking behavior is changed to align with the standard behavior present in Kubernetes.

Here we can see that the proxy no longer redirects the traffic to the lo interface, but instead forwards it to the application on eth0. As a result, the standard behavior of Kubernetes is retained, but we still get all the benefits of Istio. This change allows Istio to get closer to its goal of being a drop-in transparent proxy that works with existing workloads with zero configuration. Additionally, it avoids unintended exposure of applications binding only to lo.

Traffic Interception (Outbound traffic)

iptable analysis

Outbound Traffic Routing

  1. OUTPUT chain forwarding traffic to ISTIO_OUTPUT chain
  2. Traffic that sends non-localhost requests and is istio proxy user space is forwarded to ISTIO_REDIRECT chain
  3. ISTIO_REDIRECT chain is directly redirected to the 15001 outlet traffic port monitored by envoy
  4. After a series of export traffic governance actions, envoy continues to send response data, which will be intercepted by netfilter and forwarded to the export traffic OUTPUT chain
  5. OUTPUT chain forwarding traffic to ISTIO_OUTPUT chain
  6. The traffic is directly returned to the POSTROUTING and then reaches the Reviews service.

Inbound Traffic Routing

  1. The product page service sends a TCP connection request to the reviews service
  2. The request enters the Pod kernel space where the reviews service is located, is intercepted by netfilter, passes through the PREROUTING chain, and then forwarded to the ISTIO_INBOUND chain
  3. ISTIO_INBOUND chain is defined by this rule — A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT intercepts and forwards to ISTIO_IN_REDIRECT
  4. ISTIO_IN_REDIRECT chain is directly redirected to the 15006 inlet traffic port monitored by the envoy.
  5. After a series of inbound traffic governance actions, the envoy sends a TCP connection request to review the service. This step belongs to outbound traffic for the envoy and will be intercepted by netfilter and forwarded to the outbound traffic OUTPUT chain
  6. OUTPUT chain forwarding traffic to ISTIO_OUTPUT chain
  7. The destination is localhost, which can’t be matched to the forwarding rule chain. RETURN directly to the next chain.
  8. The request from sidecar arrives at the review service 9080 port

References:

https://istio.io/
https://www.solo.io/blog/
https://istio.io/latest/blog/
https://github.com/istio
https://github.com/istio/istio/wiki/Design-Doc-Links
https://github.com/istio/old_pilot_repo/blob/master/doc/design.md

Disclaimer

--

--

dineshkumarr.ramasamy@gmail.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store