The Ops Community ⚙️: Roman Balashevich

Pixie: an X-ray Machine for Kubernetes Traffic

Roman Balashevich — Wed, 07 Sep 2022 21:47:40 +0000

0. Introduction and challenges

Practice shows that it is not enough to develop a useful application or service and deploy it in Kubernetes - you also need to be sure that they work stably.

Observability is tooling or a technical solution that allows teams to actively debug their system. Observability is based on exploring properties and patterns not defined in advance.

Such task in decentralized systems has always been difficult.
The variable and dynamic nature of workloads and resources is the fundamental reason why Kubernetes observability is so difficult. Rather than dealing with a single server, we now have to deal with an unknown number of them (due to autoscaling). We now have numerous distributed services rather than a single monolithic application. Dealing with latency, distributed transactions, errors, and other issues grew more difficult.

The greater the level of abstraction in a decentralized system, the more difficult it is to reason about, debug, and troubleshoot it.

Consider making an HTTP(s) call to an API running on a cloud provider's Kubernetes cluster. A simplified sequence diagram (drawn by Piotr Zaniewski) is shown below, with crucial parts highlighted:

🎨 A full-size image is available here.

Things can go wrong at any point in this communication chain, performance can deteriorate, security issues can arise, and so on. For operational success, knowing what's going on in the cluster and having thorough insights into every step of the communication chain is critical.

And good news is that we have Pixie, another great CNCF project, for this observability task today!

I. Why Pixie is a fast and precise tool

Pixie is an open source (3.4k⭐) observability platform built by New Relic. It is very convenient to use Pixie to view the high-level state of your cluster (service maps, cluster resources, application traffic) and also drill down into more detailed views (pod state, flame graphs, individual full-body application requests).

Pixie uses eBPF, which stands for extended Berkeley Packet Filter, to automatically collect telemetry data such as full-body requests, resource and network metrics, application profiles, and more.

It collects, stores and queries all telemetry data locally in the cluster. Providing a granular level of observability is possible thanks to eBPF that makes the kernel programmable in a safe and performant way.

Rather than relying on gauges and static counters exposed by the operating system, eBPF allows for the generation of visibility events and the collection and in-kernel aggregation of custom metrics based on a broad range of potential sources.

Let’s take a look at how this works for application request tracing.

When Pixie is deployed to the nodes in your cluster, it deploys eBPF kernel probes that are set up to trigger on the Linux syscalls used for networking. When your application makes any network-related syscalls -- such as send() and recv() -- Pixie's eBPF probes snoop the data and send it to Pixie’s edge module (PEM, an agent).

The edge module parses the data according to the detected protocol and stores the data in tables locally on the node. Looking ahead, can be mentioned that these data tables can then be queried and visualized using the Pixie API, CLI or web-based UI.

Btw, for your attention, CNCF's Liz Rice has told interesting things about eBPF's advantages which are meaningful in our cloud field.

This increases the depth of visibility that might be attained and decreases the overall system overhead dramatically. This is achieved by collecting only the required visibility data and by producing histograms and similar data structures at the source of the event, rather than depending on the export of samples.

Pixie supports EKS, GKE and AKS. The tool is modest on resources, it uses less than 5% of cluster CPU, and in most cases even less than 2%.

II. Does it work with a local cluster?

🚦 The short answer is "Yes". For local development, creators recommend using Minikube with a VM driver (kvm2 in Linux, Kernel-based Virtual Machine driver). Unfortunately for the article's author, K3s/K3d clusters are not well-supported by Pixie.

Well, we'll start with this: install Minikube, launch a cluster and deploy in it something [painfully familiar, but] popular. More on that below.

Installing Minikube on Ubuntu

⚠️ To use VM drivers, verify that your system has virtualization support enabled:

egrep -q 'vmx|svm' /proc/cpuinfo && echo yes || echo no

If the above command outputs no, ensure that your BIOS has hardware virtualization enabled.

Proper installation of KVM (and libvirt which is among the requirements) are highly specific to each Linux distribution. It is mandatory to have 2 virtual CPUs or more, at least 2 GB of free memory and enough of free disk space. You may find Ubuntu related instructions here. Some author's notes may be useful on it.

You need to ensure that your username is added to libvirt and kvm groups!

$ sudo adduser `id -un` libvirt
Adding user '<username>' to group 'libvirt' ...
$ sudo adduser `id -un` kvm
Adding user '<username>' to group 'kvm' ...

Do not forget to enable and start libvirt daemon:

$ sudo systemctl enable libvirtd
$ sudo systemctl start libvirtd

To make kvm2 the default driver:

$ minikube config set driver kvm2

Start Minikube:

$ minikube start --driver=kvm2 --cni=flannel --cpus=2 --memory=4000 -p=pixie-cluster
☺️  [pixie-cluster] minikube v1.25.2 on Linuxmint 20.2
✨  Using the kvm2 driver based on user configuration
💾  Downloading driver docker-machine-driver-kvm2:
    > docker-machine-driver-kvm2-...: 65 B / 65 B [----------] 100.00% ? p/s 0s
    > docker-machine-driver-kvm2-...: 11.62 MiB / 11.62 MiB  100.00% 24.49 MiB
💿  Downloading VM boot image ...
    > minikube-v1.25.2.iso.sha256: 65 B / 65 B [-------------] 100.00% ? p/s 0s
    > minikube-v1.25.2.iso: 237.06 MiB / 237.06 MiB  100.00% 34.47 MiB p/s 7.1s
👍  Starting control plane node pixie-cluster in cluster pixie-cluster
🔥  Creating kvm2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
🐋  Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
    ▪️ kubelet.housekeeping-interval=5m
    ▪️ Generating certificates and keys ...
    ▪️ Booting up control plane ...
    ▪️ Configuring RBAC rules ...
🔗  Configuring Flannel (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪️ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "pixie-cluster" cluster and "default" namespace by default

A cluster name followed by -p is not mandatory. Set for convenience, but if set, it is used everywhere.

⚠️ If you're getting socket permission error

Failed to connect socket to '/var/run/libvirt/libvirt-sock'

try to reboot so the new group libvirt was actually visible for my user. You can achieve the same without rebooting by running

$ exec sudo su -l $USER

III. Deploying Pixie

☁️ Web UI is hosted in the cloud. Visit product page and sign up to obtain API key which will allow you to use Pixie.

Pixie's CLI is the fastest and easiest way to deploy Pixie. You can also deploy Pixie using YAML or Helm. You can use these steps to install Pixie to one or more clusters.

To deploy Pixie using the CLI:

# Deploy the Pixie Platform in your K8s cluster (No OLM present on cluster).
px deploy
# Deploy the Pixie Platform in your K8s cluster (OLM already exists on cluster).
px deploy --deploy_olm=false
# Deploy Pixie with a specific memory limit (2Gi is the default, 1Gi is the minimum recommended)
px deploy --pem_memory_limit=1Gi

Let's see what Pixie's deployment flow outputs.

$ px deploy --pem_memory_limit=1Gi
Pixie CLI
Running Cluster Checks:
 ✔️    Kernel version > 4.14.0
 ✔️    Cluster type is supported
 ✔️    K8s version > 1.16.0
 ✔️    Kubectl > 1.10.0 is present
 ✔️    User can create namespace
 ✔️    Cluster type is in list of known supported types
Installing Vizier version: 0.11.1
Generating YAMLs for Pixie
Deploying Pixie to the following cluster: pixie-cluster
Is the cluster correct? (y/n) [y] : y
Found 1 nodes
 ✔️    Installing OLM CRDs
 ✔️    Deploying OLM
 ✔️    Deploying Pixie OLM Namespace
 ✔️    Installing Vizier CRD
 ✔️    Deploying OLM Catalog
 ✔️    Deploying OLM Subscription
 ✔️    Creating namespace
 ✔️    Deploying Vizier
 ✔️    Waiting for Cloud Connector to come online
Waiting for Pixie to pass healthcheck
 ✔️    Wait for PEMs/Kelvin
 ⠇    Wait for healthcheck
 ✔️    Wait for PEMs/Kelvin
 ✔️    Wait for healthcheck
==> Next Steps:
Run some scripts using the px cli. For example:
- px script list : to show pre-installed scripts.
- px run px/service_stats : to run service info for sock-shop demo application (service selection coming soon!).
Check out our docs: https://docs.withpixie.ai:443.
Visit : https://work.withpixie.ai:443 to use Pixie's UI.

IV. Prepare the payload

As a payload and for greater proximity to the people, the author suggests deploying a WordPress site and a MySQL database. Both applications use PersistentVolumes and PersistentVolumeClaims to store data.

🐘 You may decrease storage: 20Gi value in YAMLs down to 2Gi, for example.

Verify that PVs are provisioned and bound (it can take up to a few minutes) and pods are running. Then run the following command to get the IP Address for the WordPress Service:

$ kubectl get svc
NAME              TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
kubernetes        ClusterIP      10.96.0.1      <none>        443/TCP        14h
wordpress         LoadBalancer   10.105.33.77   <pending>     80:30140/TCP   5h31m
wordpress-mysql   ClusterIP      None           <none>        3306/TCP       5h31m

⚠️ Minikube can only expose Services through NodePort. The EXTERNAL-IP is always pending.

$ minikube service wordpress --url -p=pixie-cluster

Copy the IP address (i.e.http://192.168.39.74:30140) and load the page in your browser to view your site.

You should see the WordPress set up page similar to the following screenshot:

⚠️ Do not leave your WordPress installation on this page.
Proceed with WordPress installation as usual - by creating a username and password, updating plugins and themes etc. Thus, it will be possible to accumulate a database of requests and traffic for analysis. Refreshing the page in the browser (with F5 key) helps with this, too.

V. Pixie's web UI basic concepts

The interface can be conditionally divided into

a main dropdown menu containing px script to execute,
a panel for visualizing services in a cluster,
reporting and logging panels,
a hidden information panel that saves screen space and slides up, and
a sidebar menu.

The general view will be something like this:

🎨 A full-size image is available here.

Elements in the service visualization panel (top) can be dragged, and you can also use the mouse wheel to zoom.

In fact, the entire web UI is a display of the scripts' output with various parameters and objects within the cluster.

VI. List of scripts

$ px script list
Pixie CLI
Table ID: script_list
  NAME                                DESCRIPTION                                         
  bpftrace/capable                    Capable                                             
  bpftrace/dc_snoop                   DC Snoop                                            
  bpftrace/exec_snoop                 Exec Snoop                                          
  bpftrace/md_flush                   MD Flush                                            
  bpftrace/nap_time                   NAP Time                                            
  bpftrace/oom_kill                   OOM Killer                                          
  bpftrace/sync_snoop                 Sync Snoop                                          
  bpftrace/tcp_drops                  TCP drops                                           
  bpftrace/tcp_retransmits            TCP retransmissions                                 
  px/agent_status                     Get agent status.                                   
  px/cluster                          Cluster Overview                                    
  px/cql_data                         Sample CQL Data                                     
  px/cql_stats                        CQL Pod LET metrics                                 
  px/dns_data                         Raw DNS Data                                        
  px/dns_flow_graph                   DNS Flow Graph                                      
  px/dns_query_summary                DNS Queries Overview                                
  px/funcs                            Get a list all of all funcs available in Pixie.     
  px/http_data                        HTTP Data                                           
  px/http_data_filtered               Sample HTTP Data                                    
  px/http_post_requests               Sample HTTP POST Data                               
  px/http_request_stats               HTTP Requests Statistics by Service                 
  px/inbound_conns                    Inbound Connections                                 
  px/ip                               Traffic to/from IP                                  
  px/jvm_data                         JVM stats                                           
  px/jvm_stats                        JVM Stats per Pod                                   
  px/kafka_consumer_rebalancing       Kafka Consumer Rebalancing Events                   
  px/kafka_data                       Kafka messages                                      
  px/kafka_overview                   Kafka Overview                                      
  px/kafka_producer_consumer_latency  Kafka producer-consumer latency                     
  px/largest_http_request             Largest HTTP Request                                
  px/most_http_data                   Pod, Endpoint pair with most HTTP Data Transferred  
  px/mysql_data                       MySQL Data                                          
  px/mysql_flow_graph                 MySQL Flow Graph                                    
  px/mysql_stats                      MySQL Pod LET metrics                               
  px/namespace                        Namespace Overview                                  
  px/namespaces                       Namespaces Overview                                 
  px/nats_data                        NATS data                                           
  px/net_flow_graph                   Network Flow                                        
  px/network_stats                    Network Stats                                       
  px/node                             Node overview                                       
  px/nodes                            Nodes overview                                      
  px/outbound_conns                   Outbound Connections                                
  px/perf_flamegraph                  Flamegraph                                          
  px/pgsql_data                       Postgres Data                                       
  px/pgsql_flow_graph                 PostgreSQL Flow Graph                               
  px/pgsql_stats                      PostgreSQL Pod LET metrics                          
  px/pid_memory_usage                 Memory Usage of Processes.                          
  px/pixie_quality_metrics            Metrics that sample Pixie's collectors              
  px/pod                              Pod Overview                                        
  px/pod_edge_stats                   Pod to Pod SLAs                                     
  px/pod_lifetime_resource            Total resource usage over pod lifetime.             
  px/pod_memory_usage                 Memory Usage of Processes                           
  px/pods                             Pods Overview                                       
  px/redis_data                       Redis RPC messages                                  
  px/redis_flow_graph                 Redis Flow Graph                                    
  px/redis_stats                      Redis Pod LET metrics                               
  px/schemas                          Get all the table schemas available in the system   
  px/service                          Service Overview                                    
  px/service_edge_stats               Basic Service to Service SLAs                       
  px/service_resource_usage           Resource Usage of Services.                         
  px/service_stats                    Basic service SLAs.                                 
  px/services                         Services Overview                                   
  px/slow_http_requests               Sample of Slow Requests                             
  px/sql_queries                      Normalized SQL Queries                              
  px/sql_query                        SQL Query Params                                    
  px/tcp_drops                        TCP drops                                           
  px/tcp_retransmits                  TCP retransmissions                                 
  px/tracepoint_status                Tracepoint Overview                                 
  px/upids                            UPIDs                                               
  pxbeta/pii_cluster_egress           Traffic containing PII leaving the cluster          
  pxbeta/service_endpoint             Endpoint Overview                                   
  pxbeta/service_endpoints            Endpoints Overview                                  
  pxbeta/vfs_snoop                    VFS Write/Delete Snoop                              
  sotw/dns_external_fqdn_list         Lists external FQDNs from DNS traffic.              
  sotw/dns_queries_filtered           All DNS queries filtered by a specific query nameTTP Data                                           
  px/http_data_filtered               Sample HTTP Data                                    
  px/http_post_requests               Sample HTTP POST Data                               
  px/http_request_stats               HTTP Requests Statistics by Service                 
  px/inbound_conns                    Inbound Connections                                 
  px/ip                               Traffic to/from IP                                  
  px/jvm_data                         JVM stats                                           
  px/jvm_stats                        JVM Stats per Pod                                   
  px/kafka_consumer_rebalancing       Kafka Consumer Rebalancing Events                   
  px/kafka_data                       Kafka messages                                      
  px/kafka_overview                   Kafka Overview                                      
  px/kafka_producer_consumer_latency  Kafka producer-consumer latency                     
  px/largest_http_request             Largest HTTP Request                                
  px/most_http_data                   Pod, Endpoint pair with most HTTP Data Transferred  
  px/mysql_data                       MySQL Data                                          
  px/mysql_flow_graph                 MySQL Flow Graph                                    
  px/mysql_stats                      MySQL Pod LET metrics                               
  px/namespace                        Namespace Overview                                  
  px/namespaces                       Namespaces Overview                                 
  px/nats_data                        NATS data                                           
  px/net_flow_graph                   Network Flow                                        
  px/network_stats                    Network Stats                                       
  px/node                             Node overview                                       
  px/nodes                            Nodes overview                                      
  px/outbound_conns                   Outbound Connections                                
  px/perf_flamegraph                  Flamegraph                                          
  px/pgsql_data                       Postgres Data                                       
  px/pgsql_flow_graph                 PostgreSQL Flow Graph                               
  px/pgsql_stats                      PostgreSQL Pod LET metrics                          
  px/pid_memory_usage                 Memory Usage of Processes.                          
  px/pixie_quality_metrics            Metrics that sample Pixie's collectors              
  px/pod                              Pod Overview                                        
  px/pod_edge_stats                   Pod to Pod SLAs                                     
  px/pod_lifetime_resource            Total resource usage over pod lifetime.             
  px/pod_memory_usage                 Memory Usage of Processes                           
  px/pods                             Pods Overview                                       
  px/redis_data                       Redis RPC messages                                  
  px/redis_flow_graph                 Redis Flow Graph                                    
  px/redis_stats                      Redis Pod LET metrics                               
  px/schemas                          Get all the table schemas available in the system   
  px/service                          Service Overview                                    
  px/service_edge_stats               Basic Service to Service SLAs                       
  px/service_resource_usage           Resource Usage of Services.                         
  px/service_stats                    Basic service SLAs.                                 
  px/services                         Services Overview                                   
  px/slow_http_requests               Sample of Slow Requests                             
  px/sql_queries                      Normalized SQL Queries                              
  px/sql_query                        SQL Query Params                                    
  px/tcp_drops                        TCP drops                                           
  px/tcp_retransmits                  TCP retransmissions                                 
  px/tracepoint_status                Tracepoint Overview                                 
  px/upids                            UPIDs                                               
  pxbeta/pii_cluster_egress           Traffic containing PII leaving the cluster          
  pxbeta/service_endpoint             Endpoint Overview                                   
  pxbeta/service_endpoints            Endpoints Overview                                  
  pxbeta/vfs_snoop                    VFS Write/Delete Snoop                              
  sotw/dns_external_fqdn_list         Lists external FQDNs from DNS traffic.              
  sotw/dns_queries_filtered           All DNS queries filtered by a specific query name

VII. How about practice? Popular queries

With Pixie, observability may include:

🐞 Full-body HTTP request

The involved sripts are px/http_data_filtered and `px/http_request_stats.

🎨 A full-size image is available here.

💡 Substituting HTTP error codes such as 500 you may investigate problematic services or pods.

🐞 Application profiles

The involved scripts are px/namespace and px/pod.

🐞 Network health metrics

The involved scripts are px/net_flow_graph, px/dns_flow_graph, px/tcp_drops.

🐞 Infra health metrics

The involved scripts is px/nodes.

🎨 A full-size image is available here.

🐞 Database query profiles

The involved scripts are px/mysql_stats and px/sql_queries.

🎨 A full-size image is available here.

Query latencies are visible at a glance, too:

Based on these data, you can think further, to optimize the operation of the DBMS, but this is a completely different story that goes beyond the scope of this article.

VII. Conclusion

Pixie is one of a handful of observability tools that offer eBPF or kernel-level observability. Other ones are Cilium, see author's previous article, and CVF.

With a single CLI command, developers are able to see all of their applications’ metrics, events, logs, and traces. Pixie’s technology removes the need to add instrumentation code, set up ad hoc dashboards, or move data off of the cluster, saving developers valuable time so they can focus instead on building better software.

Skip the code changes and get instant Kubernetes observability! Good luck.

Cilium: eBPF powered CNI, a NOS Solution for Modern Clouds

Roman Balashevich — Fri, 29 Jul 2022 18:59:00 +0000

This is a repost of my latest article of July 2022 written for Otomato on Dev.to.

NOS stands for Networking, Observability, and Security.

Some details are good to be repeated

Well, you could know that Kubernetes uses iptables for kube-proxy, the component which implements services and load balancing by DNAT iptables rules, and also the most of CNI plugins are using iptables for Network Policies.

Performance suffers when there is a lot of traffic or when there have been many changes made to the iptables rules in a system. As the number of services increases, measurements show unexpected latency and decreased performance.

To overcome these troubles, eBPF technology was invented. It had been eventually implemented in Linux kernel 3.18.

Making full use of eBPF requires at least Linux 4.4 or above!

e[xtended]BPF, as we know it today, was created by Alexei Starovoitov and Daniel Borkmann, who are still maintainers but are joined by over a hundred contributors. Looking ahead, it is important to note that Daniel now works for Isovalent.

eBPF, which stands for extended Berkeley Packet Filter, is a linux kernel feature that allows to automatically collect network telemetry data such as full-body requests, resource and network metrics, application profiles, and more.

eBPF mechanics are useful when we need to collect, query and proceed with all telemetry data in the cluster. Providing a granular level of observability is possible thanks to eBPF that makes the kernel programmable in a safe and performant way.

With the advent of eBPF, you could add logic to the kernel from user-space rather than altering the kernel code. This technique is significantly safer because to eBPF, in addition to being simplified. By implementing safety checks, the eBPF verification process ensures that the eBPF code you load into your kernel is secure to execute.

For instance, it ensures and enforces:

your program runs in a limited amount of CPU time (no indefinite loops);
your program does not crash or produce harmful, fatal bugs;
your eBPF code is being loaded by a process with the necessary permissions;
your code has a size restriction;
no unreachable code is permitted.

So, think of it as a lightweight, fully-sandboxed virtual machine (VM) inside the Linux kernel. eBPF programs are event based, and are executed on a specific hook, such as network events, system calls, function entries, and kernel trace points.

Popular Projects Powered by eBPF

Pixie, application troubleshooting platform for Kubernetes
Cilium by Isovalent, eBPF Networking, Security and Observability for Kubernetes
Falco , cloud-native runtime security
Tracee by Aqua, runtime security and forensics
Calico by Tigera, a direct competitor to Cilium as the CNI.

What is Cilium? What it features?

In order to fulfill the networking, security, and visibility requirements of container workloads, Cilium is an open source project built on top of eBPF. On top of eBPF, it offers a high-level abstraction.

What Kubernetes and container runtimes are to the Linux kernel's namespaces, cgroups, and seccomp, Cilium is to the eBPF: the appropriate abstraction layer above it.

The Least Privilege security paradigm for network policy

Implementing the Least Privilege security when your K8s pods communicate with one another is a recommended practice. The fundamental K8s Network Policies (which function at L3/L4) are effective, but Cilium Network Policies allow you to improve upon them (operate broader, at L3-L7).

In the age of K8s and microservices, this can be quite helpful because monitoring and regulating network traffic with metadata (such as IPs and ports) doesn't add much value. As services arrive and leave, IPs and ports are always changing.

The approach's flexibility is that you can manage the traffic with Cilium by using Pod, HTTP, gRPC, Kafka, DNS, and other metadata.

For instance, you can create HTTP rules that specify the route, header, and request methods that let a particular Pod to make a specific API call. Another example is creating DNS rules based on FQDN to restrict access to only certain domains. Because of this, we are able to establish security policies that are more beneficial and practical for our real-world use-cases.

With Cilium v1.12 you may even install a route which informs other endpoints that the Pod is now unreachable (i.e., when a Pod is deleted)!

Multi-Cluster Connectivity & Load Balancing

Cilium enables K8s pods to communicate and be discovered across K8s clusters by utilizing a cluster mesh. High Availability and Multi-Cloud are a few use scenarios (connecting K8s clusters across cloud providers).

With Cilium, services are now capable of understanding topology and affinity. This means that instead of balancing evenly across load-balancing backends, you can choose to prefer backends only in the local or remote clusters.

kube-proxy may be changed to Cilium for eBPF; iptables, which is being superseded by eBPF, is used by kube-proxy.

As a result, performance is significantly enhanced by this adjustment.

Cilium's “underwater” features

Between-pod transparent encryption. Support for IPsec and Wireguard protocols
Network Performance Improvement
Infra Scalability
Enhanced visibility of traffic flows, including L7 protocols, in addition to IPs and ports.
Monitoring and better network fault visibility between your microservices Metrics that work with Prometheus are offered by Cilium.

Cilium CNI supports Layer 7 (L7) policies. It competes with Istio on such feature. Cilium’s Layer 7 policy is simple to use with its own Envoy & proxylib based filter.

Observability matters!

Cilium team also offers Hubble (yes, the name is the same as the famous far space crawling telescope's one, but for clouds), which is a fully distributed networking and security observability platform for cloud native workloads. Hubble is open source software and built on top of Cilium and eBPF to enable deep visibility into the communication and behavior of services as well as the networking infrastructure in a completely transparent manner.

Only pods that are a part of the Cilium agent's network can have policies enforced on them. Unmanaged pods are frequently the outcome of delivering workloads to a cluster before installing or running Cilium, which can lead to the platform missing any recently produced pods in your environment. To identify which pods are not managed by the platform, you can compare a node's total pod count with the number of Cilium-managed pods on it (endpoint state).

According to Cilium's endpoint lifecycle, the metric's state tag offers further details about the stage that a set of endpoints are in. This allows you to keep track of how each of your pods is doing when Cilium applies and modifies network restrictions. The agent may be refreshing its networking configuration, for instance, if several endpoints have the regenerating status. It's crucial to keep an eye on pods in this condition to make sure they receive the required policy updates.

And no one understood Cilium based monitoring's nuances better than Datadog team. The illustration belongs to them.

Hubble's metrics, in contrast to Cilium metrics, allow you to keep an eye on the connectivity and security of the network in which your Cilium-managed Kubernetes pods are operating.

“How can I benefit from this?”

Well, the author's conviction: if useless to the masses, no new technology is needed!

GCP GKE

☁️ Google had already introduced so-called GKE Dataplane V2, an opinionated data plane that harnesses the power of eBPF and Cilium, in mid-2021. They also use Dataplane V2 to bring Kubernetes Network Policy logging to Google Kubernetes Engine (GKE).

To try out Kubernetes Network Policy logging for yourself, create a new GKE cluster with Dataplane V2 using the following command:

gcloud container clusters create <cluster-name> \
    --enable-dataplane-v2 \
    --enable-ip-alias \
    --release-channel rapid \
    {--region region-name | --zone zone-name}

The configuration of the Cilium agent and the Cilium Network Policy determines whether an endpoint accepts traffic from a source or not. Network policy enforcement is built into GKE Dataplane V2. You do not need to enable network policy enforcement in clusters that use GKE Dataplane V2.

AWS EKS

☁️ If you are on AWS, know the fact EKS Anywhere uses Cilium for pod networking and security.

Every form of Pod to Pod communication is by default permitted within a Kubernetes cluster. Although it might encourage experimentation, this flexibility is not seen as secure. You can limit network traffic between pods (often referred to as East/West traffic) and between pods and external services using Kubernetes network policies. The levels 3 and 4 of the OSI model are where Kubernetes network policies operate. Network policies can also employ IP addresses, port numbers, protocol numbers, or a combination of these to identify source and destination pods in addition to pod selectors and labels. Tigera's Calico is an open source policy engine that is compatible with EKS.

When combined with Istio, Calico enables expanded network policies with a broader set of functionality, including support for layer 7 rules, such as HTTP, in addition to providing the entire set of Kubernetes network policy features. The Cilium developers, Isovalent, have also expanded the network policies to offer a limited amount of support for layer 7 rules, such as HTTP. For limiting communication between Kubernetes Services/Pods and resources that run inside or outside of your VPC, Cilium additionally supports DNS hostnames.

Azure AKS

☁️ If you are on Azure, you may know AKS has only officially supported two CNI’s, Kubenet and Azure CNI. In April 2022 when they announced the ability to create an AKS cluster with no CNI. This means you can deploy any CNI you would like. In this blog post, British MVPs from Pixel Robots show how to create an AKS cluster with no CNI and then... deploy Cilium!

OpenShift

☁️ OpenShift: Cilium can also be installed on traditional VMs or bare-metal servers. OpenShift platform teams are able to implement label-based controls on the communication between application pods and external nodes by permitting VMs or bare-metal servers to join the Cilium cluster. On the other hand, the external nodes will have access to the cluster services and be able to resolve cluster names.

k3s & minikube

☁️ Eventually, you can play with Cilium on k3s, or even on minikube, just in the garage. 🚜

Moreover, by learning well with this chic Network Policy editor from Cilium, you will be able to expand the functionality of your laboratory stage. Manifests which designed there can be downloaded.

Hands-on labs on Instruqt®

You may try these interactive labs to learn more about the Cilium technology. You can learn and experiment in each lab's specialized live environment.

Good wishes & who else to read

Deploy with care! Strive to have gurus by your side!

The author wish to thank Yarel Maman, Johann Rehberger, Chris Mutchler, Lin Sun, Arthur Chiao, Glen Yu and Richard Hooper for their contribution to community.

P.S.: Hardcore devs may find some technical details here and here, on L7 parsing with proxylib in Golang.