Building a MultiCluster Environment with Cilium on BareMetal Kubernetes Cluster: A Comprehensive Guide

Building a MultiCluster Environment with Cilium on BareMetal Kubernetes Cluster: A Comprehensive Guide

Multi-Cluster (Cluster Mesh)

Cluster mesh extends the networking datapath across multiple clusters. It allows endpoints in all connected clusters to communicate while providing full policy enforcement. Load-balancing is available via Kubernetes annotations.

Setting up Cluster Mesh A Cilium MultiCluster setup involves deploying Cilium, a powerful networking and security solution for Kubernetes, across multiple Kubernetes clusters. This setup enables communication and workload deployment across clusters, providing benefits such as fault tolerance, disaster recovery, and improved performance. Here's an explanation of various aspects of Cilium MultiCluster

Architecture:

In a MultiCluster architecture with Cilium, you have multiple Kubernetes clusters deployed across different geographic regions, data centers, or cloud providers.

Each Kubernetes cluster runs Cilium as its CNI (Container Network Interface) plugin to manage networking, security, and observability for containerized workloads.

Connectivity:

Cilium facilitates connectivity between pods and services deployed across different Kubernetes clusters.

It achieves this through mechanisms like cluster federation, network peering, or VPN tunnels, depending on the chosen MultiCluster setup.

Service Discovery:

Cilium provides service discovery capabilities across MultiCluster environments, allowing pods in one cluster to discover and communicate with services deployed in other clusters.

This enables seamless communication between microservices distributed across multiple clusters.

Network Policies:

Cilium allows the enforcement of network policies across MultiCluster deployments, ensuring that traffic between clusters adheres to security and compliance requirements.

Network policies define rules for traffic filtering, segmentation, and access control based on various criteria such as IP addresses, ports, and labels.

Identity-Aware Networking:

Cilium supports identity-aware networking across MultiCluster environments, allowing granular control over communication based on workload identities.

Workload identities, such as Kubernetes service accounts or custom attributes, can be used to enforce fine-grained access control policies between clusters.

Observability:

Cilium provides comprehensive observability features for MultiCluster environments, including real-time network visibility, metrics collection, and distributed tracing.

Operators can monitor network traffic, performance metrics, and security events across clusters to ensure operational efficiency and security.

Security:

Cilium enhances security in MultiCluster environments by enforcing security policies, encrypting inter-cluster communication, and providing threat detection capabilities.

It protects against network-based attacks, ensures data confidentiality, and enables secure communication between clusters.

Scalability and Performance:

Cilium is designed for scalability and performance in MultiCluster deployments, leveraging eBPF (extended Berkeley Packet Filter) technology for efficient packet processing and low-latency networking.

It supports large-scale deployments spanning multiple clusters while maintaining high throughput and low latency for inter-cluster communication.

Cilium MultiCluster offers a robust networking and security solution for Kubernetes environments spanning multiple clusters. It enables seamless communication, service discovery, and security enforcement across clusters, empowering organizations to build resilient, scalable, and secure distributed applications.

Setting up Cluster Mesh This is a step-by-step guide on how to build a mesh of Kubernetes clusters by connecting them together, enable pod-to-pod connectivity across all clusters, define global services to load-balance between clusters.

Prerequisites

2 Kubernetes Clusters

Cluster-A

root@master:~# kubectl get nodes
NAME                   STATUS   ROLES           AGE   VERSION
master.arobyte.tech    Ready    control-plane   58m   v1.26.0
worker1.arobyte.tech   NotReady    <none>          41m   v1.26.0
worker2.arobyte.tech   NotReady    <none>          40m   v1.26.0

Cluster-B

root@devmaster:~# kubectl get nodes
NAME                      STATUS   ROLES           AGE     VERSION
devmaster.arobyte.tech    Ready    control-plane   3h57m   v1.26.0
devworker1.arobyte.tech   NotReady    <none>          39m     v1.26.0
devworker2.arobyte.tech   NotReady    <none>          38m     v1.26.0

Let's Install Cilium on Both The Cluster

Cluster-A

root@master:~# kubectl config use-context master

root@master:~# helm repo add cilium https://helm.cilium.io/

helm install cilium cilium/cilium --version 1.15.1 \
   --namespace kube-system \
   --set nodeinit.enabled=true \
   --set kubeProxyReplacement=partial \
   --set hostServices.enabled=false \
   --set externalIPs.enabled=true \
   --set nodePort.enabled=true \
   --set hostPort.enabled=true \
   --set cluster.name=master \
   --set cluster.id=1

root@master:~# kubectl get nodes
NAME                   STATUS   ROLES           AGE   VERSION
master.arobyte.tech    Ready    control-plane   58m   v1.26.0
worker1.arobyte.tech   Ready    <none>          41m   v1.26.0
worker2.arobyte.tech   Ready    <none>          40m   v1.26.0

Cluster-B

root@master:~# kubectl config use-context devmaster

root@master:~# helm install cilium cilium/cilium --version 1.15.1 \
   --namespace kube-system \
   --set nodeinit.enabled=true \
   --set kubeProxyReplacement=partial \
   --set hostServices.enabled=false \
   --set externalIPs.enabled=true \
   --set nodePort.enabled=true \
   --set hostPort.enabled=true \
   --set cluster.name=devmaster \
   --set cluster.id=2

root@master:~# kubectl get pods -n metallb-system
NAME                                 READY   STATUS    RESTARTS   AGE
metallb-controller-9fcb75cf5-8s5w4   1/1     Running   0          35m
metallb-speaker-v769s                4/4     Running   0          35m
metallb-speaker-vfg62                4/4     Running   0          35m
metallb-speaker-www9n                4/4     Running   0          35m

Metallb Load Balancer Must be installed on Both the Cluster For Load balancer.

Cluster-A

root@master:~# kubectl get pods -n metallb-system
NAME                                 READY   STATUS    RESTARTS   AGE
metallb-controller-9fcb75cf5-mds7w   1/1     Running   0          30m
metallb-speaker-cfsw4                4/4     Running   0          30m
metallb-speaker-nblvz                4/4     Running   0          30m
metallb-speaker-sr68w                4/4     Running   0          30m

Cluster-B

root@devmaster:~# kubectl get pods -n metallb-system
NAME                                 READY   STATUS    RESTARTS   AGE
metallb-controller-9fcb75cf5-8s5w4   1/1     Running   0          35m
metallb-speaker-v769s                4/4     Running   0          35m
metallb-speaker-vfg62                4/4     Running   0          35m
metallb-speaker-www9n                4/4     Running   0          35m

Now We can see on Cluster-A and Cluster-B all the nodes are in Ready state after installing Cilium CNI

Enable Cilium Multicluster on Both Clusters

root@master:~# cilium clustermesh enable --context master --service-type LoadBalancer
root@master:~# cilium clustermesh status --context master --wait
✅ Service "clustermesh-apiserver" of type "LoadBalancer" found
✅ Cluster access information is available:
  - 172.16.16.150:2379
⌛ Waiting (0s) for deployment clustermesh-apiserver to become ready: only 0 of 1 replicas are available

root@master:~# cilium clustermesh enable --context devmaster --service-type LoadBalancer
root@master:~# cilium clustermesh status --context master --wait
✅ Service "clustermesh-apiserver" of type "LoadBalancer" found
✅ Cluster access information is available:
  - 172.16.16.150:2379
✅ Deployment clustermesh-apiserver is ready
🔌 No cluster connected
🔀 Global services: [ min:0 / avg:0.0 / max:0 ]

root@master:~# cilium clustermesh status --context devmaster --wait
✅ Service "clustermesh-apiserver" of type "LoadBalancer" found
✅ Cluster access information is available:
  - 172.17.17.150:2379
✅ Deployment clustermesh-apiserver is ready
🔌 No cluster connected
🔀 Global services: [ min:0 / avg:0.0 / max:0 ]

We can see cluster mesh installed Successfully on both Clusters

Lets connect both clusters together

root@master:~# cilium clustermesh connect --context master \
   --destination-context devmaster
✅ Detected Helm release with Cilium version 1.15.1
✨ Extracting access information of cluster devmaster...
🔑 Extracting secrets from cluster devmaster...
ℹ️  Found ClusterMesh service IPs: [172.17.17.150]
✨ Extracting access information of cluster master...
🔑 Extracting secrets from cluster master...
ℹ️  Found ClusterMesh service IPs: [172.16.16.150]
⚠️ Cilium CA certificates do not match between clusters. Multicluster features will be limited!
ℹ️ Configuring Cilium in cluster 'master' to connect to cluster 'devmaster'
ℹ️ Configuring Cilium in cluster 'devmaster' to connect to cluster 'master'
✅ Connected cluster master and devmaster!

Let’s verify the status of the Cilium cluster mesh once again

root@master:~# cilium clustermesh status --context master --wait
✅ Service "clustermesh-apiserver" of type "LoadBalancer" found
✅ Cluster access information is available:
  - 172.16.16.150:2379
✅ Deployment clustermesh-apiserver is ready
✅ All 3 nodes are connected to all clusters [min:1 / avg:1.0 / max:1]
🔌 Cluster Connections:
  - devmaster: 3/3 configured, 3/3 connected
🔀 Global services: [ min:0 / avg:0.0 / max:0 ]


root@master:~# cilium clustermesh status --context devmaster --wait
✅ Service "clustermesh-apiserver" of type "LoadBalancer" found
✅ Cluster access information is available:
  - 172.17.17.150:2379
✅ Deployment clustermesh-apiserver is ready
✅ All 3 nodes are connected to all clusters [min:1 / avg:1.0 / max:1]
🔌 Cluster Connections:
  - master: 3/3 configured, 3/3 connected
🔀 Global services: [ min:0 / avg:0.0 / max:0 ]

Verify the Kubernetes Service with Cilium Mesh Control Plane

root@master:~# kubectl config use-context devmaster
root@master:~# kubectl get svc -A | grep clustermesh
kube-system      clustermesh-apiserver           LoadBalancer   10.98.126.70     172.17.17.150   2379:32200/TCP           34m
kube-system      clustermesh-apiserver-metrics   ClusterIP      None             <none>          9962/TCP,9963/TCP        34m

root@master:~# kubectl config use-context master
Switched to context "master".
root@master:~# kubectl get svc -A | grep clustermesh
kube-system      clustermesh-apiserver           LoadBalancer   10.105.111.27    172.16.16.150   2379:31285/TCP           37m
kube-system      clustermesh-apiserver-metrics   ClusterIP      None             <none>          9962/TCP,9963/TCP        37m

Validate the connectivity by running the connectivity test in multi-cluster mode

root@master:~# cilium connectivity test --context master --multi-cluster devmaster
ℹ️  Monitor aggregation detected, will skip some flow validation steps
✨ [master] Creating namespace cilium-test for connectivity check...
✨ [devmaster] Creating namespace cilium-test for connectivity check...
✨ [master] Deploying echo-same-node service...
✨ [master] Deploying echo-other-node service...
✨ [master] Deploying DNS test server configmap...
✨ [devmaster] Deploying DNS test server configmap...
✨ [master] Deploying same-node deployment...
✨ [master] Deploying client deployment...
✨ [master] Deploying client2 deployment...
✨ [devmaster] Deploying echo-other-node service...
✨ [devmaster] Deploying other-node deployment...
✨ [host-netns] Deploying master daemonset...
✨ [host-netns] Deploying devmaster daemonset...
✨ [host-netns-non-cilium] Deploying master daemonset...
ℹ️  Skipping tests that require a node Without Cilium
⌛ [master] Waiting for deployment cilium-test/client to become ready...
⌛ [master] Waiting for deployment cilium-test/client2 to become ready...
⌛ [master] Waiting for deployment cilium-test/echo-same-node to become ready...
⌛ [devmaster] Waiting for deployment cilium-test/echo-other-node to become ready...
⌛ [master] Waiting for pod cilium-test/client-65847bf96-2mqtd to reach DNS server on cilium-test/echo-same-node-66f8958597-mbr7s pod...
⌛ [master] Waiting for pod cilium-test/client2-85585bdd-bnn77 to reach DNS server on cilium-test/echo-same-node-66f8958597-mbr7s pod...
⌛ [master] Waiting for pod cilium-test/client-65847bf96-2mqtd to reach DNS server on cilium-test/echo-other-node-5db9b7bbbc-lvsqj pod...
⌛ [master] Waiting for pod cilium-test/client2-85585bdd-bnn77 to reach DNS server on cilium-test/echo-other-node-5db9b7bbbc-lvsqj pod...
⌛ [master] Waiting for pod cilium-test/client-65847bf96-2mqtd to reach default/kubernetes service...
⌛ [master] Waiting for pod cilium-test/client2-85585bdd-bnn77 to reach default/kubernetes service...
⌛ [master] Waiting for Service cilium-test/echo-other-node to become ready...
⌛ [master] Waiting for Service cilium-test/echo-other-node to be synchronized by Cilium pod kube-system/cilium-6qfz6
⌛ [master] Waiting for Service cilium-test/echo-same-node to become ready...
⌛ [master] Waiting for Service cilium-test/echo-same-node to be synchronized by Cilium pod kube-system/cilium-6qfz6
⌛ [master] Waiting for DaemonSet cilium-test/host-netns-non-cilium to become ready...
⌛ [master] Waiting for DaemonSet cilium-test/host-netns to become ready...
⌛ [devmaster] Waiting for DaemonSet cilium-test/host-netns to become ready...
ℹ️  Skipping IPCache check
🔭 Enabling Hubble telescope...
⚠️  Unable to contact Hubble Relay, disabling Hubble telescope and flow validation: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:4245: connect: connection refused"
ℹ️  Expose Relay locally with:
   cilium hubble enable
   cilium hubble port-forward&
ℹ️  Cilium version: 1.15.1
🏃 Running 66 tests ...
[=] Test [no-unexpected-packet-drops] [1/66]

  [-] Scenario [no-unexpected-packet-drops/no-unexpected-packet-drops]
  🟥 Found unexpected packet drops:
{
  "labels": {
    "direction": "EGRESS",
    "reason": "Unsupported L2 protocol"
  },
  "name": "cilium_drop_count_total",
  "value": 428
}
[=] Test [no-policies] [2/66]
.....................
  [-] Scenario [no-policies/pod-to-pod]
  [.] Action [no-policies/pod-to-pod/curl-ipv4-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-same-node-66f8958597-mbr7s (10.0.2.109:8080)]
  [.] Action [no-policies/pod-to-pod/curl-ipv4-1: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-other-node-5db9b7bbbc-lvsqj (10.0.2.163:8080)]
  [.] Action [no-policies/pod-to-pod/curl-ipv4-2: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-same-node-66f8958597-mbr7s (10.0.2.109:8080)]
  [.] Action [no-policies/pod-to-pod/curl-ipv4-3: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-other-node-5db9b7bbbc-lvsqj (10.0.2.163:8080)]
  [-] Scenario [no-policies/pod-to-world]
  [.] Action [no-policies/pod-to-world/http-to-one.one.one.one-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> one.one.one.one-http (one.one.one.one:80)]
  [.] Action [no-policies/pod-to-world/https-to-one.one.one.one-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> one.one.one.one-https (one.one.one.one:443)]
  [.] Action [no-policies/pod-to-world/https-to-one.one.one.one-index-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> one.one.one.one-https-index (one.one.one.one:443)]
  [.] Action [no-policies/pod-to-world/http-to-one.one.one.one-1: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> one.one.one.one-http (one.one.one.one:80)]
  [.] Action [no-policies/pod-to-world/https-to-one.one.one.one-1: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> one.one.one.one-https (one.one.one.one:443)]
  [.] Action [no-policies/pod-to-world/https-to-one.one.one.one-index-1: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> one.one.one.one-https-index (one.one.one.one:443)]
  [-] Scenario [no-policies/pod-to-cidr]
  [.] Action [no-policies/pod-to-cidr/external-1111-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> external-1111 (1.1.1.1:443)]
  [.] Action [no-policies/pod-to-cidr/external-1111-1: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> external-1111 (1.1.1.1:443)]
  [.] Action [no-policies/pod-to-cidr/external-1001-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> external-1001 (1.0.0.1:443)]
  [.] Action [no-policies/pod-to-cidr/external-1001-1: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> external-1001 (1.0.0.1:443)]
  [-] Scenario [no-policies/client-to-client]
  [.] Action [no-policies/client-to-client/ping-ipv4-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/client2-85585bdd-bnn77 (10.0.2.148:0)]
  [.] Action [no-policies/client-to-client/ping-ipv4-1: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/client-65847bf96-2mqtd (10.0.2.95:0)]
  [-] Scenario [no-policies/pod-to-service]
  [.] Action [no-policies/pod-to-service/curl-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-other-node (echo-other-node.cilium-test:8080)]
  [.] Action [no-policies/pod-to-service/curl-1: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-same-node (echo-same-node.cilium-test:8080)]
  [.] Action [no-policies/pod-to-service/curl-2: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-same-node (echo-same-node.cilium-test:8080)]
  [.] Action [no-policies/pod-to-service/curl-3: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-other-node (echo-other-node.cilium-test:8080)]
  [-] Scenario [no-policies/pod-to-hostport]
  [.] Action [no-policies/pod-to-hostport/curl-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-other-node-5db9b7bbbc-lvsqj (172.17.17.101:4000)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://172.17.17.101:4000" failed: error with exec request (pod=cilium-test/client-65847bf96-2mqtd, container=client): command terminated with exit code 28
  ℹ️  curl output:


  [.] Action [no-policies/pod-to-hostport/curl-1: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-same-node-66f8958597-mbr7s (172.16.16.102:4000)]
  [.] Action [no-policies/pod-to-hostport/curl-2: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-same-node-66f8958597-mbr7s (172.16.16.102:4000)]
  [.] Action [no-policies/pod-to-hostport/curl-3: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-other-node-5db9b7bbbc-lvsqj (172.17.17.101:4000)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://172.17.17.101:4000" failed: error with exec request (pod=cilium-test/client2-85585bdd-bnn77, container=client2): command terminated with exit code 28
  ℹ️  curl output:


  [-] Scenario [no-policies/pod-to-host]
  [.] Action [no-policies/pod-to-host/ping-ipv4-internal-ip: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> 192.168.0.114 (192.168.0.114:0)]
  [.] Action [no-policies/pod-to-host/ping-ipv4-internal-ip: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> 172.16.16.101 (172.16.16.101:0)]
  [.] Action [no-policies/pod-to-host/ping-ipv4-internal-ip: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> 172.16.16.102 (172.16.16.102:0)]
  [.] Action [no-policies/pod-to-host/ping-ipv4-internal-ip: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> 192.168.0.114 (192.168.0.114:0)]
  [.] Action [no-policies/pod-to-host/ping-ipv4-internal-ip: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> 172.16.16.101 (172.16.16.101:0)]
  [.] Action [no-policies/pod-to-host/ping-ipv4-internal-ip: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> 172.16.16.102 (172.16.16.102:0)]
  [-] Scenario [no-policies/host-to-pod]
  [.] Action [no-policies/host-to-pod/curl-ipv4-0: cilium-test/host-netns-gnqtv (172.16.16.101) -> cilium-test/echo-same-node-66f8958597-mbr7s (10.0.2.109:8080)]
  [.] Action [no-policies/host-to-pod/curl-ipv4-1: cilium-test/host-netns-gnqtv (172.16.16.101) -> cilium-test/echo-other-node-5db9b7bbbc-lvsqj (10.0.2.163:8080)]
  [.] Action [no-policies/host-to-pod/curl-ipv4-2: cilium-test/host-netns-rswsn (172.17.17.102) -> cilium-test/echo-same-node-66f8958597-mbr7s (10.0.2.109:8080)]
  [.] Action [no-policies/host-to-pod/curl-ipv4-3: cilium-test/host-netns-rswsn (172.17.17.102) -> cilium-test/echo-other-node-5db9b7bbbc-lvsqj (10.0.2.163:8080)]
  [.] Action [no-policies/host-to-pod/curl-ipv4-4: cilium-test/host-netns-xfkhb (172.17.17.101) -> cilium-test/echo-same-node-66f8958597-mbr7s (10.0.2.109:8080)]
  [.] Action [no-policies/host-to-pod/curl-ipv4-5: cilium-test/host-netns-xfkhb (172.17.17.101) -> cilium-test/echo-other-node-5db9b7bbbc-lvsqj (10.0.2.163:8080)]
  [.] Action [no-policies/host-to-pod/curl-ipv4-6: cilium-test/host-netns-5x8wz (172.16.16.102) -> cilium-test/echo-same-node-66f8958597-mbr7s (10.0.2.109:8080)]
  [.] Action [no-policies/host-to-pod/curl-ipv4-7: cilium-test/host-netns-5x8wz (172.16.16.102) -> cilium-test/echo-other-node-5db9b7bbbc-lvsqj (10.0.2.163:8080)]
  [-] Scenario [no-policies/pod-to-external-workload]
[=] Test [no-policies-extra] [3/66]
.
  [-] Scenario [no-policies-extra/pod-to-remote-nodeport]
  [.] Action [no-policies-extra/pod-to-remote-nodeport/curl-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-other-node (echo-other-node.cilium-test:8080)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://192.168.0.114:32184" failed: error with exec request (pod=cilium-test/client-65847bf96-2mqtd, container=client): command terminated with exit code 7
  ℹ️  curl output:


  [.] Action [no-policies-extra/pod-to-remote-nodeport/curl-1: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-other-node (echo-other-node.cilium-test:8080)]
  [.] Action [no-policies-extra/pod-to-remote-nodeport/curl-2: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-same-node (echo-same-node.cilium-test:8080)]
  [.] Action [no-policies-extra/pod-to-remote-nodeport/curl-3: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-same-node (echo-same-node.cilium-test:8080)]
  [.] Action [no-policies-extra/pod-to-remote-nodeport/curl-4: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-other-node (echo-other-node.cilium-test:8080)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://192.168.0.114:32184" failed: error with exec request (pod=cilium-test/client2-85585bdd-bnn77, container=client2): command terminated with exit code 7
  ℹ️  curl output:


  [.] Action [no-policies-extra/pod-to-remote-nodeport/curl-5: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-other-node (echo-other-node.cilium-test:8080)]
  [.] Action [no-policies-extra/pod-to-remote-nodeport/curl-6: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-same-node (echo-same-node.cilium-test:8080)]
  [.] Action [no-policies-extra/pod-to-remote-nodeport/curl-7: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-same-node (echo-same-node.cilium-test:8080)]
  [-] Scenario [no-policies-extra/pod-to-local-nodeport]
  [.] Action [no-policies-extra/pod-to-local-nodeport/curl-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-other-node (echo-other-node.cilium-test:8080)]
  [.] Action [no-policies-extra/pod-to-local-nodeport/curl-1: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-same-node (echo-same-node.cilium-test:8080)]
  [.] Action [no-policies-extra/pod-to-local-nodeport/curl-2: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-other-node (echo-other-node.cilium-test:8080)]
  [.] Action [no-policies-extra/pod-to-local-nodeport/curl-3: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-same-node (echo-same-node.cilium-test:8080)]
[=] Test [allow-all-except-world] [4/66]
................
[=] Test [client-ingress] [5/66]
------------
----------------
---------------
-------------------

Deploying a Simple Example Service

In Cluster-A, deploy:

root@master:~# kubectl config use-context master
Switched to context "master"
root@master:~# kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.15.1/examples/kubernetes/clustermesh/global-service-example/cluster1.yaml
service/rebel-base created
deployment.apps/rebel-base created
configmap/rebel-base-response created
deployment.apps/x-wing created

In Cluster-B, deploy:

root@master:~# kubectl config use-context devmaster
Switched to context "devmaster".
root@master:~# kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.15.1/examples/kubernetes/clustermesh/global-service-example/cluster2.yaml
service/rebel-base created
deployment.apps/rebel-base created
configmap/rebel-base-response created
deployment.apps/x-wing created

Let's validate the deployment on both the clusters

Cluster-A

root@master:~# kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
rebel-base-5f575fcc5d-46sx4   1/1     Running   0          15m
rebel-base-5f575fcc5d-62nqv   1/1     Running   0          15m
x-wing-58c9c6d949-lgv7j       1/1     Running   0          15m
x-wing-58c9c6d949-vqqxv       1/1     Running   0          15m

Cluster-B

root@devmaster:~# kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
rebel-base-5f575fcc5d-8jfpl   1/1     Running   0          13m
rebel-base-5f575fcc5d-cnw8x   1/1     Running   0          13m
x-wing-58c9c6d949-5nbfh       1/1     Running   0          13m
x-wing-58c9c6d949-tbwfq       1/1     Running   0          13m

From either cluster, access the global service

root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}

We will see replies from pods in both clusters.

In Cluster-A, add service.cilium.io/shared="false" to existing global service

root@master:~# kubectl annotate service rebel-base service.cilium.io/shared="false" --overwrite
service/rebel-base annotated

From Cluster-A, access the global service one more time

root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}

We will still see replies from pods in both clusters

From Cluster-B, access the global service again

root@master:~# kubectl annotate service rebel-base service.cilium.io/shared-
service/rebel-base annotated
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}

We will see replies from pods only from cluster 2, as the global service in cluster 1 is no longer shared

In Cluster-A, remove service.cilium.io/shared annotation of existing global service

root@master:~# kubectl config use-context master
Switched to context "master".
root@master:~# kubectl annotate service rebel-base service.cilium.io/shared-
service/rebel-base annotated

From either cluster, access the global service

root@master:~# kubectl config use-context master
Switched to context "master".
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}

root@master:~# kubectl config use-context devmaster
Switched to context "devmaster".
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}

We will see replies from pods in both clusters again