Introduction to KubeArmor KubeArmor is an open-source security solution designed specifically for Kubernetes environments. Developed by AhnLab, a leading cybersecurity company, KubeArmor provides fine-grained container-aware security policies to enforce access control, system call filtering, and network policies.

Architecture Overview

The Need for Kubernetes Security As organizations increasingly adopt Kubernetes for their containerized workloads, ensuring the security of these environments becomes crucial. Traditional security measures are often inadequate in containerized environments due to their dynamic nature and the large attack surface they present. KubeArmor addresses these challenges by providing granular security controls tailored to Kubernetes deployments.

Key Features of KubeArmor

Container-Aware Security Policies: KubeArmor allows administrators to define security policies at the container level, ensuring that each workload operates within its designated security context.

System Call Filtering: By intercepting and filtering system calls made by containers, KubeArmor can prevent unauthorized actions and enforce security policies in real-time.

Network Policies: KubeArmor enables administrators to define network policies to control inbound and outbound traffic between containers, helping to prevent lateral movement and unauthorized access.

Audit Logging: KubeArmor logs all security-related events, providing administrators with visibility into container activities and potential security threats.

How KubeArmor Works KubeArmor operates by deploying an agent as a DaemonSet within the Kubernetes cluster. This agent intercepts system calls made by containers and enforces security policies defined by the administrator. By leveraging the Linux Security Module (LSM) framework, KubeArmor integrates seamlessly with the underlying operating system, ensuring minimal performance overhead.

Install kubearmor

root@master:~# curl -sfL http://get.kubearmor.io/ | sudo sh -s -- -b /usr/local/bin
kubearmor/kubearmor-client info checking GitHub for latest tag
kubearmor/kubearmor-client info found version: 1.2.1 for v1.2.1/linux/amd64
kubearmor/kubearmor-client info installed /usr/local/bin/karmor
root@master:~# karmor install
🛡       Installed helm release : kubearmor-operator
😄      KubeArmorConfig created
⌚️      This may take a couple of minutes
ℹ️       Waiting for KubeArmor Snitch to run: —

Verify the Installation

root@master:~# kubectl get all -n kubearmor
NAME                                            READY   STATUS    RESTARTS   AGE
pod/kubearmor-apparmor-containerd-98c2c-6k2mc   1/1     Running   0          44m
pod/kubearmor-apparmor-containerd-98c2c-hhgww   1/1     Running   0          3m19s
pod/kubearmor-apparmor-containerd-98c2c-m2974   1/1     Running   0          45m
pod/kubearmor-controller-575b4b46c5-jnppw       2/2     Running   0          3m10s
pod/kubearmor-operator-5bcfb76b4f-8kkwp         1/1     Running   0          47m
pod/kubearmor-relay-6b59fbf77f-wqp7r            1/1     Running   0          46m

NAME                                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
service/kubearmor                              ClusterIP   10.97.110.240    <none>        32767/TCP   46m
service/kubearmor-controller-metrics-service   ClusterIP   10.103.151.164   <none>        8443/TCP    46m
service/kubearmor-controller-webhook-service   ClusterIP   10.111.41.204    <none>        443/TCP     46m

NAME                                                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                                                                                                                                            AGE
daemonset.apps/kubearmor-apparmor-containerd-98c2c   3         3         3       3            3           kubearmor.io/btf=yes,kubearmor.io/enforcer=apparmor,kubearmor.io/runtime=containerd,kubearmor.io/seccomp=yes,kubearmor.io/socket=run_containerd_containerd.sock,kubernetes.io/os=linux   45m

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kubearmor-controller   1/1     1            1           46m
deployment.apps/kubearmor-operator     1/1     1            1           47m
deployment.apps/kubearmor-relay        1/1     1            1           46m

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/kubearmor-controller-575b4b46c5   1         1         1       3m10s
replicaset.apps/kubearmor-controller-64b5b9d54b   0         0         0       46m
replicaset.apps/kubearmor-operator-5bcfb76b4f     1         1         1       47m
replicaset.apps/kubearmor-relay-6b59fbf77f        1         1         1       46m

Deploy nginx app in default namespace

root@master:~# kubectl create deployment nginx --image=nginx
deployment.apps/nginx created

root@master:~# kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
nginx-748c667d99-qjjxn   1/1     Running   0          34s
root@master:~# POD=$(kubectl get pod -l app=nginx -o name)
root@master:~# echo $POD
pod/nginx-748c667d99-qjjxn

Use Case: Deny Installation of package Management Tool Creating a policy to deny the execution

root@master:~# cat <<EOF | kubectl apply -f -
apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: block-pkg-mgmt-tools-exec
spec:
  selector:
    matchLabels:
      app: nginx
  process:
    matchPaths:
    - path: /usr/bin/apt
    - path: /usr/bin/apt-get
  action:
    Block
EOF
kubearmorpolicy.security.kubearmor.com/block-pkg-mgmt-tools-exec created

Lets Verify installing package

root@master:~# kubectl exec -it $POD -- bash -c "apt update && apt install masscan"
bash: line 1: /usr/bin/apt: Permission denied
command terminated with exit code 126

Use Case: Deny Access to Service Token To mitigate the risk posed by default service account tokens being mounted in every pod, even if not utilized by applications, you can restructure your Kubernetes cluster's service account setup. Policy to restrict access

root@master:~# cat <<EOF | kubectl apply -f -
apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: block-service-access-token-access
spec:
  selector:
    matchLabels:
      app: nginx
  file:
    matchDirectories:
    - dir: /run/secrets/kubernetes.io/serviceaccount/
      recursive: true
  action:
    Block
EOF
kubearmorpolicy.security.kubearmor.com/block-service-access-token-access created

Verify the service token access

root@nginx-748c667d99-qjjxn:/# kubectl exec -it $POD -- bash
root@nginx-748c667d99-qjjxn:/# curl https://$KUBERNETES_PORT_443_TCP_ADDR/api --insecure --header "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)"
cat: /run/secrets/kubernetes.io/serviceaccount/token: Permission denied
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/api\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403

Use Case: Least Permission Policy

root@master:~# kubectl annotate ns default kubearmor-file-posture=block --overwrite
namespace/default annotated
root@master:~# cat <<EOF | kubectl apply -f -
apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: only-allow-nginx-exec
spec:
  selector:
    matchLabels:
      app: nginx
  file:
    matchDirectories:
    - dir: /
      recursive: true
  process:
    matchPaths:
    - path: /usr/sbin/nginx
    - path: /bin/bash
  action:
    Allow
EOF
kubearmorpolicy.security.kubearmor.com/only-allow-nginx-exec created

Verify the Access

root@master:~# kubectl exec -it $POD -- bash -c "chroot"
bash: line 1: /usr/sbin/chroot: Permission denied
command terminated with exit code 126

Benefits of Using KubeArmor Enhanced Security Posture: By enforcing fine-grained security policies at the container level, KubeArmor reduces the attack surface and mitigates the risk of unauthorized access and data breaches.

Compliance and Auditing: KubeArmor's audit logging capabilities help organizations demonstrate compliance with regulatory requirements and industry standards by providing detailed records of security-related events.

Operational Efficiency: With KubeArmor, administrators can centrally manage security policies across their Kubernetes clusters, streamlining security operations and reducing manual overhead.

Conclusion In today's threat landscape, securing Kubernetes environments is paramount for organizations seeking to harness the benefits of containerization without compromising on security. KubeArmor offers a robust solution for enhancing Kubernetes security by providing fine-grained security controls tailored to containerized workloads. By leveraging KubeArmor's capabilities, organizations can mitigate security risks, achieve compliance, and ensure the integrity of their Kubernetes deployments.

Introduction

Cert-manager creates TLS certificates for workloads in your Kubernetes or OpenShift cluster and renews the certificates before they expire. Cert-manager can obtain certificates from a variety of certificate authorities, including: Let's Encrypt, HashiCorp Vault, Venafi and private PKI.

In this blog I am using Hashicorp Vault as Certificate Issuer with Cert-Manager

Login to Vault( I am running vault in my cluster)

root@master:~/vault# kubectl exec -it vault-0 -- /bin/sh

Enable the PKI secrets engine at its default path.

/ $ vault secrets enable pki
Success! Enabled the pki secrets engine at: pki/

Configure the max lease time-to-live

/ $ vault secrets tune -max-lease-ttl=8760h pki
Success! Tuned the secrets engine at: pki/

Generate a self-signed certificate

/ $ vault write pki/root/generate/internal \
>     common_name=arobyte.tech \
>     ttl=8760h
Key              Value
---              -----
certificate      -----BEGIN CERTIFICATE-----
MIIDODCCAiCgAwIBAgIUekLUNWVLV3am8DTRk33Y9KX0t8kwDQYJKoZIhvcNAQEL
BQAwFzEVMBMGA1UEAxMMYXJvYnl0ZS50ZWNoMB4XDTI0MDMxMzE1NDU0NVoXDTI1
MDMxMzE1NDYxNVowFzEVMBMGA1UEAxMMYXJvYnl0ZS50ZWNoMIIBIjANBgkqhkiG

-----END CERTIFICATE-----
expiration       1741880775
issuing_ca       -----BEGIN CERTIFICATE-----
MIIDODCCAiCgAwIBAgIUekLUNWVLV3am8DTRk33Y9KX0t8kwDQYJKoZIhvcNAQEL
BQAwFzEVMBMGA1UEAxMMYXJvYnl0ZS50ZWNoMB4XDTI0MDMxMzE1NDU0NVoXDTI1
zw4bj+X2hQyMqu5QHdFF4n58s9I9M5oq9IIBlMqxQqQdN79UirJc/LTk71roOKi7
PD1A3HmuNnWt04+0f8maI9txbUToWq15t8d5zBoM85sF2AGc04OmQmXvL+cGqImJ
9+RIo+iKIJnLiAMt
-----END CERTIFICATE-----
serial_number    7a:42:d4:35:65:4b:57:76:a6:f0:34:d1:93:7d:d8:f4:a5:f4:b7:c9

Configure the PKI secrets engine certificate issuing and certificate revocation list (CRL) endpoints to use the Vault service in the default namespace.

/ $ vault write pki/config/urls \
>     issuing_certificates="http://vault.vault.svc.cluster.local:8200/v1/pki/ca" \
>     crl_distribution_points="http://vault.vault.svc.cluster.local:8200/v1/pki/crl"
Success! Data written to: pki/config/urls

Configure a role named arobyte-role-tech that enables the creation of certificates robyte.tech domain with any subdomains.

/ $ vault write pki/roles/arobyte-role-tech \
>     allowed_domains=arobyte.tech \
>     allow_subdomains=true \
>     max_ttl=72h
Success! Data written to: pki/roles/arobyte-role-tech

Create a policy named pki that enables read access to the PKI secrets engine paths.

/ $ vault policy write pki - <<EOF
> path "pki*"                        { capabilities = ["read", "list"] }
> path "pki/sign/arobyte-role-com"    { capabilities = ["create", "update"] }
> path "pki/issue/arobyte-role-com"   { capabilities = ["create"] }
> EOF
Success! Uploaded policy: pki

Enable the Kubernetes authentication method.

/ $ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

Configure the Kubernetes authentication method to use location of the Kubernetes API.

/ $ vault write auth/kubernetes/config \
>     kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
Success! Data written to: auth/kubernetes/config

Create a Kubernetes authentication role named issuer that binds the pki policy with a Kubernetes service account named issuer

/ $ vault write auth/kubernetes/role/issuer \
>     bound_service_account_names=issuer \
>     bound_service_account_namespaces=default \
>     policies=pki \
>     ttl=20m
Success! Data written to: auth/kubernetes/role/issuer

Lets Install Cert-manager

root@master:~# kubectl create namespace cert-manager

Install Jetstack's cert-manager's version 1.12.3 resources.

root@master:~# kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v1.12.3/cert-manager.crds.yaml
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created

root@master:~# helm repo add jetstack https://charts.jetstack.io
"jetstack" has been added to your repositories
root@master:~# helm repo update
...Successfully got an update from the "jetstack" chart repository
Update Complete. ⎈Happy Helming!⎈

Install the cert-manager

root@master:~# helm install cert-manager \
    --namespace cert-manager \
    --version v1.12.3 \
  jetstack/cert-manager

NAME: cert-manager
LAST DEPLOYED: Wed Mar 13 21:26:19 2024
NAMESPACE: cert-manager
STATUS: deployed

Check the status

root@master:~# kubectl get pods --namespace cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-65dfbdf7d6-qp5fk              1/1     Running   0          3m44s
cert-manager-cainjector-79f5dbffcf-lh4d6   1/1     Running   0          3m44s
cert-manager-webhook-77b984cc67-8nxhh      1/1     Running   0          3m44s

Create a service account named issuer within the default namespace.

root@master:~# kubectl create serviceaccount issuer
serviceaccount/issuer created

Create a secret definition

root@master:~# cat >> issuer-secret.yaml <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: issuer-token
  annotations:
    kubernetes.io/service-account.name: issuer
type: kubernetes.io/service-account-token
EOF
root@master:~# kubectl apply -f issuer-secret.yaml
secret/issuer-token created

root@master:~# kubectl get secrets
NAME                 TYPE                                  DATA   AGE
issuer-token   kubernetes.io/service-account-token   3      7s

Define an Issuer, named vault-issuer, that sets Vault as a certificate issuer.

root@master:~# cat > vault-issuer.yaml <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault-issuer
  namespace: default
spec:
  vault:
    server: http://vault.vault.svc.cluster.local:8200
    path: pki/sign/arobyte-role-tech
    auth:
      kubernetes:
        mountPath: /v1/auth/kubernetes
        role: issuer
        secretRef:
          name: issuer-token
          key: token
EOF
root@master:~# kubectl apply --filename vault-issuer.yaml
issuer.cert-manager.io/vault-issuer created

Create the arobyte-tech certificate

root@master:~# cat arobyte-tech-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: arobyte-tech
  namespace: default
spec:
  secretName: arobyte-role-tech
  issuerRef:
    name: vault-issuer
  commonName: www.arobyte.tech
  dnsNames:
  - www.arobyte.tech
root@master:~# kubectl apply --filename arobyte-tech-cert.yaml

View the details of the arobyte-tech certificate

root@master:~# kubectl describe certificate.cert-manager arobyte-tech -n default
Name:         arobyte-tech
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate


Events:
  Type    Reason     Age    From                                       Message
  ----    ------     ----   ----                                       -------
  Normal  Issuing    2m55s  cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal  Generated  2m55s  cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "arobyte-tech-gwkjp"
  Normal  Requested  2m54s  cert-manager-certificates-request-manager  Created new CertificateRequest resource "arobyte-tech-fdvjr"
  Normal  Issuing    27s    cert-manager-certificates-issuing          The certificate has been successfully issued

The certificate reports that it has been issued successfully.

Verify the Certificates

root@master:~# kubectl get certificate
NAME           READY   SECRET              AGE
arobyte-tech   True    arobyte-role-tech   95m


root@master:~# kubectl describe secrets arobyte-role-tech
Name:         arobyte-role-tech
Namespace:    default
Labels:       controller.cert-manager.io/fao=true
Annotations:  cert-manager.io/alt-names: www.arobyte.tech
              cert-manager.io/certificate-name: arobyte-tech
              cert-manager.io/common-name: www.arobyte.tech
              cert-manager.io/ip-sans:
              cert-manager.io/issuer-group:
              cert-manager.io/issuer-kind: Issuer
              cert-manager.io/issuer-name: vault-issuer
              cert-manager.io/uri-sans:

Type:  kubernetes.io/tls

Data
====
tls.key:  1675 bytes
ca.crt:   1176 bytes
tls.crt:  1419 bytes

root@master:~# kubectl describe certificate arobyte-tech
Name:         arobyte-tech
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:

Spec:
  Common Name:  www.arobyte.tech
  Dns Names:
    www.arobyte.tech
  Issuer Ref:
    Name:       vault-issuer
  Secret Name:  arobyte-role-tech
Status:
  Conditions:
    Last Transition Time:  2024-03-13T17:43:48Z
    Message:               Certificate is up to date and has not expired
    Observed Generation:   1
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2024-03-16T17:43:47Z
  Not Before:              2024-03-13T17:43:17Z
  Renewal Time:            2024-03-15T17:43:37Z
  Revision:                1
Events:                    <none>

Multi-Cluster (Cluster Mesh)

Cluster mesh extends the networking datapath across multiple clusters. It allows endpoints in all connected clusters to communicate while providing full policy enforcement. Load-balancing is available via Kubernetes annotations.

Setting up Cluster Mesh A Cilium MultiCluster setup involves deploying Cilium, a powerful networking and security solution for Kubernetes, across multiple Kubernetes clusters. This setup enables communication and workload deployment across clusters, providing benefits such as fault tolerance, disaster recovery, and improved performance. Here's an explanation of various aspects of Cilium MultiCluster

Architecture:

In a MultiCluster architecture with Cilium, you have multiple Kubernetes clusters deployed across different geographic regions, data centers, or cloud providers.

Each Kubernetes cluster runs Cilium as its CNI (Container Network Interface) plugin to manage networking, security, and observability for containerized workloads.

Connectivity:

Cilium facilitates connectivity between pods and services deployed across different Kubernetes clusters.

It achieves this through mechanisms like cluster federation, network peering, or VPN tunnels, depending on the chosen MultiCluster setup.

Service Discovery:

Cilium provides service discovery capabilities across MultiCluster environments, allowing pods in one cluster to discover and communicate with services deployed in other clusters.

This enables seamless communication between microservices distributed across multiple clusters.

Network Policies:

Cilium allows the enforcement of network policies across MultiCluster deployments, ensuring that traffic between clusters adheres to security and compliance requirements.

Network policies define rules for traffic filtering, segmentation, and access control based on various criteria such as IP addresses, ports, and labels.

Identity-Aware Networking:

Cilium supports identity-aware networking across MultiCluster environments, allowing granular control over communication based on workload identities.

Workload identities, such as Kubernetes service accounts or custom attributes, can be used to enforce fine-grained access control policies between clusters.

Observability:

Cilium provides comprehensive observability features for MultiCluster environments, including real-time network visibility, metrics collection, and distributed tracing.

Operators can monitor network traffic, performance metrics, and security events across clusters to ensure operational efficiency and security.

Security:

Cilium enhances security in MultiCluster environments by enforcing security policies, encrypting inter-cluster communication, and providing threat detection capabilities.

It protects against network-based attacks, ensures data confidentiality, and enables secure communication between clusters.

Scalability and Performance:

Cilium is designed for scalability and performance in MultiCluster deployments, leveraging eBPF (extended Berkeley Packet Filter) technology for efficient packet processing and low-latency networking.

It supports large-scale deployments spanning multiple clusters while maintaining high throughput and low latency for inter-cluster communication.

Cilium MultiCluster offers a robust networking and security solution for Kubernetes environments spanning multiple clusters. It enables seamless communication, service discovery, and security enforcement across clusters, empowering organizations to build resilient, scalable, and secure distributed applications.

Setting up Cluster Mesh This is a step-by-step guide on how to build a mesh of Kubernetes clusters by connecting them together, enable pod-to-pod connectivity across all clusters, define global services to load-balance between clusters.

Prerequisites

2 Kubernetes Clusters

Cluster-A

root@master:~# kubectl get nodes
NAME                   STATUS   ROLES           AGE   VERSION
master.arobyte.tech    Ready    control-plane   58m   v1.26.0
worker1.arobyte.tech   NotReady    <none>          41m   v1.26.0
worker2.arobyte.tech   NotReady    <none>          40m   v1.26.0

Cluster-B

root@devmaster:~# kubectl get nodes
NAME                      STATUS   ROLES           AGE     VERSION
devmaster.arobyte.tech    Ready    control-plane   3h57m   v1.26.0
devworker1.arobyte.tech   NotReady    <none>          39m     v1.26.0
devworker2.arobyte.tech   NotReady    <none>          38m     v1.26.0

Let's Install Cilium on Both The Cluster

Cluster-A

root@master:~# kubectl config use-context master

root@master:~# helm repo add cilium https://helm.cilium.io/

helm install cilium cilium/cilium --version 1.15.1 \
   --namespace kube-system \
   --set nodeinit.enabled=true \
   --set kubeProxyReplacement=partial \
   --set hostServices.enabled=false \
   --set externalIPs.enabled=true \
   --set nodePort.enabled=true \
   --set hostPort.enabled=true \
   --set cluster.name=master \
   --set cluster.id=1

root@master:~# kubectl get nodes
NAME                   STATUS   ROLES           AGE   VERSION
master.arobyte.tech    Ready    control-plane   58m   v1.26.0
worker1.arobyte.tech   Ready    <none>          41m   v1.26.0
worker2.arobyte.tech   Ready    <none>          40m   v1.26.0

Cluster-B

root@master:~# kubectl config use-context devmaster

root@master:~# helm install cilium cilium/cilium --version 1.15.1 \
   --namespace kube-system \
   --set nodeinit.enabled=true \
   --set kubeProxyReplacement=partial \
   --set hostServices.enabled=false \
   --set externalIPs.enabled=true \
   --set nodePort.enabled=true \
   --set hostPort.enabled=true \
   --set cluster.name=devmaster \
   --set cluster.id=2

root@master:~# kubectl get pods -n metallb-system
NAME                                 READY   STATUS    RESTARTS   AGE
metallb-controller-9fcb75cf5-8s5w4   1/1     Running   0          35m
metallb-speaker-v769s                4/4     Running   0          35m
metallb-speaker-vfg62                4/4     Running   0          35m
metallb-speaker-www9n                4/4     Running   0          35m

Metallb Load Balancer Must be installed on Both the Cluster For Load balancer.

Cluster-A

root@master:~# kubectl get pods -n metallb-system
NAME                                 READY   STATUS    RESTARTS   AGE
metallb-controller-9fcb75cf5-mds7w   1/1     Running   0          30m
metallb-speaker-cfsw4                4/4     Running   0          30m
metallb-speaker-nblvz                4/4     Running   0          30m
metallb-speaker-sr68w                4/4     Running   0          30m

Cluster-B

root@devmaster:~# kubectl get pods -n metallb-system
NAME                                 READY   STATUS    RESTARTS   AGE
metallb-controller-9fcb75cf5-8s5w4   1/1     Running   0          35m
metallb-speaker-v769s                4/4     Running   0          35m
metallb-speaker-vfg62                4/4     Running   0          35m
metallb-speaker-www9n                4/4     Running   0          35m

Now We can see on Cluster-A and Cluster-B all the nodes are in Ready state after installing Cilium CNI

Enable Cilium Multicluster on Both Clusters

root@master:~# cilium clustermesh enable --context master --service-type LoadBalancer
root@master:~# cilium clustermesh status --context master --wait
✅ Service "clustermesh-apiserver" of type "LoadBalancer" found
✅ Cluster access information is available:
  - 172.16.16.150:2379
⌛ Waiting (0s) for deployment clustermesh-apiserver to become ready: only 0 of 1 replicas are available

root@master:~# cilium clustermesh enable --context devmaster --service-type LoadBalancer
root@master:~# cilium clustermesh status --context master --wait
✅ Service "clustermesh-apiserver" of type "LoadBalancer" found
✅ Cluster access information is available:
  - 172.16.16.150:2379
✅ Deployment clustermesh-apiserver is ready
🔌 No cluster connected
🔀 Global services: [ min:0 / avg:0.0 / max:0 ]

root@master:~# cilium clustermesh status --context devmaster --wait
✅ Service "clustermesh-apiserver" of type "LoadBalancer" found
✅ Cluster access information is available:
  - 172.17.17.150:2379
✅ Deployment clustermesh-apiserver is ready
🔌 No cluster connected
🔀 Global services: [ min:0 / avg:0.0 / max:0 ]

We can see cluster mesh installed Successfully on both Clusters

Lets connect both clusters together

root@master:~# cilium clustermesh connect --context master \
   --destination-context devmaster
✅ Detected Helm release with Cilium version 1.15.1
✨ Extracting access information of cluster devmaster...
🔑 Extracting secrets from cluster devmaster...
ℹ️  Found ClusterMesh service IPs: [172.17.17.150]
✨ Extracting access information of cluster master...
🔑 Extracting secrets from cluster master...
ℹ️  Found ClusterMesh service IPs: [172.16.16.150]
⚠️ Cilium CA certificates do not match between clusters. Multicluster features will be limited!
ℹ️ Configuring Cilium in cluster 'master' to connect to cluster 'devmaster'
ℹ️ Configuring Cilium in cluster 'devmaster' to connect to cluster 'master'
✅ Connected cluster master and devmaster!

Let’s verify the status of the Cilium cluster mesh once again

root@master:~# cilium clustermesh status --context master --wait
✅ Service "clustermesh-apiserver" of type "LoadBalancer" found
✅ Cluster access information is available:
  - 172.16.16.150:2379
✅ Deployment clustermesh-apiserver is ready
✅ All 3 nodes are connected to all clusters [min:1 / avg:1.0 / max:1]
🔌 Cluster Connections:
  - devmaster: 3/3 configured, 3/3 connected
🔀 Global services: [ min:0 / avg:0.0 / max:0 ]


root@master:~# cilium clustermesh status --context devmaster --wait
✅ Service "clustermesh-apiserver" of type "LoadBalancer" found
✅ Cluster access information is available:
  - 172.17.17.150:2379
✅ Deployment clustermesh-apiserver is ready
✅ All 3 nodes are connected to all clusters [min:1 / avg:1.0 / max:1]
🔌 Cluster Connections:
  - master: 3/3 configured, 3/3 connected
🔀 Global services: [ min:0 / avg:0.0 / max:0 ]

Verify the Kubernetes Service with Cilium Mesh Control Plane

root@master:~# kubectl config use-context devmaster
root@master:~# kubectl get svc -A | grep clustermesh
kube-system      clustermesh-apiserver           LoadBalancer   10.98.126.70     172.17.17.150   2379:32200/TCP           34m
kube-system      clustermesh-apiserver-metrics   ClusterIP      None             <none>          9962/TCP,9963/TCP        34m

root@master:~# kubectl config use-context master
Switched to context "master".
root@master:~# kubectl get svc -A | grep clustermesh
kube-system      clustermesh-apiserver           LoadBalancer   10.105.111.27    172.16.16.150   2379:31285/TCP           37m
kube-system      clustermesh-apiserver-metrics   ClusterIP      None             <none>          9962/TCP,9963/TCP        37m

Validate the connectivity by running the connectivity test in multi-cluster mode

root@master:~# cilium connectivity test --context master --multi-cluster devmaster
ℹ️  Monitor aggregation detected, will skip some flow validation steps
✨ [master] Creating namespace cilium-test for connectivity check...
✨ [devmaster] Creating namespace cilium-test for connectivity check...
✨ [master] Deploying echo-same-node service...
✨ [master] Deploying echo-other-node service...
✨ [master] Deploying DNS test server configmap...
✨ [devmaster] Deploying DNS test server configmap...
✨ [master] Deploying same-node deployment...
✨ [master] Deploying client deployment...
✨ [master] Deploying client2 deployment...
✨ [devmaster] Deploying echo-other-node service...
✨ [devmaster] Deploying other-node deployment...
✨ [host-netns] Deploying master daemonset...
✨ [host-netns] Deploying devmaster daemonset...
✨ [host-netns-non-cilium] Deploying master daemonset...
ℹ️  Skipping tests that require a node Without Cilium
⌛ [master] Waiting for deployment cilium-test/client to become ready...
⌛ [master] Waiting for deployment cilium-test/client2 to become ready...
⌛ [master] Waiting for deployment cilium-test/echo-same-node to become ready...
⌛ [devmaster] Waiting for deployment cilium-test/echo-other-node to become ready...
⌛ [master] Waiting for pod cilium-test/client-65847bf96-2mqtd to reach DNS server on cilium-test/echo-same-node-66f8958597-mbr7s pod...
⌛ [master] Waiting for pod cilium-test/client2-85585bdd-bnn77 to reach DNS server on cilium-test/echo-same-node-66f8958597-mbr7s pod...
⌛ [master] Waiting for pod cilium-test/client-65847bf96-2mqtd to reach DNS server on cilium-test/echo-other-node-5db9b7bbbc-lvsqj pod...
⌛ [master] Waiting for pod cilium-test/client2-85585bdd-bnn77 to reach DNS server on cilium-test/echo-other-node-5db9b7bbbc-lvsqj pod...
⌛ [master] Waiting for pod cilium-test/client-65847bf96-2mqtd to reach default/kubernetes service...
⌛ [master] Waiting for pod cilium-test/client2-85585bdd-bnn77 to reach default/kubernetes service...
⌛ [master] Waiting for Service cilium-test/echo-other-node to become ready...
⌛ [master] Waiting for Service cilium-test/echo-other-node to be synchronized by Cilium pod kube-system/cilium-6qfz6
⌛ [master] Waiting for Service cilium-test/echo-same-node to become ready...
⌛ [master] Waiting for Service cilium-test/echo-same-node to be synchronized by Cilium pod kube-system/cilium-6qfz6
⌛ [master] Waiting for DaemonSet cilium-test/host-netns-non-cilium to become ready...
⌛ [master] Waiting for DaemonSet cilium-test/host-netns to become ready...
⌛ [devmaster] Waiting for DaemonSet cilium-test/host-netns to become ready...
ℹ️  Skipping IPCache check
🔭 Enabling Hubble telescope...
⚠️  Unable to contact Hubble Relay, disabling Hubble telescope and flow validation: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:4245: connect: connection refused"
ℹ️  Expose Relay locally with:
   cilium hubble enable
   cilium hubble port-forward&
ℹ️  Cilium version: 1.15.1
🏃 Running 66 tests ...
[=] Test [no-unexpected-packet-drops] [1/66]

  [-] Scenario [no-unexpected-packet-drops/no-unexpected-packet-drops]
  🟥 Found unexpected packet drops:
{
  "labels": {
    "direction": "EGRESS",
    "reason": "Unsupported L2 protocol"
  },
  "name": "cilium_drop_count_total",
  "value": 428
}
[=] Test [no-policies] [2/66]
.....................
  [-] Scenario [no-policies/pod-to-pod]
  [.] Action [no-policies/pod-to-pod/curl-ipv4-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-same-node-66f8958597-mbr7s (10.0.2.109:8080)]
  [.] Action [no-policies/pod-to-pod/curl-ipv4-1: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-other-node-5db9b7bbbc-lvsqj (10.0.2.163:8080)]
  [.] Action [no-policies/pod-to-pod/curl-ipv4-2: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-same-node-66f8958597-mbr7s (10.0.2.109:8080)]
  [.] Action [no-policies/pod-to-pod/curl-ipv4-3: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-other-node-5db9b7bbbc-lvsqj (10.0.2.163:8080)]
  [-] Scenario [no-policies/pod-to-world]
  [.] Action [no-policies/pod-to-world/http-to-one.one.one.one-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> one.one.one.one-http (one.one.one.one:80)]
  [.] Action [no-policies/pod-to-world/https-to-one.one.one.one-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> one.one.one.one-https (one.one.one.one:443)]
  [.] Action [no-policies/pod-to-world/https-to-one.one.one.one-index-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> one.one.one.one-https-index (one.one.one.one:443)]
  [.] Action [no-policies/pod-to-world/http-to-one.one.one.one-1: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> one.one.one.one-http (one.one.one.one:80)]
  [.] Action [no-policies/pod-to-world/https-to-one.one.one.one-1: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> one.one.one.one-https (one.one.one.one:443)]
  [.] Action [no-policies/pod-to-world/https-to-one.one.one.one-index-1: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> one.one.one.one-https-index (one.one.one.one:443)]
  [-] Scenario [no-policies/pod-to-cidr]
  [.] Action [no-policies/pod-to-cidr/external-1111-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> external-1111 (1.1.1.1:443)]
  [.] Action [no-policies/pod-to-cidr/external-1111-1: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> external-1111 (1.1.1.1:443)]
  [.] Action [no-policies/pod-to-cidr/external-1001-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> external-1001 (1.0.0.1:443)]
  [.] Action [no-policies/pod-to-cidr/external-1001-1: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> external-1001 (1.0.0.1:443)]
  [-] Scenario [no-policies/client-to-client]
  [.] Action [no-policies/client-to-client/ping-ipv4-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/client2-85585bdd-bnn77 (10.0.2.148:0)]
  [.] Action [no-policies/client-to-client/ping-ipv4-1: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/client-65847bf96-2mqtd (10.0.2.95:0)]
  [-] Scenario [no-policies/pod-to-service]
  [.] Action [no-policies/pod-to-service/curl-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-other-node (echo-other-node.cilium-test:8080)]
  [.] Action [no-policies/pod-to-service/curl-1: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-same-node (echo-same-node.cilium-test:8080)]
  [.] Action [no-policies/pod-to-service/curl-2: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-same-node (echo-same-node.cilium-test:8080)]
  [.] Action [no-policies/pod-to-service/curl-3: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-other-node (echo-other-node.cilium-test:8080)]
  [-] Scenario [no-policies/pod-to-hostport]
  [.] Action [no-policies/pod-to-hostport/curl-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-other-node-5db9b7bbbc-lvsqj (172.17.17.101:4000)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://172.17.17.101:4000" failed: error with exec request (pod=cilium-test/client-65847bf96-2mqtd, container=client): command terminated with exit code 28
  ℹ️  curl output:


  [.] Action [no-policies/pod-to-hostport/curl-1: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-same-node-66f8958597-mbr7s (172.16.16.102:4000)]
  [.] Action [no-policies/pod-to-hostport/curl-2: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-same-node-66f8958597-mbr7s (172.16.16.102:4000)]
  [.] Action [no-policies/pod-to-hostport/curl-3: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-other-node-5db9b7bbbc-lvsqj (172.17.17.101:4000)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://172.17.17.101:4000" failed: error with exec request (pod=cilium-test/client2-85585bdd-bnn77, container=client2): command terminated with exit code 28
  ℹ️  curl output:


  [-] Scenario [no-policies/pod-to-host]
  [.] Action [no-policies/pod-to-host/ping-ipv4-internal-ip: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> 192.168.0.114 (192.168.0.114:0)]
  [.] Action [no-policies/pod-to-host/ping-ipv4-internal-ip: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> 172.16.16.101 (172.16.16.101:0)]
  [.] Action [no-policies/pod-to-host/ping-ipv4-internal-ip: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> 172.16.16.102 (172.16.16.102:0)]
  [.] Action [no-policies/pod-to-host/ping-ipv4-internal-ip: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> 192.168.0.114 (192.168.0.114:0)]
  [.] Action [no-policies/pod-to-host/ping-ipv4-internal-ip: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> 172.16.16.101 (172.16.16.101:0)]
  [.] Action [no-policies/pod-to-host/ping-ipv4-internal-ip: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> 172.16.16.102 (172.16.16.102:0)]
  [-] Scenario [no-policies/host-to-pod]
  [.] Action [no-policies/host-to-pod/curl-ipv4-0: cilium-test/host-netns-gnqtv (172.16.16.101) -> cilium-test/echo-same-node-66f8958597-mbr7s (10.0.2.109:8080)]
  [.] Action [no-policies/host-to-pod/curl-ipv4-1: cilium-test/host-netns-gnqtv (172.16.16.101) -> cilium-test/echo-other-node-5db9b7bbbc-lvsqj (10.0.2.163:8080)]
  [.] Action [no-policies/host-to-pod/curl-ipv4-2: cilium-test/host-netns-rswsn (172.17.17.102) -> cilium-test/echo-same-node-66f8958597-mbr7s (10.0.2.109:8080)]
  [.] Action [no-policies/host-to-pod/curl-ipv4-3: cilium-test/host-netns-rswsn (172.17.17.102) -> cilium-test/echo-other-node-5db9b7bbbc-lvsqj (10.0.2.163:8080)]
  [.] Action [no-policies/host-to-pod/curl-ipv4-4: cilium-test/host-netns-xfkhb (172.17.17.101) -> cilium-test/echo-same-node-66f8958597-mbr7s (10.0.2.109:8080)]
  [.] Action [no-policies/host-to-pod/curl-ipv4-5: cilium-test/host-netns-xfkhb (172.17.17.101) -> cilium-test/echo-other-node-5db9b7bbbc-lvsqj (10.0.2.163:8080)]
  [.] Action [no-policies/host-to-pod/curl-ipv4-6: cilium-test/host-netns-5x8wz (172.16.16.102) -> cilium-test/echo-same-node-66f8958597-mbr7s (10.0.2.109:8080)]
  [.] Action [no-policies/host-to-pod/curl-ipv4-7: cilium-test/host-netns-5x8wz (172.16.16.102) -> cilium-test/echo-other-node-5db9b7bbbc-lvsqj (10.0.2.163:8080)]
  [-] Scenario [no-policies/pod-to-external-workload]
[=] Test [no-policies-extra] [3/66]
.
  [-] Scenario [no-policies-extra/pod-to-remote-nodeport]
  [.] Action [no-policies-extra/pod-to-remote-nodeport/curl-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-other-node (echo-other-node.cilium-test:8080)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://192.168.0.114:32184" failed: error with exec request (pod=cilium-test/client-65847bf96-2mqtd, container=client): command terminated with exit code 7
  ℹ️  curl output:


  [.] Action [no-policies-extra/pod-to-remote-nodeport/curl-1: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-other-node (echo-other-node.cilium-test:8080)]
  [.] Action [no-policies-extra/pod-to-remote-nodeport/curl-2: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-same-node (echo-same-node.cilium-test:8080)]
  [.] Action [no-policies-extra/pod-to-remote-nodeport/curl-3: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-same-node (echo-same-node.cilium-test:8080)]
  [.] Action [no-policies-extra/pod-to-remote-nodeport/curl-4: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-other-node (echo-other-node.cilium-test:8080)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://192.168.0.114:32184" failed: error with exec request (pod=cilium-test/client2-85585bdd-bnn77, container=client2): command terminated with exit code 7
  ℹ️  curl output:


  [.] Action [no-policies-extra/pod-to-remote-nodeport/curl-5: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-other-node (echo-other-node.cilium-test:8080)]
  [.] Action [no-policies-extra/pod-to-remote-nodeport/curl-6: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-same-node (echo-same-node.cilium-test:8080)]
  [.] Action [no-policies-extra/pod-to-remote-nodeport/curl-7: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-same-node (echo-same-node.cilium-test:8080)]
  [-] Scenario [no-policies-extra/pod-to-local-nodeport]
  [.] Action [no-policies-extra/pod-to-local-nodeport/curl-0: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-other-node (echo-other-node.cilium-test:8080)]
  [.] Action [no-policies-extra/pod-to-local-nodeport/curl-1: cilium-test/client-65847bf96-2mqtd (10.0.2.95) -> cilium-test/echo-same-node (echo-same-node.cilium-test:8080)]
  [.] Action [no-policies-extra/pod-to-local-nodeport/curl-2: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-other-node (echo-other-node.cilium-test:8080)]
  [.] Action [no-policies-extra/pod-to-local-nodeport/curl-3: cilium-test/client2-85585bdd-bnn77 (10.0.2.148) -> cilium-test/echo-same-node (echo-same-node.cilium-test:8080)]
[=] Test [allow-all-except-world] [4/66]
................
[=] Test [client-ingress] [5/66]
------------
----------------
---------------
-------------------

Deploying a Simple Example Service

In Cluster-A, deploy:

root@master:~# kubectl config use-context master
Switched to context "master"
root@master:~# kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.15.1/examples/kubernetes/clustermesh/global-service-example/cluster1.yaml
service/rebel-base created
deployment.apps/rebel-base created
configmap/rebel-base-response created
deployment.apps/x-wing created

In Cluster-B, deploy:

root@master:~# kubectl config use-context devmaster
Switched to context "devmaster".
root@master:~# kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.15.1/examples/kubernetes/clustermesh/global-service-example/cluster2.yaml
service/rebel-base created
deployment.apps/rebel-base created
configmap/rebel-base-response created
deployment.apps/x-wing created

Let's validate the deployment on both the clusters

Cluster-A

root@master:~# kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
rebel-base-5f575fcc5d-46sx4   1/1     Running   0          15m
rebel-base-5f575fcc5d-62nqv   1/1     Running   0          15m
x-wing-58c9c6d949-lgv7j       1/1     Running   0          15m
x-wing-58c9c6d949-vqqxv       1/1     Running   0          15m

Cluster-B

root@devmaster:~# kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
rebel-base-5f575fcc5d-8jfpl   1/1     Running   0          13m
rebel-base-5f575fcc5d-cnw8x   1/1     Running   0          13m
x-wing-58c9c6d949-5nbfh       1/1     Running   0          13m
x-wing-58c9c6d949-tbwfq       1/1     Running   0          13m

From either cluster, access the global service

root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}

We will see replies from pods in both clusters.

In Cluster-A, add service.cilium.io/shared="false" to existing global service

root@master:~# kubectl annotate service rebel-base service.cilium.io/shared="false" --overwrite
service/rebel-base annotated

From Cluster-A, access the global service one more time

root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}

We will still see replies from pods in both clusters

From Cluster-B, access the global service again

root@master:~# kubectl annotate service rebel-base service.cilium.io/shared-
service/rebel-base annotated
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}

We will see replies from pods only from cluster 2, as the global service in cluster 1 is no longer shared

In Cluster-A, remove service.cilium.io/shared annotation of existing global service

root@master:~# kubectl config use-context master
Switched to context "master".
root@master:~# kubectl annotate service rebel-base service.cilium.io/shared-
service/rebel-base annotated

From either cluster, access the global service

root@master:~# kubectl config use-context master
Switched to context "master".
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}

root@master:~# kubectl config use-context devmaster
Switched to context "devmaster".
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
root@master:~# kubectl exec -ti deployment/x-wing -- curl rebel-base
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}

We will see replies from pods in both clusters again

Service Mesh

Service Mesh is a dedicated infrastructure layer that controls service-to-service communication over a network. This method enables separate parts of an application to communicate with each other. Service meshes appear commonly in concert with cloud-based applications, containers and microservices. A service mesh controls the delivery of service requests in an application. Common features provided by a service mesh include service discovery, load balancing, encryption and failure recovery. High availability is also common through the use of software controlled by APIs rather than through hardware. Service meshes can make service-to-service communication fast, reliable and secure.

How a service mesh works

A service mesh architecture uses a proxy instance called a sidecar in whichever development paradigm is in use, typically containers and/or microservices. In a microservice application, a sidecar attaches to each service. In a container, the sidecar attaches to each application container, VM or container orchestration unit, such as a Kubernetes pod. Sidecars can handle tasks abstracted from the service itself, such as monitoring and security.

Service instances, sidecars and their interactions make up what is called the data plane in a service mesh. A different layer called the control plane manages tasks such as creating instances, monitoring and implementing policies for network management and security. Control planes can connect to a CLI or a GUI interface for application management.

Why adopt a service mesh?

An application structured in a microservices architecture might comprise dozens or hundreds of services, all with their own instances that operate in a live environment. It's a big challenge for developers to keep track of which components must interact, monitor their health and performance and make changes to a service or component if something goes wrong.

A service mesh enables developers to separate and manage service-to-service communications in a dedicated infrastructure layer. As the number of microservices involved with an application increases, so do the benefits of using a service mesh to manage and monitor them.

Key features of a service mesh A service mesh framework typically provides many capabilities that make containerized and microservices communications more reliable, secure and observable.

Reliability. Managing communications through sidecar proxies and the control plane improves efficiency and reliability of service requests, policies and configurations. Specific capabilities include load balancing and fault injection.

Observability. Service mesh frameworks can provide insights into the behavior and health of services. The control plane can collect and aggregate telemetry data from component interactions to determine service health, such as traffic and latency, distributed tracing and access logs. Third-party integration with tools, such as Prometheus, Elasticsearch and Grafana, enables further monitoring and visualization.

Security. Service mesh can automatically encrypt communications and distribute security policies, including authentication and authorization, from the network to the application and individual microservices. Centrally managing security policies through the control plane and sidecar proxies helps keep up with increasingly complex connections within and between distributed applications.

Linkerd Service Mesh Linkerd is a service mesh for Kubernetes. It makes running services easier and safer by giving you runtime debugging, observability, reliability, and security—all without requiring any changes to your code.

How it works Linkerd works by installing a set of ultralight, transparent “micro-proxies” next to each service instance. These proxies automatically handle all traffic to and from the service. Because they’re transparent, these proxies act as highly instrumented out-of-process network stacks, sending telemetry to, and receiving control signals from, the control plane. This design allows Linkerd to measure and manipulate traffic to and from your service without introducing excessive latency.

Installation

Setup Validate your Kubernetes setup by running:

root@master:~# kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.26.0
Kustomize Version: v4.5.7
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

Install the CLI

root@master:~# curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install | sh
Downloading linkerd2-cli-edge-24.2.5-linux-amd64...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 52.5M  100 52.5M    0     0  1485k      0  0:00:36  0:00:36 --:--:-- 1796k
Download complete!

Validating checksum...
Checksum valid.

Linkerd edge-24.2.5 was successfully installed 🎉


*******************************************
* This script is deprecated and no longer *
* installs stable releases.               *
*                                         *
* The latest edge release has been        *
* installed. In the future, please use    *
*   run.linkerd.io/install-edge           *
* for this behavior.                      *
*                                         *
* For stable releases, please see         *
*  https://linkerd.io/releases/           *
*******************************************

Add the linkerd CLI to your path with:

  export PATH=$PATH:/root/.linkerd2/bin

root@master:~# export PATH=$HOME/.linkerd2/bin:$PATH

root@master:~# linkerd version
Client version: edge-24.2.5
Server version: unavailable

You should see the CLI version, and also Server version: unavailable. This is because you haven’t installed the control plane on your cluster. Don’t worry—we’ll fix that soon enough. Make sure that your Linkerd version and Kubernetes version are compatible by checking Linkerd’s supported Kubernetes versions.

Validate Kubernetes Cluster

root@master:~# linkerd check --pre
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version

pre-kubernetes-setup
--------------------
√ control plane namespace does not already exist
√ can create non-namespaced resources
√ can create ServiceAccounts
√ can create Services
√ can create Deployments
√ can create CronJobs
√ can create ConfigMaps
√ can create Secrets
√ can read Secrets
√ can read extension-apiserver-authentication configmap
√ no clock skew detected

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

Status check results are √

Install Linkerd CRD onto cluster

root@master:~# linkerd install --crds | kubectl apply -f -
Rendering Linkerd CRDs...
Next, run `linkerd install | kubectl apply -f -` to install the control plane.

customresourcedefinition.apiextensions.k8s.io/authorizationpolicies.policy.linkerd.io created
customresourcedefinition.apiextensions.k8s.io/httproutes.policy.linkerd.io created
customresourcedefinition.apiextensions.k8s.io/meshtlsauthentications.policy.linkerd.io created
customresourcedefinition.apiextensions.k8s.io/networkauthentications.policy.linkerd.io created
customresourcedefinition.apiextensions.k8s.io/serverauthorizations.policy.linkerd.io created
customresourcedefinition.apiextensions.k8s.io/servers.policy.linkerd.io created
customresourcedefinition.apiextensions.k8s.io/serviceprofiles.linkerd.io created
customresourcedefinition.apiextensions.k8s.io/httproutes.gateway.networking.k8s.io created
customresourcedefinition.apiextensions.k8s.io/externalworkloads.workload.linkerd.io created

Install Linkerd onto cluster

root@master:~# linkerd install | kubectl apply -f -
namespace/linkerd created
clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-identity created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-identity created
serviceaccount/linkerd-identity created
clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-destination created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-destination created
serviceaccount/linkerd-destination created
secret/linkerd-sp-validator-k8s-tls created
validatingwebhookconfiguration.admissionregistration.k8s.io/linkerd-sp-validator-webhook-config created
secret/linkerd-policy-validator-k8s-tls created
validatingwebhookconfiguration.admissionregistration.k8s.io/linkerd-policy-validator-webhook-config created
clusterrole.rbac.authorization.k8s.io/linkerd-policy created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-destination-policy created
role.rbac.authorization.k8s.io/remote-discovery created
rolebinding.rbac.authorization.k8s.io/linkerd-destination-remote-discovery created
role.rbac.authorization.k8s.io/linkerd-heartbeat created
rolebinding.rbac.authorization.k8s.io/linkerd-heartbeat created
clusterrole.rbac.authorization.k8s.io/linkerd-heartbeat created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-heartbeat created
serviceaccount/linkerd-heartbeat created
clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-proxy-injector created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-proxy-injector created
serviceaccount/linkerd-proxy-injector created
secret/linkerd-proxy-injector-k8s-tls created
mutatingwebhookconfiguration.admissionregistration.k8s.io/linkerd-proxy-injector-webhook-config created
configmap/linkerd-config created
role.rbac.authorization.k8s.io/ext-namespace-metadata-linkerd-config created
secret/linkerd-identity-issuer created
configmap/linkerd-identity-trust-roots created
service/linkerd-identity created
service/linkerd-identity-headless created
deployment.apps/linkerd-identity created
service/linkerd-dst created
service/linkerd-dst-headless created
service/linkerd-sp-validator created
service/linkerd-policy created
service/linkerd-policy-validator created
deployment.apps/linkerd-destination created
cronjob.batch/linkerd-heartbeat created
deployment.apps/linkerd-proxy-injector created
service/linkerd-proxy-injector created
secret/linkerd-config-overrides created

Check Linkerd Status

root@master:~# linkerd check 

Install the Demo App

root@master:~# curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/emojivoto.yml | kubectl apply -f -
namespace/emojivoto created
serviceaccount/emoji created
serviceaccount/voting created
serviceaccount/web created
service/emoji-svc created
service/voting-svc created
service/web-svc created
deployment.apps/emoji created
deployment.apps/vote-bot created
deployment.apps/voting created
deployment.apps/web created

Validate the App

root@master:~# kubectl get all -n emojivoto
NAME                            READY   STATUS    RESTARTS   AGE
pod/emoji-5f844f4cb5-wv7d4      1/1     Running   0          3m12s
pod/vote-bot-5c64bd6898-bqwqf   1/1     Running   0          3m12s
pod/voting-7b77644858-slxk6     1/1     Running   0          3m11s
pod/web-58fb4f6fb6-9rp92        1/1     Running   0          3m11s

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/emoji-svc    ClusterIP   10.111.202.78   <none>        8080/TCP,8801/TCP   3m13s
service/voting-svc   ClusterIP   10.111.34.59    <none>        8080/TCP,8801/TCP   3m13s
service/web-svc      ClusterIP   10.102.200.28   <none>        80/TCP              3m12s

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/emoji      1/1     1            1           3m12s
deployment.apps/vote-bot   1/1     1            1           3m12s
deployment.apps/voting     1/1     1            1           3m12s
deployment.apps/web        1/1     1            1           3m11s

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/emoji-5f844f4cb5      1         1         1       3m12s
replicaset.apps/vote-bot-5c64bd6898   1         1         1       3m12s
replicaset.apps/voting-7b77644858     1         1         1       3m12s
replicaset.apps/web-58fb4f6fb6        1         1         1       3m11s

Lets Verify the App

root@master:~# kubectl -n emojivoto port-forward svc/web-svc 8080:80
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
^Croot@master:~kubectl -n emojivoto port-forward svc/web-svc 8080:80 --address 0.0.0.0.0
Forwarding from 0.0.0.0:8080 -> 8080
Handling connection for 8080

If you click around Emojivoto, you might notice that it’s a little broken! For example, if you try to vote for the donut emoji, you’ll get a 404 page.

Now to debug the issue we need to check the respective pod logs. Now lets see how mesh service help us to troubleshoot the issue

Lets inject the linkerd to mesh the Emojivoto application

We can do this on a live application without downtime, thanks to Kubernetes’s rolling deploys

root@master:~kubectl get -n emojivoto deploy -o yaml | linkerd inject - | kubectl apply -f -
deployment "emoji" injected
deployment "vote-bot" injected
deployment "voting" injected
deployment "web" injected

deployment.apps/emoji configured
deployment.apps/vote-bot configured
deployment.apps/voting configured
deployment.apps/web configured

We have used linkerd inject to inject mesh in the application

Congratulations! You’ve now added Linkerd to an application! Just as with the control plane, it’s possible to verify that everything is working the way it should on the data plane side. Check your data plane with:

root@master:~# linkerd -n emojivoto check --proxy
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks contains all node podCIDRs
√ cluster networks contains all pods
√ cluster networks contains all services

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is used

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days
√ policy-validator webhook has valid cert
√ policy-validator cert is valid for at least 60 days

linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
√ control plane proxies are up-to-date
√ control plane proxies and cli versions match

linkerd-data-plane
------------------
√ data plane namespace exists
√ data plane proxies are ready
√ data plane is up-to-date
√ data plane and cli versions match
√ data plane pod labels are configured correctly
√ data plane service labels are configured correctly
√ data plane service annotations are configured correctly
√ opaque ports are properly annotated

Status check results are √

Explore Linkerd

Let’s install the viz extension, which will install an on-cluster metric stack and dashboard.

root@master:~linkerd viz install | kubectl apply -f - -
namespace/linkerd-viz created
clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-viz-metrics-api created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-metrics-api created
serviceaccount/metrics-api created
clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-viz-prometheus created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-prometheus created
serviceaccount/prometheus created
clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-viz-tap created
clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-viz-tap-admin created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-tap created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-tap-auth-delegator created
serviceaccount/tap created
rolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-tap-auth-reader created
secret/tap-k8s-tls created
apiservice.apiregistration.k8s.io/v1alpha1.tap.linkerd.io created
role.rbac.authorization.k8s.io/web created
rolebinding.rbac.authorization.k8s.io/web created
clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-viz-web-check created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-web-check created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-web-admin created
clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-viz-web-api created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-web-api created
serviceaccount/web created
service/metrics-api created
deployment.apps/metrics-api created
server.policy.linkerd.io/metrics-api created
authorizationpolicy.policy.linkerd.io/metrics-api created
meshtlsauthentication.policy.linkerd.io/metrics-api-web created
networkauthentication.policy.linkerd.io/kubelet created
configmap/prometheus-config created
service/prometheus created
deployment.apps/prometheus created
server.policy.linkerd.io/prometheus-admin created
authorizationpolicy.policy.linkerd.io/prometheus-admin created
service/tap created
deployment.apps/tap created
server.policy.linkerd.io/tap-api created
authorizationpolicy.policy.linkerd.io/tap created
clusterrole.rbac.authorization.k8s.io/linkerd-tap-injector created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-tap-injector created
serviceaccount/tap-injector created
secret/tap-injector-k8s-tls created
mutatingwebhookconfiguration.admissionregistration.k8s.io/linkerd-tap-injector-webhook-config created
service/tap-injector created
deployment.apps/tap-injector created
server.policy.linkerd.io/tap-injector-webhook created
authorizationpolicy.policy.linkerd.io/tap-injector created
networkauthentication.policy.linkerd.io/kube-api-server created
service/web created
deployment.apps/web created
serviceprofile.linkerd.io/metrics-api.linkerd-viz.svc.cluster.local created
serviceprofile.linkerd.io/prometheus.linkerd-viz.svc.cluster.local created

Once you’ve installed the extension, let’s validate everything one last time

root@master:~# linkerd check

Lets Access the dashboard

root@master:~# linkerd viz dashboard --address 0.0.0.0
Linkerd dashboard available at:
http://0.0.0.0:50750
Opening Linkerd dashboard in the default browser

Click around, explore

Pluto Pluto is a utility to help users find deprecated Kubernetes apiVersions in their code repositories and their helm releases.

Purpose Kubernetes Deprecation Policy Kubernetes is a large system with many components and many contributors. As with any such software, the feature set naturally evolves over time, and sometimes a feature may need to be removed. This could include an API, a flag, or even an entire feature. To avoid breaking existing users, Kubernetes follows a deprecation policy for aspects of the system that are slated to be removed.

Deprecating parts of the API Since Kubernetes is an API-driven system, the API has evolved over time to reflect the evolving understanding of the problem space. The Kubernetes API is actually a set of APIs, called "API groups", and each API group is independently versioned. API versions fall into 3 main tracks, each of which has different policies for deprecation

v1 GA (generally available, stable)

v1beta1 Beta (pre-release)

v1alpha1 Alpha (experimental)

We can use pluto in our CICD pipeline to verify our existing helm templates and keep on updating the templates removing the deprecated API versions.

Lets install Pluto and See how it work

root@master:~# wget https://github.com/FairwindsOps/pluto/releases/download/v5.19.0/pluto_5.19.0_linux_amd64.tar.gz
root@master:~# tar -xvzf pluto_5.19.0_linux_amd64.tar.gz
root@master:~# cp pluto /usr/local/bin/
root@master:~# pluto version
Version:5.19.0 Commit:37f1ee128fd56136729e3491c399bd86e7690e15

Lets Run Pluto to Detect Deprecated version in cluster

root@master:~# pluto detect-helm -owide
There were no resources found with known deprecated apiVersions.


Want more? Automate Pluto for free with Fairwinds Insights!
 🚀 https://fairwinds.com/insights-signup/pluto 🚀

Check only one single namespace

root@master:~# pluto detect-helm -n metallb-system -owide
There were no resources found with known deprecated apiVersions.


Want more? Automate Pluto for free with Fairwinds Insights!
 🚀 https://fairwinds.com/insights-signup/pluto 🚀

Check helm nginx-ingress on local server

root@master:~# tree nginx-ingress
nginx-ingress
├── Chart.yaml
├── ci
│   ├── daemonset-customconfig-values.yaml
│   ├── daemonset-customnodeport-values.yaml
│   ├── daemonset-headers-values.yaml
│   ├── daemonset-internal-lb-values.yaml
│   ├── daemonset-nodeport-values.yaml
│   ├── daemonset-tcp-udp-configMapNamespace-values.yaml
│   ├── daemonset-tcp-udp-values.yaml
│   ├── daemonset-tcp-values.yaml
│   ├── deamonset-default-values.yaml
│   ├── deamonset-metrics-values.yaml
│   ├── deamonset-psp-values.yaml
│   ├── deamonset-webhook-and-psp-values.yaml
│   ├── deamonset-webhook-values.yaml
│   ├── deployment-autoscaling-values.yaml
│   ├── deployment-customconfig-values.yaml
│   ├── deployment-customnodeport-values.yaml
│   ├── deployment-default-values.yaml
│   ├── deployment-headers-values.yaml
│   ├── deployment-internal-lb-values.yaml
│   ├── deployment-metrics-values.yaml
│   ├── deployment-nodeport-values.yaml
│   ├── deployment-psp-values.yaml
│   ├── deployment-tcp-udp-configMapNamespace-values.yaml
│   ├── deployment-tcp-udp-values.yaml
│   ├── deployment-tcp-values.yaml
│   ├── deployment-webhook-and-psp-values.yaml
│   └── deployment-webhook-values.yaml
├── OWNERS
├── README.md
├── templates
│   ├── addheaders-configmap.yaml
│   ├── admission-webhooks
│   │   ├── job-patch
│   │   │   ├── clusterrolebinding.yaml
│   │   │   ├── clusterrole.yaml
│   │   │   ├── job-createSecret.yaml
│   │   │   ├── job-patchWebhook.yaml
│   │   │   ├── psp.yaml
│   │   │   ├── rolebinding.yaml
│   │   │   ├── role.yaml
│   │   │   └── serviceaccount.yaml
│   │   └── validating-webhook.yaml
│   ├── clusterrolebinding.yaml
│   ├── clusterrole.yaml
│   ├── controller-configmap.yaml
│   ├── controller-daemonset.yaml
│   ├── controller-deployment.yaml
│   ├── controller-hpa.yaml
│   ├── controller-metrics-service.yaml
│   ├── controller-poddisruptionbudget.yaml
│   ├── controller-prometheusrules.yaml
│   ├── controller-psp.yaml
│   ├── controller-rolebinding.yaml
│   ├── controller-role.yaml
│   ├── controller-serviceaccount.yaml
│   ├── controller-service-internal.yaml
│   ├── controller-servicemonitor.yaml
│   ├── controller-service.yaml
│   ├── controller-webhook-service.yaml
│   ├── default-backend-deployment.yaml
│   ├── default-backend-hpa.yaml
│   ├── default-backend-poddisruptionbudget.yaml
│   ├── default-backend-psp.yaml
│   ├── default-backend-rolebinding.yaml
│   ├── default-backend-role.yaml
│   ├── default-backend-serviceaccount.yaml
│   ├── default-backend-service.yaml
│   ├── _helpers.tpl
│   ├── NOTES.txt
│   ├── proxyheaders-configmap.yaml
│   ├── tcp-configmap.yaml
│   └── udp-configmap.yaml
└── values.yaml

5 directories, 71 files


WARNING: This chart is deprecated
There were no resources found with known deprecated apiVersions.


Want more? Automate Pluto for free with Fairwinds Insights!
 🚀 https://fairwinds.com/insights-signup/pluto 🚀

Verify API in installed workload in cluster

root@master:~# pluto detect-api-resources -owide
There were no resources found with known deprecated apiVersions.

Want more? Automate Pluto for free with Fairwinds Insights!
 🚀 https://fairwinds.com/insights-signup/pluto 🚀

Please refer Pluto Commands to learn more.......

Goldilocks

Goldilocks is a utility that can help you identify a starting point for resource requests and limits.

Goldilocks used kubernetes vertical-pod-autoscaler in recommendation mode to give suggestion for resource requests on each of our apps. This tool creates a VPA for each workload in a namespace and then queries them for information. Once your VPAs are in place, you'll see recommendations appear in the Goldilocks dashboard

To make goldilocks work , VPA must be installed in your cluster

Lets See Goldilocks in function

Assuming VPA is running in your cluster, lets install goldilocks

Lest verify VPS is up and running in Cluster

root@master:~# kubectl --namespace=kube-system get pods|grep vpa
vpa-admission-controller-68dccb8777-hlt7j          1/1     Running   0                22m
vpa-recommender-6b8dc459d8-gxhvf                   1/1     Running   0                22m
vpa-updater-6498cd765-r9qg4                        1/1     Running   0                22m

root@master:~# helm repo add fairwinds-stable https://charts.fairwinds.com/stable
root@master:~# kubectl create namespace goldilocks
root@master:~# helm install goldilocks --namespace goldilocks fairwinds-stable/goldilocks

Lets Verify the Pods and Service running in goldilocks namespace

root@master:~# kubectl get pods -n goldilocks
NAME                                    READY   STATUS    RESTARTS   AGE
goldilocks-controller-cccfb89f8-6wrls   1/1     Running   0          17m
goldilocks-dashboard-7744d9bddd-82rzb   1/1     Running   0          17m
goldilocks-dashboard-7744d9bddd-tmjjj   1/1     Running   0          17m

root@master:~# kubectl get svc -n goldilocks
NAME                   TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)        AGE
goldilocks-dashboard   LoadBalancer   10.97.78.212   172.16.16.158   80:30407/TCP   17m

Now we have to assign label goldilocks.fairwinds.com/enabled=true to respective namespace which need to monitored by Goldilocks

root@master:~# kubectl label ns default goldilocks.fairwinds.com/enabled=true

Now Let's Access the Goldilocks Dashboard from Browser and if every thing configured as expected, we will see recommendation on dashboard for workloads running in default namespace.

Prepare node Storage Provisioner must be Install In Kubernetes to provision dynamic PVC

Install Vault In Kubernetes - Clone the Vault Repository

#git clone https://github.com/gittest20202/vault.git 

- Traverse To Vault Repo

#cd vault

- Install the RBAC for the Vault

ClusterRoles: Kubernetes ClusterRoles are entities that have been assigned certain special permissions. ServiceAccounts: Kubernetes ServiceAccounts are identities assigned to entities such as pods to enable their interaction with the Kubernetes APIs using the role’s permissions. ClusterRoleBindings: ClusterRoleBindings are entities that provide roles to accounts i.e. they grant permissions to service accounts.

#kubectl create ns vault

#cat rbac.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault
  namespace: vault

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-server-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault
  namespace: vault

# kubectl create -f rbac.yaml

- Create Vault ConfigMaps

#cat configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: vault-config
  namespace: vault
data:
  extraconfig-from-values.hcl: |-
    disable_mlock = true
    ui = true

    listener "tcp" {
      tls_disable = 1
      address = "[::]:8200"
      cluster_address = "[::]:8201"
    }
    storage "file" {
      path = "/vault/data"
    }

#kubectl apply -f configmap.yaml

• disable_mlock: Executing mlock syscall prevents memory from being swapped to
• disk: This option disables the server from executing the mlock syscall.
• ui: Enables the built-in web UI.
• listener: Configures how Vault is listening for API requests.
• storage: Configures the storage backend where Vault data is stored.

- Deploy Vault Services

Services in Kubernetes are the objects that pods use to communicate with each other. ClusterIP type services are usually used for inter-pod communication.

There are two types of ClusterIP services - Headless Services - Services

Normal Kubernetes services act as load balancers and follow round-robin logic to distribute loads. Headless services don’t act like load balancers. Also, normal services are assigned IPs by Kubernetes whereas Headless services are not. For the vault server, we will create a headless service for internal usage. It will be very useful when we scale the vault to multiple replicas. A non-headless service will be created for UI as we want to load balance requests to the replicas when accessing the UI. Vault exposes its UI at port 8200. We will use a non-headless service of type NodePort as we want to access this endpoint from outside Kubernetes Cluster.

#cat services.yaml
---
# Service for Vault Server
apiVersion: v1
kind: Service
metadata:
  name: vault
  namespace: vault
  labels:
    app.kubernetes.io/name: vault
    app.kubernetes.io/instance: vault
  annotations:
spec:
  type: LoadBalancer  
  publishNotReadyAddresses: true
  ports:
    - name: http
      port: 8200
      targetPort: 8200
      nodePort: 32000
    - name: https-internal
      port: 8201
      targetPort: 8201
  selector:
    app.kubernetes.io/name: vault
    app.kubernetes.io/instance: vault
    component: server

---
# Headless Service
apiVersion: v1
kind: Service
metadata:
  name: vault-internal
  namespace: vault
  labels:
    app.kubernetes.io/name: vault
    app.kubernetes.io/instance: vault
  annotations:
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  ports:
    - name: "http"
      port: 8200
      targetPort: 8200
    - name: https-internal
      port: 8201
      targetPort: 8201
  selector:
    app.kubernetes.io/name: vault
    app.kubernetes.io/instance: vault
    component: server

#kubectl apply -f services.yaml

- Deploy Vault StatefulSet

#cat statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: vault
  namespace: vault
  labels:
    app.kubernetes.io/name: vault
    app.kubernetes.io/instance: vault
spec:
  serviceName: vault-internal
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: vault
      app.kubernetes.io/instance: vault
      component: server
  template:
    metadata:
      labels:
        app.kubernetes.io/name: vault
        app.kubernetes.io/instance: vault
        component: server
    spec:
      serviceAccountName: vault
      securityContext:
        runAsNonRoot: true
        runAsGroup: 1000
        runAsUser: 100
        fsGroup: 1000
      volumes:
        - name: config
          configMap:
            name: vault-config
        - name: home
          emptyDir: {}
      containers:
        - name: vault          
          image: hashicorp/vault:1.8.0
          imagePullPolicy: IfNotPresent
          command:
          - "/bin/sh"
          - "-ec"
          args: 
          - |
            cp /vault/config/extraconfig-from-values.hcl /tmp/storageconfig.hcl;
            [ -n "${HOST_IP}" ] && sed -Ei "s|HOST_IP|${HOST_IP?}|g" /tmp/storageconfig.hcl;
            [ -n "${POD_IP}" ] && sed -Ei "s|POD_IP|${POD_IP?}|g" /tmp/storageconfig.hcl;
            [ -n "${HOSTNAME}" ] && sed -Ei "s|HOSTNAME|${HOSTNAME?}|g" /tmp/storageconfig.hcl;
            [ -n "${API_ADDR}" ] && sed -Ei "s|API_ADDR|${API_ADDR?}|g" /tmp/storageconfig.hcl;
            [ -n "${TRANSIT_ADDR}" ] && sed -Ei "s|TRANSIT_ADDR|${TRANSIT_ADDR?}|g" /tmp/storageconfig.hcl;
            [ -n "${RAFT_ADDR}" ] && sed -Ei "s|RAFT_ADDR|${RAFT_ADDR?}|g" /tmp/storageconfig.hcl;
            /usr/local/bin/docker-entrypoint.sh vault server -config=/tmp/storageconfig.hcl     
          securityContext:
            allowPrivilegeEscalation: false
          env:
            - name: HOSTNAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: VAULT_ADDR
              value: "http://127.0.0.1:8200"
            - name: VAULT_API_ADDR
              value: "http://$(POD_IP):8200"
            - name: SKIP_CHOWN
              value: "true"
            - name: SKIP_SETCAP
              value: "true"
            - name: VAULT_CLUSTER_ADDR
              value: "https://$(HOSTNAME).vault-internal:8201"
            - name: HOME
              value: "/home/vault"
          volumeMounts:
            - name: data
              mountPath: /vault/data  
            - name: config
              mountPath: /vault/config
            - name: home
              mountPath: /home/vault
          ports:
            - containerPort: 8200
              name: http
            - containerPort: 8201
              name: https-internal
            - containerPort: 8202
              name: http-rep
          readinessProbe:
            exec:
              command: ["/bin/sh", "-ec", "vault status -tls-skip-verify"]
            failureThreshold: 2
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 3
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
             storage: 1Gi


#kubectl apply -f statefulset.yaml

- Unseal & Initialise Vault

#kubectl exec vault-0 -- vault operator init -key-shares=1 -key-threshold=1 -format=json > keys.json

#VAULT_UNSEAL_KEY=$(cat keys.json | jq -r ".unseal_keys_b64[]")
#echo $VAULT_UNSEAL_KEY

#VAULT_ROOT_KEY=$(cat keys.json | jq -r ".root_token")
#echo $VAULT_ROOT_KEY

#kubectl exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY

- Login & Access Vault UI

#kubectl exec vault-0 -- vault login $VAULT_ROOT_KEY

• Creating Vault Secrets

#kubectl exec -it vault-0 -- /bin/sh

- Create secrets

#vault secrets enable -version=2 -path="demo-app" kv
#vault kv put demo-app/user01 name=devopscube
#vault kv get demo-app/user01

- Create a Policy

#vault policy write demo-policy - <<EOH
path "demo-app/*" {
  capabilities = ["read"]
}
EOH

#vault policy list

- Enable Vault Kubernetes Authentication Method

#vault auth enable kubernetes
#vault write auth/kubernetes/config token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
#exit

- Create ServiceAccount

#kubectl create serviceaccount vault-auth

- Login Back to Vault Pod

#kubectl exec vault-0 -- vault login $VAULT_ROOT_KEY
#vault write auth/kubernetes/role/webapp \
        bound_service_account_names=vault-auth \
        bound_service_account_namespaces=default \
        policies=demo-policy \
        ttl=72h

- Deploy A Pod

#cat pod.yaml
---
apiVersion: v1
kind: Pod
metadata:
  name: vault-client
  namespace: default
spec:
  containers:
  - image: nginx:latest
    name: nginx
  serviceAccountName: vault-auth

#kubectl create -f pod.yaml

#kubectl exec -it vault-client /bin/bash

#jwt_token=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

#echo $jwt_token

- Install External Secret Operator

#helm repo add external-secrets https://charts.external-secrets.io
#helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace 

#kubectl get po -n external-secrets

- Clone The Repository

#git clone https://github.com/gittest20202/external-secrets.git

# cd ExternalSecrets

#cat cluster-secret-store.yaml
apiVersion: external-secrets.io/v1alpha1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "http://vault-server-internal.vault:8200"
      path: "secret"
      version: "v1"
      auth:
        tokenSecretRef:
          name: "vault-token"
          key: "token"
          namespace: vault

#echo -n "Vault Secret Token" |base64

#cat vault-token-secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-token
  namespace: vault
data:
  token: <base64-generated-token"

- Apply The Files

#kubectl apply -f cluster-secret-store.yaml
#kubectl apply -f vault-token-secret.yaml

- Create External Secret

#cat external-pullsecret-cluster.yaml
apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: pullsecret-cluster-sno01
  namespace: sno01
spec:
  refreshInterval: "15s"
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: pullsecret-cluster-sno01
  data:
  - secretKey: .dockerconfigjson
    remoteRef:
      key: secret/pullsecret
      property: dockerconfigjson

#kubectl apply -f external-pullsecret-cluster.yaml

- Login to Vault Pod

#kubectl exec vault-0 -- vault login $VAULT_ROOT_KEY
#vault secrets enable -path=secret/ kv
#vault kv put secret/pullsecret dockerconfigjson='{"auths":{"cloud.openshift.com":{"auth":"3BlbnNoaWZ0LXJl==","email":"example@redhat.com"},"quay.io":{"auth":"ZZMVhJRUJUR1I3WUwxN05VMQ==","email":"example@redhat.com"},"registry.connect.redhat.com":{"auth":"3BlbnNoaWZ0LXJl==","email":"example@redhat.com"},"registry.redhat.io":{"auth":"==","email":"example@redhat.com"}}}'

- Apply The Pull-Secret

#kubectl apply -f apply -f external-pullsecret-cluster.yaml

- Validation

#kubectl get externalsecret -n default pullsecret-cluster-sno01
#kubectl get secrets -n default pullsecret-cluster-sno01
#kubectl secrets -n default pullsecret-cluster-sno01 -o yaml 
apiVersion: v1
data:
  .dockerconfigjson: eyJhdXRocyI6eyJjbG91ZC5vcGVuc2hpZnQuY29tIjp7ImF1dGgiOiIzQmxibk5vYVdaMExYSmw9PSIsImVtYWlsIjoiZXhhbXBsZUByZWRoYXQuY29tIn0sInF1YXkuaW8iOnsiYXV0aCI6IlpaTVZoSlJVSlVSMUkzV1V3eE4wNVZNUT09IiwiZW1haWwiOiJleGFtcGxlQHJlZGhhdC5jb20ifSwicmVnaXN0cnkuY29ubmVjdC5yZWRoYXQuY29tIjp7ImF1dGgiOiIzQmxibk5vYVdaMExYSmw9PSIsImVtYWlsIjoiZXhhbXBsZUByZWRoYXQuY29tIn0sInJlZ2lzdHJ5LnJlZGhhdC5pbyI6eyJhdXRoIjoiPT0iLCJlbWFpbCI6ImV4YW1wbGVAcmVkaGF0LmNvbSJ9fX0=

#echo -n "eyJhdXRocyI6eyJjbG91ZC5vcGVuc2hpZnQuY29tIjp7ImF1dGgiOiIzQmxibk5vYVdaMExYSmw9PSIsImVtYWlsIjoiZXhhbXBsZUByZWRoYXQuY29tIn0sInF1YXkuaW8iOnsiYXV0aCI6IlpaTVZoSlJVSlVSMUkzV1V3eE4wNVZNUT09IiwiZW1haWwiOiJleGFtcGxlQHJlZGhhdC5jb20ifSwicmVnaXN0cnkuY29ubmVjdC5yZWRoYXQuY29tIjp7ImF1dGgiOiIzQmxibk5vYVdaMExYSmw9PSIsImVtYWlsIjoiZXhhbXBsZUByZWRoYXQuY29tIn0sInJlZ2lzdHJ5LnJlZGhhdC5pbyI6eyJhdXRoIjoiPT0iLCJlbWFpbCI6ImV4YW1wbGVAcmVkaGF0LmNvbSJ9fX0=" | base64 --decode

Introduction

Kube-VIP, or Kubernetes Virtual IP, is a project that provides high availability for Kubernetes services. It does so by managing a virtual IP (VIP) that floats between nodes in a Kubernetes cluster, ensuring that if a node hosting a service goes down, the IP address associated with that service is quickly reassigned to another healthy node. This helps in maintaining uninterrupted access to the service.

Here's a high-level overview of the architecture of Kube-VIP:

Components:

VIP Manager: This component is responsible for managing the virtual IP address. It monitors the health of nodes and services in the cluster and reassigns the virtual IP as necessary.

Health Check Monitor: Monitors the health of nodes and services. It checks the health status of nodes and services using various mechanisms like HTTP probes, TCP probes, etc.

Leader Election: To ensure that only one instance of the VIP manager is active at any given time, Kube-VIP utilizes leader election mechanisms provided by Kubernetes or an external consensus mechanism like etcd.

IPVS or IPTables: Kube-VIP can utilize either IPVS (IP Virtual Server) or IPTables to manage network traffic and route it to the appropriate backend pods.

Operation:

Initialization: During initialization, Kube-VIP is deployed on each node in the Kubernetes cluster. It configures the network interfaces to manage the virtual IP. Monitoring: The health check monitor continuously monitors the health of nodes and services by periodically sending probes. Failover: If a node or service becomes unhealthy, the VIP manager initiates failover by reassigning the virtual IP to a healthy node. This is done by updating the ARP (Address Resolution Protocol) tables or IPTables rules to redirect traffic to the new node. Recovery: Once the failed node or service is restored to a healthy state, the VIP manager can revert the changes and return the virtual IP to its original state.

Integration with Kubernetes: Kube-VIP integrates seamlessly with Kubernetes using custom resources and controllers. It leverages Kubernetes primitives to manage the virtual IP address and ensure high availability of services. Customization: Kube-VIP allows for customization of various parameters such as health check intervals, failure detection thresholds, and failover strategies to suit the specific requirements of the cluster.

In This Blog I am going to discuss how can we use kube-vip to configure Kubernetes in HA Mode with 2 master node and 3 worker node.

My Infrastructure Login to One Control node

root@devmaster1:~# export VIP=172.17.17.110
root@devmaster1:~#export INTERFACE=vboxnet0(Interface where your node IP is assigned)
root@devmaster1:~# KVVERSION=$(curl -sL https://api.github.com/repos/kube-vip/kube-vip/releases | jq -r ".[0].name")
root@devmaster1:~# alias kube-vip="ctr image pull ghcr.io/kube-vip/kube-vip:$KVVERSION; ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:$KVVERSION vip /kube-vip"

Now Run the below command to generate a kube-vip.yaml file

root@devmaster1:~#kube-vip manifest pod \
--interface $INTERFACE \
--address $VIP \
--controlplane \
--services \
--arp \
--leaderElection | tee /etc/kubernetes/manifests/kube-vip.yaml

kube-vip.yaml file will get store in /etc/kubernetes/manifests/ and when we will run kubeadm command, kubelet will spin up a static pod.

Now let's run kubeadm command to install the cluster

root@devmaster1:~# kubeadm init --control-plane-endpoint="172.17.17.110:6443" --upload-certs --apiserver-advertise-address=172.17.17.1 --pod-network-cidr=192.168.0.0/16 

I0213 20:22:14.645006 26476 version.go:256] remote version is much newer: v1.29.1; falling back to: stable-1.26 [init] Using Kubernetes version: v1.26.13 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [devmaster1.homecluster.store kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.17.17.1 172.17.17.110] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [devmaster1.homecluster.store localhost] and IPs [172.17.17.1 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [devmaster1.homecluster.store localhost] and IPs [172.17.17.1 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 6.565268 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace [upload-certs] Using certificate key: 3de2839a68ee0b02f4d947ba7a8d361595847fb9b449b3a311742cda620c6723 [mark-control-plane] Marking the node devmaster1.homecluster.store as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node devmaster1.homecluster.store as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule] [bootstrap-token] Using token: z9ci87.eb8vkv02assothnx [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of the control-plane node running the following command on each as root:

Save the Generated Join commands to Join the Second Master and Worker Nodes to cluster

Export the kubeconfig file and verify the devmaster1 cluster status.

root@devmaster1:~# export KUBECONFIG=/etc/kubernetes/admin.conf
root@devmaster1:~# kubectl get nodes
NAME                           STATUS     ROLES           AGE   VERSION
devmaster1.homecluster.store   Ready      control-plane   39m   v1.26.0

Now Lets Login to devmaster2 node and run the Join Command

root@devmaster2:~# kubeadm join 172.17.17.110:6443 --token xxxxxxx
--discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxx  --control-plane
--certificate-key xxxxxxxxxxxxxxxxxxxxxxx --apiserver-advertise-
address=172.17.17.100

[preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [preflight] Running pre-flight checks before initializing the new control plane instance [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace [download-certs] Saving the certificates to the folder: "/etc/kubernetes/pki" [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [devmaster2 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.17.17.100 172.17.17.110] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [devmaster2 localhost] and IPs [172.17.17.100 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [devmaster2 localhost] and IPs [172.17.17.100 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Valid certificates and keys now exist in "/etc/kubernetes/pki" [certs] Using the existing "sa" key [kubeconfig] Generating kubeconfig files [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [check-etcd] Checking that the etcd cluster is healthy [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... [etcd] Announced new etcd member joining to the existing etcd cluster [etcd] Creating static Pod manifest for "etcd" [etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s The 'update-status' phase is deprecated and will be removed in a future release. Currently it performs no operation [mark-control-plane] Marking the node devmaster2 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node devmaster2 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule] This node has joined the cluster and a new control plane instance was created: * Certificate signing request was sent to apiserver and approval was received. * The Kubelet was informed of the new secure connection details. * Control plane label and taint were applied to the new node. * The Kubernetes control plane instances scaled up. * A new etcd member was added to the local/stacked etcd cluster. To start administering your cluster from this node, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Run 'kubectl get nodes' to see this node join the cluster.

Login Back to devmaster1 and run kubectl get nodes command

root@devmaster1:~# kubectl get nodes
NAME                           STATUS     ROLES           AGE   VERSION
devmaster1.homecluster.store   Ready      control-plane   48m   v1.26.0
devmaster2.homecluster.store   NotReady   control-plane   41m   v1.26.0

Lets install Calico CNI to make devmaster2 node to ready state.

root@devmaster1:~# kubectl --kubeconfig=/etc/kubernetes/admin.conf create -f https://docs.projectcalico.org/v3.15/manifests/calico.yaml

configmap/calico-config created customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created clusterrole.rbac.authorization.k8s.io/calico-node created clusterrolebinding.rbac.authorization.k8s.io/calico-node created daemonset.apps/calico-node created serviceaccount/calico-node created deployment.apps/calico-kube-controllers created serviceaccount/calico-kube-controllers created

Verify the Status

root@devmaster1:~# kubectl get nodes
NAME                           STATUS     ROLES           AGE   VERSION
devmaster1.homecluster.store   Ready      control-plane   48m   v1.26.0
devmaster2.homecluster.store  Ready      control-plane   41m   v1.26.0  

Login on all 3 Worker Nodes and run the Join Command

root@devworker1:~# kubeadm join 172.17.17.110:6443 --token xxxxxxxxxxxxxx \
 --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxx

[preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

Login to devmaster1.homecluster.store and verify

root@devmaster1:~# kubectl get nodes
NAME                           STATUS   ROLES           AGE     VERSION
devmaster1.homecluster.store   Ready    control-plane   117m    v1.26.0
devmaster2.homecluster.store   Ready    control-plane   110m    v1.26.0
devworker1.homecluster.store   Ready    <none>          5m10s   v1.26.0
devworker2.homecluster.store   Ready    <none>          3m      v1.26.0
devworker3.homecluster.store   Ready    <none>          1m40s   v1.26.0

We can see now cluster is up and running with 2 Master Node and 3 Worker Nodes and listening over VIP 172.17.17.100

Verify the current running CNI

root@devmaster:~# kubectl get pods -n kube-system | grep calico
calico-kube-controllers-5dd4b7dfd9-j7mfj              1/1     Running   6 (46d ago)      51d
calico-node-lctql                                     1/1     Running   3 (46d ago)      71d
calico-node-nx7lx                                     1/1     Running   0                2m34s
calico-node-xdmm7                                     1/1     Running   2 (46d ago)      71d

Introduction In the realm of Kubernetes networking, the choice of Container Network Interface (CNI) plays a crucial role in determining performance, security, and scalability. While Calico has long been a popular choice for networking solutions, Cilium has emerged as a compelling alternative, offering advanced features and robust performance enhancements. In this guide, we'll explore the process of migrating from Calico to Cilium CNI, highlighting the benefits and steps involved in making the transition.

Why Migrate from Calico to Cilium? Before delving into the migration process, it's essential to understand why organizations might consider shifting from Calico to Cilium CNI. While Calico provides solid networking capabilities, Cilium offers several key advantages

Enhanced Performance: Cilium leverages eBPF (extended Berkeley Packet Filter) technology to provide efficient packet processing, resulting in lower latency and improved throughput compared to traditional networking solutions.

Advanced Security: Cilium offers powerful security features, including Layer 7 application-aware security policies, transparent encryption, and network visibility, enabling organizations to strengthen their defense against cyber threats.

Native Integration with Service Mesh: Cilium seamlessly integrates with popular service mesh solutions like Istio, enabling enhanced observability, traffic management, and security within Kubernetes environments.

Rich Feature Set: Cilium provides a wide range of features, including network policy enforcement, load balancing, DNS-based service discovery, and more, empowering organizations to build highly resilient and scalable infrastructure.

Migration Process: Now, let's dive into the steps involved in migrating from Calico to Cilium CNI

Install the cilium binary on linux machine

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

Install Cilium

root@devmaster:~# cilium install --version 1.15.0

Verify the Cilium installation

Verify the Cilium Connectivity

root@devmaster:~# cilium connectivity test
ℹ️  Monitor aggregation detected, will skip some flow validation steps
✨ [kubernetes] Creating namespace cilium-test for connectivity check...
✨ [kubernetes] Deploying echo-same-node service...
✨ [kubernetes] Deploying DNS test server configmap...
✨ [kubernetes] Deploying same-node deployment...
✨ [kubernetes] Deploying client deployment...
✨ [kubernetes] Deploying client2 deployment...
✨ [kubernetes] Deploying client3 deployment...
✨ [kubernetes] Deploying echo-other-node service...
✨ [kubernetes] Deploying other-node deployment...
✨ [host-netns] Deploying kubernetes daemonset...
✨ [host-netns-non-cilium] Deploying kubernetes daemonset...
ℹ️  Skipping tests that require a node Without Cilium
⌛ [kubernetes] Waiting for deployment cilium-test/client to become ready...
⌛ [kubernetes] Waiting for deployment cilium-test/client2 to become ready...
⌛ [kubernetes] Waiting for deployment cilium-test/echo-same-node to become ready...
⌛ [kubernetes] Waiting for deployment cilium-test/client3 to become ready...
⌛ [kubernetes] Waiting for deployment cilium-test/echo-other-node to become ready...
⌛ [kubernetes] Waiting for pod cilium-test/client-65847bf96-ctw2m to reach DNS server on cilium-test/echo-same-node-56dfd8bd85-hd72q pod...
⌛ [kubernetes] Waiting for pod cilium-test/client2-85585bdd-cjvnw to reach DNS server on cilium-test/echo-same-node-56dfd8bd85-hd72q pod...
⌛ [kubernetes] Waiting for pod cilium-test/client3-54d97dc775-rzlm6 to reach DNS server on cilium-test/echo-same-node-56dfd8bd85-hd72q pod...
⌛ [kubernetes] Waiting for pod cilium-test/client3-54d97dc775-rzlm6 to reach DNS server on cilium-test/echo-other-node-7b76c5bbf9-rzt5f pod...
⌛ [kubernetes] Waiting for pod cilium-test/client-65847bf96-ctw2m to reach DNS server on cilium-test/echo-other-node-7b76c5bbf9-rzt5f pod...
⌛ [kubernetes] Waiting for pod cilium-test/client2-85585bdd-cjvnw to reach DNS server on cilium-test/echo-other-node-7b76c5bbf9-rzt5f pod...
⌛ [kubernetes] Waiting for pod cilium-test/client-65847bf96-ctw2m to reach default/kubernetes service...
⌛ [kubernetes] Waiting for pod cilium-test/client2-85585bdd-cjvnw to reach default/kubernetes service...
⌛ [kubernetes] Waiting for pod cilium-test/client3-54d97dc775-rzlm6 to reach default/kubernetes service...
⌛ [kubernetes] Waiting for Service cilium-test/echo-other-node to become ready...
⌛ [kubernetes] Waiting for Service cilium-test/echo-other-node to be synchronized by Cilium pod kube-system/cilium-cwrxr
⌛ [kubernetes] Waiting for Service cilium-test/echo-other-node to be synchronized by Cilium pod kube-system/cilium-jwpck
⌛ [kubernetes] Waiting for Service cilium-test/echo-same-node to become ready...
⌛ [kubernetes] Waiting for Service cilium-test/echo-same-node to be synchronized by Cilium pod kube-system/cilium-cwrxr
⌛ [kubernetes] Waiting for Service cilium-test/echo-same-node to be synchronized by Cilium pod kube-system/cilium-jwpck
⌛ [kubernetes] Waiting for NodePort 172.17.17.101:31784 (cilium-test/echo-other-node) to become ready...
⌛ [kubernetes] Waiting for NodePort 172.17.17.101:31656 (cilium-test/echo-same-node) to become ready...
⌛ [kubernetes] Waiting for NodePort 192.168.0.117:31784 (cilium-test/echo-other-node) to become ready...
⌛ [kubernetes] Waiting for NodePort 192.168.0.117:31656 (cilium-test/echo-same-node) to become ready...
⌛ [kubernetes] Waiting for NodePort 172.17.17.102:31784 (cilium-test/echo-other-node) to become ready...
⌛ [kubernetes] Waiting for NodePort 172.17.17.102:31656 (cilium-test/echo-same-node) to become ready...
⌛ [kubernetes] Waiting for DaemonSet cilium-test/host-netns-non-cilium to become ready...
⌛ [kubernetes] Waiting for DaemonSet cilium-test/host-netns to become ready...
ℹ️  Skipping IPCache check
🔭 Enabling Hubble telescope...
⚠️  Unable to contact Hubble Relay, disabling Hubble telescope and flow validation: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:4245: connect: connection refused"
ℹ️  Expose Relay locally with:
   cilium hubble enable
   cilium hubble port-forward&
ℹ️  Cilium version: 1.15.0
🏃 Running 64 tests ...

Create a per-node config that will instruct Cilium to “take over” CNI networking on the node. Initially, this will apply to no nodes; you will roll it out gradually via the migration process.

root@devmaster:~# cat <<EOF | kubectl apply --server-side -f -
apiVersion: cilium.io/v2alpha1
kind: CiliumNodeConfig
metadata:
  namespace: kube-system
  name: cilium-default
spec:
  nodeSelector:
    matchLabels:
      io.cilium.migration/cilium-default: "true"
  defaults:
    write-cni-conf-when-ready: /host/etc/cni/net.d/05-cilium.conflist
    custom-cni-conf: "false"
    cni-chaining-mode: "none"
    cni-exclusive: "true"
EOF
ciliumnodeconfig.cilium.io/cilium-default serverside-applied

Select a node to be migrated. It is not recommended to start with a control-plane node.

root@devmaster:~# NODE="devworker2.homecluster.store"
root@devmaster:~# kubectl cordon $NODE
node/devworker2.homecluster.store cordoned
root@devmaster:~# kubectl drain --ignore-daemonsets $NODE
node/devworker2.homecluster.store already cordoned
root@devmaster:~# kubectl get nodes
NAME                           STATUS                     ROLES           AGE   VERSION
devmaster.homecluster.store    Ready                      control-plane   72d   v1.26.0
devworker1.homecluster.store   Ready                      <none>          72d   v1.26.0
devworker2.homecluster.store   Ready,SchedulingDisabled   <none>          72d   v1.26.0

Label the node. This causes the CiliumNodeConfig to apply to this node.

root@devmaster:~# kubectl label node $NODE --overwrite "io.cilium.migration/cilium-default=true"
node/devworker2.homecluster.store labeled

Restart Cilium. This will cause it to write its CNI configuration file.

root@devmaster:~# kubectl -n kube-system delete pod --field-selector spec.nodeName=$NODE -l k8s-app=cilium
pod "cilium-jvc7v" deleted
root@devmaster:~# kubectl -n kube-system rollout status ds/cilium -w
daemon set "cilium" successfully rolled out

Restart the Node

root@devmaster:~# ssh root@172.17.17.102
root@devworker2:~# init 6

Validate that the node has been successfully migrated.

root@devmaster:~# cilium status --wait
kubectl get -o wide node $NODE
kubectl -n kube-system run --attach --rm --restart=Never verify-network \
  --overrides='{"spec": {"nodeName": "'$NODE'", "tolerations": [{"operator": "Exists"}]}}' \
  --image ghcr.io/nicolaka/netshoot:v0.8 -- /bin/bash -c 'ip -br addr && curl -s -k https://$KUBERNETES_SERVICE_HOST/healthz && echo'

Uncordon the node.

root@devmaster:~# kubectl uncordon $NODE
root@devmaster:~# kubectl get -o wide node $NODE

Once you are satisfied everything has been migrated successfully, select another unmigrated node in the cluster and repeat these steps.

Delete the Calico CNI

root@devmaster:~# kubectl delete crd $(kubectl get crd | grep calico | awk '{print $1}')

root@devmaster:~# kubectl delete -n kube-system deployment calico-kube-controllers

root@devmaster:~# kubectl delete -n kube-system daemonset calico-node

Verify Calico CNI get Deleted

root@devmaster:~# kubectl get pods -n kube-system
NAME                                                  READY   STATUS    RESTARTS         AGE
cilium-cwrxr                                          1/1     Running   0                15m
cilium-dr7r5                                          1/1     Running   0                5m39s
cilium-jwpck                                          1/1     Running   0                13m
cilium-operator-5b7fb7b87d-f2v97                      1/1     Running   0                6m4s
coredns-787d4945fb-65hzf                              1/1     Running   0                6m4s
coredns-787d4945fb-lfkw4                              1/1     Running   0                38m
etcd-devmaster.homecluster.store                      1/1     Running   4 (46d ago)      72d
kube-apiserver-devmaster.homecluster.store            1/1     Running   4                72d
kube-controller-manager-devmaster.homecluster.store   1/1     Running   13 (2d19h ago)   72d
kube-proxy-4gtbg                                      1/1     Running   2 (46d ago)      72d
kube-proxy-dh9z4                                      1/1     Running   2 (46d ago)      72d
kube-proxy-mtjjl                                      1/1     Running   3 (46d ago)      72d
kube-scheduler-devmaster.homecluster.store            1/1     Running   13 (2d19h ago)   72d

Reboot all the nodes and Check Nodes Status

root@devmaster:~# kubectl get nodes
NAME                           STATUS   ROLES           AGE   VERSION
devmaster.homecluster.store    Ready    control-plane   72d   v1.26.0
devworker1.homecluster.store   Ready    <none>          72d   v1.26.0
devworker2.homecluster.store   Ready    <none>          72d   v1.26.0


root@devmaster:~# kubectl get pods -n metallb-system
NAME                          READY   STATUS    RESTARTS         AGE
controller-586bfc6b59-pcq87   1/1     Running   8 (5m45s ago)    65d
speaker-2zwg4                 1/1     Running   3 (3m7s ago)     72d
speaker-8n84l                 1/1     Running   12 (6m36s ago)   72d
speaker-zdxv6                 1/1     Running   3 (5m45s ago)    72d

Setting up Hubble Observability

root@devmaster# cilium hubble enable
root@devmaster# cilium status

Install the Hubble Client

root@devmaster:/etc/cni/net.d# HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
HUBBLE_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then HUBBLE_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
sha256sum --check hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum
sudo tar xzvfC hubble-linux-${HUBBLE_ARCH}.tar.gz /usr/local/bin
rm hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 17.0M  100 17.0M    0     0  3692k      0  0:00:04  0:00:04 --:--:-- 5802k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100    92  100    92    0     0     53      0  0:00:01  0:00:01 --:--:--   219
hubble-linux-amd64.tar.gz: OK
hubble

Validate Hubble API Access

root@devmaster# cilium hubble port-forward&

Now you can validate that you can access the Hubble API via the installed CLI

root@devmaster:~# hubble status
Healthcheck (via localhost:4245): Ok
Current/Max Flows: 12,285/12,285 (100.00%)
Flows/s: 150.13
Connected Nodes: 3/3

Query the flow API and look for flows

root@devmaster:~# hubble observe

Enable the Hubble UI

root@devmaster:~# cilium hubble enable --ui

Creating diagrams as code has become a popular practice, especially in the context of infrastructure as code (IaC) and documentation. In this blog, we'll explore some of the popular tools for creating diagrams as code and discuss how they can be used in different scenarios.

[- Diagram] (https://diagrams.mingrammer.com/docs/getting-started/installation) Using Diagram we can create diagrams for multiple environment like AWS,AZURE,GCP,Kubernetes.

Requirements

It requires Python 3.6 or higher, check your Python version first. It uses Graphviz to render the diagram, so you need to install Graphviz to use diagrams. After installing graphviz (or already have it), install the diagrams.

# pip install diagrams

(On Ubuntu)
# apt install graphviz

(On RHEL)
# yum install graphviz

Create Diagram For Kubernetes

from diagrams import Cluster, Diagram, Node, Edge
from diagrams.k8s.compute import Pod
from diagrams.k8s.compute import Deploy
from diagrams.k8s.network import Ing
from diagrams.k8s.group import NS
from diagrams.k8s.podconfig import Secret
from diagrams.k8s.storage import PVC
from diagrams.k8s.rbac import CRole
from diagrams.k8s.rbac import CRB

with Diagram("Kubernetes Cluster", show=False):
  with Cluster("Kubernetes"):
    with Cluster("Rbac"):
      rbac = CRB("")
      with Cluster("Role"):
        role = CRole("")
    with Cluster("App"):
      ns = NS("")
      with Cluster("Ingress"):
        ingress = Ing("")
        with Cluster("Secret"):
          secrets = Secret("")
        with Cluster("App"):
          deploy = Deploy("")
          with Cluster("Pods"):
            pod = Pod("")
        with Cluster("PVC"):
           pvc = PVC("")
  rbac >> role >> ns
  ns >> deploy >> pod >> pvc
  pod >> secrets
  deploy >> ingress

- Mermaid Mermaid is a JavaScript-based diagramming and charting tool that allows users to create diagrams and flowcharts using a simple and human-readable text-based syntax. It is particularly popular for its integration with Markdown, making it easy to embed diagrams directly into documentation, README files, or other text-based formats.

graph LR;
 IpBlockS([IpBlock])-. Traffic Out From <br> The Cluster .->|![Ingress Image](images/ingress.png)| ingress;
 PodNetworkS([PodNetwork])-. Traffic From <br> PodNetwork  .->|![Ingress Image](images/ingress.png)| ingress;
 NameSpaceNetworkS([NameSpaceNetwork])-. Traffic From <br> NameSpaceNetwork  .->|![Ingress Image](images/ingress.png)| ingress;
 ingress .->|routing <br> rule|namespace[namespace];
 subgraph cluster
 ingress;
 namespace .->|routing <br> rule|egress[Egress];
 end
 egress[Egress]-. Traffic Out To <br> The Cluster  .->IpBlockD([IpBlock]);
 egress[Egress]-. Traffic To <br> PodNetwork  .->PodNetworkD([PodNetwork]);
 egress[Egress]-. Traffic To <br> NameSpaceNetwork  .->NameSpaceNetworkD([NameSpaceNetwork]);
 classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000;
 classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff;
 classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5;
 class ingress,namespace,egress k8s;
 class client plain;
 class cluster cluster;

- PlantUML PlantUML is an open-source tool that allows users to create Unified Modeling Language (UML) diagrams using a simple and human-readable text-based syntax. UML diagrams are widely used in software development to visually represent different aspects of a system's architecture, design, and behavior. PlantUML makes it easy to express complex UML diagrams in a concise and maintainable manner.

@startyaml
!theme lightgray
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
@endyaml

We have a variety of tools at our disposal, but I've identified these specific ones for drawing diagrams. I've seamlessly integrated them into my GitLab CI/CD pipeline. I encourage you to give them a try; they prove to be highly effective for creating and managing our diagrams.

Hope you will like this blog and start using it in your CI/CD pipelines.

kpt stands for Kuberentes Package Toolkit. It is a set of tools for working with and managing Kubernetes manifests as packages. kpt helps you to organize, customize, share, and manage Kubernetes manifests in a more modular and reusable way.

Key features of kpt include:

Package Management: kpt allows you to organize your Kubernetes configuration files into packages. A package is a directory containing one or more Kubernetes manifests, and it can be versioned and shared.

Declarative Configuration: It encourages a declarative approach to configuration, where you describe the desired state of your Kubernetes resources, making it easier to manage configurations across different environments.

Resource Configuration: kpt provides commands to work with and transform Kubernetes resources. This includes adding, updating, or removing fields from manifests.

GitOps Workflow: It aligns with the GitOps approach, where changes to your infrastructure are driven by Git commits. You can use kpt to fetch, update, and apply changes to your Kubernetes manifests stored in Git repositories.

Template Functions: kpt supports template functions that allow you to parameterize and customize your manifests based on different environments or requirements.

Resource Composition: You can compose and customize your Kubernetes manifests by using kpt functions and tools.

System Requirements

• KPT must be installed

 #wget https://github.com/GoogleContainerTools/kpt/releases/download/v1.0.0-beta.44/kpt_linux_amd64
 #chmod +x kpt_linux_amd64
 #cp kpt_linux_amd64 /usr/local/bin/kpt
 #kpt version
 1.0.0-beta.44

• Git must be installed

# git version
git version 2.40.1

• Kubernetes cluster

root@master:~/nginx# kubectl get nodes
NAME                        STATUS   ROLES           AGE   VERSION
master.homecluster.store    Ready    control-plane   38d   v1.26.0
worker1.homecluster.store   Ready    <none>          38d   v1.26.0
worker2.homecluster.store   Ready    <none>          38d   v1.26.0

kpt is fully integrated with Git and enables forking, rebasing and versioning a package of configuration using the underlying Git version control system.

First, let’s fetch the kpt package from Git to your local filesystem:

# kpt pkg get https://github.com/GoogleContainerTools/kpt/package-examples/nginx@v0.9
# cd nginx

kpt pkg commands provide the functionality for working with packages on Git and on your local filesystem.

#kpt pkg tree
Package "nginx"
├── [Kptfile]  Kptfile nginx
├── [deployment.yaml]  Deployment my-nginx
└── [svc.yaml]  Service my-nginx-svc

Apply the Package

kpt live commands provide the functionality for deploying packages to a Kubernetes cluster.

Initialize the kpt package:

 # kpt live init

Apply the resources to the cluster:

#root@master:~/nginx/nginx# kubectl get pods
no resource found
# kpt live apply --reconcile-timeout=15m
inventory update started
inventory update finished
apply phase started
service/my-nginx-svc apply successful
deployment.apps/my-nginx apply successful
apply phase finished
reconcile phase started
service/my-nginx-svc reconcile successful
deployment.apps/my-nginx reconcile pending
deployment.apps/my-nginx reconcile successful
reconcile phase finished
inventory update started
inventory update finished
apply result: 2 attempted, 2 successful, 0 skipped, 0 failed
reconcile result: 2 attempted, 2 successful, 0 skipped, 0 failed, 0 timed out

root@master:~/nginx/nginx# kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
my-nginx-66f8758855-57q7v   1/1     Running   0          81s
my-nginx-66f8758855-5zj88   1/1     Running   0          81s
my-nginx-66f8758855-lbpmq   1/1     Running   0          81s
my-nginx-66f8758855-zp6nm   1/1     Running   0          81s

Delete the package from the cluster:

# kpt live destroy
delete phase started
deployment.apps/my-nginx delete successful
service/my-nginx-svc delete successful
delete phase finished
reconcile phase started
deployment.apps/my-nginx reconcile successful
service/my-nginx-svc reconcile successful
reconcile phase finished
inventory update started
inventory update finished
delete result: 2 attempted, 2 successful, 0 skipped, 0 failed
reconcile result: 2 attempted, 2 successful, 0 skipped, 0 failed, 0 timed out