K3S Thing: Make Traefik Forward Real Client IP

Background

After migrating test environment applications to K3S cluster, application gets wrong client IP. The IP Address the application gets is cluster node’s cni0 network interface ip address. Before migration, applications sit behind Nginx. Nginx will put client information to X-Forwarded-* request headers and forward to applications.

Traefik Proxy is default ingress controller in K3S cluster. It takes role of Nginx, and certainly it will add X-Forwarded-* headers, so why forward wrong client IP.

Relate github issues and traefik forum topics:

• Please enable proxy protocol in k3s ingress
• Getting Real Client IP with k3s
• How to set X-Real-Ip when using NodePort service?
• Cannot get client IP in X-Real-IP
• Client ip on kubernetes with loadbalancer

Analysis

Traefik Proxy is deploy as load balancer in K3S. So network flow will be

When network packages arrived at Traefik, network package is already SNATed and lost original client connection information.

Solution

To make traefik get real client IP, make network packages arrived at Traefik not SNATed. According to Kubernetes Using Source IP document, add set service.spec.externalTrafficPolicy to Local. Then kube-proxy will forward network packages to local node pod only. It means traefik pod should be schedule to all nodes. To do this, change traefik deployment to DaemonSet deployment. Then helm values.yaml will be

additionalArguments:
  - "--log.level=INFO"
  - --entrypoints.web.http.redirections.entryPoint.to=:443
  - --entrypoints.web.http.redirections.entryPoint.scheme=https
  - "--entrypoints.websecure.http.tls"
  - "--ping"
  - "--metrics.prometheus"

deployment:
  kind: DaemonSet

service:
  spec:
    externalTrafficPolicy: Local

After doing this, traefik will forward correct client IP. Network flow will change to

No LoadBalancer Solution

On 2021-06-28 morning, our production system get lots of source NATed client ips. Only partial client IPs source NATed. All of nodes in K3S cluster have partial NATed client IPs. So the externalTrafficPolicy: Local is not stable. The stable way is let traefik listen 80 and 443 ports directly.

additionalArguments:
  - "--log.level=INFO"
  - "--ping"
  - "--metrics.prometheus"

deployment:
  kind: DaemonSet

# no load balancer
service:
  enabled: false

# use host network for directly port listening
hostNetwork: true

ports:
  web:
    port: 80
    expose: true
    protocol: TCP
    exposePort: 80
    redirectTo: websecure
  websecure:
    port: 443
    expose: true
    protocol: TCP
    exposePort: 443
    tls:
      enable: true

# run as root to 80 and 443 port listen permission
securityContext:
  capabilities:
    add:
      - NET_BIND_SERVICE
  runAsNonRoot: false
  runAsUser: 0