monitoring AWS Load Balancer Controller in AWS EKS

“Suddenly”, but AWS ALB Controller is good at metrics for Prometheus.

Using a standard URI /metrics.

To begin with, let’s check that there are metrics in the pod.

Find under:

kk -n cube-system get pod | grape hour

aws-load-balancer-controller-7bdcf75789-wk4bt   1/1     Running   0          41h

Forward a port to it:

kk port-forward -n kube-system pods/aws-load-balancer-controller-7bdcf75789-wk4bt 8080:8080

Forwarding from 127.0.0.1:8080 -> 8080

Forwarding from [::1]:8080 -> 8080

And we try from a working machine:

curl -s localhost:8080/metrics | head

HELP aws_api_call_duration_seconds Perceived latency from when your code makes an SDK call, includes retries

TYPE aws_api_call_duration_seconds histogram

aws_api_call_duration_seconds_bucket{operation=”AuthorizeSecurityGroupIngress”,service=”EC2″,le=”0.005″} 0

aws_api_call_duration_seconds_bucket{operation=”AuthorizeSecurityGroupIngress”,service=”EC2″,le=”0.01″} 0

aws_api_call_duration_seconds_bucket{operation=”AuthorizeSecurityGroupIngress”,service=”EC2″,le=”0.025″} 0

aws_api_call_duration_seconds_bucket{operation=”AuthorizeSecurityGroupIngress”,service=”EC2″,le=”0.05″} 0

aws_api_call_duration_seconds_bucket{operation=”AuthorizeSecurityGroupIngress”,service=”EC2″,le=”0.1″} 0

aws_api_call_duration_seconds_bucket{operation=”AuthorizeSecurityGroupIngress”,service=”EC2″,le=”0.25″} 29

aws_api_call_duration_seconds_bucket{operation=”AuthorizeSecurityGroupIngress”,service=”EC2″,le=”0.5″} 39

aws_api_call_duration_seconds_bucket{operation=”AuthorizeSecurityGroupIngress”,service=”EC2″,le=”1″} 39

Super – there are metrics. It remains to collect them.

We have our own Prometheus on each AWS Elastic Kubernetes cluster, which is launched and configured through the Prometheus Operator.

Prometheus instances in Kubernetes are connected to a central server through Prometheus federation – it collects metrics from them.

What we need is to create a ServiceMonitor for Prometheus in Kubernetes, using which Prometheus will start collecting metrics from the desired endpoint, and then send them to the central Prometheus, where we will make alerts and dashboards in Grafana.

We already have several custom ServiceMonitors:

kk -n monitoring get servicemonitors

NAME                                                 AGE

backend-redis                                        225d

devops-rabbitmq-servicemonitor                       80d

ALB Controller Service

AWS ALB Contoller deploys from the manifest with Deployment, and creates only a sub. If you install it from the Helm chart, the Service itself is probably already there, and even, possibly, ServiceMonitor.

In our case, for the test, we will create a new Service that will forward traffic to the pod with ALB Controller:

---
apiVersion: v1
kind: Service
metadata:
  name: aws-alb-controller-svc
  namespace: kube-system
  labels:
    app.kubernetes.io/instance: aws-load-balancer-controller
    app.kubernetes.io/name: aws-load-balancer-controller
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: aws-load-balancer-controller
  ports:
    - name: http
      protocol: TCP
      port: 8080
      targetPort: 8080

Service is created in kube-system Namespace, where our ALB Controller lives, and in it .spec.selector use labels из ALB Controller Deployment:

kk -n kube-system get deploy aws-load-balancer-controller –show-labels

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE    LABELS

aws-load-balancer-controller   1/1     1            1           156d   app.kubernetes.io/instance=aws-load-balancer-controller,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=aws-load-balancer-controller,app.kubernetes.io/version=v2.2.2,helm.sh/chart=aws-load-balancer-controller-1.2.5

ALB Controller ServiceMonitor

Next, we describe the ServiceMonitor:

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  namespace: monitoring
  labels:
    serviceapp: aws-alb-controller-servicemonitor
    release: prometheus
  name: aws-alb-controller-servicemonitor
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 15s
    port: http
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
    matchLabels:
      app.kubernetes.io/name: aws-load-balancer-controller

Applying the manifest:

kk apply -f alb-svc-mo-check.yaml

service/aws-alb-controller-svc created

servicemonitor.monitoring.coreos.com/aws-alb-controller-servicemonitor created

Let’s check the Service:

kk -n kube-system get svc aws-alb-controller-svc

NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE

aws-alb-controller-svc   ClusterIP   172.20.158.27 8080/TCP   1m

Forward a port to this service:

kk -n kube-system port-forward services/aws-alb-controller-svc 8081:8080

Forwarding from 127.0.0.1:8081 -> 8080

Checking the availability of metrics:

curl -s localhost:8081/metrics | head

HELP aws_api_call_duration_seconds Perceived latency from when your code makes an SDK call, includes retries

TYPE aws_api_call_duration_seconds histogram

aws_api_call_duration_seconds_bucket{operation=”AuthorizeSecurityGroupIngress”,service=”EC2″,le=”0.005″} 0

aws_api_call_duration_seconds_bucket{operation=”AuthorizeSecurityGroupIngress”,service=”EC2″,le=”0.01″} 0

aws_api_call_duration_seconds_bucket{operation=”AuthorizeSecurityGroupIngress”,service=”EC2″,le=”0.025″} 0

aws_api_call_duration_seconds_bucket{operation=”AuthorizeSecurityGroupIngress”,service=”EC2″,le=”0.05″} 0

aws_api_call_duration_seconds_bucket{operation=”AuthorizeSecurityGroupIngress”,service=”EC2″,le=”0.1″} 0

aws_api_call_duration_seconds_bucket{operation=”AuthorizeSecurityGroupIngress”,service=”EC2″,le=”0.25″} 29

aws_api_call_duration_seconds_bucket{operation=”AuthorizeSecurityGroupIngress”,service=”EC2″,le=”0.5″} 39

aws_api_call_duration_seconds_bucket{operation=”AuthorizeSecurityGroupIngress”,service=”EC2″,le=”1″} 39

Checking in Kubernetes Prometheus – Status > Targets:

And metrics:

And in the central Prometheus:

It remains to come up with alerts and a dashboard for Grafana.

Amazon web services,HOWTO’s,Monitoring,Prometheus,Virtualization,AWS,AWS Load Balancer,Kubernetes,monitoring,

#monitoring #AWS #Load #Balancer #Controller #AWS #EKS

Leave a Comment

Your email address will not be published. Required fields are marked *