Advanced Configurations for Prometheus Monitoring in K8s Using CTO.ai

Monitoring is a crucial aspect of maintaining a healthy, stable, and efficient Kubernetes (K8s) environment. Utilizing Prometheus for monitoring your Kubernetes cluster can provide real-time insights and metrics into the system, aiding in prompt troubleshooting and system optimization. This blog delves deep into setting up advanced configurations for Prometheus monitoring in a K8s environment using CTO.aiI for continuous integration and deployment.

Prerequisites

Before we dive into the advanced configurations, ensure you have the following setups:

• CTO.ai account
• Helm installed in your K8s cluster
• Prometheus and Grafana are installed in your K8s cluster
• Kubernetes Cluster: A functioning Kubernetes cluster, set up and configured with kubectl.
• Helm: Helm package manager installed in your system.

Setting up CTO.ai EKS Workflow

Start by connecting your GitHub account and installing the EKS EC2 ASG workflow we support,  and choose the repository where your Kubernetes configurations are stored.

git clone [email protected]:workflows-sh/aws-eks-ec2-asg-cdk.git

cd aws-eks-ec2-asg-cdk

Next, set up and add your secret keys, AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_ACCOUNT_NUMBER, and GITHUB_TOKEN with write permissions to the project secret settings in CTO.ai.

After cloning the repo from GitHub, run and build your Workflow using ops build -b . and deploy your infrastructure to AWS.

Enter the name of your environment. You can use dev as the name of your environment. You can also use Prod or Stage, depending on what you want.

• Your workflow will start deploying and creating your resources on AWS using CloudFormation

• After deploying your AWS EC2 and Elastic Kubernetes workflow, you can see your stack directly on AWS CloudFormation. In your CloudFormation Stack, you can see your AWS resources created: Dev-AWS-EKS-ASG-Provider, AWS-EKS-EC2-ASG Resource, Dev-AWS-EKS-EC2-ASG, Sample-App-AWS-EKS, CDKToolkit, Dev-Sample-App-AWS-EKS-EC2-ASG.

Installing Prometheus Operator

Installing the Prometheus Operator using Helm is a straightforward and manageable method. Begin by adding the Prometheus community Helm chart:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts


helm repo update

Next, create a values.yaml file to hold your custom configurations and install the Prometheus Operator:

helm install prometheus prometheus-community/prometheus

• View your Helm Chart with the helm list command in your terminal.  
  

• Back in your terminal, you can view your Prometheus pods using the kubectl get pods command.
  

• Prometheus is now accessible for your workloads. Next, get your service using kubectl get svc so you can access your Prometheus server from your localhost web UI.
  

• Port-Forward into your service using kubectl port-forward service/prometheus-server 9090:80
  

Setup RBAC

Create the necessary RBAC roles and role bindings to grant Prometheus the necessary permissions. Create a file named rbac.yaml with the following content:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: prometheus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
  - apiGroups: [""]
    resources:
      - nodes
      - nodes/proxy
      - services
      - endpoints
      - pods
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources:
      - configmaps
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: prometheus

Apply the RBAC configurations using kubectl apply -f rbac.yaml

Create a Configuration File

Create a Prometheus configuration file prometheus-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
  namespace: prometheus
data:
  prometheus.yml: |-
    global:
      scrape_interval: 15s
    scrape_configs:
      - job_name: 'kubernetes-apiservers'
        kubernetes_sd_configs:
        - role: endpoints
        scheme: https
        tls_config:
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
        - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
          action: keep
          regex: default;kubernetes;https

And apply the configuration using kubectl apply -f prometheus-config.yaml

Deploy Prometheus

Create a deployment file prometheus-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-deployment
  namespace: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-server
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      serviceAccountName: prometheus
      containers:
        - name: prometheus
          image: prom/prometheus
          args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus/"
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/
      volumes:
        - name: prometheus-config-volume
          configMap:
            name: prometheus-server-conf

Deploy Prometheus: kubectl apply -f prometheus-deployment.yaml

Expose Prometheus

Expose Prometheus using a service or ingress. Here, we will create a service:

kubectl -n prometheus expose deployment prometheus-deployment --type=NodePort --name=prometheus-service

You can access the dashboard using the node IP and the NodePort.

View Alert Manager and active alerts.

Unlock the Full Potential of Prometheus Monitoring in K8s with CTO.ai: Your Advanced Configuration Guide

Taking advantage of advanced configurations in Prometheus not only facilitates more detailed and nuanced monitoring of your Kubernetes environments but also empowers teams to proactively manage the system health.

Ready to unlock the power of CTO.ai for your team? Schedule your consultation now with one of our experts today!