Monitoring is a crucial aspect of maintaining a healthy, stable, and efficient Kubernetes (K8s) environment. Utilizing Prometheus for monitoring your Kubernetes cluster can provide real-time insights and metrics into the system, aiding in prompt troubleshooting and system optimization. This blog delves deep into setting up advanced configurations for Prometheus monitoring in a K8s environment using CTO.aiI for continuous integration and deployment.
Prerequisites
Before we dive into the advanced configurations, ensure you have the following setups:
- CTO.ai account
- Helm installed in your K8s cluster
- Prometheus and Grafana are installed in your K8s cluster
- Kubernetes Cluster: A functioning Kubernetes cluster, set up and configured with
kubectl
. - Helm: Helm package manager installed in your system.
Setting up CTO.ai EKS Workflow
Start by connecting your GitHub account and installing the EKS EC2 ASG workflow we support, and choose the repository where your Kubernetes configurations are stored.
git clone [email protected]:workflows-sh/aws-eks-ec2-asg-cdk.git
cd aws-eks-ec2-asg-cdk
Next, set up and add your secret keys, AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_ACCOUNT_NUMBER
, and GITHUB_TOKEN
with write permissions to the project secret settings in CTO.ai.
After cloning the repo from GitHub, run and build your Workflow using ops build -b .
and deploy your infrastructure to AWS.
Enter the name of your environment. You can use dev as the name of your environment. You can also use Prod or Stage, depending on what you want.
- Your workflow will start deploying and creating your resources on AWS using CloudFormation
- After deploying your AWS EC2 and Elastic Kubernetes workflow, you can see your stack directly on AWS CloudFormation. In your CloudFormation Stack, you can see your AWS resources created:
Dev-AWS-EKS-ASG-Provider
,AWS-EKS-EC2-ASG Resource
,Dev-AWS-EKS-EC2-ASG
,Sample-App-AWS-EKS
,CDKToolkit
,Dev-Sample-App-AWS-EKS-EC2-ASG
.
Installing Prometheus Operator
Installing the Prometheus Operator using Helm is a straightforward and manageable method. Begin by adding the Prometheus community Helm chart:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
Next, create a values.yaml file to hold your custom configurations and install the Prometheus Operator:
helm install prometheus prometheus-community/prometheus
- View your Helm Chart with the
helm list
command in your terminal.
- Back in your terminal, you can view your Prometheus pods using the
kubectl get pods
command.
- Prometheus is now accessible for your workloads. Next, get your service using
kubectl get svc
so you can access your Prometheus server from your localhost web UI.
- Port-Forward into your service using
kubectl port-forward service/prometheus-server 9090:80
Setup RBAC
Create the necessary RBAC roles and role bindings to grant Prometheus the necessary permissions. Create a file named rbac.yaml
with the following content:
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
namespace: prometheus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups: [""]
resources:
- nodes
- nodes/proxy
- services
- endpoints
- pods
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources:
- configmaps
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: prometheus
Apply the RBAC configurations using kubectl apply -f rbac.yaml
Create a Configuration File
Create a Prometheus configuration file prometheus-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-server-conf
namespace: prometheus
data:
prometheus.yml: |-
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
scheme: https
tls_config:
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;https
And apply the configuration using kubectl apply -f prometheus-config.yaml
Deploy Prometheus
Create a deployment file prometheus-deployment.yaml
:
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus-deployment
namespace: prometheus
spec:
replicas: 1
selector:
matchLabels:
app: prometheus-server
template:
metadata:
labels:
app: prometheus-server
spec:
serviceAccountName: prometheus
containers:
- name: prometheus
image: prom/prometheus
args:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus/"
ports:
- containerPort: 9090
volumeMounts:
- name: prometheus-config-volume
mountPath: /etc/prometheus/
volumes:
- name: prometheus-config-volume
configMap:
name: prometheus-server-conf
Deploy Prometheus: kubectl apply -f prometheus-deployment.yaml
Expose Prometheus
Expose Prometheus using a service or ingress. Here, we will create a service:
kubectl -n prometheus expose deployment prometheus-deployment --type=NodePort --name=prometheus-service
You can access the dashboard using the node IP and the NodePort.
View Alert Manager and active alerts.
Unlock the Full Potential of Prometheus Monitoring in K8s with CTO.ai: Your Advanced Configuration Guide
Taking advantage of advanced configurations in Prometheus not only facilitates more detailed and nuanced monitoring of your Kubernetes environments but also empowers teams to proactively manage the system health.
Ready to unlock the power of CTO.ai for your team? Schedule your consultation now with one of our experts today!
Comments