How Can You Use Prometheus Queries to Retrieve CPU and Memory Usage in Kubernetes Pods?

To get CPU and memory usage in Kubernetes pods using Prometheus queries, we can use special metrics from the Prometheus monitoring system. We can run queries like sum(rate(container_cpu_usage_seconds_total[5m])) by (pod) for CPU and sum(container_memory_usage_bytes) by (pod) for memory. These queries give us real-time information about how much resources we use. This helps us watch the performance and how well our Kubernetes applications use resources.

In this article, we will look at best ways to use Prometheus queries for monitoring CPU and memory in Kubernetes pods. We will talk about important metrics to check, how to write Prometheus queries for CPU and memory, how to show these metrics in Grafana, and how to set alerts for best performance. Here is a short list of what we will talk about:

How to Use Prometheus Queries to Get CPU and Memory Usage in Kubernetes Pods
What Metrics Should You Check in Kubernetes Pods with Prometheus Queries?
How to Write Prometheus Queries for CPU Usage in Kubernetes Pods?
How to Write Prometheus Queries for Memory Usage in Kubernetes Pods?
How to Show CPU and Memory Metrics from Prometheus in Grafana?
How to Set Alerts for CPU and Memory Usage in Kubernetes Pods with Prometheus?
Frequently Asked Questions

For more knowledge on Kubernetes and how to manage it, you can look at related articles like What is Kubernetes and How Does it Simplify Container Management? and How Do I Monitor My Kubernetes Cluster?.

What Metrics Should We Monitor in Kubernetes Pods with Prometheus Queries?

When we use Prometheus queries to check Kubernetes pods, we need to focus on important metrics. These metrics give us a look into how our applications perform and how healthy they are. Here are the main metrics we should watch:

CPU Usage:
- Metric: container_cpu_usage_seconds_total
- Description: This shows the total CPU time that containers use. We need to keep an eye on this to understand CPU use over time.
- Query Example:
```
rate(container_cpu_usage_seconds_total{namespace="your-namespace"}[5m])
```
Memory Usage:
- Metric: container_memory_usage_bytes
- Description: This tells us the current memory that the container uses. We have to check memory to prevent out-of-memory errors.
- Query Example:
```
container_memory_usage_bytes{namespace="your-namespace"}
```
Memory Limits:
- Metric: kube_pod_container_resource_limits_memory_bytes
- Description: This shows the memory limits set for the containers. This metric helps us ensure our pods do not use more memory than they should.
- Query Example:
```
kube_pod_container_resource_limits_memory_bytes{namespace="your-namespace"}
```
CPU Limits:
- Metric: kube_pod_container_resource_limits_cpu_cores
- Description: This metric shows the CPU limits for each container. It is important for making sure our applications do not use more resources than we give them.
- Query Example:
```
kube_pod_container_resource_limits_cpu_cores{namespace="your-namespace"}
```
Network Usage:
- Metric: container_network_receive_bytes_total and container_network_transmit_bytes_total
- Description: These metrics track how many bytes the container receives and sends over the network.
- Query Example:
```
rate(container_network_receive_bytes_total{namespace="your-namespace"}[5m])
rate(container_network_transmit_bytes_total{namespace="your-namespace"}[5m])
```
Restart Count:
- Metric: kube_pod_container_status_restarts_total
- Description: This counts how many times a container has restarted. A high number of restarts can show problems with the application.
- Query Example:
```
kube_pod_container_status_restarts_total{namespace="your-namespace"}
```
Disk I/O:
- Metric: container_fs_reads_bytes_total and container_fs_writes_bytes_total
- Description: These metrics track how many bytes the container reads from and writes to the filesystem.
- Query Example:
```
rate(container_fs_reads_bytes_total{namespace="your-namespace"}[5m])
rate(container_fs_writes_bytes_total{namespace="your-namespace"}[5m])
```

These metrics give us a full view of how resources are used and the health of applications in Kubernetes pods. By watching these metrics through Prometheus queries, we can keep our performance good and find problems quickly in our Kubernetes setup. For more information on Kubernetes monitoring, we can check out how to monitor my Kubernetes cluster.

How to Write Prometheus Queries for CPU Usage in Kubernetes Pods?

We can get CPU usage data for Kubernetes pods using Prometheus queries. We can use some built-in metrics from kube-state-metrics and cAdvisor. The queries below can help us get the data we need.

Current CPU Usage for All Pods
This query gets the current CPU usage for all running pods in the Kubernetes cluster:
```
sum(rate(container_cpu_usage_seconds_total{namespace="your-namespace"}[5m])) by (pod)
```
CPU Usage for a Specific Pod
To check CPU usage for a specific pod, we should change your-pod-name and your-namespace to the right values:
```
sum(rate(container_cpu_usage_seconds_total{pod="your-pod-name", namespace="your-namespace"}[5m]))
```
CPU Usage Over Time
To see the CPU usage over a time period like the last hour, we can use this query:
```
sum(rate(container_cpu_usage_seconds_total{namespace="your-namespace"}[1h])) by (pod)
```
Percentage of CPU Usage
If we want to find out the percentage of CPU usage compared to the total CPU capacity of the node, we can use this query:
```
sum(rate(container_cpu_usage_seconds_total{namespace="your-namespace"}[5m])) by (pod) / sum(kube_node_status_allocatable_cpu_cores) * 100
```

Alerting for High CPU Usage
To create an alert for high CPU usage, like when it is more than 80%, we can use this query:

sum(rate(container_cpu_usage_seconds_total{namespace="your-namespace"}[5m])) by (pod) / sum(kube_node_status_allocatable_cpu_cores) > 0.8

We should run these queries in the Prometheus query interface. We can also put them in our monitoring dashboards like Grafana to see the data. We need to change the namespace and pod names based on our environment. For more details on monitoring, we can check out how to monitor a Kubernetes application with Prometheus and Grafana.

How to Write Prometheus Queries for Memory Usage in Kubernetes Pods

To get memory usage data for Kubernetes pods with Prometheus, we mainly use the container_memory_usage_bytes metric. This metric tells us the memory usage of containers in bytes. Here are some simple examples of how we can write Prometheus queries for this.

Basic Memory Usage Query for All Pods:

sum(container_memory_usage_bytes) by (pod)

Memory Usage for a Specific Pod: Change <pod_name> to the name of your pod.
```
sum(container_memory_usage_bytes{pod="<pod_name>"})
```
Memory Usage for Pods in a Specific Namespace: Change <namespace> to the namespace you want.
```
sum(container_memory_usage_bytes{namespace="<namespace>"}) by (pod)
```
Memory Usage for Containers: If we want to see memory usage for each container in the pods:
```
sum(container_memory_usage_bytes) by (pod, container)
```
Average Memory Usage Over a Time Window: To find the average memory usage in the last 5 minutes:
```
avg(rate(container_memory_usage_bytes[5m])) by (pod)
```

Memory Usage Percentage: To calculate memory usage as a percentage of the total memory available:

(sum(container_memory_usage_bytes) / sum(kube_pod_container_resource_requests_memory_bytes)) * 100

Filtering by Container Name: To see memory usage for a specific container in a pod:
```
sum(container_memory_usage_bytes{pod="<pod_name>", container="<container_name>"})
```
Alerting Based on Memory Usage: We can set alerts for high memory usage by using a query with a threshold:
```
sum(container_memory_usage_bytes{namespace="<namespace>"}) by (pod) > <threshold_value>
```

These queries help us to check memory usage in our Kubernetes pods with Prometheus. We can also show these metrics in Grafana to get better insights into our application’s performance. For more information on how to monitor Kubernetes applications with Prometheus and Grafana, check out how to monitor a Kubernetes application with Prometheus and Grafana.

How to Visualize CPU and Memory Metrics from Prometheus in Grafana?

To see CPU and memory metrics from Prometheus in Grafana, we can follow these steps:

Set Up Grafana: First, we need to make sure Grafana is installed and running. We can check the official Grafana installation guide for help.
Add Prometheus as Data Source:
- We open Grafana and log in.
- Then, we go to Configuration and click on Data Sources.
- Next, we click on Add data source and choose Prometheus.
- In the URL field, we enter the Prometheus server URL. For example, we can use http://prometheus-server:9090.
- Finally, we click Save & Test to check if the connection works.
Create a Dashboard:
- We go to Dashboards and then New Dashboard.
- After that, we click on Add new panel.
Query CPU Metrics:
- In the panel editor, we set the data source to Prometheus.
- We can use this query to see CPU usage for all pods:
```
sum(rate(container_cpu_usage_seconds_total{container_name!="POD"}[5m])) by (namespace, pod)
```
- We can change the visualization type, like graph or gauge, if we want.
Query Memory Metrics:
- Now, we add another panel for memory usage.
- We can use this query to see memory usage:
```
sum(container_memory_usage_bytes{container_name!="POD"}) by (namespace, pod)
```
- We choose a good visualization format for this.
Customize Your Panel:
- In the panel settings, we can change titles, legends, and how things look.
- We set the refresh rate to update the data regularly.
Save the Dashboard:
- After adding all the metrics we want, we click on Save in the top right corner to keep our dashboard.

This setup helps us to monitor and see CPU and memory usage from Kubernetes pods using Prometheus and Grafana. If we want to learn more about monitoring Kubernetes clusters, we can check this article on how to monitor my Kubernetes cluster.

How to Set Up Alerts for CPU and Memory Usage in Kubernetes Pods with Prometheus?

To set up alerts for CPU and memory usage in Kubernetes pods using Prometheus, we need to configure alerting rules in Prometheus. Alerts will send notifications when resource usage goes over set limits. Here’s how we can do this:

Define Alerting Rules: We create a YAML file for our alerting rules. This file will say when to alert us about CPU and memory usage.

groups:
  - name: kubernetes-alerts
    rules:
      - alert: HighCPUUsage
        expr: sum(rate(container_cpu_usage_seconds_total{job="kubelet", image!=""}[5m])) by (pod_name) > 0.8
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High CPU usage detected on pod {{ $labels.pod_name }}"
          description: "Pod {{ $labels.pod_name }} is using more than 80% CPU for the last 5 minutes."

      - alert: HighMemoryUsage
        expr: sum(container_memory_usage_bytes{job="kubelet", image!=""}) by (pod_name) / sum(container_spec_memory_limit_bytes{job="kubelet", image!=""}) by (pod_name) > 0.9
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High Memory usage detected on pod {{ $labels.pod_name }}"
          description: "Pod {{ $labels.pod_name }} is using more than 90% of its memory limit for the last 5 minutes."

Load Alerting Rules: We need to make sure Prometheus loads our alerting rules. We change the prometheus.yml configuration file to add our rules file.
```
rule_files:
  - '/etc/prometheus/rules/*.yaml'  # Change the path if needed
```
Deploy Prometheus: If we have not set up Prometheus yet, we need to deploy it in our Kubernetes cluster. We can use the Prometheus Operator or Helm to make it easier.

Example with Helm:
```
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/prometheus
```

Configure Alertmanager: To handle alerts, we set up Alertmanager to get notifications. We update the Alertmanager configuration file (alertmanager.yml) to add notification channels.

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: 'slack'

receivers:
  - name: 'slack'
    slack_configs:
      - api_url: 'YOUR_SLACK_WEBHOOK_URL'
        channel: '#alerts'

Test Alerts: After we set up the configuration, we can use the Prometheus UI to check if the alerts are working. We can also create high resource usage in our pods to see if the alerts go off like we expect.

By doing these steps, we can set up alerts for CPU and memory usage in Kubernetes pods using Prometheus. This helps us to monitor resource use and act quickly if there are problems. For more details on monitoring in Kubernetes, we can check this guide on monitoring a Kubernetes application with Prometheus and Grafana.

Frequently Asked Questions

1. What is Prometheus and how does it work with Kubernetes?

Prometheus is a tool that helps us monitor and alert. It is open-source and is used a lot in Kubernetes. It collects data about metrics from set targets at certain times. Then, it stores this data in a time-series database. Prometheus has a good query language. This helps us get and change time-series data. It is great for checking CPU and memory use in Kubernetes pods. For more details, we can check how to monitor your Kubernetes cluster.

2. How can I access CPU and memory metrics in Prometheus for Kubernetes pods?

To get CPU and memory metrics from Kubernetes pods, we can use Prometheus queries. We target metrics like container_cpu_usage_seconds_total for CPU and container_memory_usage_bytes for memory. By writing specific Prometheus queries, we can monitor and see resource use. This helps us make performance better.

3. What are the best practices for setting up Prometheus monitoring in Kubernetes?

Best practices for using Prometheus in Kubernetes include using Helm to deploy Prometheus. This makes management easier. We should also set up service discovery to find targets automatically. Using resource limits on pods helps stop resource overuse. Also, adding Grafana for visualization can make our monitoring better. It helps us create useful dashboards.

4. How do I create alerts for CPU and memory usage in Prometheus?

To make alerts for CPU and memory in Prometheus, we need to set alerting rules in the configuration file. We can use expressions like increase(container_cpu_usage_seconds_total[5m]) > threshold to set off alerts when use goes over a limit. When we connect Prometheus with Alertmanager, we can handle alerts better and send notifications.

5. Can I visualize CPU and memory metrics from Prometheus in Grafana?

Yes, we can use Grafana to see CPU and memory metrics from Prometheus. By adding Prometheus as a data source in Grafana, we can make custom dashboards. These dashboards show real-time metrics for our Kubernetes pods. This helps us see what is happening and find any performance problems easily. For more steps, we can check how to monitor a Kubernetes application with Prometheus and Grafana.