Monitoring with Prometheus & Grafana

A system is not truly complete without observability. When services start throwing errors or running out of memory, you must discover the failure before your customers do. Monitoring provides visibility into infrastructure and application internals.

The combo of Prometheus (metrics storage and scraping engine) and Grafana (dashboard visualizer) has become the gold standard DevOps observability stack.

Monitoring System Architecture

Prometheus uses a pull-based design. Rather than applications pushing metrics (which can overwhelm servers), Prometheus periodically scrapes structured text data from targets over HTTP endpoints.

Exposing Metrics

To monitor an application, it must expose its statistics on a URL (usually /metrics).

• Node Exporter: An agent run on Linux hosts to expose CPU, Memory, Disk, and Network metrics.
• Application Instrumentation: Developers use Prometheus SDK client libraries to publish custom counters and gauges (e.g. HTTP request count, latency).

Here is a sample scraped raw format from a /metrics endpoint:

# HELP http_requests_total Total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="post",code="200"} 1043
http_requests_total{method="get",code="500"} 12

Configuring Prometheus

Prometheus is configured via a YAML file, specifying target scrape jobs:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alert.rules.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

  - job_name: 'backend-api'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['api-service:8080']

Alerting Rules

Prometheus evaluates metrics against alert criteria. If a threshold is crossed, it forwards warnings to Alertmanager:

groups:
  - name: instance_alerts
    rules:
      - alert: HostHighCpuLoad
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Host high CPU load (instance {{ $labels.instance }})"
          description: "CPU load is > 85% for 5 minutes."

These alerts can trigger webhooks to send Slack messages, PagerDuty incidents, or system emails, closing the automation loop.