Monitoring

Introduction to Grafana

Visualize metrics, logs, and traces

Grafana is an open-source analytics and visualization platform. It allows you to query, visualize, alert on, and explore your metrics, logs, and traces regardless of where they're stored.

Why Grafana?#

Multi-source support#

Connect to dozens of data sources including Prometheus, Elasticsearch, InfluxDB, and cloud platforms.

Powerful dashboards#

Create dynamic, reusable dashboards with a rich set of visualization options.

Alerting#

Define alert rules across multiple data sources and receive notifications via email, Slack, PagerDuty, and more.

Extensibility#

Extend Grafana with plugins for additional data sources, panels, and apps.

Core concepts#

Data sources#

Data sources are the backends that Grafana queries for metrics, logs, or traces.

Common data sources:

  • Prometheus
  • Elasticsearch
  • Loki
  • InfluxDB
  • MySQL/PostgreSQL
  • CloudWatch

Dashboards#

Dashboards are collections of panels organized into rows. Each panel displays data visualized in a specific way.

Panels#

Panels are the basic visualization building blocks. Types include:

  • Time series graphs
  • Stat panels
  • Gauges
  • Tables
  • Heatmaps
  • Logs panels

Variables#

Variables enable dynamic dashboards that can be filtered or scoped:

1
$environment: production, staging, development
2
$service: api, web, worker

Creating dashboards#

Basic dashboard structure#

1
{
2
  "title": "Application Metrics",
3
  "panels": [
4
    {
5
      "title": "Request Rate",
6
      "type": "timeseries",
7
      "targets": [
8
        {
9
          "expr": "rate(http_requests_total[5m])",
10
          "legendFormat": "{{method}} {{endpoint}}"
11
        }
12
      ]
13
    }
14
  ]
15
}

Dashboard as code#

Use Grafonnet or Terraform to manage dashboards as code:

1
// dashboard.jsonnet
2
local grafana = import 'grafonnet/grafana.libsonnet';
3
local dashboard = grafana.dashboard;
4
local prometheus = grafana.prometheus;
5
6
dashboard.new(
7
  'My Dashboard',
8
  schemaVersion=27,
9
)
10
.addPanel(
11
  grafana.graphPanel.new(
12
    'Request Rate',
13
    datasource='Prometheus',
14
  )
15
  .addTarget(
16
    prometheus.target('rate(http_requests_total[5m])')
17
  ),
18
  gridPos={x: 0, y: 0, w: 12, h: 8}
19
)

Alerting#

Alert rules#

Define conditions that trigger alerts:

1
groups:
2
  - name: app-alerts
3
    rules:
4
      - alert: HighErrorRate
5
        expr: |
6
          sum(rate(http_requests_total{status=~"5.."}[5m])) /
7
          sum(rate(http_requests_total[5m])) > 0.05
8
        for: 5m
9
        labels:
10
          severity: critical
11
        annotations:
12
          summary: High error rate detected

Notification channels#

Configure where alerts are sent:

  • Email
  • Slack
  • PagerDuty
  • Opsgenie
  • Webhooks

Provisioning#

Automate Grafana configuration with provisioning files:

1
# datasources.yml
2
apiVersion: 1
3
datasources:
4
  - name: Prometheus
5
    type: prometheus
6
    url: http://prometheus:9090
7
    isDefault: true
8
9
  - name: Loki
10
    type: loki
11
    url: http://loki:3100
1
# dashboards.yml
2
apiVersion: 1
3
providers:
4
  - name: default
5
    folder: ''
6
    type: file
7
    options:
8
      path: /var/lib/grafana/dashboards

Integration with DevOps Hub#

Visualize DevOps Hub pipeline metrics in Grafana:

  1. Add DevOps Hub as a data source
  2. Import the DevOps Hub dashboard template
  3. Configure variables for your projects
1
datasources:
2
  - name: DevOps Hub
3
    type: prometheus
4
    url: https://metrics.assistance.bg
5
    basicAuth: true
6
    basicAuthUser: $DEVOPSHUB_PROJECT_ID
7
    secureJsonData:
8
      basicAuthPassword: $DEVOPSHUB_API_KEY

Next steps#