--- name: sre-dashboards description: Design and operationalize SRE dashboards that surface reliability, latency, error, saturation, and capacity signals across services. category: devops risk: critical source: https://github.com/BagelHole/DevOps-Security-Agent-Skills source_repo: BagelHole/DevOps-Security-Agent-Skills source_type: community date_added: '2026-09-20' license: MIT license_source: https://github.com/BagelHole/DevOps-Security-Agent-Skills/blob/main/LICENSE compatibility: Requires the relevant platform CLIs (kubectl, helm, terraform, git, CI runners) and authorized access to the target environment. Docs-only; helper scripts and templates not bundled. metadata: author: devops-skills version: '1.0' --- # SRE Dashboards Build dashboards that help teams detect, triage, and prevent reliability incidents. ## When to Use This Skill Use this skill when: - Defining service-level dashboards for production systems - Tracking SLO health and error-budget burn - Creating incident command-center views - Standardizing dashboard patterns across teams ## Prerequisites - Metrics pipeline (Prometheus, OpenTelemetry, or vendor equivalent) - Logs/traces linked to services and environments - Agreed service taxonomy (team, service, tier, environment) ## Dashboard Architecture Structure dashboards in layers: 1. **Executive Reliability View**: SLO attainment, incident counts, MTTR trends. 2. **Service Health View**: RED/USE metrics, dependency health, release markers. 3. **Deep-Dive View**: Per-endpoint latency, resource saturation, error categories. Keep each view answer-oriented: - *Are customers impacted?* - *What changed?* - *Where is the bottleneck?* ## Core SRE Panels ### Golden Signals - **Latency**: p50/p95/p99 request duration by endpoint - **Traffic**: request throughput and queue depth - **Errors**: 5xx rate, failed jobs, timeout ratio - **Saturation**: CPU, memory, disk I/O, thread/connection pool exhaustion ### SLO Panels - Current SLI value (rolling windows: 5m, 1h, 24h, 30d) - Error-budget remaining (%) - Burn-rate panels (fast and slow windows) - Multi-window burn alert status ### Change Correlation - Deployment markers and config-change annotations - Feature flag state overlays - Upstream/downstream dependency error rates ## Example PromQL Snippets ```promql # API error rate (%) 100 * sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) ``` ```promql # p95 latency by route histogram_quantile(0.95, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])) ) ``` ```promql # 5-minute burn rate for an illustrative 99.9% availability SLO ( sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) ) / (1 - 0.999) ``` Burn rate divides the bad-event ratio by the allowed bad-event ratio `(1 - SLO)`, not by another window’s observed errors. Compute the 1h window separately and combine short/long threshold conditions for an alert. Scope both numerator and denominator to the same service/environment; distinguish missing/no-traffic data from healthy zero errors. See [Google SRE alerting guidance](https://sre.google/workbook/alerting-on-slos/). ## Operational Guidelines - Use consistent color semantics (green=healthy, yellow=degrading, red=breach) - Label units explicitly (ms, req/s, %, cores) - Default time windows to incident-friendly ranges (15m, 1h, 6h, 24h) - Minimize panel count per dashboard to reduce cognitive load - Add runbook links directly in panel descriptions ## Troubleshooting ### Panel appears flat or empty - Verify label cardinality and filters (`service`, `env`, `region`) - Confirm scrape/ingest latency is within expected range - Check metric rename regressions after instrumentation updates ### High cardinality slows dashboards - Aggregate by stable dimensions (`service`, `route_group`) instead of raw IDs - Use recording rules for expensive percentile and ratio queries - Split deep-dive dashboards from NOC summary dashboards ## Related Skills - prometheus-grafana (`prometheus-grafana`) - Dashboard implementation and PromQL - opentelemetry (`opentelemetry`) - Standardized telemetry instrumentation - alerting-oncall (`alerting-oncall`) - Reliability alert routing and escalation - agent-observability (`agent-observability`) - AI workload reliability telemetry ## Limitations - Guidance executes against real environments: confirm target, blast radius, and rollback plan before applying anything. - Never deploy to production without explicit approval. Docs-only import: upstream scripts and templates not bundled. ### Example ```bash git status && git diff --stat kubectl diff -f manifest.yaml ``` > Adapted from [BagelHole/DevOps-Security-Agent-Skills](https://github.com/BagelHole/DevOps-Security-Agent-Skills) (MIT); frontmatter, When to Use/Limitations, and safety boundaries added for upstream compliance. Docs-only import: helper scripts and templates not bundled. --- Source attribution: BagelHole/DevOps-Security-Agent-Skills contributors, via Agentic Awesome Skills Original project: https://github.com/BagelHole/DevOps-Security-Agent-Skills License: MIT + CC-BY-4.0 License information: https://creativecommons.org/licenses/by/4.0/ Original upstream instructions; downloaded from Dave AI. ## Upstream license notice MIT License Copyright (c) 2026 Toby Miller Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Dave AI resource metadata Creator / publisher: BagelHole/DevOps-Security-Agent-Skills contributors, via Agentic Awesome Skills Original project: https://github.com/BagelHole/DevOps-Security-Agent-Skills Attribution basis: Recorded upstream repository Fields of work: Development & Engineering, Data & Analytics Roles: DevOps / SRE, Business Intelligence Analyst Specialties: Specialty not specified Tools: Tool not specified Country applicability: Country not specified Classification basis: Dave AI task classification from package title, description, and declared metadata Instruction SHA-256: 22b3bf320329940c6ec20ef8d39485b2e57eac495bf53ca3e7267ee809697d84 Metadata reviewed: 2026-10-27 This metadata supersedes earlier Dave AI catalog attribution. Original instruction and supporting files are preserved.