---
name: incident-runbook-generator
description: Creates step-by-step incident response runbooks for common outages with actions, owners, rollback procedures, and communication templates. Use for "incident runbook", "outage response", "incident management", or "on-call procedures".
---

# Incident Runbook Generator

Create actionable runbooks for common incidents.

## Runbook Template

````markdown
# Runbook: Database Connection Pool Exhausted

**Severity:** P1 (Critical)
**Estimated Time to Resolve:** 15-30 minutes
**Owner:** Database Team (On-call)

## Symptoms

- Application errors: "connection pool exhausted"
- Increased API latency (>5s)
- Failed health checks
- CloudWatch alarm: `DatabaseConnectionsHigh`

## Detection

- Alert: DatabaseConnectionPoolExhausted
- Metrics: `active_connections > max_connections * 0.9`
- Logs: "Error: connect ETIMEDOUT"

## Immediate Actions (5 min)

1. **Verify the issue**
   ```bash
   # Check current connections
   SELECT count(*) FROM pg_stat_activity;
   ```
````

2. **Identify long-running queries**

   ```sql
   SELECT pid, now() - pg_stat_activity.query_start AS duration, query
   FROM pg_stat_activity
   WHERE state = 'active'
   ORDER BY duration DESC
   LIMIT 10;
   ```

3. **Kill blocking queries** (if safe)
   ```sql
   SELECT pg_terminate_backend(pid)
   FROM pg_stat_activity
   WHERE state = 'idle in transaction'
   AND now() - state_change > interval '5 minutes';
   ```

## Mitigation (10 min)

1. **Scale up connection pool** (temporary)

   ```bash
   # Update RDS parameter group
   aws rds modify-db-parameter-group \
     --db-parameter-group-name prod-params \
     --parameters "ParameterName=max_connections,ParameterValue=200"
   ```

2. **Restart application** (if needed)

   ```bash
   kubectl rollout restart deployment/api
   ```

3. **Monitor recovery**
   ```bash
   watch -n 5 'psql -c "SELECT count(*) FROM pg_stat_activity;"'
   ```

## Root Cause Investigation

Check for:

- [ ] Recent deployment (new code with connection leaks)
- [ ] Traffic spike (legitimate or DDoS)
- [ ] Slow queries holding connections
- [ ] Connection pool configuration too small
- [ ] Application not releasing connections

## Rollback Steps

If caused by deployment:

```bash
# Rollback to previous version
kubectl rollout undo deployment/api

# Verify
kubectl rollout status deployment/api
```

## Communication Template

**Initial (within 5 min):**

```
🚨 INCIDENT: Database connection pool exhausted
Status: Investigating
Impact: API errors and slowness
ETA: 15-30 min
Next update: 10 min
```

**Update (every 10 min):**

```
UPDATE: Killed long-running queries
Status: Mitigating
Impact: Still degraded, improving
Actions: Scaling connection pool
Next update: 10 min
```

**Resolution:**

```
✅ RESOLVED: Database connections normalized
Duration: 25 minutes
Root cause: Connection leak in v2.3.4
Fix: Rolled back to v2.3.3
Follow-up: Bug fix PR #1234
Postmortem: [link]
```

## Prevention

- [ ] Add connection pool metrics to dashboards
- [ ] Implement connection timeout (30s)
- [ ] Add connection leak detection in tests
- [ ] Set up pre-deployment load testing
- [ ] Review connection pool sizing

## Related Runbooks

- Database High CPU
- Slow Database Queries
- Application OOM

```

## Output Checklist

- [ ] Symptoms documented
- [ ] Detection criteria
- [ ] Step-by-step actions
- [ ] Owner assigned
- [ ] Rollback procedure
- [ ] Communication templates
- [ ] Prevention measures
ENDFILE
```


---
Source attribution: patricio0312rev
Original project: https://github.com/patricio0312rev/skillset/blob/49ab171bdf44f8dae0a1c9812f5f703c6183ea0b/templates/performance/incident-runbook-generator/SKILL.md
License: MIT
License information: https://opensource.org/license/mit
Original upstream instructions; downloaded from Dave AI.

## Upstream license notice

MIT License

Copyright (c) 2026 Patricio Marroquin

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.



# Dave AI resource metadata

Creator / publisher: patricio0312rev
Original project: https://github.com/patricio0312rev/skillset/blob/49ab171bdf44f8dae0a1c9812f5f703c6183ea0b/templates/performance/incident-runbook-generator/SKILL.md
Attribution basis: Recovered original package and original project license at a pinned revision

Fields of work: Development & Engineering, Operations, Customer Support
Roles: DevOps / SRE, Operations Manager, Technical Support
Specialties: DevOps
Tools: AWS
Country applicability: Country not specified
Classification basis: Source catalog labels
Instruction SHA-256: a13a5dcb665b672ea0d419b5549c7dcddaf0d1a0d888af69ea1f36cbd6efe377
Metadata reviewed: 2026-10-27
This metadata supersedes earlier Dave AI catalog attribution. Original instruction and supporting files are preserved.