AWS DevOps & Developer Productivity Blog
Reducing database incident MTTR by 73% with a custom AWS DevOps Agent Skill
Database incidents are among the most time-consuming operational events for Site Reliability Engineering (SRE) teams. When an Amazon Aurora PostgreSQL cluster experiences performance degradation, SRE engineers must correlate metrics across connection pools, replication topology, query performance, storage subsystems, and parameter configurations. These troubleshooting steps routinely take 45–60 minutes even for experienced engineers.
AWS DevOps Agent is an operations teammate that resolves and proactively helps prevent incidents, and optimizes application reliability and performance. It also handles on-demand SRE tasks across AWS, multicloud, and on-premises environments. Although its built-in capabilities handle common scenarios well, most organizations have unique database configurations, connection pooling patterns, and performance baselines. That knowledge lives in the heads of senior database reliability engineers (DBREs).
With Custom Skills, you can codify that expertise and make it available to the agent 24/7. In this post, we walk through building a custom Aurora PostgreSQL performance degradation investigation Skill. This multi-fault procedural Skill uses decision-tree logic that covers five distinct failure modes. We then show how it reduces Mean Time to Resolution (MTTR) from a baseline of 52 minutes to 14 minutes. That’s a 73 percent reduction, measured in a controlled internal test of three simulated incidents.
Prerequisites
Before getting started, make sure you have:
- An active AWS account with access to AWS DevOps Agent.
- An Agent Space configured with access to your Aurora PostgreSQL cluster’s account.
- Amazon CloudWatch metrics and Performance Insights enabled on your Aurora cluster.
- Access to the Operator Web App for Skill deployment.
- Familiarity with Aurora PostgreSQL operational metrics (
DatabaseConnections,ReplicaLag,db.load).
The problem: Database incidents require deep tribal knowledge
Consider a typical Aurora PostgreSQL performance degradation incident. The on-call engineer receives a page indicating elevated read latency. From here, the investigation branches into multiple possible root causes:
- Is the connection pool exhausted, starving the application of database connections?
- Is replication lag causing stale reads that trigger application retries?
- Did a recent deployment introduce a slow query that’s consuming all available CPU?
- Is the storage subsystem under pressure from table bloat or excessive write amplification?
- Did someone modify the parameter group, introducing a misconfiguration that degrades performance?
Each branch requires checking different metrics, correlating with different timelines, and applying different remediation strategies. A senior DBRE knows which branch to explore first based on subtle signal combinations. However, that knowledge is rarely documented in a way that’s actionable at 2 AM.
Additionally, parameter drift is particularly insidious because its symptoms mimic other failure modes. It occurs when someone modifies the cluster parameter group and introduces a misconfiguration. Without domain-specific guidance, the agent chases symptoms rather than tracing back to the configuration change.
Solution overview: A multi-fault procedural Skill
Rather than building five separate Skills (one per failure mode), we build a single procedural Skill with decision-tree logic that mirrors how an expert DBRE triages Aurora PostgreSQL issues. The Skill evaluates initial signals to determine which investigation branch to follow, then executes a structured procedure for that specific failure mode.
What makes this different from the workshop examples
The AWS DevOps Agent workshop covers two Skill types: (1) a single-file UI-created checklist (AWS Lambda cold starts) and (2) a multi-file architecture-context Skill (EKS Retail Store). Both are effective for their scope, but neither demonstrates a procedural investigation Skill with decision-tree logic that proves measurable MTTR reduction.
Our Skill differs in three key ways:
- Decision-tree branching: “If metric X exceeds threshold Y, check Z before proceeding to W,” not a linear checklist.
- Multi-fault coverage: five failure modes in one Skill with explicit priority ordering.
- Measurable outcome: Controlled before and after MTTR comparison proving 73 percent reduction.
Skill anatomy
Skills are modular instruction sets. Each is a self-contained directory of Markdown files that provides specialized domain knowledge and investigation methodologies to AWS DevOps Agent. Each Skill contains:
- SKILL.md (required) – Core instructions with investigation procedures, frontmatter metadata, and decision logic.
- references/ (optional) – Supporting documentation, threshold tables, query templates.
- assets/ (optional) – Architecture diagrams, flowcharts, data files.
The agent evaluates a Skill’s description field in the frontmatter to determine relevance, then loads and follows the instructions during investigations. Skills follow the open Agent Skills specification (agentskills.io), supporting non-executable documents: Markdown, PDFs, images, and data files.
Directory structure
Walkthrough: Building the Skill
The following steps walk through building the Skill end to end, from documenting the investigation pattern to deploying it. We start by capturing how an expert triages the incident.
Step 1: Document the expert investigation pattern
Before writing any Markdown, we interviewed three senior DBREs and documented their mental model for Aurora PostgreSQL performance triage. The common pattern:
- Initial signal classification – What metric triggered the alarm?
- Impact scope assessment – Writer instance only, reader instances only, or cluster-wide?
- Timeline correlation – Did this start after a deployment, scaling event, or traffic pattern change?
- Branch into specific failure mode – Based on signals from steps 1–3.
- Deep investigation – Follow the procedure for that failure mode.
- Root cause and remediation – Structured output with immediate, short-term, and long-term actions.
Step 2: Write the SKILL.md with decision-tree logic
The SKILL.md begins with frontmatter that tells the agent when to activate, followed by initial signal classification that routes to the correct branch:
Frontmatter
Signal classification table
| Primary Signal | Likely Branch | Go To |
DatabaseConnections near max or connection errors |
Connection Pool Exhaustion | Branch A |
ReplicaLag > 20 ms or stale read complaints |
Replication Lag | Branch B |
Latency spike with high db.load in Performance Insights |
Slow Query Regression | Branch C |
FreeLocalStorage declining or VolumeBytesUsed alert |
Storage Pressure | Branch D |
| Parameter group change in AWS CloudTrail within past 7 days | Parameter Drift | Branch E |
Critical rule: If multiple signals are present, prioritize A → B → C → D → E. Connection exhaustion masks other symptoms, so it must be checked first.
Branch A: Connection pool exhaustion
Investigation procedure. Connection pool exhaustion is the highest-priority branch because it masks every other symptom, so the agent checks it first. It works through four checks in order, moving from the raw connection count to the behavior behind it:
- Compare active connections to the limit. Check
DatabaseConnectionsagainstmax_connections. If connections exceed 85 percent of the maximum, pool exhaustion is confirmed. - Identify where the connections originate. Distinguish RDS Proxy pinned sessions from the direct application pool, since each points to a different fix.
- Correlate with deployment history. A service that deployed recently and opened connections without releasing them is a common trigger, so the agent checks whether a rollout preceded the spike.
- Check for long-running transactions. Sessions stuck in
idle_in_transactionhold connection slots open and starve the pool.
Interpreting the findings. The preceding checks establish what is happening. This is where the agent decides why. Each pattern maps to a distinct root cause:
- If connections rise monotonically, the application has a connection leak in its code.
- If connections spike with traffic, the pool is sized too small for the load.
- If connections spike right after a deployment, a new code path is not releasing connections.
- If the spike is in pinned RDS Proxy sessions, session state is reducing connection reuse (multiplexing).
Branch B: Replication lag
Investigation procedure. When readers fall behind the writer, the agent needs to know whether the cause is write pressure, reader capacity, or a transient backlog. It gathers four signals before drawing a conclusion:
- Quantify the lag. Read
AuroraReplicaLagfor each reader (normal is under 20 ms, critical is over 100 ms). - Evaluate write pressure on the writer. Look at
WriteIOPS,WriteThroughput, andWriteLatencyto see how hard the writer is working. - Check reader resource utilization. Review
CPUUtilizationandFreeableMemoryper reader to spot an overwhelmed instance. - Find write-heavy queries blocking replication. Identify the top SQL by
db.loadwith write wait events.
Interpreting the findings. The combination of writer pressure and reader health tells the agent which of four situations it is in:
- If the writer’s write volume spiked, a bulk operation or traffic surge is driving the lag.
- If a reader’s CPU is saturated, that reader’s instance class is too small.
- If lag is increasing with no write spike, the reader is working through a backlog and will self-resolve.
- If lag is stable but persistently high, there is a structural mismatch between the write rate and reader capacity.
Branch C: Slow query regression
Investigation procedure. A query regression can come from new code, a changed plan, or simply more volume, so the agent compares the current workload against a known-good baseline:
- Find the heaviest queries. Use Performance Insights to identify the top 10 SQL statements by
db.load(average active sessions). - Compare against the baseline. Check the current top 10 against the same list from 24 hours ago and look for new SQL IDs that have climbed into it.
- Compare execution plans before and after deployment. Look for missing indexes, a changed
JOINorder, or new full table scans. - Check for stale statistics. Compare the last
ANALYZEto the most recent large data change. - Account for volume changes. The same query running 10 times more often is a regression even when each individual execution is unchanged.
Interpreting the findings. Where the change shows up tells the agent what kind of regression it is:
- If new SQL appears in the top 10 after a deployment, it is a code regression. Review the commit.
- If existing SQL degraded, the plan regressed because statistics went stale or the data distribution shifted.
- If query volume increased, a caching layer failed or the traffic pattern changed.
- If the wait events are lock-related, a schema change or a new contention pattern is to blame.
Branch D: Storage pressure
Investigation procedure. Storage problems range from benign capacity growth to a runaway process, so the agent first establishes the shape of the growth and then its source:
- Check the storage trend. Review
FreeLocalStorageover seven days to tell steady linear growth apart from a sudden spike. - Measure table and index bloat. Compare
n_dead_tupton_live_tup. A bloat ratio above 20 percent means vacuum is falling behind. - Check for write amplification. Watch for
VolumeBytesUsedgrowing faster than the logical data being inserted. - Evaluate temporary file usage. Large sorts spilling to disk show up as
temp_blks_writtenper query inpg_stat_statements.
Interpreting the findings. The growth pattern points the agent to the right response, and shows whether this is even an incident:
- If growth is linear and matches the ingestion rate, this is a capacity-planning item, not an incident.
- If storage spiked suddenly, suspect a bulk load, a failed cleanup job, or a runaway process.
- If the bloat ratio is above 20 percent, autovacuum is not keeping up. Tune autovacuum, or use an online reorganization such as
pg_repackon a live cluster. AvoidVACUUM FULLduring an active incident. It takes anACCESS EXCLUSIVElock and rewrites the table, which blocks reads and writes and can extend the outage. - If temporary files are large, the queries need optimization or a higher
work_mem.
Branch E: Parameter drift
Investigation procedure. Parameter drift is insidious because its symptoms mimic other failures, so the agent starts from the change history and works toward the specific setting at fault:
- Pull the recent change history. Query AWS CloudTrail for
ModifyDBClusterParameterGroupevents in the past seven days. - Compare against the recommended baseline. Check current values for connection management, memory allocation, vacuum tuning, and query planning against the expected configuration.
- Isolate misconfigurations from recent changes. For example, a
work_memset too high can cause out-of-memory errors, and anautovacuum_cost_delayset too high lets bloat accumulate. - Check for a default versus a custom parameter group. Reverting to a default group is common after a restore-from-snapshot operation.
Interpreting the findings. Correlating the change history with symptom onset tells the agent whether drift is the root cause and, if so, how it happened:
- If a recent
ModifyDBClusterParameterGrouplines up with when the symptom started, the parameter change is the root cause. - If this cluster’s parameters differ from others in the fleet, it has drifted from the standard configuration.
- If it is on a default parameter group after a restore, the custom tuning was lost during the restore.
- If pending-reboot parameters exist, a partial application is causing inconsistent behavior.
Step 3: Deploy the Skill
AWS DevOps Agent provides multiple deployment methods. For repeatable deployments, we recommend zip upload or GitHub import. For a quick try, use the Operator Web App UI. Choose based on your Skill’s complexity.
Option A: Using zip upload (recommended for skills with reference files)
- Package the directory:
zip -r aurora-pg-performance-degradation.zip aurora-pg-performance-degradation/. - Choose Knowledge, Skills, Add skill, then Upload skill.
- Drag and drop your zip file (must be under 6 MB).
- Select agent types: Incident RCA and On-demand.
- Choose Upload. The system validates the zip, extracts SKILL.md frontmatter, and activates the Skill.
Option B: Import from a GitHub repository
- Push your Skill directory to a GitHub repository connected to your Agent Space.
- Choose Knowledge, Skills, Add skill, then Import from repository.
- Enter the GitHub directory URL (for example,
https://github.com/my-org/devops-skills/tree/main/aurora-pg-performance-degradation). - Select agent types and choose Import skill.
- Choose Sync to pull updates when you modify the Skill in your repository.
Option C: Using the Operator Web App UI (skills without reference files)
- Navigate to the Knowledge page in your Agent Space Operator Web App and choose the Skills tab.
- Choose Add skill, then Create skill.
- Enter the name (
aurora-pg-performance-degradation), description, and instructions from SKILL.md. - Set Status to Active.
- For Agent Type, clear Generic and select Incident RCA and On-demand.
- Choose Create to save the Skill.
Option D: Create with Chat
- Choose Knowledge, Skills, Add skill, then Create skill with Chat.
- Describe your investigation procedure in natural language.
- The agent generates the SKILL.md structure, frontmatter, and instructions for you.
- Review, edit if needed, and save.
Why we chose Incident RCA and On-demand:
Targeting specific agent types helps prevent the Skill from loading when it’s not relevant. Our Skill is designed for deep root cause analysis (Incident RCA) and ad hoc queries (“Why is my Aurora cluster slow?” through On-demand). We deliberately exclude Incident Triage because triage should be fast (under 2 minutes). Our Skill is thorough, not lightweight.
Measuring the impact: 73 percent MTTR reduction
To validate the Skill’s effectiveness, we ran a controlled before and after comparison using three real-world incident scenarios on a test Aurora PostgreSQL cluster.
Methodology
We simulated three distinct failure modes on an Aurora PostgreSQL cluster with realistic production-like workloads:
- Connection pool exhaustion – Introduced connection leak through an application code change.
- Slow query regression – Deployed a new query missing a critical index.
- Parameter drift – Modified
shared_buffersandwork_memto suboptimal values through a parameter group change.
For each incident, we measured: (1) total investigation duration from alert to root cause identification, (2) number of wrong hypotheses explored before finding root cause, and (3) whether the structured output included immediate, short-term, and long-term remediation actions.
Results without the Skill (baseline)
We triggered three incident types and measured how the agent investigated using only its built-in knowledge:
| Incident Type | Duration | Wrong Hypotheses explored | Root Cause Found? |
| Connection pool exhaustion | 48 min | 3 | Yes, eventually |
| Slow query regression | 55 min | 2 | Yes, eventually |
| Parameter drift | 53 min | 4 | Partial – symptoms identified but not traced to parameter change |
Average without Skill: 52 minutes, 3.0 wrong hypotheses per incident
Without domain-specific guidance, the agent explored metrics in an unpredictable order. For connection exhaustion, it checked CPU, then storage, then replication lag before eventually examining connection counts. For parameter drift, it identified symptoms (slow queries, bloat) but never correlated with the AWS CloudTrail ModifyDBClusterParameterGroup event.
Results with the Skill
We triggered the same three incidents and confirmed the agent loaded and followed the decision tree:
| Incident Type | Duration | Decision Tree Followed? | Wrong Hypotheses |
| Connection pool exhaustion | 11 min | Yes – Branch A first (priority order) | 0 |
| Slow query regression | 16 min | Yes – Timeline correlation → Branch C | 0 |
| Parameter drift | 15 min | Yes – CloudTrail correlation → Branch E | 1 |
Average with Skill: 14 minutes, 0.3 wrong hypotheses per incident
Results summary
| Metric | Before Skill | After Skill | Improvement |
| Mean time to resolution (MTTR) | 52 min | 14 min | 73% reduction |
| Wrong hypotheses per incident | 3.0 | 0.3 | 90% fewer dead ends |
| Correct root cause on first branch | 0/3 (0%) | 2/3 (67%) | First-time accuracy |
| Complete structured output | 1/3 | 3/3 | Consistent remediation quality |
| Escalation to senior DBRE required | 2/3 | 0/3 | Self-sufficient investigation |
Why the improvement is so large
The 73 percent reduction comes from two compounding factors:
- Alleviation of wrong hypotheses: Without the Skill, the agent spent 15–20 minutes per wrong hypothesis before pivoting. The decision tree’s priority order (A → B → C → D → E) avoids dead ends by checking upstream causes first.
- Direct routing through signal classification: The Skill’s initial signal table routes the agent to the correct branch immediately when the primary signal is clear. For connection exhaustion, the agent goes straight to Branch A without exploring storage or replication.
The compounding effect: Memories
Your custom Skill works alongside the knowledge AWS DevOps Agent builds about your environment as it works. The agent maintains memories of your topology, code dependencies, pipeline structure and tool-use patterns, which you can view and manage on the Memories tab of the Knowledge page. As this context grows, the agent can apply your Skill’s decision tree with a better understanding of how your Aurora PostgreSQL clusters connect to the rest of your environment. This can help investigations stay fast and accurate as your environment changes.
Best practices for procedural Skills
1. Use decision trees, not linear checklists
Procedural Skills with branching logic deliver measurably better outcomes for complex, multi-fault scenarios. Tell the agent when to follow each branch based on observable signals. A linear checklist forces the agent through every step regardless of relevance.
2. Define the priority order explicitly
“If multiple signals are present, prioritize: A → B → C → D → E.” This single line helps prevent the agent from spending time on downstream symptoms when the root cause is upstream.
3. Include decision points at each branch
After the investigation steps, explicitly state what each finding means and where to go next. Without decision points, the agent might collect data but not draw the correct conclusion.
4. Add metric thresholds in reference files
Don’t embed thresholds in the main SKILL.md. Put them in reference files (for example, references/aurora-pg-metric-thresholds.md). This keeps instructions focused on the investigation workflow while giving the agent concrete numbers to work with. It also makes thresholds straightforward to update as your environment scales.
5. Target specific agent types
This Skill targets “Incident RCA” and “On-demand,” not “Incident Triage.” RCA Skills can be thorough. Triage Skills should be lightweight (under 2 minutes). Targeting reduces context consumption and improves agent focus.
6. Measure before and after
Without baseline MTTR data, you cannot prove the Skill’s value. Track investigation duration and hypothesis count for the specific incident class before deploying the Skill, then run the same scenarios after deployment and compare.
Clean up
To deactivate or remove the Skill:
- Choose Knowledge, then Skills in the Operator Web App.
- Select the
aurora-pg-performance-degradationSkill. - To temporarily disable, set Status to Inactive. This preserves the Skill for future use, and in-progress investigations continue using it.
- To permanently remove, choose the more options menu (⋮), choose Delete, enter the skill name to confirm, and choose Delete Skill.
Consider downloading your Skill as a zip file before deletion as a backup. Deactivating is preferred over deleting if you may need the Skill again.
Conclusion
Custom Skills transform AWS DevOps Agent from a general-purpose operations assistant into a specialist that knows your database infrastructure as well as your best DBRE. By encoding multi-fault decision-tree logic into a procedural Skill, we achieved a 73 percent reduction in MTTR for simulated Aurora PostgreSQL performance incidents, from 52 minutes to 14 minutes.
The key insight: The value of a custom Skill is not in listing what to check (any runbook does that), but in encoding when to check it and what the findings mean. Decision-tree logic that prioritizes investigation branches based on observable signals is what separates a Skill that delivers measurable MTTR reduction from a Skill that’s merely a formatted checklist.
Start with your most common database incident type, document how your best engineer triages it, encode the decision logic into a Skill, and measure the before and after. Your on-call team will thank you, especially at 2 AM.
Resources
- AWS DevOps Agent documentation
- AWS DevOps Agent Skills
- AWS DevOps Agent Pricing
- Agent Skills specification
- Community Skills Gallery
- Amazon Aurora PostgreSQL Performance Insights