AWS Database Blog Official Database Blog of Amazon Web Services Mon, 11 Dec 2017 17:12:47 +0000 en-US hourly 1 Amazon RDS Under the Hood: Multi-AZ Mon, 11 Dec 2017 17:12:47 +0000 0542d603dce635de6974868c426edca77920d85f AWS customers bet their businesses on their data store and highly available access to it. For these customers, Multi-AZ configurations provide an easy-to-use solution for high availability (HA). When you enable Multi-AZ, Amazon RDS maintains a redundant and consistent standby copy of your data. If you encounter problems with the primary copy, Amazon RDS automatically […] <p>AWS customers bet their businesses on their data store and highly available access to it. For these customers, Multi-AZ configurations provide an easy-to-use solution for high availability (HA).</p> <p>When you enable Multi-AZ, <a href="">Amazon RDS</a> maintains a redundant and consistent standby copy of your data. If you encounter problems with the primary copy, Amazon RDS automatically switches to the standby copy to provide continued availability to the data. The two copies are maintained in different Availability Zones (AZs), hence the name “Multi-AZ.” Having separate Availability Zones greatly reduces the likelihood that both copies will concurrently be affected by most types of disturbances. Proper management of the data, simple reconfiguration, and reliable user access to the copies are key to addressing the high availability requirements that customer environments demand.</p> <p>This post describes Amazon RDS Multi-AZ configurations for <a href="">MySQL</a>, <a href="">MariaDB</a>, <a href="">PostgreSQL</a>, and <a href="">Oracle</a> database instances. Amazon RDS for <a href="">SQL Server</a> and Amazon RDS for <a href="">Amazon Aurora</a> use a different technology stack to provide Multi-AZ capabilities.</p> <p><span style="text-decoration: underline"><strong>Basic design</strong></span><br /> The Multi-AZ feature is implemented using a replication layer installed between the database application and the <a href="">Amazon EBS</a> volumes. This layer handles application read and write requests and applies them in an environment where two discrete EBS volume copies are maintained—one accessed locally and one accessed remotely.</p> <p>During normal operation, there are two active <a href="">Amazon EC2</a> instances with the replication layer installed. Each instance manages one EBS volume with a full copy of the data. Configuration binds the two instances and their volumes as a Multi-AZ database instance. The replication layers are in direct communication with each other over a TCP connection.</p> <p>At any moment in time, each instance is assigned a specific role. One is the <em>primary</em>, and it exposes an external endpoint through which users access their data. The other is the <em>standby</em>, and it acts as a secondary instance that synchronously writes all data that it receives from the primary. Database write operations result in the data being properly written to both volumes before a successful response is sent back to the calling application. However, read operations are always performed through the primary EBS volume. Because the database server process is not running on the standby instance, it does not expose an external endpoint. Consequently, its copy of the data is not available to users.</p> <p>To improve availability, Multi-AZ tries to consistently ensure that one of the instances is in the primary role, providing access to its copy of the data. If there is an availability issue, the standby instance can automatically be promoted to the primary role and availability can be restored through redirection. This event is referred to as a <em>failover</em>. The previous primary, if it’s still up and running, is demoted to the standby role.</p> <p>Redirection to the new primary instance is provided through DNS. The relevant records in the results from client DNS queries have very low time-to-live values. It is intended to inhibit long-term caching of the name-to-address information. This causes the client to refresh the information sooner in the failover process, picking up the DNS redirection changes more quickly.</p> <p>The following diagram depicts a Multi-AZ instance that is running in its normal connected state.</p> <p><img class="aligncenter size-full wp-image-2294" src="" alt="" width="900" height="450" /></p> <p style="text-align: center">Figure 1: Multi-AZ instance</p> <p>The database application (<strong>DB APP</strong>, shown in yellow) uses DNS (shown in orange) to obtain the address information for the current external endpoint that is providing access to the data.</p> <p>There are two RDS DB instances in this Multi-AZ instance: the primary instance (shown on the left side, in green) and the standby instance (shown on the right side, in blue). In this example, DNS is directing the application to the primary instance <strong>EC2 #1</strong>, serving the primary copy of the data <strong>EBS #1</strong> that is available in <strong>Availability Zone #1</strong>. The replication layers of the two EC2 instances are connected. Write operations that the application issues also result in writes to the second instance (path shown in gray).</p> <p>Generally, failover events are rare, but they do occur. For situations in which Amazon RDS detects problems, the failover is initiated by automation. You can also manually trigger failover events through the Amazon RDS API.</p> <p>The replication layer has limited visibility above itself and is therefore incapable of making some of the more strategic decisions. For example, it doesn’t know about such things as user connectivity issues, local or regional outages, or the state of its EC2 peer that may have unexpectedly gone silent. For this reason, the two instances are monitored and managed by an external observer that has access to more critical information and periodically queries the instances for status. When appropriate, the observer takes action to ensure that availability and performance requirements are met.</p> <p>The availability and durability improvements provided by Multi-AZ come at a minimal performance cost. In the normal use case, the replication layers are connected, and synchronous write operations to the standby EBS volume occur. The standby instance and volume are in a distinct and geographically distant Availability Zone. Assessment shows increases in database commit latencies of between 2 ms and 5 ms. However, the actual impact on real-world use cases is highly workflow-dependent. Most customer Multi-AZ instances show a minor impact on performance, if any at all.</p> <p>This design enables AWS to provide a Service Level Agreement (SLA) that exceeds 99.95 percent availability to customer data. To learn more, see the <a href="">Amazon RDS Service Level Agreement</a>.</p> <p><span style="text-decoration: underline"><strong>Intricacies of the implementation</strong></span><br /> You might think that the design of a volume replication facility is rather simple and straightforward. However, the actual implementation is fairly complex. This is because it must account for all the predicaments that two networked, discrete instances and volumes might find themselves in, inside a constantly changing and sometimes disrupted environment.</p> <p>Normal ongoing replication assumes that everything is in reasonable working order and is performing well: The EC2 instances are available, regular instance monitoring is functional, the EBS volumes are available, and the network is performing as expected. But what happens when one or more of these pieces is misbehaving? Let’s look at some of the issues that could arise and how they are addressed.</p> <p><strong>Connectivity issues and synchronization</strong><br /> Occasionally the primary and standby instances are not connected to each other, either due to a problem or a deliberate administrative action. Ongoing replication is not possible, and waiting a long time for connectivity to be restored is not acceptable. When connectivity is lost or deliberately discontinued, the instances momentarily pause, waiting for a decision to be made by the observer. When the observer detects this condition, it directs an available instance to assume the primary role and to proceed on its own without replication. There is now only one current copy of the data, and the other copy is becoming increasingly out of date.</p> <p>Connectivity issues are usually investigated, and the problem is often quickly corrected. If the issue persists beyond a minimum amount of time, it triggers an attention for operator intervention. It is therefore expected that the majority of connectivity issues will be relatively short-lived conditions, and the two instances will soon have connectivity restored. When connectivity is restored, the volumes must be resynchronized before returning to the normal, ongoing replication state.</p> <p>The resynchronization process ensures that both copies of the data are restored to a consistent state. In an effort to reduce the time needed for resynchronization, the primary keeps track of blocks that are modified while the two instances are disconnected. When resynchronizing occurs, only those modifications need to be sent from the primary instance to the standby instance, which speeds up the process.</p> <p><strong>Fault tolerance in a dynamic environment</strong><br /> AWS is a large-scale, highly dynamic environment, and Amazon RDS Multi-AZ is designed to step in and take action when software and hardware disruptions occur.</p> <p>In the event of a disruption, instance or volume availability problems are the most usual case, and they are predominantly resolved by performing a simple failover operation. This restores availability through the standby instance and volume.</p> <p>In the unlikely event that a volume experiences a failure, it is replaced with a new one. The process of replacement begins with securing a snapshot of the surviving volume. This is mainly for durability reasons, but it also helps improve the performance of the subsequent resynchronization of the volumes. The instance is then connected to the new volume and the volume is hydrated from the snapshot. Upon completion, the volumes are resynchronized and replication is restored.</p> <p>Instance or volume replacement might also be an option in situations where a component exhibits behaviors outside the norm. For example, a substantial or prolonged increase in latency or reduction in bandwidth can indicate an issue with the location of the path to the resource. Replacement is expected to be a permanent solution in such situations. Note that a replacement can impact performance, so it is only performed when necessary.</p> <p>There could be situations in which an entire AWS Region or Availability Zone is affected—for example, during extreme weather or a widespread power outage. During these situations, special attention is given to ensure that Multi-AZ instances remain available. Care must be taken not to escalate a problem into a more serious situation. The observer uses Region availability information to pause unnecessary automated recovery actions while the underlying issue gets resolved.</p> <p><span style="text-decoration: underline"><strong>Conclusion</strong></span><br /> Amazon RDS Multi-AZ configurations improve the availability and durability of customer data. With automated monitoring for problem detection and subsequent corrective action to restore availability in the event of a disruption, Multi-AZ ensures that your data remains intact. For more information, see <a href="">Amazon RDS High Availability</a>.</p> <hr /> <h3>About the Author</h3> <p><strong><img class="size-full wp-image-2298 alignleft" src="" alt="" width="108" height="143" />John Gemignani is a principal software engineer in RDS at Amazon Web Services.</strong></p> Part 2 – Role of the DBA When Moving to Amazon RDS: Automation Mon, 04 Dec 2017 18:41:02 +0000 027886eaff058f7a918de6e029dd49934cabe2ba In Part 1 of this blog series, I talked about how Amazon Relational Database Service (Amazon RDS) can help change the focus of your role as a database administrator (DBA) from routine, time-consuming tasks to project work that helps the business move faster. In this post, I discuss how you can push that advantage one step further and use AWS tools to do more through automation. An important aspect of being an effective DBA when your business is running at top speed is using code and automation whenever you can. AWS provides tools for you to make this easier. <p>In <a href="">Part 1 of this blog series</a>, I talked about how <a href="">Amazon Relational Database Service (Amazon RDS)</a> can help change the focus of your role as a database administrator (DBA) from routine, time-consuming tasks to project work that helps the business move faster. Spending more time focused on controlling access to the database, helping application teams draft and apply changes to database structures, and performing reactive and proactive performance tuning are important tasks that more directly contribute to the business bottom line.</p> <p><span style="text-decoration: underline"><strong>Automation tips</strong></span><br /> In this post, I discuss how you can push that advantage one step further and use AWS tools to do more through automation. An important aspect of being an effective DBA when your business is running at top speed is using code and automation whenever you can. AWS provides tools for you to make this easier.</p> <p>Many of the examples in this post use the&nbsp;<a href="">AWS Command Line Interface</a>&nbsp;(AWS CLI) to work with Amazon RDS. The AWS CLI is a unified tool to manage your&nbsp;AWS services. With just one tool to download and configure, you can control multiple&nbsp;AWS services from the&nbsp;command line&nbsp;and automate them through scripts. All the code examples used in this post are available to download from this <a href="">GitHub repository</a>.</p> <p>If you’d like to skip ahead and see the code in action, go to <a href="#case-study">Case Study: Amazon RDS accelerates unit testing</a>.</p> <p><strong>Database creation</strong><br /> The basic building block of Amazon RDS is the DB instance. Your Amazon RDS DB instance is similar to your on-premises Microsoft SQL Server. After you create your SQL Server DB instance, you can add one or more custom databases to it. For a guided walkthrough on creating and connecting to a SQL Server database, you can follow the example in the <a href="">Amazon RDS documentation</a>.</p> <p>In the following example command, you create a DB instance running the SQL Server database engine SE named&nbsp;sqltest with a&nbsp;db.m4.large instance class, 500 GB of storage, a standby database in another Availability Zone, and automated backups enabled with a retention period of <em>seven</em> days.</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-bash">aws rds create-db-instance --db-instance-identifier &quot;&lt;your-instance-identifier&gt;&quot; \ --allocated-storage 500 --db-instance-class db.m4.large \ --engine sqlserver-se --engine-version &quot;12.00.4422.0.v1&quot; \ --master-username &quot;&lt;your-user&gt;&quot; --master-user-password &quot;&lt;your-password&gt;&quot; \ --backup-retention-period 7 --license-model license-included </code></pre> </div> <p><span id="more-2202"></span></p> <p><em><strong>Stopping and starting your database</strong></em><br /> If you use a DB instance intermittently, and you want to resume where you last left your data, you can stop your Amazon RDS instance temporarily to save money. If you have development or test instances that are used only during working hours, you can shut them down overnight and then start them up when you return to the office. You can automate this process with a little bit of orchestration and a few calls to the API. For additional details, review the <a href="">Stop Instance</a> documentation.</p> <p><strong>Parameter groups</strong><br /> You manage your DB engine configuration through the use of parameters in a DB parameter group. DB parameter groups act as a container for engine configuration values that are applied to one or more DB instances. When you have a custom parameter group, it is easy to associate it with an instance by specifying it when you create your instance. A default DB parameter group is created if you create a DB instance without specifying a customer-created DB parameter group.</p> <p><strong>Option groups</strong><br /> Some DB engines offer additional features that make it easier to manage data and databases, and to provide additional security for your database. Amazon RDS uses <em>option groups</em> to enable and configure these features. An&nbsp;option group&nbsp;can specify features, called <em>options</em>, that are available for a particular Amazon RDS DB instance. Options can have settings that specify how the option works. When you associate a DB instance with an option group, the specified options and option settings are enabled for that DB instance.</p> <p>The following command example uses the AWS CLI to modify an existing database <strong>sqltest</strong> to use a new option group <strong>sqlserver-se-12-native-backup </strong>and applies the change immediately. The option group I used in the example is one that uses native backup and restore to Amazon S3 within Amazon RDS for SQL Server. You can read more about the option in the blog post <a href="">Amazon RDS for SQL Server Support for Native Backup/Restore to Amazon S3</a>.</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-bash">aws rds modify-db-instance --db-instance-identifier “&lt;your-instance-identifier&gt;“ \ --option-group-name “sqlserver-se-12-native-backup” \ --apply-immediately</code></pre> </div> <p><strong>Monitoring</strong><br /> Monitoring is an important part of maintaining the reliability, availability, and performance of Amazon RDS and your AWS solutions. Amazon RDS provides metrics in real time to <a href="">Amazon CloudWatch</a>. Amazon CloudWatch is a monitoring service for AWS Cloud resources and the applications you run on AWS. Standard CloudWatch monitoring includes built-in metrics for your DB instance that are visible using the console, AWS CLI, or API for no additional charge.</p> <p>AWS also gives you the option to enable <a href="">Enhanced Monitoring</a> for your Amazon RDS instance. Enhanced Monitoring provides additional metrics, increased granularity, and per-process information. The option is available when you create or modify your instance, and it does not require a restart. When you enable Enhanced Monitoring for your instance, you can choose the granularity, and you specify an IAM role that is authorized to collect the information. The enhanced metrics are published to CloudWatch Logs under a log group named <em><strong>RDSOSMetrics</strong></em><em>, and they are also available in the Amazon RDS monitoring section of the AWS Management Console</em>. The enhanced monitoring data contains numerous metrics that you can use to review data that is valuable for performance monitoring.</p> <p><img class="aligncenter size-full wp-image-2205" src="" alt="" width="1371" height="450" /></p> <p>For the metrics and alerts that follow in this post, I focus on standard monitoring in Amazon CloudWatch.</p> <p><em><strong>Metrics</strong></em><br /> Amazon RDS sends CloudWatch data on hundreds of metrics. For the purposes of this blog, I mention just a few of them (<code>WriteThroughput</code>, <code>WriteLatency</code>, and <code>ReadLatency</code>). For a complete list of metrics available for monitoring, open the <em>CloudWatch</em> console and choose <strong>Metrics</strong>, <strong>RDS</strong>.</p> <p><img class="aligncenter size-full wp-image-2206" src="" alt="" width="1431" height="490" /></p> <p>You can also watch the performance of your database over time through the CloudWatch console. It can help you spot trends and understand the performance baseline of your database.</p> <p><img class="aligncenter size-full wp-image-2207" src="" alt="" width="1527" height="276" /></p> <p><em><strong>Alarms: preparation</strong></em><br /> You can create a CloudWatch alarm that sends an <a href="">Amazon Simple Notification Service (SNS)</a> message when the alarm changes state. An alarm watches a single metric over a time period that you specify. It performs one or more actions based on the value of the metric relative to a given threshold over a number of time periods. The action is a notification sent to an Amazon SNS topic or Auto Scaling policy.</p> <p>This means that before you create an alarm, you should create a topic and subscribe to it using your email address. The following command example creates an SNS topic.</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-bash">aws sns create-topic --name “app-dba-notification”</code></pre> </div> <p>The topic <a href="">Amazon Resource Name (ARN)</a> is returned as the output of the command.</p> <p><img class="aligncenter size-full wp-image-2208" src="" alt="" width="1834" height="156" /></p> <p>You can use the output ARN and any valid email address to subscribe to your new SNS topic.</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-bash">aws sns subscribe --topic-arn &lt;your-arn-here&gt; \ --protocol email --notification-endpoint &lt;your-email&gt; </code></pre> </div> <p>The subscription must be confirmed before messages can be sent to your email address. In your email application, open the message from AWS Notifications and confirm your subscription. Your web browser displays a confirmation response from Amazon Simple Notification Service.</p> <p><img class="aligncenter size-full wp-image-2209" src="" alt="" width="1898" height="504" /></p> <p><em><strong>Alarms: creating</strong></em><br /> Alarms invoke actions for sustained state changes only. CloudWatch alarms do not invoke actions simply because they are in a particular state. The state must have changed and been maintained for a specified number of periods. You can use the <a href="">AWS CLI</a> to create an alarm for the metric <code>WriteThroughput</code> measuring over a period of five minutes (300 seconds) for two periods when the value meets or exceeds 48,000.</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-bash">aws cloudwatch put-metric-alarm --alarm-name &quot;Write throughput&quot; \ --alarm-description &quot;write-throughput&quot; --actions-enabled \ --alarm-actions &quot;&lt;your-arn&gt;&quot;\ --metric-name &quot;WriteThroughput&quot; --namespace &quot;AWS/RDS&quot; --statistic &quot;Maximum&quot; \ --dimensions &quot;Name=DBInstanceIdentifier,Value=&lt;your-instance-identifier&gt;&quot; \ --period 300 --evaluation-periods 2 --threshold 48000 \ --comparison-operator GreaterThanOrEqualToThreshold \ --treat-missing-data notBreaching</code></pre> </div> <p>Now when your database write throughput exceeds your threshold value for a period of 10 minutes, CloudWatch sends a message to your SNS topic, and you receive an email. When you navigate to the console via the link in the email, you can clearly see which alarms need attention.</p> <p><img class="aligncenter size-full wp-image-2210" src="" alt="" width="919" height="193" /></p> <p>Your alert is also clearly visible on your instance in the Amazon RDS console.</p> <p><img class="aligncenter size-full wp-image-2212" src="" alt="" width="1264" height="263" /></p> <p>To read more about monitoring options and best practices for monitoring Amazon RDS, see <a href="">Monitoring Amazon RDS</a>.</p> <p><strong>Backup and restore</strong><br /> You can back up and restore DB instances using automated or manual snapshots in Amazon RDS. You can restore to any point in time during your backup retention period, or share a copy of your database with another AWS Region or another account.</p> <p><em><strong>Backing up a database</strong></em><br /> Amazon RDS creates automated backups of your DB instance during the backup window of your DB instance. Amazon RDS saves the automated backups of your DB instance according to the backup retention period that you specify. If necessary, you can recover your database to any point in time during the backup retention period.</p> <p>You can also back up your DB instance manually by creating a DB snapshot. In this example, I use the snapshot to restore the database to the same AWS Region and account. Amazon RDS also allows you to copy snapshots to other Regions and share with other accounts. The flexibility to copy and share snapshots from one AWS Region to another, or with another account, makes it easy to deploy new database copies.</p> <p>When you create a DB snapshot, you need to identify which DB instance you are going to back up. Then give your DB snapshot a name so that you can restore from it later.</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-bash">aws rds create-db-snapshot --db-snapshot-identifier &quot;&lt;your-snapshot-name&gt;&quot; \ --db-instance-identifier “&lt;your-instance-identifier&gt;” </code></pre> </div> <p><em><strong>Restoring a database</strong></em><br /> Amazon RDS creates a storage volume snapshot of your DB instance, backing up the entire DB instance and not just individual databases. You can create a DB instance by restoring from this DB snapshot. When you restore the DB instance, you provide the name of the DB snapshot to restore from. Then provide a name for the new DB instance that is created from the restore. You can use one of the automated snapshots to restore your database, or you can use a manual snapshot that you have taken.</p> <p><em><strong>Restoring a database to a point in time</strong></em><br /> Using the automated snapshots, you can restore to any point in time during your backup retention period. To determine the latest restorable time for a DB instance, use the AWS CLI&nbsp;<a href="">describe-db-instances</a>&nbsp;command and look at the value returned in the&nbsp;<code>LatestRestorableTime</code>&nbsp;field for the DB instance.</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-bash">aws rds describe-db-instances --db-instance-identifier “&lt;your-instance-identifier&gt;”</code></pre> </div> <p>The latest restorable time for a DB instance is typically within five minutes of the current time.</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-bash">aws rds restore-db-instance-to-point-in-time \ --source-db-instance-identifier “&lt;your-instance-identifier&gt;” \ --target-db-instance-identifier “&lt;your-target-identifier&gt;” \ --restore-time 2017-10-04T00:00:00.000Z</code></pre> </div> <p>The example command restores the database <strong>sqltest</strong> to a new database <strong>sqltest-copy</strong> at the restore time of <strong>October 4, 2017, midnight UTC</strong>. I can then point my application at the new database. For more information about point-in-time recovery of an RDS database, see <a href="">Restoring a DB Instance to a Specified Time</a>.</p> <p>You can also read more about <a href="">Backing Up and Restoring Amazon RDS DB Instances</a> in the Amazon RDS documentation.</p> <p><strong>Orchestration</strong><br /> The AWS CLI is a great way to quickly provision, change, or remove resources. You can use it to create collections of resources, but you have to manage the dependencies between the resources yourself. <a href="">AWS CloudFormation</a> is a service that gives developers and businesses an easy way to create a collection of related AWS resources and provision them in an orderly and predictable fashion.</p> <p>AWS CloudFormation automates and simplifies the task of repeatedly and predictably creating groups of related resources that power your applications. An important advantage of AWS CloudFormation is that it allows you to&nbsp;automate&nbsp;service provisioning steps in a fairly simple way. There is no extra charge for AWS CloudFormation; you pay only for the AWS resources that you launch.</p> <p>The following image shows an excerpt from an AWS CloudFormation template that is used for the case study in the next section. The template restores an Amazon RDS database from a snapshot and creates five alarms for that new instance. The expectation for this template is that all resources are successfully created, or they are all rolled back. The full template is available in the <a href="">code repository</a> for this blog post.</p> <div id="case-study"></div> <p><img class="aligncenter size-full wp-image-2213" src="" alt="" width="1692" height="574" /></p> <p><span style="text-decoration: underline"><strong>Case study: Amazon RDS accelerates unit testing</strong></span><br /> Now that we’ve reviewed all the ways that you can add automation to your administration tasks, let’s look at how you can put these concepts to use to solve a complex testing problem.</p> <p>If you have products that run against customer databases, testing new application code changes against various database configurations in a timely fashion can be a challenge. The following matrix shows just two configurations of SQL Server against six machine types. One customer was doing this type of testing for multiple database engines, and multiple parameter configurations.</p> <p><img class="aligncenter size-full wp-image-2214" src="" alt="" width="1430" height="167" /></p> <p>Before migrating to Amazon RDS, the customer used physical servers and performed configuration changes manually for each test. The customer further managed data consistency by using the engine native backup and restore.</p> <p>As the size of the test data bloomed to just under 400 GB, the restore process was taking longer and longer, and the management time was increasing. With once-a-week testing, the DBA estimated that the time to install software, configure parameters, and manage the restorations for the SQL Server test fleet was conservatively taking an average of three to five hours a week.</p> <p>When the customer moved to Amazon RDS, managing the testing process became a much simpler operation. The DBA did three things in preparation for automated testing:</p> <ol> <li>Created several parameter groups in Amazon RDS for the various SQL Server parameter configurations that they commonly test.</li> <li>Created a base Amazon RDS snapshot with a large set of pristine test data so that tests run on consistent data each time no matter the configuration.</li> <li>Drafted an AWS CloudFormation template that would restore the snapshot to a new database with three important CloudWatch alarms.</li> </ol> <p><img class="aligncenter size-full wp-image-2215" src="" alt="" width="1359" height="650" /></p> <p>By setting up these few key pieces now, you can launch all tests whenever new code hits the test systems.</p> <p>For each configuration in the text matrix, the following steps occur.</p> <ul> <li>The stack is launched.</li> <li>Application tests are run.</li> <li>Outcomes are recorded, data changes are captured, and any potential alerts are triggered.</li> <li>The stack is deleted.</li> </ul> <p>For the SQL Server test matrix as defined, 12 stacks are created and destroyed each test cycle, and they are all managed by an automated code pipeline.</p> <p>Now that the process is fully automated, the DBA spends less than five minutes a month adding or changing new parameter groups. Those three to five hours a week are now spent helping the application teams increase their SQL code efficiency. The more efficient code has helped them certify smaller CPU and memory footprints for their customers.</p> <p><strong>Note:</strong> For more information about <a href="">SQL Server Audit</a> or <a href="">c2 audit mode Server Configuration</a>, consult your SQL Server documentation. For more information about automation topics, a great place to start is the <a href="">AWS DevOps Blog</a>.</p> <p><span style="text-decoration: underline"><strong>Summary</strong></span><br /> With Amazon RDS, DBAs can focus less on routine, time-consuming tasks, and spend more time on the tasks that directly contribute to the business bottom line. And you can take it a step further by using code to automate any remaining regular tasks. AWS tools like the AWS CLI make managing services and automating everyday tasks even easier, leaving you more time to help move the business forward.</p> <hr /> <h3>About the Author</h3> <p><strong><img class="size-full wp-image-1883 alignleft" src="" alt="" width="120" height="120" />Wendy Neu has worked as a Data Architect with Amazon since January 2015.</strong> Prior to joining Amazon, she worked as a consultant in Cincinnati, OH helping customers integrate and manage their data from different unrelated data sources.</p> Migrate Delimited Files from Amazon S3 to an Amazon DynamoDB NoSQL Table Using AWS Database Migration Service and AWS CloudFormation Mon, 27 Nov 2017 17:25:55 +0000 13b22e03917ddb829f12190e02a60976f8228665 Introduction Recently, AWS Database Migration Services (AWS DMS) added support for using Amazon S3 as a source for your database migration. This new support means that you can now load data in comma-separated value (CSV) format from S3 into any supported target, whether or not the target has native S3 support. In most cases, when […] <p><span style="text-decoration: underline"><strong>Introduction</strong></span><br /> Recently, <a href="">AWS Database Migration Services (AWS DMS)</a> added support for using <a href="">Amazon S3</a> as a source for your database migration. This new support means that you can now load data in comma-separated value (CSV) format from S3 into any supported target, whether or not the target has native S3 support.</p> <p>In most cases, when you are migrating to a new database, you have access to your source database and can use the database directly as a source. Sometimes, however, you might not have access to the source directly. In other cases, the source is really old, or possibly unsupported. In these cases, if you can export the data in CSV format, you can still migrate, or replatform, your data.</p> <p>In this blog post, we show you how to use Amazon S3 as a source and push a file with 4 million rows of data into <a href="">Amazon DynamoDB</a> using AWS DMS. We use <a href="">AWS CloudFormation</a> to create all of the resources, and initiate the task. You can find all of the code samples used today in <a href="">this repository</a>.</p> <p><span style="text-decoration: underline"><strong>Prerequisites</strong></span><br /> Before you begin working with DMS, you need at least two <a href="">AWS Identity and Access Management (IAM)</a> service roles with sufficient permissions to access the resources in your account. These are the <code>dms-cloudwatch-logs-role</code> for pushing logs to Amazon CloudWatch and the <code>dms-vpc-role</code> for use by the DMS service. For more information on these, see <a href="">Creating the IAM Roles to Use with the AWS CLI and AWS DMS API</a> in the DMS documentation.</p> <p>When you are working with S3 as a source, check that the S3 bucket is in the same AWS Region as the replication instance you are using to migrate. In addition, the AWS account you are using for the migration must have read access to the source bucket. Finally, the role assigned to the user account creating the migration task must have S3 permissions for <code>GetObject</code> and <code>ListBucket</code>.</p> <p>When working with a DynamoDB database as a target for AWS DMS, make sure that your IAM role allows AWS DMS to assume and grant access to the DynamoDB tables that are being migrated into. If you are using a separate role, make sure that you allow the DMS service to perform <code>AssumeRole</code>. The user account creating the migration task must be able to perform the DynamoDB actions <code>PutItem</code>, <code>CreateTable</code>, <code>DescribeTable</code>, <code>DeleteTable</code>, <code>DeleteItem</code>, and <code>ListTables</code>.</p> <p>For the examples in this blog post, we use a role with the S3, DynamoDB, and DMS permissions listed preceding, as shown in the following image.<br /> <img class="aligncenter size-full wp-image-2147" src="" alt="" width="1429" height="629" /></p> <p><span style="text-decoration: underline"><strong>Test data</strong></span><br /> You can work through the examples in this post easily using test data readily available on the web. IMDB datasets are available for noncommercial or personal use from a requester pays S3 bucket. When you use your account and access keys to download a file from this bucket, your account is charged for the data transfer and request costs.</p> <p>As you work through all of the examples in this post, you can use the IMDB file title.basics.tsv.gz downloaded from <code>imdb-datasets/documents/v1/current/</code>. The repository includes a short shell script that downloads the file and pushes it to the S3 bucket of your choice. Simply change the <code>BUCKETKEY</code> variable to your bucket and the <code>AWSPROFILE</code> variable to your profile name (possibly <strong>default</strong>). As an alternative, modify the code samples in the repository to work with files of your choice. The following image shows the shell script that downloads the file from IMDB.<br /> <img class="aligncenter size-full wp-image-2148" src="" alt="" width="1239" height="493" /></p> <p>For the examples in this blog post, you work with one of the IMDB files, <strong>title.basics.csv</strong>. The file contains about 4 million rows and is roughly 367.6 MB. You load it directly from your S3 bucket into a target DynamoDB table, which you can precreate. By precreating the table, you can provision higher than the default write throughput for faster loading of the data.</p> <p><span id="more-2144"></span></p> <p><span style="text-decoration: underline"><strong>S3 as a source</strong></span><br /> To use S3 as a source for DMS, the source data files must be in CSV format. To load the file title.basics.csv from your S3 bucket, you need to provide a few things to DMS. These are a JSON mapping for the table, the bucket name, and a role with sufficient permissions to access that bucket.</p> <p><strong>JSON mapping</strong><br /> In the table mapping JSON file, for the first property, you need to tell DMS how many tables you are loading. In this example, you load only one table, so enter the value of 1.</p> <p>The second property is <code>Tables</code>, and it contains an array of table definitions that matches that counter. To load your single table title_basics, tell DMS your table name, the path to the table from the S3 bucket, the name of your table owner, and how many columns in the table. You also provide a description of those columns. The following figure shows the sample JSON with the <code>TableColumns</code> property collapsed.<br /> <img class="aligncenter size-full wp-image-2151" src="" alt="" width="1432" height="330" /></p> <p>In the <code>TableColumns</code> property, you define each of the columns that correspond to your .csv file. In the example, there are nine string type columns of varying length that make up the file and ultimately the table. There are common properties among the columns that include name, type, and length.</p> <p>For the column that you want to use as the partition key, you specify two additional properties indicating that the column required is nullable (<code>ColumnNullable</code>) and that it serves as the partition key (<code>ColumnIsPK</code>). The following screenshot shows title_basics column definitions.<br /> <img class="aligncenter size-full wp-image-2152" src="" alt="" width="1670" height="440" /></p> <p><strong>Creating the S3 endpoint by using the console</strong><br /> If you are going to configure your S3 endpoint from the console, navigate to <strong>Database Migration Services</strong>, <strong>Endpoints</strong>, and choose the blue <strong>Create endpoint</strong> button. You can then enter the following information:</p> <ul> <li>The source</li> <li>A name for your endpoint</li> <li>S3 as the source engine</li> <li>The Amazon Resource Name (ARN) of the service role with access to the S3 bucket</li> <li>The S3 bucket name</li> <li>The JSON table structure that you defined</li> </ul> <p>The following shows a completed console form example.<br /> <img class="aligncenter size-full wp-image-2153" src="" alt="" width="1431" height="650" /></p> <p>The title.basics.csv file is actually a tab-delimited file. For DMS to interpret it correctly, you need to open the <strong>Advanced</strong> property section and enter the extra connection attributes value of <code>csvDelimiter=/t</code> into the box. For more information about advanced properties, see <a href="">Extra Connection Attributes for S3 as a Source for AWS DMS</a> in the DMS documentation. The extra connection attributes box is shown following.<br /> <img class="aligncenter size-full wp-image-2154" src="" alt="" width="1430" height="142" /></p> <p><strong>Creating the S3 endpoint by using a template</strong><br /> To use CloudFormation to create the same resource that we just created from the console preceding, you enter the information into a <code>Resource</code> section of your template using the type <code>AWS::DMS::Endpoint</code>. In the template, you don’t have to include a name for your endpoint. Instead, you can have it inherit the name from the stack resource label. You can read more about CloudFormation in <a href="">Getting Started with AWS CloudFormation: Learn Template Basics</a> in the CloudFormation documentation.</p> <p>In your template, you have to provide all of the other information that is required in the console, including the table definition, service role ARN, bucket name, and CSV delimiter. You enter the table definition as escaped JSON in a single line. The following screenshot shows a CloudFormation template with an S3 endpoint.<br /> <img class="aligncenter size-full wp-image-2155" src="" alt="" width="1730" height="666" /></p> <p><span style="text-decoration: underline"><strong>DynamoDB as a target</strong></span><br /> When AWS DMS creates tables on an Amazon DynamoDB target endpoint, it sets several Amazon DynamoDB parameter values. The cost for the table creation depends on the amount of data and the number of tables to be migrated.</p> <p>When AWS DMS sets Amazon DynamoDB parameter values for a migration task, the default value for the Read Capacity Units (RCU) parameter is set to 200.</p> <p>The Write Capacity Units (WCU) parameter value is also set, but its value depends on several other settings. The default value for the WCU parameter is 200. To read more about how DMS calculates WCU and how to influence it, see <a href="">Using an Amazon DynamoDB Database as a Target for AWS Database Migration Service</a> in the DMS documentation.</p> <p><strong>Creating the DynamoDB endpoint by using the console</strong><br /> If you are going to configure your S3 endpoint from the console, navigate to <strong>Database Migration Services</strong>, <strong>Endpoints</strong>, and choose the blue <strong>Create endpoint</strong> button. You can then enter the following information:</p> <ul> <li>A name for your endpoint</li> <li>The target</li> <li>DynamoDB as the target engine</li> <li>The ARN of the service role with permissions for DynamoDB</li> </ul> <p>The following shows a completed console form example.<br /> <img class="aligncenter size-full wp-image-2156" src="" alt="" width="1432" height="296" /></p> <p><strong>Creating the DynamoDB endpoint by using a template</strong><br /> Just as with the S3 endpoint example, to use CloudFormation you enter the information into another <code>Resource</code> section of your template using the type <code>AWS::DMS::Endpoint</code>. Your endpoint name inherits its name from the stack resource label.</p> <p>You need to provide the service role ARN, the engine name as <code>DYNAMODB</code>, and the endpoint type of <code>target</code>. An example is shown following.<br /> <img class="aligncenter size-full wp-image-2157" src="" alt="" width="1736" height="518" /></p> <p><span style="text-decoration: underline"><strong>CloudFormation orchestration</strong></span><br /> AWS CloudFormation makes deploying a set of Amazon Web Services (AWS) resources as simple as submitting a template. A&nbsp;<em>template</em>&nbsp;is a simple text file that describes a&nbsp;<strong><em>stack</em></strong>, a collection of AWS resources that you want to deploy together as a group. You use the template to define all the AWS resources you want in your stack.</p> <p>For this example, in addition to the endpoints that you created, you also need a replication instance and a migration task. CloudFormation takes the template and creates all of the resources specified in the right order and starts the migration task.</p> <p>In its simplest form, after you’ve completed the prerequisites and uploaded your files, a migration is essentially four steps:</p> <ol> <li>Create a replication instance</li> <li>Create a source endpoint</li> <li>Create a target endpoint</li> <li>Create and run a replication task</li> </ol> <p>These four pieces are what your CloudFormation template contains in the Resources section:</p> <ol> <li>DMSReplicationInstance</li> <li>DMSEndpointS3</li> <li>DMSEndpointDynamoDB</li> <li>DMSTaskMigration</li> </ol> <p>The following screenshot shows the CloudFormation template structure.<br /> <img class="aligncenter size-full wp-image-2158" src="" alt="" width="1724" height="320" /></p> <p><strong>Creating a replication instance by using a template</strong><br /> To specify a replication instance resource in a CloudFormation template, use the type <code>AWS::DMS::ReplicationInstance</code>. You specify how much allocated storage that you want the replication instance to have, how big you want the instance to be, and whether you want the instance to be publicly accessible. An example is shown following.<br /> <img class="aligncenter size-full wp-image-2159" src="" alt="" width="1678" height="558" /></p> <p>You also need to indicate the security group that you want the replication instance to use. For more information, see <a href="">Setting Up a Network for Database Migration</a> in the DMS documentation.</p> <p><strong>Creating a replication task by using a template</strong><br /> The migration task is where all of your work is done. The task takes the replication instance and the two endpoints and uses those resources to tell DMS how to move your data and what settings to use.</p> <p>In the following image, the <code>replicationInstanceArn</code>, <code>SourceEndpointArn</code>, and <code>TargetEndpointArn</code> have <code>Ref:</code> values that point to the other <code>Resource</code> entries within the template. CloudFormation detects these names and when the resources are instantiated, uses the resulting ARN values to build the DMS task.<br /> <img class="aligncenter size-full wp-image-2160" src="" alt="" width="1754" height="614" /></p> <p>The replication task settings, and the table mappings, are pulled in by using an escaped JSON string. You can find both the formatted raw JSON strings and the escaped versions in the repository with the rest of the code samples for you to review. There’s also a simple Python script that takes the formatted JSON files and transform them into escaped one-liners for pulling into CloudFormation templates. If you want to make changes to the JSON settings, you might find it easier to change them in the formatted file and then transform them.</p> <p><strong>Working with parameters and metadata by using a template</strong><br /> Let’s add just a quick word about parameters used in the CloudFormation template example. Both the parameter section and the metadata section are optional in a CloudFormation template. However, parameters can help significantly simplify items that commonly change in a template. Metadata can simplify how a template is visually laid out on the screen after upload.</p> <p>CloudFormation template parameters are used in this example to identify the values that change when you run this template in your account. Items like the security group for the replication instance, the S3 bucket name, and the service role ARN to use for accessing resources change, because they are specific to your account. Additionally, providing a list of replication instance sizes and a default value helps make it easier to see at a glance what choices are available.</p> <p>An example of CloudFormation parameters is shown following.<br /> <img class="aligncenter size-full wp-image-2161" src="" alt="" width="1858" height="796" /></p> <p>CloudFormation metadata information is used in this template to help organize the information on the console screen for your user. An example is shown following.<br /> <img class="aligncenter size-full wp-image-2162" src="" alt="" width="1696" height="576" /></p> <p><strong>Creating a stack</strong><br /> Creating a stack from your CloudFormation template is as easy as uploading your template to S3 through the console, filling in a few values, and then choosing <strong>Create</strong>.</p> <p>Start by opening the CloudFormation console and choosing the blue <strong>Create stack</strong> button. Next, choose <strong>Upload a template to S3</strong>, and choose the file <code>cloudformation-dms-migration-s3-dynamodb.yaml</code> from your local drive. Your template loads on the screen and looks similar to the following image.<br /> <img class="aligncenter size-full wp-image-2163" src="" alt="" width="1431" height="637" /></p> <p>When you’ve completed the form with the proper values for instance size, security group, S3 bucket name, and service role ARN, you can then choose <strong>Next</strong>, <strong>Next</strong>, and <strong>Create</strong> to launch your stack. The stack instantiates your resources, starts your task, and migrates your data. Your stack indicates when all resources have been launched both overall and in the <strong>Events</strong> panel, shown following.<br /> <img class="aligncenter size-full wp-image-2164" src="" alt="" width="1428" height="300" /></p> <p>Once your stack indicates that creation is complete, you can observe your task execution either in the DMS console or the CloudWatch console. The DMS console is shown following.<br /> <img class="aligncenter size-full wp-image-2165" src="" alt="" width="1430" height="212" /></p> <p>A completed load in the CloudWatch logs for the DMS task shows that more than 4 million rows were successfully loaded. The CloudWatch task completion message is shown following.<br /> <img class="aligncenter size-full wp-image-2166" src="" alt="" width="1430" height="128" /></p> <p><span style="text-decoration: underline"><strong>Conclusion</strong></span><br /> Now that you have a CloudFormation template for loading data using DMS from S3, you can extend this example to load your own data. Add multiple tables to your <code>ExternalTableDefinition</code> and let DMS load multiple tables at one time. Change the target endpoint from DynamoDB to <a href="">Amazon Aurora with PostgreSQL compatibility</a>, or to <a href="">Amazon Redshift</a> or another DMS target type, simply by swapping out the resources in your template. Using S3 as a source for DMS, you can get delimited data from just about anywhere and push quickly it to any number of target engines.</p> <hr /> <h3>About the Author</h3> <p><strong><img class="size-full wp-image-1883 alignleft" src="" alt="" width="120" height="120" />Wendy Neu has worked as a Data Architect with Amazon since January 2015.</strong> Prior to joining Amazon, she worked as a consultant in Cincinnati, OH helping customers integrate and manage their data from different unrelated data sources.</p> New, Memory-Optimized Amazon EC2 Instance Types Drive Database Workloads Wed, 22 Nov 2017 17:36:44 +0000 2428bbbfa6596ec660eb40319ab79c9a5f753180 Perfectly sized instances for maximizing Microsoft SQL Server Standard In September, Amazon Web Services announced availability of the new Amazon EC2 x1e.32xlarge instance type with 128 vCPU and 3,904 GiB of memory. Since that announcement, we have heard from customers that they want more instance configuration choices with fewer vCPUs, while maintaining a high ratio […] <p><strong>Perfectly sized instances for maximizing Microsoft SQL Server Standard</strong></p> <p>In September, Amazon Web Services announced availability of the new <a href="">Amazon EC2</a> x1e.32xlarge instance type with 128 vCPU and 3,904 GiB of memory.</p> <p>Since that announcement, we have heard from customers that they want more instance configuration choices with fewer vCPUs, while maintaining a high ratio to memory. Last week, the introduction of the <a href="">x1e family</a> was completed with the <a href="">announcement</a> of general availability of five new sizes—x1e.xlarge, x1e.2xlarge, x1e.4xlarge, x1e.8xlarge, and x1e.16xlarge. These are currently available in the US East (N. Virginia), US West (Oregon), EU (Ireland), Asia Pacific (Tokyo), and Asia Pacific (Sydney) regions.</p> <p>With the highest memory per vCPU among Amazon EC2 instance types and one of the lowest prices per GiB of memory, the new x1e instance sizes are well suited for high-performance databases, in-memory databases, and other memory-optimized enterprise applications.</p> <p>Bill Ramos, director of technical product management at DB Best, has this to say about the new x1e instance types:</p> <blockquote> <p>The new x1e memory-optimized instance family hits a sweet spot with our SQL Server 2016/2017 customers who want in-memory computing using Standard Edition. With the new SQL Server 2017 read-scale availability groups, our customers can perform data warehouse like queries using in-memory clustered columnstore performance while running the online transaction processing on the primary replica using Standard Edition as well. It’s great that you can start with a 4 core, 122 GiB system using SQL Server 2017 Standard Edition and then scale up as needed. With the 8 core, 244 GiB system, customers can run their SQL Server database instance with 128 GiB with Analysis Services using another 64 Gib and still have room for other applications all using Standard Edition.</p> </blockquote> <p>The smallest member of the x1e family (x1e.xlarge) has 4 vCPU and 122 GiB memory, and the x1e.2xlarge has 8 vCPU and 244 GiB memory. These are ideal candidates for Microsoft SQL Server Standard Edition. With the x1e.2xlarge instance type, you can allocate the maximum allowed memory (128 GB) for SQL Server Standard database engine and still have enough remaining for Analysis, Integration, or Reporting Services (SSAS, SSIS, or SSRS).</p> <p>These new instance types, along with the SIOS DataKeeper <a href="">Quick Start</a> introduced in May and the <a href="">price reduction</a> announced in July for SQL Server Standard on Amazon EC2, bring more affordable SQL Server high availability to AWS customers. With the lowest price per GiB of memory among Amazon EC2 instance types, you can take advantage of not only large amount of memory, but also optimize your spend. For example, the on-demand price for an x1e.xlarge instance running Microsoft Windows and SQL Server Standard Edition is just 61 percent more than the cost of an r4.xlarge instance but with four times more memory. For more pricing comparisons, take a look at the AWS <a href="">Simple Monthly Calculator</a>.</p> <p>With x1e instances, you still get all the licensing options for running SQL Server on Amazon EC2. With the <em>license included SQL Server option</em>, you can avoid making a long-term purchase and don’t have to deal with true-ups, software compliance audits, or Software Assurance. This option works well if you prefer to avoid buying licenses and want to upgrade from an older version of SQL Server.</p> <p>You can also choose the <em>License Mobility option</em><strong>.</strong> With this option, you can use your active Software Assurance agreement to bring your existing licenses to EC2 without needing dedicated infrastructure.</p> <p>You can choose to <em>bring you own licenses</em> to the new X1e instances and take advantage of your existing license investment while further optimizing your upgrade costs. You can run SQL Server on&nbsp;<a href="">EC2 Dedicated Instances</a>&nbsp;or&nbsp;<a href="">EC2 Dedicated Hosts</a>, with the potential to reduce operating costs by licensing SQL Server on a per-core basis. You can bring your own SQL Server 2017 licenses (BYOL) to EC2 Windows, EC2 Linux instances, or to Docker containers running in Amazon EC2.</p> <hr /> <h3>About the Author</h3> <p><strong><img class="size-full wp-image-2269 alignleft" src="" alt="" width="103" height="137" />Tom Staab is a partner solutions architect at Amazon Web Services.</strong> He works with our customers to provide guidance and technical assistance on database projects, helping them improving the value of their solutions when using AWS.</p> Audit Amazon Aurora Database Logs for Connections, Query Patterns, and More, using Amazon Athena and Amazon QuickSight Mon, 20 Nov 2017 17:57:27 +0000 a87910fe56261997a282984f2cd6f43e64b17be0 Amazon Aurora offers a high-performance advanced auditing feature that logs detailed database activity to the database audit logs in Amazon CloudWatch. If you are using Aurora 1.10.1 or greater, you can use advanced auditing to meet regulatory or compliance requirements by capturing eligible events like tables queried, queries issued, and connections and disconnections. You can […] <p>Amazon Aurora offers a high-performance <a href="">advanced auditing</a> feature that logs detailed database activity to the database audit logs in <a href="">Amazon CloudWatch</a>. If you are using Aurora 1.10.1 or greater, you can use advanced auditing to meet regulatory or compliance requirements by capturing eligible events like tables queried, queries issued, and connections and disconnections. You can also use advanced auditing in tandem with <a href="">Amazon Athena</a> and <a href="">Amazon QuickSight</a> for easy, low-cost reporting on the logs when you write the logs to <a href="">Amazon S3</a>.</p> <p>The previous blog post <a href="">Monitoring Amazon Aurora Audit Events with Amazon CloudWatch</a> showed you how to send Aurora audit logs to CloudWatch for continuously monitoring activity in your DB clusters. &nbsp;This post shows how to do these things:</p> <ul> <li>Export your Aurora audit logs from CloudWatch to S3.</li> <li>Use Amazon Athena for historical analysis of events in your DB clusters.</li> <li>Build rich visualization with Amazon QuickSight.</li> </ul> <p><img class="aligncenter size-full wp-image-2127" src="" alt="" width="1239" height="402" /></p> <p><span style="text-decoration: underline"><strong>Prerequisites</strong></span><br /> As prerequisites, we recommend the following:</p> <ul> <li><a href="">Amazon Aurora Cluster</a> running in your AWS account configured with Aurora auditing and sending data to CloudWatch. For more information, see the AWS Database blog post <a href="">Monitoring Amazon Aurora Audit Events With Amazon CloudWatch</a>, or the <a href="">Aurora documentation</a>.</li> <li>Familiarity with <a href="">Amazon Athena</a>. For more information, see the <a href="">Athena documentation</a>.</li> <li>Familiarity with <a href="">Amazon QuickSight</a>. For more information, see <a href="">Amazon QuickSight documentation</a>.</li> </ul> <p><span id="more-2124"></span></p> <p><span style="text-decoration: underline"><strong>Getting started</strong></span><br /> On your Amazon Aurora cluster, you enable the collection of audit logs by setting several DB cluster parameters. When advanced auditing is enabled, you can use it to log any combination of the events in the table following. For more information, see the blog post <a href="">Monitoring Amazon Aurora Audit Events with Amazon CloudWatch</a>. Choose one or more of the events following for advanced auditing.</p> <table style="width: 100%" border="1px solid black"> <tbody> <tr> <td><strong>Event</strong></td> <td>Definition</td> </tr> <tr> <td>CONNECT</td> <td>Logs both successful and failed connections and also disconnections. This event includes user information.</td> </tr> <tr> <td>QUERY</td> <td>Logs all queries in plain text, including queries that fail due to syntax or permission errors.</td> </tr> <tr> <td>QUERY_DCL</td> <td>Similar to the QUERY event, but returns only data control language (DCL) queries (GRANT, REVOKE, and so on).</td> </tr> <tr> <td>QUERY_DDL</td> <td>Similar to the QUERY event, but returns only data definition language (DDL) queries (CREATE, ALTER, and so on).</td> </tr> <tr> <td>QUERY_DML</td> <td>Similar to the QUERY event, but returns only data manipulation language (DML) queries (INSERT, UPDATE, and so on).</td> </tr> <tr> <td>TABLE</td> <td>Logs the tables that were affected by query execution.</td> </tr> </tbody> </table> <p><span style="text-decoration: underline"><strong>Viewing logs</strong></span><br /> Recent logs are still available in the AWS RDS Management Console, but now you can capture and keep logs for extended periods of time. This functionality means that even when you are writing thousands of transactions per second, the data is preserved in CloudWatch for as long as you choose to keep it. For information about how to change your log retention setting, see <a href="">Change Log Data Retention in CloudWatch Logs</a> in the CloudWatch documentation.</p> <p><strong>Console</strong><br /> To view the log data in the console, choose one of your Aurora cluster instances and then choose <strong>Logs</strong> to open the list of log files for the instance. From the list, look for the audit logs and choose either <strong>view</strong>, <strong>watch</strong>, or <em><strong>download</strong></em>. <strong>View</strong> shows you the entire log on your screen. <strong>Watch</strong> displays a tail of the log showing the most recent events. <strong>Download</strong> opens a link from which you can save logs as a file on your computer to view in an editor.</p> <p><img class="aligncenter size-full wp-image-2129" src="" alt="" width="1429" height="416" /></p> <p>When you view an audit log, you can see the identity of the user running the query and the timestamp of the query execution. Depending on the setting you chose for the audit, a log might also include the detailed SQL code for the operation among other columns. For detailed information about each of the columns output to the audit logs, see the <a href="">Aurora auditing documentation</a>.</p> <p><img class="aligncenter size-full wp-image-2130" src="" alt="" width="1432" height="726" /></p> <p><strong>CloudWatch</strong><br /> Audit logs in Aurora are rotated when they reach 100 MB. With your audit logs in CloudWatch, you gain control over how long you retain your logs in CloudWatch Logs. By default, your logs are stored indefinitely in CloudWatch. You can specify how long you want CloudWatch to retain your logs. To do so, open the CloudWatch console, choose&nbsp;<strong>Logs</strong>, locate your log group, and then choose&nbsp;Expire <strong>Event After</strong>. From here, you can select a retention period that’s suitable for you.</p> <p><img class="aligncenter size-full wp-image-2131" src="" alt="" width="1431" height="214" /></p> <p>When your logs are available in CloudWatch Logs, you can search across your logs or a subset of logs by specific terms, phrases, or values. To do so, on the&nbsp;CloudWatch&nbsp;console choose&nbsp;<strong>Logs</strong>, and then locate and select your log group. Choose&nbsp;<strong>Search Log Group</strong>&nbsp;to search across all your logs. You can also select specific log streams using the check boxes.</p> <p><img class="aligncenter size-full wp-image-2132" src="" alt="" width="1054" height="274" /></p> <p>Log files are in UTF-8 encoding. Logs are written in multiple files, the number of which varies based on instance size. To see the latest events, you can sort by the time value when you are searching through the logs.</p> <p><span style="text-decoration: underline"><strong>Storing logs in S3</strong></span><br /> Amazon CloudWatch Logs is a great feature for continuous monitoring, storing, and accessing your log files. You can use the CloudWatch Logs console to view the audit logs periodically and do searching and filtering. To look at patterns in your logs, and perform deeper historical analysis with Athena, you can extract the data from the CloudWatch logs <strong>message</strong> field. You can do so using either the API or the AWS CLI and store the result in S3. Find detailed information about how to download the logs, and what options to use, in the <a href="">log access section</a> of the CloudWatch documentation.</p> <p>You can use the CLI to download the log files, extract the message portion of the logs, and then push the result to S3. Once the message payload is in S3, you can query it with Athena.</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-bash"><strong>#!/usr/bin/env bash</strong> REGION=&quot;us-west-2&quot; BUCKETNAME=&quot;99990227-uniquebucket&quot; # Set the log group name that you are interested in LOG_GROUP_NAME=&quot;/aws/rds/cluster/mylogs-databasecluster-2q941argqxxl/audit&quot; echo &quot;current date of run $CURRENTDATE&quot; # create a local directory to download the files into mkdir -p templogs cd templogs/ # download all files that match audit/ for FILENAME in `aws logs describe-log-streams --log-group-name $LOG_GROUP_NAME --region $REGION | jq -r '.logStreams[].logStreamName'`; do echo &quot; downloading... $FILENAME&quot; # get the data from each log file and format the message payload so Athena can easily parse as CSV aws logs get-log-events --log-group-name $LOG_GROUP_NAME --log-stream-name $FILENAME | jq -r '.events[].message' | awk -F''\''' -v OFS='' '{ for (i=2; i&lt;=NF; i+=2) gsub(&quot;[,]&quot;, &quot; &quot;, $i);$1=$1 } 1' &gt;&gt;$FILENAME done # upload the files to s3 for the instance aws s3 sync . s3://$BUCKETNAME/$LOG_GROUP_NAME/ --region us-east-1 cd .. rm -rf templogs/ echo &quot;upload to S3 complete&quot;</code></pre> </div> <p><strong>Note:</strong> The previous example code uploads the log message extracts to an S3 bucket with a key that corresponds to the log group name.</p> <p><span style="text-decoration: underline"><strong>Querying logs directly in S3 with Athena</strong></span><br /> To look at all the logs in aggregate in your S3 bucket, you can use Amazon Athena. If you are new to Amazon Athena, you can find a good introductory post on the AWS Blog in <a href="">Amazon Athena—Interactive SQL Queries for Data in Amazon S3</a>.</p> <p><strong>Set up a table in Athena</strong><br /> To set up an Athena query for this dataset, start by opening Amazon Athena from the console. On the <strong>Query Editor</strong> screen, you can enter the following Hive DDL syntax into the query window to create your table.</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-sql">CREATE EXTERNAL TABLE IF NOT EXISTS auditsaurora.audit_records ( `timestamp` bigint, `serverhost` string, `username` string, `host` string, `connectionid` string, `queryid` int, `operation` string, `mydatabase` string, `object` string, `retcode` int ) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'serialization.format' = ',', 'field.delim' = ',' ) LOCATION 's3://99990227-uniquebucket/loggroupname' TBLPROPERTIES ('has_encrypted_data'='false');</code></pre> </div> <p><strong>Query data with Athena</strong><br /> Your log entries are not in sequential order and might be duplicated between extraction runs if you extract logs frequently over time. To put the logs in time order, you can use the timestamp value. To get a list of users and operations that have run against the cluster, issue a <em>distinct</em> query against the data.</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-sql">SELECT DISTINCT queryid, username, operation, retcode, object FROM auditsaurora.audit_records ORDER BY timestamp DESC;</code></pre> </div> <p>The results of this previous query show which of your users have accessed the cluster most recently, what objects they read, and what queries they issued.</p> <p><img class="aligncenter size-full wp-image-2133" src="" alt="" width="1223" height="540" /></p> <p><span style="text-decoration: underline"><strong>Visualizing log data with Amazon QuickSight</strong></span><br /> Now that you have the raw data from the logs in S3 and Amazon Athena, you can visualize your audit records through Amazon QuickSight in just a few clicks. If you are new to Amazon QuickSight, you can find a good introductory post on the AWS News Blog in <a href="">Amazon QuickSight – Fast &amp; Easy to Use Business Intelligence for Big Data at 1/10th the Cost of Traditional Solutions</a>.</p> <p>To visualize your data, open the Amazon QuickSight dashboard and choose <strong>New analysis</strong> and then <strong>New data set</strong>. You can choose <strong>Athena</strong> as your data source, enter a name for this data source, and then choose <strong>Create data source</strong>. Choose the database and the table you created in Athena, and then choose <strong>Select</strong>. Choose either <strong>SPICE</strong> or <strong>Direct query</strong> and then <strong>Edit/Preview data</strong> to enter a query rather than just a raw select from the table.</p> <p><img class="aligncenter size-full wp-image-2134" src="" alt="" width="1190" height="612" /></p> <p>On the preview pane, choose <strong>Switch to custom SQL tool</strong>, and then enter the query to retrieve the distinct list of records from the data source. Give your query a name, and then choose <strong>Save &amp; visualize</strong> from the top bar.</p> <p><img class="aligncenter size-full wp-image-2135" src="" alt="" width="1429" height="285" /></p> <p>To look at the count of records by operation and users, you add <strong>username</strong> and then add <strong>operation</strong> to the visualization. You can then use filtering to reduce the focus down to the users and events that you are interested in for analysis. The following resulting graph example shows that the user has issued 199 queries against the cluster since auditing was started.</p> <p><img class="aligncenter size-full wp-image-2136" src="" alt="" width="1429" height="729" /></p> <p><span style="text-decoration: underline"><strong>Summary</strong></span><br /> Amazon Aurora’s advanced auditing feature is a powerful solution that you can combine with other AWS services to gain visibility into user activity on your database. By writing advanced auditing logs from your Amazon Aurora cluster to CloudWatch, you can store them as long as you require them. When the data is in S3, it can be automatically tiered into lower-cost, longer-term storage using lifecycle policies. With Amazon Athena and Amazon QuickSight, you have lightweight, limited-cost query and reporting tools for data like logs that you access infrequently.</p> <hr /> <h3>About the Author</h3> <p><strong><img class="size-full wp-image-1883 alignleft" src="" alt="" width="120" height="120" />Wendy Neu has worked as a Data Architect with Amazon since January 2015.</strong> Prior to joining Amazon, she worked as a consultant in Cincinnati, OH helping customers integrate and manage their data from different unrelated data sources.</p> Migration Validation (Part 2) – Introducing Data Validation in AWS Database Migration Service Fri, 17 Nov 2017 21:26:19 +0000 4798fae2e1cd5c9ddf18ad280e45b10c13d72432 AWS Database Migration Service (AWS DMS) helps you migrate databases to AWS quickly and securely. You can migrate your data to and from most widely used commercial and open source databases, such as Oracle, Microsoft SQL Server, and PostgreSQL. The service supports homogeneous migrations such as Oracle to Oracle, and also heterogeneous migrations between different […] <p><a href="">AWS Database Migration Service (AWS DMS)</a> helps you migrate databases to AWS quickly and securely. You can migrate your data to and from most widely used commercial and open source databases, such as Oracle, Microsoft SQL Server, and PostgreSQL. The service supports homogeneous migrations such as Oracle to Oracle, and also heterogeneous migrations between different database platforms, such as Oracle to PostgreSQL or MySQL to Oracle. In addition, it now provides a mechanism to validate the data after the migration.</p> <p>This post gives you a quick overview of how you can create migration tasks that use the data validation feature. You can create these tasks by using the AWS DMS console.</p> <p>The AWS DMS migration process consists of setting up a replication instance, source and target endpoints, and a replication task. The replication task runs on the replication instance and migrates data from the source endpoint to the target endpoint.</p> <p>To create migration tasks, you can use the AWS Management Console or the AWS Command Line Interface (AWS CLI). If you are a first-time AWS CLI user, we recommend that you read the following documentation and get accustomed to configuring and using the AWS CLI:</p> <ol> <li><a href="">Create an AWS Identity and Access Management (IAM) user</a></li> <li><a href="">Set up required permissions for the IAM user</a></li> <li><a href="">Set up the roles required to use AWS DMS</a></li> <li><a href="">Set up the AWS CLI</a></li> </ol> <p><span style="text-decoration: underline"><strong>Problem/use case</strong></span><br /> AWS DMS supports migrating homogeneous and heterogeneous databases. But many customers are looking for a way to validate the data after the migration.&nbsp;They are replicating their production databases. Before doing the cut over, they want to have high confidence in their migration by comparing the data between the source and target.</p> <p>Customers use continuous replication to replicate data into different databases (for example, Oracle to PostgreSQL). Because it is their production system, they want to ensure that the data is being migrated without any loss or corruption.</p> <p><span style="text-decoration: underline"><strong>Solution</strong></span><br /> Now AWS DMS provides a way to validate replicated data between two databases using a data validation feature.</p> <p>To validate the replicated data, AWS DMS compares the data between the source and target to ensure that the data is the same. It makes appropriate queries to the source and the target to retrieve the data. If the data volume is large, AWS DMS also adopts a strategy to split the data into smaller manageable units and compare them. AWS DMS splits the table into multiple smaller groups of contiguous rows based on the primary key, which is known as a <em>partition</em>. It then compares the data at the partition level and exposes results of the final comparison to you. This partitioning approach helps AWS DMS compare a predefined amount of data at any point in time.</p> <p>You can use the results of the comparison to determine whether a significant difference exists between the source and target. You can enable this data validation feature when you create the AWS DMS task on the AWS DMS console.</p> <p><span style="text-decoration: underline"><strong>Creating an AWS DMS data validation task using the console</strong></span><br /> To enable the data validation feature using the AWS DMS console, when you create an AWS DMS task, select <strong>Enable validation</strong> under <strong>Task Settings</strong>.</p> <p><img class="aligncenter size-full wp-image-2223" src="" alt="" width="1075" height="1065" /></p> <p>On the <strong>Table statistics</strong> tab, you can check the result of the data validation during full load and change processing.</p> <p><img class="aligncenter size-full wp-image-2224" src="" alt="DMS Data Validation" width="1775" height="438" /></p> <p>To understand the results of the data validation, see the next section.</p> <p><span style="text-decoration: underline"><strong>Checking results of the data validation</strong></span><br /> The following metrics describe the results of data validation.</p> <p><strong>Validation state</strong></p> <ul> <li><strong>Validated:</strong> Indicates the records that are currently in sync between the source and target.</li> <li><strong>Mismatched records:</strong> Indicates the number of unmatched records after applying all known updates to the target. A future update on the source might change a record from unmatched to matched.</li> <li><strong>ValidationPendingRecords:</strong> Indicates how many records are yet to be validated.</li> <li><strong>ValidationFailedRecords:</strong> Indicates how many records failed the validation, which means that the values are not matching between the source and target.</li> <li><strong>ValidationSuspendedRecords: </strong>Indicates how many records are suspended. This can happen when rows are too busy and continue to be modified, so AWS DMS can’t compare them.</li> </ul> <p>You can also enable data validation using AWS CLI commands. To do this, add the following validation settings to the task settings JSON:</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-bash">&quot;ValidationSettings&quot;: { &quot;EnableValidation&quot;: true } </code></pre> </div> <p><img class="aligncenter size-full wp-image-2225" src="" alt="" width="858" height="229" /></p> <p>After you start the AWS DMS task with data validation set using the AWS CLI, you can view the validation results at a table level using the AWS CLI <code>DescribeTableStatistics</code> API call.</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-bash">$aws dms describe-table-statistics --replication-task-arn arn:aws:dms:us-west-2:aws-account-id:task:5VXX7BZB5XLUKAYQTTSLZKTISY</code></pre> </div> <p><img class="aligncenter size-full wp-image-2226" src="" alt="" width="936" height="853" /></p> <p>Failure details of the validation are stored on the target database in a table named <code class="lang-bash">aws_dms_validation_failures</code>. This table is similar to the use of the <code class="lang-bash">aws_dms_exceptions</code> table for storing exception details in applying the DML. Basically, this table stores the failure type, primary key value for a single failed record, or the start or end key values for a group of failed records.</p> <p><span style="text-decoration: underline"><strong>Validation failure table definitions</strong></span><br /> The following table shows the structure of the <code class="lang-bash">aws_dms_validation_failures</code> table.</p> <table style="width: 100%" border="1px solid black"> <tbody> <tr> <td><strong>Column</strong></td> <td><strong>Description</strong></td> </tr> <tr> <td>TaskIdentifier</td> <td>The identifier of the task.</td> </tr> <tr> <td>TableName</td> <td>The name of the table.</td> </tr> <tr> <td>SchemaName</td> <td>The name of the schema.</td> </tr> <tr> <td>RuleId</td> <td>The identifier of the validation rule.</td> </tr> <tr> <td>StartKey</td> <td>The primary key for row record type. If this is a range key, it is the start of the range. It is represented as a JSON string; the key in the JSON is the column name, and the value is the corresponding value. The JSON contains all columns in case of composite keys. If the primary key is a binary object, its value is represented as a base64-encoded string in the JSON.</td> </tr> <tr> <td>EndKey</td> <td>Similar to the start key, except it is present only for the Range record type and is the end key of the range.</td> </tr> <tr> <td>RecordType</td> <td>Can be either Row or Range.</td> </tr> <tr> <td>RecordCount</td> <td>Represents the total number of records in the range. It doesn’t mean that they are not in sync or it can’t compare them; it just means that out of these records, some of them (we don’t know exactly how many of them) are out of sync or can’t be compared. For row records type, the value is always 1.</td> </tr> <tr> <td>Failure Type</td> <td>Can be either OutofSync or CannotCompare.</td> </tr> </tbody> </table> <p>When you find mismatched records, it’s important to understand what caused it and resolve any issues. A mismatch can happen for several reasons:</p> <ul> <li>An update failing to apply to the target due to constraint violations, type conversion issues, etc.</li> <li>A direct update in the target</li> <li>Other unknown reasons</li> </ul> <p>We can reload the table that reports <code class="lang-bash">OutOfSync </code>records. Generally, if <code class="lang-bash">OutOfSync </code>records are not caused by direct updates in the target or other known issues such as a constraint violation, you should engage the AWS Support team for further investigation.</p> <p><span style="text-decoration: underline"><strong>Limitations</strong></span><br /> Note the following limitations when you are working with the AWS DMS data validation feature:</p> <ul> <li>Validation works only for tables with a primary key/unique index.</li> <li>If the target database is modified outside of AWS DMS, validation might fail to report the discrepancies accurately.</li> <li>If the same row or set of rows are modified continuously, validation can’t validate those busy rows, and you must check the busy rows themselves.</li> <li>You might not be able to compare all different types of data. For example, AWS DMS might not support comparing blob types as they are, or the comparison of some real/float numbers, etc.</li> <li>One of the current limitations with primary key constraints is that the length must be less than 1,024.</li> </ul> <p><span style="text-decoration: underline"><strong>Summary</strong></span><br /> Data validation in AWS DMS is a great feature that can help you gain a high level of confidence in your migrations. It enables you to figure out how much of your data has successfully been replicated and validated. Data validation provides metrics, such as the total number of replicated rows, how many rows were compared, how many are in sync, how many are out of sync, and how many rows are not comparable because of inflight changes.</p> <p>AWS DMS also provides a new migration assessment feature to help you validate your settings before running your migration tasks. For more information, see <a href="">Migration Validation (Part 1) – Introducing Migration Assessment in AWS Database Migration Service</a>.</p> <p>Good luck, and happy migrating!</p> <hr /> <h3>About the Author</h3> <p><strong><img class="size-full wp-image-2219 alignleft" src="" alt="" width="104" height="141" />Mahesh Kansara is a cloud support engineer at Amazon Web Services.</strong> He works with our customers to provide guidance and technical assistance on various database and analytical projects, helping them improving the value of their solutions when using AWS.</p> Migration Validation (Part 1) – Introducing Migration Assessment in AWS Database Migration Service Fri, 17 Nov 2017 21:22:15 +0000 d9346cdf4546df312813ec6800fca8d86e33d956 We are excited to announce a new feature that provides a pre-migration checklist in AWS Database Migration Service (AWS DMS). AWS DMS does a great job of helping you move your data between multiple supported sources and targets. However, migrations can be difficult, especially when you’re moving from one database engine to another (known as […] <p>We are excited to announce a new feature that provides a pre-migration checklist in <a href="">AWS Database Migration Service (AWS DMS)</a>. AWS DMS does a great job of helping you move your data between multiple supported sources and targets. However, migrations can be difficult, especially when you’re moving from one database engine to another (known as a <em>heterogeneous</em> migration). Multiple factors like resources (on the source, target, and replication instances), the network, and the nature of the data play a big part in determining the success of your database migration. Even if you’re doing a “simple” lift and shift, you can face many issues that might complicate the process.</p> <p>There can also be limitations related to using AWS DMS. For example, AWS DMS might not support the data types in the tables involved in the migration. Or the prerequisites for the source or target database engine might not have been followed correctly. With the introduction of this feature, AWS DMS extends the managed service capability by providing you with a pre-migration task assessment. This capability helps you validate migration task settings before initiating your task.</p> <p>This post discusses what aspects the migration assessment feature addresses and how you can use it before starting your migration.</p> <p><span style="text-decoration: underline"><strong>Pre-migration checklist</strong></span><br /> The success of the data migration process is dependent on many variables—for example, the network connection between the components, the database permissions, and the source data that is being migrated. Some migration processes end with data that is only partially migrated. Some result in failures due to incorrect configurations on the source or target database engines or other problems that affect the success of the migration.</p> <p>The task assessment report indicates any inaccuracies or problems that might occur during the migration. It helps identify problems early in the process so that you can tackle them accordingly.</p> <p>The pre-migration task assessment is a list of tests. It performs validation checks of the different migration components and provides a list of warnings and errors. The objective of the checklist is to identify issues with the migration process before it starts running and to inform you of migration errors that are caused by data type issues between the source and the target.</p> <p><span style="text-decoration: underline"><strong>Migration assessments</strong></span><br /> With the current release of this feature, the task assessment report includes information about unsupported and partially supported data types in AWS DMS. The assessment process scans all the source data types in the tables that are to be migrated. It then compares this list with the supported data types for this type of database and generates a report. This report indicates the data types that are not supported or that are partially supported for the present database migration.</p> <p><img class="aligncenter size-full wp-image-2252" src="" alt="" width="1430" height="639" /></p> <p><span id="more-2251"></span></p> <p>To review the detailed task assessment report on the AWS DMS console, choose <strong>Open</strong>. The following image shows a sample report:</p> <p><img class="aligncenter size-full wp-image-2254" src="" alt="" width="952" height="732" /></p> <p>These results are categorized as follows:</p> <ul> <li>Not supported <ul> <li>Data types that AWS DMS does not support. Those columns of data won’t be migrated.</li> </ul> </li> <li>Partially supported <ul> <li>Data types that AWS DMS supports, but when it tries to migrate them, the target data type mapping might not match the source. For example, when you move from PostgreSQL to Oracle, the <code class="lang-sql">TEXT </code>data type is mapped as <code class="lang-sql">NCLOB </code>on the target and not <code class="lang-sql">CLOB </code>or <code class="lang-sql">BLOB</code>.</li> <li>Data types that AWS DMS can migrate, but with partial success. For example, when you move from PostgreSQL to PostgreSQL, JSONB migration works fine if the data in this column is fewer than 255 characters.</li> </ul> </li> </ul> <p>Additionally, the results are generated and stored as follows:</p> <ul> <li>The task assessment process creates an <a href="">Amazon S3</a> bucket in the customer account with the name <code class="lang-bash">dms-&lt;account-number&gt;-&lt;uniqueidentifier&gt;</code>.</li> <li>Each task assessment creates a folder with the task name and a subfolder with the date of the assessment.</li> <li>The data file includes a list of data structures in JSON format for each unsupported data type.</li> <li>The results file starts with a summary that includes a list of the unsupported data types and the column count of each one.</li> </ul> <p><span style="text-decoration: underline"><strong>Enable task assessment on the console</strong></span><br /> As mentioned before, the assessment report is generated before the task is run. On the AWS DMS console, you enable the feature when you create the task.</p> <p><img class="aligncenter size-full wp-image-2256" src="" alt="" width="1517" height="802" /></p> <p>When creating the task, under <strong>Post Creation</strong>, choose <strong>Task assessment</strong>. The following report is generated on the <strong>Assessment results</strong> tab:</p> <p><img class="aligncenter size-full wp-image-2257" src="" alt="" width="1827" height="316" /></p> <p>After you review the report and make the necessary corrections, choose <strong>Start task </strong>to initiate the migration process.</p> <p><span style="text-decoration: underline"><strong>Enable task assessment with the AWS CLI</strong></span><br /> If you use the AWS CLI, the task must already be created. If you use the AWS CLI for reviewing the pre-migration checklist, when you create the task, have it in the <strong>Start task </strong>mode. Then use the following command to start the assessment:</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-bash">aws dms start-replication-task-assessment --replication-task-arn <strong>&lt;task_arn&gt;</strong></code></pre> </div> <p>Use the following command to review the generated assessment report:</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-bash">aws dms describe-replication-task-assessment --replication-task-arn <strong>&lt;task_arn&gt;</strong></code></pre> </div> <p>The preceding command results in the following output:</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-bash">{ &quot;ReplicationTaskAssessmentResults&quot;: [ { &quot;ReplicationTaskIdentifier&quot;: &quot;mytask123&quot;, &quot;AssessmentStatus&quot;: &quot;issues found&quot;, &quot;S3ObjectUrl&quot;: &quot;https://dms... &quot;, &quot;AssessmentResultsFile&quot;: &quot;mytask123/2017-11-10-16-31&quot;, &quot;ReplicationTaskArn&quot;: &quot;task_arn&quot;, &quot;AssessmentResults&quot;: &quot;{\&quot;summary\&quot;:{\&quot;task-name\&quot;:\&quot;mytask123\&quot;,\&quot;not-supported\&quot;:{\&quot;data-types\&quot;:[],\&quot;column-count\&quot;:0},\&quot;partially-supported\&quot;:{\&quot;data-types\&quot;:[\&quot;enum\&quot;],\&quot;column-count\&quot;:1}},\&quot;types\&quot;:[{\&quot;data-type\&quot;:\&quot;enum\&quot;,\&quot;support-level\&quot;:\&quot;partially-supported\&quot;,\&quot;schemas\&quot;:[{\&quot;schema-name\&quot;:\&quot;employees\&quot;,\&quot;tables\&quot;:[{\&quot;table-name\&quot;:\&quot;employees\&quot;,\&quot;columns\&quot;:[\&quot;gender\&quot;]}]}]}]}&quot;, &quot;ReplicationTaskLastAssessmentDate&quot;: 1510331499.0 } ], &quot;BucketName&quot;: &quot;dms-&lt;customer_account_number&gt;-uniqueid&quot; }</code></pre> </div> <p>This command fetches the results from the S3 bucket where AWS DMS stores the assessment results and displays the output accordingly. If you run multiple task assessment reports, it will output all the reports.</p> <p><span style="text-decoration: underline"><strong>Summary</strong></span><br /> Database migration projects can be difficult, but the benefits of migrating your database to the cloud are significantly greater than the challenges migrations can present. Using the migration assessment feature can help you predict issues with a migration before you start. It also helps keep you informed about what to expect from your migrations. The checklist generated by this feature helps you assess and make the necessary changes required by AWS DMS and helps put your migrations on the right track.</p> <p>Thanks for reading! Have a successful database migration!</p> <hr /> <h3>About the Author</h3> <p><strong><img class="size-full wp-image-2008 alignleft" src="" alt="" width="117" height="158" />Abhinav Singh is a database engineer in Database Migration Service at Amazon Web Services.</strong> He works with our customers to provide guidance and technical assistance on database migration projects, helping them improving the value of their solutions when using AWS.</p> New AWS DMS and AWS Snowball Integration Enables Mass Database Migrations and Migrations of Large Databases Fri, 17 Nov 2017 19:22:05 +0000 a0072fc3543d3f2f2bca15cb7bea59ce01e3f868 Before reading this blog post, we recommend that you look at the&nbsp;AWS DMS,&nbsp;AWS SCT&nbsp;and&nbsp;AWS Snowball blogs and get to know these services. More than 40,000 databases have been migrated to AWS using AWS Database Migration Service (AWS DMS), either as a one-time migration or with ongoing replication. AWS Database Migration Service (AWS DMS) and AWS […] <p><em>Before reading this blog post, we recommend that you look at the&nbsp;</em><a href=""><em>AWS DMS</em></a><em>,&nbsp;</em><a href=""><em>AWS SCT</em></a><em>&nbsp;and&nbsp;</em><a href=""><em>AWS Snowball</em></a> <em>blogs and get to know these services.</em></p> <p>More than 40,000 databases have been migrated to AWS using <a href="">AWS Database Migration Service</a> (AWS DMS), either as a one-time migration or with ongoing replication. AWS Database Migration Service (AWS DMS) and <a href="">AWS Schema Conversion Tool</a> (AWS SCT) significantly simplify and expedite the database migration process in a low-cost, highly available manner.</p> <p>At some point in any migration, however, the bandwidth of your network becomes a limiting factor. It’s simple physics. If you want to move 5 terabytes from your on-premises network to the AWS Cloud over your 10-Gbps network… no problem. Increase that by an order of magnitude or two, or work with a slow, busy network. Suddenly you can spend days, weeks, or months waiting for your data. Maybe you are only trying to move a 500-GB database. But your network is painfully slow, because you are in a remote location or because there are geospecific network challenges at the time of the migration. Or perhaps you have many smaller databases to migrate that together add up to a significant size.</p> <p>Another common scenario that can hinder or delay a database migration project is the lack of outside access to the database itself. You might find yourself all set to start AWS DMS on your source database, only to find that your corporate network policy doesn’t allow access to the database from outside your corporate network.</p> <p>These scenarios and others like them are where <a href="">AWS Snowball Edge</a> and its brand-new integration with AWS DMS come in.</p> <p>AWS Snowball Edge is a service and also a physical storage appliance from AWS that enables you to move petabytes of data to AWS. It helps eliminate challenges that you can encounter with large-scale data transfers, including high network costs, long transfer times, and security concerns. You know that you can order a book or a Crock-Pot on Amazon Prime, knowing it will show up at your door&nbsp;two days later. Similarly, you can order several AWS Snowball Edge appliances from your AWS Management Console. They show up at your data center a few days later, each with a secure capacity of 100 TB.</p> <p>Combining these powerful services, AWS announced today the AWS DMS integration with AWS Snowball Edge, so that you can more easily move large database workloads to AWS.</p> <p>Following is an architecture diagram showing various components involved in this integration. It shows how to fully migrate the source database to the target database on AWS, including replication of ongoing changes on the source database.</p> <p><img class="aligncenter size-full wp-image-2237" src="" alt="" width="2524" height="972" /></p> <p><span id="more-2232"></span></p> <p>Some of the salient features of this integration architecture are the following:</p> <ul> <li>You get the ability to physically attach a secure, hardened device directly inside your data center rather than opening ports from the outside.</li> <li>You can now move very large databases from on-premises to AWS Cloud.</li> <li>This integration provides a “push” model to migrate databases instead of a “pull” model.</li> <li>You can migrate one or more databases using the same AWS Snowball Edge device without a need to upgrade your network infrastructure and consume dedicated bandwidth.</li> <li>While migrating these multiterabyte and multipetabyte databases to AWS, your on-premises databases remain online. They can be decommissioned when the AWS Snowball Edge appliance is shipped back to AWS and is automatically loaded onto your target Amazon RDS– or Amazon EC2–based database.</li> <li>You can migrate existing data (one time) or optionally perform ongoing data replication to the target database.</li> </ul> <p>A few notes about working with AWS Snowball Edge and AWS DMS:</p> <ul> <li>The version of AWS Snowball required for integration with AWS DMS is <a href="">AWS Snowball Edge</a>.</li> <li>Currently, you need a Linux host to run the DMS Snowball agent.</li> <li>The AWS Snowball Edge must be on the same network as your source database.</li> </ul> <p>You can use the steps following to migrate a database or multiple databases using the new integration of AWS DMS and AWS Snowball Edge.</p> <p><span style="text-decoration: underline"><strong>Preparation</strong></span><br /> Preparation involves setting up prerequisites, creating an Amazon S3 bucket, and getting and configuring your AWS Snowball Edge.</p> <p><strong>Prerequisites</strong><br /> As prerequisites, you must set up the source and target databases. To do so, look at the documentation for the AWS DMS <a href="">source configuration</a> and <a href="">target configuration</a>.</p> <p><strong>Step 1: Create an Amazon S3 bucket (staging S3)</strong><br /> When you’ve set up the source and target databases as described in the documentation, you create a bucket in <a href="">Amazon S3</a>. This bucket is also called the “staging S3.”</p> <p>This bucket acts as a temporary staging area for existing data and ongoing transactions during a database migration process.</p> <p>When database migration is complete and cutover to the target database is done, you can delete this staging S3 bucket.</p> <p>This bucket should be in the same AWS Region as the target database. Also, AWS DMS and AWS SCT need <a href="">AWS Identity and Access Management (IAM)</a> roles to access this bucket.</p> <p>For more information, see <a href="">Prerequisites When Using S3 as a Source for AWS DMS</a> in the AWS DMS documentation.</p> <p><strong>Step 2: Order and configure the AWS Snowball Edge</strong><br /> Next, you create an AWS Snowball job through the AWS Management Console and order your AWS Snowball Edge appliance. As part of this step, you specify the Amazon S3 bucket (staging S3) you created in the previous step.</p> <p>When your AWS Snowball Edge appliance arrives, configure it on your local network following the steps mentioned in the <a href="">Getting Started section</a> of the AWS Snowball Edge documentation.</p> <p>When AWS Snowball Edge device is connected to your network, you install the Snowball client. You then unlock the Snowball by downloading the manifest file and an unlock code from the AWS Management Console, as shown in the following command.</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-bash">snowballEdge unlock -i -m /user/tmp/manifest -u 01234-abcde-01234-ABCDE-01234</code></pre> </div> <p>Run the following command on the Snowball client to get the local access key and local secret key to use following.</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-bash">snowballEdge credentials -i -m /user/tmp/manifest -u 01234-abcde-01234-ABCDE-01234</code></pre> </div> <p>In the commands preceding, replace the IP address and unlock code with your AWS Snowball Edge configuration information.</p> <p><span style="text-decoration: underline"><strong>Configuration</strong></span><br /> Next, configure your migration by taking the following steps.</p> <p><strong>Step 1: Configure AWS SCT</strong><br /> In the first step of configuration, configure the global settings for AWS SCT. These settings include the AWS service profile and the database drivers for the source and target databases.</p> <p>To do so, start AWS SCT and for <strong>Settings</strong> choose <strong>Global Settings</strong>, <strong>AWS Service Profiles</strong>. The <strong>Global Settings</strong> page opens.</p> <p>Along with the AWS access key and secret key, you also need to specify the Amazon S3 bucket (the staging S3) that was created in the earlier step.</p> <p><img class="aligncenter size-full wp-image-2240" src="" alt="" width="790" height="568" /></p> <p>When the AWS service profile is configured in AWS SCT, you can use the source and target database details to create a new project in SCT. Then you can connect to both the source and target databases in this AWS SCT project.</p> <p><img class="aligncenter size-full wp-image-2241" src="" alt="" width="1217" height="583" /></p> <p><strong>Step 2: Configure the AWS DMS Replication Agent instance and install the AWS DMS Replication Agent</strong><br /> The local Linux machine where an agent runs and connects to the source database or databases to migrate data is called an <em>AWS DMS Replication Agent instance. </em>The agent process running on this instance is called an <em>AWS DMS Replication Agent.</em></p> <p>You size the Linux machine that you use depending on a couple of considerations. These considerations are the number of tasks to run on this machine and the throughput requirements for data migration from the source database to the AWS Snowball Edge device.</p> <p>The AWS DMS replication agent is delivered as a <a href="">downloadable .rpm file in the SCT package</a>. The installation steps are as follows.</p> <p>During the installation, you need to provide the port number and password. This port number and password are used in the AWS SCT UI in the next step.</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-bash">sudo rpm -i aws-schema-conversion-tool-dms-agent-&lt;version&gt;.&lt;arch&gt;.rpm tree /opt/amazon/aws-schema-conversion-tool-dms-agent/bin /opt/amazon/aws-schema-conversion-tool-dms-agent/bin ├── arep.ctl ├── arep.ctl-prev ├── ├── ├── ├── fix_permissions ├── makeconv ├── repctl ├── repctl.cfg ├── ├── replicate-local └── uconv sudo /opt/amazon/aws-schema-conversion-tool-dms-agent/bin/ Configure the AWS DMS Replication Agent Note: You will use these parameters when configuring agent in AWS Schema Conversion Tool Please provide the password for the AWS DMS Replication Agent Use minimum 8 and up to 20 alphanumeric characters with at least one digit and one capital case character Password: ******* ... [set password command] Succeeded Please provide port number the AWS DMS Replication Agent will listen on Note: You will have to configure your firewall rules accordingly Port: 8192 Starting service... ... AWS DMS Replication Agent was started You can always reconfigure the AWS DMS Replication Agent by running the script again.</code></pre> </div> <p><strong>Step 3: Install the source and target database drivers on the AWS DMS Replication Agent instance</strong><br /> The agent running on the replication instance connects to the source database to load the database transactions in AWS Snowball Edge for the target database. Thus, we need to install source and target database drivers on this instance.</p> <p>You install the ODBC drivers required for the source databases on the replication instance. For information on how to configure these drivers for specific source and target databases, see the database documentation.</p> <p>For example, to configure MySQL drivers, run the commands following.</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-bash">sudo yum install unixODBC sudo yum install mysql-connector-odbc</code></pre> </div> <p>After executing the preceding commands, make sure that the /etc/odbcinst.ini file has following contents.</p> <div class="hide-language"> <pre class="unlimited-height-code"><code class="lang-bash">cat /etc/odbcinst.ini [MySQL ODBC 5.3 Unicode Driver] Driver=/usr/lib64/ UsageCount=1 [MySQL ODBC 5.3 ANSI Driver] Driver=/usr/lib64/ UsageCount=1</code></pre> </div> <p><strong>Step 4: Configure an AWS DMS Replication instance using the console</strong><br /> For AWS DMS and AWS Snowball Edge integration, the AWS DMS replication instance is called <em>an AWS DMS remote replication instance.</em> It’s named this way because in this case, the instance is running on the AWS Cloud. This placement contrasts with that for the AWS DMS Replication Agent instance, which runs on your local Linux machine. For clarification of the two replication instances, see the architecture diagram.</p> <p>For information on how to create an AWS DMS remote replication instance using the AWS Management Console, see the <a href="">AWS DMS</a><u> blog</u> mentioned earlier or the <a href="">AWS DMS documentation</a>.</p> <p><span style="text-decoration: underline"><strong>Execution</strong></span><br /> Now that you’ve set up configuration, you can run the migration by using the following steps.</p> <p><strong>Step 1: Connect AWS SCT to the replication agent</strong><br /> In this step, you connect to the replication agent using the host name, port number, and password you provided in configuration step 3.</p> <p>In the AWS SCT user interface, navigate to <strong>View</strong>, <strong>Database Migration View (Local &amp; DMS)</strong>, and choose <strong>Register</strong>.</p> <p>Specify the IP address of the host, the port number, and the password used for AWS DMS replication agent configuration, as shown following.</p> <p><img class="aligncenter size-full wp-image-2242" src="" alt="" width="552" height="336" /></p> <p><img class="aligncenter size-full wp-image-2243" src="" alt="" width="810" height="566" /></p> <p>The replication agent creates and tests the connections to the source database, the AWS Snowball Edge device, and the staging S3 bucket. It also reports the status of the Snowball Edge and the Snowball import or export job in the AWS SCT UI.</p> <p>The AWS DMS replication agent is an independent process running on Linux and doesn’t depend on the AWS SCT.</p> <p><strong>Step 2: Create Local and DMS Tasks in AWS SCT</strong><br /> You can now create tasks on the local and remote AWS DMS replication instances. AWS DMS tasks are the actual workhorses that do the data migration.</p> <p>You create local and remote tasks in a single step from the AWS SCT UI as described following.</p> <p>First, open the context (right-click) menu for the source schema in SCT, and choose <strong>Create Local &amp; DMS Task</strong>.</p> <p>Details such as agents, replication instances, IAM roles, AWS import job names, and so on, are prepopulated from AWS SCT profile configurations and your AWS DMS resources in the AWS Region.</p> <p>Choose the appropriate agent, replication instance, migration type, and IAM role. Choose the job name, and type the Snowball IP address. Also, type the local Amazon S3 access key and local S3 secret key details obtained when you performed step 2 in the Preparation section, preceding.</p> <p><img class="aligncenter size-full wp-image-2244" src="" alt="" width="602" height="548" /></p> <p>As a result, two tasks are created, which you can see in the AWS SCT UI and DMS console:</p> <ol> <li>Local task – This task migrates existing data from the source database to the AWS Snowball Edge and also any ongoing transactions to the staging S3.</li> <li>DMS task – This task is the one that you are used to seeing in AWS DMS console. This task migrates existing data from the staging S3 to the target database. It then applies any ongoing transactions to the target database. For clarification of the two tasks, see the architecture diagram preceding.</li> </ol> <p><img class="aligncenter size-full wp-image-2245" src="" alt="" width="2108" height="432" /></p> <p><img class="aligncenter size-full wp-image-2246" src="" alt="" width="1077" height="551" /></p> <p><strong>Step 3: Test the connections, start the tasks, and monitor progress</strong><br /> You are now ready to test the connection to the source database, AWS Snowball Edge device, and staging S3 from AWS DMS Replication Agent instance. To do so, choose <strong>Test</strong> on the AWS SCT <strong>Tasks</strong> tab.</p> <p>Doing this also tests the connectivity to the staging S3 and the target database from the AWS DMS remote replication instance.</p> <p><img class="aligncenter size-full wp-image-2247" src="" alt="" width="2108" height="432" /></p> <p>Unless the test for all these tasks is successful, you can’t start the tasks.</p> <p>The AWS DMS task remains in the <strong>running</strong> state in the console until the AWS Snowball Edge device is shipped to AWS and the data is loaded into your staging S3 area.</p> <p>The following diagram shows the loaded data streams.</p> <p><img class="aligncenter size-full wp-image-2248" src="" alt="" width="963" height="498" /></p> <p>As mentioned, when the AWS Snowball Edge is attached at AWS, the AWS DMS task automatically starts loading existing data into the target database (full load). The task then applies the change data capture (CDC) logs for ongoing replication.</p> <p>When all existing data is migrated and the ongoing replication process brings both the source and target databases up to the same transaction level, you can cut over to the target database. Your applications can now point to the new database on AWS.</p> <p><img class="aligncenter size-full wp-image-2249" src="" alt="" width="1127" height="549" /></p> <p>Congratulations! You have migrated your multiterabyte database or databases to AWS using AWS DMS and AWS Snowball Edge integration.</p> <p>We also want to highlight the fact that you can migrate your database in this “push” model without using the AWS Snowball Edge device too! In this case, the local task or tasks copy the existing data from the source database to the staging S3, including the ongoing database transactions.</p> <p>The DMS tasks on the AWS DMS remote replication instance then load existing data immediately on the target database. The tasks start loading the ongoing transactions once existing data is migrated. You can also use this staging S3 flow to verify that the entire process works well by testing on a small table or two before you order your Snowball Edge.</p> <p><span style="text-decoration: underline"><strong>Summary</strong></span><br /> Many AWS features and services arise from AWS teams carefully listening to real-life customer experiences and needs. This new integration between AWS DMS and AWS Snowball Edge is an excellent example of implementing the ideas that emerge from that process. In turn, the implementation opens up new possibilities and opportunities for AWS customers.</p> <p>There are many more use cases for this feature besides migrating very large databases. During a migration, if you need compression or must deal with corporate network access policies, this integrated solution might be the tool for you. If you have limited, remote, or geographically challenged bandwidth issues, this solution might be the tool for you. Or maybe you have many databases to migrate all at once, then this solution might be the best way to accomplish your goal. Don’t hesitate to explore this solution when migrating your databases to AWS.</p> <p>For more information about this feature, read <a href="">the AWS documentation</a>. Let us know your feedback.</p> <hr /> <h3>About the Authors</h3> <p><strong><img class="size-full wp-image-2233 alignleft" src="" alt="" width="120" height="160" />Ejaz Sayyed is a partner solutions architect with the Global System Integrator (GSI) Team at Amazon Web Services.</strong> He works with the GSIs on AWS cloud adoption and helps them with solution architectures on AWS. When not at work, he likes to spend the time with his family which also includes two kids, Saad and Ayesha.</p> <p>&nbsp;</p> <p>&nbsp;</p> <p>&nbsp;</p> <p><strong><img class="size-full wp-image-2234 alignleft" src="" alt="" width="107" height="160" />Mitchell Gurspan is a senior solutions architect at Amazon Web Services.</strong> He is an AWS Certified Solutions Architect – Associate and is the author of a book on database systems. Mitchell resides in South Florida with his wife and two children. He enjoys tennis, teaches martial arts, and enjoys skiing when time allows.</p> <p>&nbsp;</p> <p>&nbsp;</p> <p>&nbsp;</p> <p>Special thanks to Alex Gershun, an AWS DMS Software Development Engineer for his inputs.</p> Get Started with Amazon Elasticsearch Service: Set CloudWatch Alarms on Key Metrics Thu, 16 Nov 2017 17:14:07 +0000 c1fcec59e0b99de8f0baaa5e0496bd9bd1edd9ac While supporting the many thousands of Amazon Elasticsearch Service (Amazon ES) domains for our customers, our team has amassed significant experience with common problems that our customers encounter—and best-practice solutions to these problems. I’ll spend the next few posts sharing these solutions. One of the most common problems that our customers have is that their […] <p>While supporting the many thousands of <a href="">Amazon Elasticsearch Service (Amazon ES)</a> domains for our customers, our team has amassed significant experience with common problems that our customers encounter—and best-practice solutions to these problems. I’ll spend the next few posts sharing these solutions.</p> <p>One of the most common problems that our customers have is that their domains are underscaled. Although there are many deeply technical reasons that this happens, we also find that many customers are not aware when they are incorrectly scaled. To ensure that you know about and can respond to problems with scale, it’s a good idea to set some alarms on your domain’s <a href="">Amazon CloudWatch</a> metrics.</p> <p>In this post, I detail the most important metrics to monitor. You can also easily extend the methodology to monitor less critical metrics.</p> <p><span style="text-decoration: underline"><strong>Critical metrics to monitor</strong></span><br /> The following are some of the most important CloudWatch metrics to monitor, along with possible solutions for issues that you might encounter in each scenario.</p> <table style="width: 100%" border="1px solid black"> <tbody> <tr> <td></td> <td>Maximum</td> <td>&gt;= 1</td> <td>1 period</td> </tr> </tbody> </table> <p>If your cluster is <strong>red</strong>, one of your primary shards and all of its replicas are not allocated. This is a “must-fix” scenario.</p> <p>A common cause for this state is a lack of free storage space on one or more of the data nodes in the cluster. In turn, a lack of free storage space prevents the service from distributing primary shards to the affected data node or nodes.</p> <p>For ways to address this issue, see the <a href="">Amazon ES documentation</a>.</p> <table style="width: 100%" border="1px solid black"> <tbody> <tr> <td>ClusterIndexWritesBlocked</td> <td>Maximum</td> <td>&gt;= 1</td> <td>1 period</td> </tr> </tbody> </table> <p>Many factors can cause a cluster to begin blocking requests. Some common factors include the following:</p> <ul> <li><code>FreeStorageSpace</code> is too low.</li> <li><code>JVMMemoryPressure</code> is too high.</li> <li><code>CPUUtilization</code> is too high.</li> </ul> <p><span id="more-2100"></span></p> <p>To alleviate this issue, consider adding more disk space or scaling your cluster.</p> <table style="width: 100%" border="1px solid black"> <tbody> <tr> <td>CPUUtilization</td> <td>Average</td> <td>&gt;= 80</td> <td>3 periods</td> </tr> <tr> <td>MasterCPUUtilization</td> <td>Average</td> <td>&gt;= 80</td> <td>3 periods</td> </tr> </tbody> </table> <p>To ensure that you aren’t exceeding your domain’s capacity, monitor your CPU on both your data nodes and your dedicated master nodes. Be sure to monitor the <strong>Average</strong> for these statistics. It’s not uncommon for a single node to peg the CPU for a short time.</p> <p>If you regularly deal with traffic spikes that push your CPUs high and then subside relatively soon after, you can also adjust the alarm’s trigger to wait longer.</p> <p>To remedy the situation, you can do any of the following:</p> <ul> <li>Reduce your traffic.</li> <li>Reduce your concurrency.</li> <li>Check your queries and remove excessive nesting in your aggregations.</li> <li>Optimize your queries.</li> <li>Scale—either horizontally or vertically.</li> </ul> <table style="width: 100%" border="1px solid black"> <tbody> <tr> <td>JVMMemoryPressure</td> <td>Maximum</td> <td>&gt;= 80</td> <td>3 periods</td> </tr> <tr> <td>MasterJVMMemoryPressure</td> <td>Maximum</td> <td>&gt;= 80</td> <td>3 periods</td> </tr> </tbody> </table> <p>The <code>JVMMemoryPressure</code> metric is the ratio of new to old memory that is allocated in the Java heap. It’s normal for this metric to oscillate between 50 and 75 percent in response to Java’s garbage collector reclaiming used heap space (lower than 50 percent is safe). If the ratio exceeds 75 percent, the garbage collector is not freeing any memory, and the cluster goes out of memory sooner or later.</p> <p>To fix this issue, allocate more memory in the cluster. You can do this by scaling vertically. Amazon ES uses half of the instance’s memory for the Java heap, up to ~32 GB. That means that you can scale vertically in instance size up to a total of 64 GB of memory. Beyond that, you can scale horizontally, distributing fewer shards per node.</p> <p>You can also examine your queries to determine whether you are using excessive memory per-query. The following are the common culprits:</p> <ul> <li>Deep paging</li> <li>Use of field data—for example, aggregations on fields with high cardinality or with deeply nested aggregations</li> </ul> <table style="width: 100%" border="1px solid black"> <tbody> <tr> <td>FreeStorageSpace</td> <td>Minimum</td> <td>&lt;= ?</td> <td>3 periods</td> </tr> </tbody> </table> <p>When your domain’s free storage space drops too low (less than 15 percent free), Elasticsearch begins relocating shards to try to balance usage. If your storage drops further, Amazon ES starts returning <code>ClusterBlockExceptions</code>. It’s a good idea to avoid these situations by setting a threshold of 25 percent. Be sure to use the <strong>Minimum</strong> free storage space, which reports the minimum free storage on any single instance in the cluster. The metric is in megabytes, so set the threshold at 25 percent of your per-instance storage.</p> <p>To remedy this situation, increase the per-node storage. If you’re using <a href="">Amazon EBS</a> volumes, increase the volume size. If you have maxed out your EBS volume size for your instance type, or if you’re using ephemeral store on your instances, increase your instance size, or add more instances to your domain.</p> <p>Alternatively, you can delete one or more of the indexes in your domain to free the storage used for those indexes.</p> <table style="width: 100%" border="1px solid black"> <tbody> <tr> <td>AutomatedSnapshotFailure</td> <td>Maximum</td> <td>&gt;= 1</td> <td>1 period</td> </tr> </tbody> </table> <p>Amazon ES takes daily snapshots of your domain. Starting with the Amazon ES support for Elasticsearch 5.5, you have access to these snapshots as well. You can also take <a href="">manual snapshots</a> of your domain. Especially if you are relying on the automatic snapshots, you’ll want to know when one fails.</p> <p>You can’t restart or retry a failed automatic snapshot. Instead, consider running a manual snapshot instead.</p> <p><span style="text-decoration: underline"><strong>How do I set an alarm?</strong></span><br /> You can find detailed information about setting CloudWatch alarms in the <a href="">Amazon CloudWatch documentation</a>.</p> <p>To set an alarm, first sign in to the <a href="">AWS Management Console</a>. Type <strong>CloudWatch</strong> in the services search bar, and then choose <strong>CloudWatch</strong> in the autocomplete menu to open the CloudWatch console.</p> <p>In the navigation pane on the left side, choose <strong>Alarms</strong>, and then choose <strong>Create Alarm</strong>.</p> <p><img class="aligncenter size-full wp-image-2101" src="" alt="" width="349" height="173" /></p> <p>In the wizard, type all or part of your domain name in the search bar. Then select the metric for the alarm, for example,</p> <p><img class="aligncenter size-full wp-image-2104" src="" alt="" width="530" height="201" /></p> <p>This opens a graph at the bottom of the page. Choose <strong>Next</strong>. Type a <strong>Name</strong> and <strong>Description </strong>for your alarm. To set the operator from the chart, use <strong>1 period</strong>. For CPUUtilization, use <strong>3 periods</strong>.</p> <p>Under <strong>Actions</strong>, you can choose an existing email distribution list or create a <strong>New</strong> list. If you choose <strong>New list</strong>, a text box appears, where you can add an email address to receive notifications. Enter a list name in the <strong>Send notification to</strong> box, add one or more email addresses (comma-separated) in the <strong>Email list</strong> box, and click <strong>Create Alarm</strong>.</p> <p><img class="aligncenter size-full wp-image-2105" src="" alt="" width="1178" height="526" /></p> <p>Repeat the process for the remaining alarms in the table. (You can reuse the list you just created or create a new one for each alarm.) CloudWatch sends a confirmation email to the addresses in the list. Be sure to confirm by following the link in the email.</p> <p><span style="text-decoration: underline"><strong>Conclusion</strong></span><br /> That’s all you need to do! If your domain starts getting into trouble, each member of the &shy;&shy;list receives an email, and you can react before a resource constraint becomes a real problem.</p> <hr /> <h3>About the Author</h3> <p><strong><img class="size-full wp-image-1921 alignleft" src="" alt="" width="112" height="153" />Jon Handler (@_searchgeek) is an AWS solutions architect specializing in search technologies.</strong> He works with our customers to provide guidance and technical assistance on database projects, helping them improving the value of their solutions when using AWS.</p> Amazon Aurora as an Alternative to Oracle RAC Mon, 13 Nov 2017 16:59:49 +0000 00720fadf90ec69112283bcc6a63cde9f48bdd2e Written by David Yahalom, CTO and co-founder of NAYA Tech—a leading database, big data, and cloud professional and consulting service provider, located in San Jose, CA. In this post, I discuss how Amazon Aurora can serve as a powerful and flexible alternative to Oracle RAC. Both Oracle RAC and Amazon Aurora are designed to provide increased high availability and performance scalability for your databases. <p><strong><em>Written by David Yahalom, </em></strong><em>CTO and co-founder of NAYA Tech—a leading database, big data, and cloud professional and consulting service provider, located in San Jose, CA. David is a certified Oracle, Apache Hadoop, and NoSQL database expert and a cloud solutions architect.</em></p> <p>Oracle Real Application Clusters (Oracle RAC) is considered to be one of the most advanced and capable technologies for enabling a highly available and scalable relational database. It is considered the default go-to standard for creating highly available and scalable Oracle databases.</p> <p>However, with the ever increasing adoption of cloud, open-source, and platform-as-a-service (PaaS) database architectures, many organizations are searching for their next-generation relational database engine. They’re looking for one that can provide similar levels of high availability and scalability to Oracle RAC, but in a Cloud/PaaS model, while maintaining the freedom of open source software.</p> <p>In this post, I discuss how <a href="">Amazon Aurora</a> can serve as a powerful and flexible alternative to Oracle RAC for certain applications. Both Oracle RAC and Amazon Aurora are designed to provide increased <strong>high availability</strong> and <strong>performance scalability</strong> for your databases. But they approach these goals from very different directions using different architectures:</p> <ul> <li>Oracle RAC uses an intricate end-to-end software stack developed by Oracle—including Oracle Clusterware and Grid Infrastructure, Oracle Automatic Storage Management (ASM), Oracle Net Listener, and the Oracle Database itself—combined with enterprise-grade storage that enables a shared-everything database cluster technology.</li> <li>Amazon Aurora simplifies the database technology stack by using AWS ecosystem components to transparently enable higher availability and performance for MySQL and PostgreSQL databases.</li> </ul> <p>So let’s dive right in!</p> <p><span style="text-decoration: underline"><strong>Preface</strong></span><br /> I’ll start this blog post with a quick disclaimer. I’m what you would call a “born and raised” Oracle DBA. My first job, 15 years ago, had me responsible for administration and developing code on production Oracle 8 databases. Since then, I’ve had the opportunity to work as a database architect and administrator with all Oracle versions up to and including Oracle 12.2. Throughout my career, I’ve delivered a lot of successful projects using Oracle as the relational database component.</p> <p>As such, I love the capabilities, features, technology, and power of Oracle Database. There’s really no denying that Oracle is still one of the most (if not <em>the </em>most) powerful and advanced relational databases in the world. Its place in the pantheon of database kings is undoubtedly safe.</p> <p>Also, please keep in mind that the Amazon Aurora and RAC architectures vary greatly and the intention of this article is not to compare the internals of Amazon Aurora vs. Oracle RAC. Instead, my goal is to look an Amazon Aurora and Oracle RAC from a <strong><em>functional</em></strong> perspective. Put forth some of the mainstream RAC use-cases and see if they can be handled by Amazon Aurora and, at the same time, leverage the inherit benefits of AWS.</p> <p><span style="text-decoration: underline"><strong>The paradigm shift</strong></span><br /> With that introduction out of the way, let’s talk about how the database industry is changing and why many customers choose to think outside the box and adopt cloud-based solutions as alternatives for commercial databases. Even applications that require the highest levels of performance and availability or those that run on high-end commercial databases can potentially now be powered by MySQL and PostgreSQL databases—albeit with some Amazon “special sauce” on top. But more on that later.</p> <p>There’s a fundamental shift happening in the database landscape as customers transition their data architectures from being monolithic—where a single, big, and feature-full, relational database powers their entire solution stack—to a more services-oriented model. The commercial database isn’t going anywhere, but, in the new model, different database technologies power different parts of the solution. This “best-of-breed” approach makes perfect sense as emerging database technologies become more mature, robust, and feature-rich.</p> <p>In addition, <a href="">there’s a huge upward trend in the adoption of cloud services</a>. Even the most traditional organization can see the benefits of tapping into the power that cloud-centric architectures can provide in performance, flexibility, high availability, and reduced total cost of ownership (TCO).</p> <p>When you combine that with the huge and <a href="">rapidly increasing adoption of open source–based relational databases</a>, you can clearly see that the database industry is moving in a different trajectory from the classic “one commercial database to rule them all” approach that was once common.</p> <p>Now, let’s dig deeper…</p> <p><span style="text-decoration: underline"><strong>Oracle RAC architecture</strong></span><br /> My first step is to flesh out the major benefits that Oracle Real Application Clusters (Oracle RAC) provides to its customers. Since this is just a blog post and not a full-blown research paper, I can’t cover every single aspect and benefit of Oracle RAC so I’m intentionally keeping the discussion at a relatively high level so it can be approachable for a wide audience and not exclusively Database Administrators.</p> <p>Oracle RAC is one of the major differentiating features for an Oracle database when compared to other relational database technologies. Oracle RAC is the technology that allows an Oracle database to provide<strong> increased levels of high availability</strong> and big <strong>performance benefits</strong>.</p> <p>Oracle RAC is an Active/Active database cluster, where multiple Oracle database servers (running Oracle Database instances, which is a collection of in-memory processes and caches) access a shared storage device that contains a single set of disk-persistent database files. This architecture is considerably different from what you usually find in a non-Oracle database cluster. Instead of each database node having its own dedicated storage, all the database nodes coordinate and share access to the same physical disks.</p> <p>All the database nodes coordinate with one another using both a dedicated network-based communication channel (known as the cluster interconnect) and a set of disk-based files.</p> <p>Public access to the cluster (from incoming applications, SQL queries, users, etc.) is performed using a set of SCAN IPs that are used to load-balance incoming sessions. In addition, each RAC cluster node has its own physical and virtual IP addresses that can be used to open a connection directly to a specific node.</p> <p><strong>Greatly simplified Oracle RAC architecture</strong><br /> <img class="aligncenter size-full wp-image-2115" src="" alt="" width="1140" height="922" /><span id="more-2113"></span></p> <p>Because of the shared nature of the RAC cluster architecture—specifically, having all nodes write to a single set of database data files on disk—the following two special coordination mechanisms were implemented to ensure that the Oracle database objects and data maintain ACID compliance:</p> <ul> <li><strong>GCS</strong> (Global Cache Services) tracks the location and the status of the database data blocks and helps guarantee data integrity for global access across all cluster nodes.</li> <li><strong>GES</strong> (Global Enqueue Services) performs concurrency control across all cluster nodes, including cache locks and the transactions.</li> </ul> <p>These services, which run as background processes on each cluster node, are essential to serialize access to shared data structures in the Oracle database.</p> <p>Shared storage is another essential component in the Oracle RAC architectures. All cluster nodes read and write data to the same physical database files stored in a disk that is accessible by all nodes. Most customers rely on high-end storage hardware to provide the shared storage capabilities required for RAC.</p> <p>In addition, Oracle provides its own software-based storage/disk management mechanism called <strong>Automatic Storage Management</strong>, or<strong> ASM</strong>. ASM is implemented as a set of special background processes that run on all cluster nodes and allow for easier management of the database storage layer.</p> <p>So, to recap, the main components of an Oracle RAC architecture include the following:</p> <ul> <li><strong>Cluster nodes: </strong>Set of one or more servers running Oracle instances, each with a collection of in-memory processes and caches.</li> <li><strong>Interconnect network: </strong>Cluster nodes communicate with one another using a dedicated “interconnect” network.</li> <li><strong>Shared storage:</strong> All cluster nodes access the same physical disks and coordinate access to a single set of database data files that contain user data. Usually handled by a combination of enterprise-grade storage with Oracle’s ASM software layer.</li> <li><strong>SCAN (Single Client Access Name):</strong> “Floating” virtual hostname/IPs providing load-balancing capabilities across cluster nodes. Naming resolution of SCAN to IP can be done via DNS or GNS (Grid Naming Service).</li> <li><strong>Virtual IPs (and Physical IPs): </strong>Each cluster node has its own dedicated IP address.</li> </ul> <p><strong>Performance and scale-out in Oracle RAC</strong></p> <p>With Oracle RAC, you can add new nodes to an existing RAC cluster without downtime. Adding more nodes to the RAC cluster increases the level of high availability that’s provided and also enhances performance.</p> <p>Although you can scale read performance easily by adding more cluster nodes, scaling write performance is a more complex subject. Technically, Oracle RAC can scale writes and reads together when adding new nodes to the cluster, but attempts from multiple sessions to modify rows that reside in the same physical Oracle block (the lowest level of logical I/O performed by the database) can cause write overhead for the requested block and affect write performance. This is well known phenomena and why RAC-Aware applications are a real thing in the real world. Concurrency is also one of the reasons and why RAC implements a “smart mastering” mechanism which attempts to reduce write-concurrency overhead. The “smart mastering” mechanism enables the database to determine which service causes which rows to be read into the buffer cache and master the data blocks only on those nodes where the service is active. Scaling writes in RAC isn’t as straightforward as scaling reads.</p> <p>With the limitations for pure write scale-out, many Oracle RAC customers choose to split their RAC clusters into multiple “services,” which are logical groupings of nodes in the same RAC cluster. By using services, you can use Oracle RAC to perform direct writes to specific cluster nodes. This is usually done in one of two ways (as shown in the diagram that follows):</p> <ol> <li>Splitting writes from different individual “modules” in the application (that is, groups of independent tables) to different nodes in the cluster. This is also known as “application partitioning” (not to be confused with database table partitions).</li> <li>In extremely un-optimized workloads with high concurrency, directing all writes to a single RAC node and load-balancing only the reads.</li> </ol> <p><img class="aligncenter size-full wp-image-2117" src="" alt="" width="1450" height="872" /></p> <p><strong>Major benefits of Oracle RAC</strong><br /> To recap, Oracle Real Application Clusters provides two major benefits that drive customer adoption:</p> <ul> <li>Multiple database nodes within a single RAC cluster provide increased high availability. No single point of failure exists from the database servers themselves. However, the shared storage requires storage-based high availability or DR solutions.</li> <li>Multiple cluster database nodes allow for scaling-out query performance across multiple servers.</li> </ul> <p><span style="text-decoration: underline"><strong>Amazon Aurora architecture</strong></span></p> <p>Aurora is Amazon’s flagship cloud database solution. When creating Amazon Aurora cluster databases, you can choose between <strong>MySQL</strong> and <strong>PostgreSQL</strong> compatibility.</p> <p>Aurora extends the “vanilla” versions of MySQL and PostgreSQL in two major ways:</p> <ul> <li>Adding enhancements to the MySQL/PostgreSQL database kernel itself to improve performance (concurrency, locking, multithreading, etc.)</li> <li>Using the capabilities of the AWS ecosystem for greater high availability, disaster recovery, and backup/recovery functionality.</li> </ul> <p>Aurora adds these enhancements without affecting the database optimizer and SQL parser. This means that these changes are completely transparent to the application. If you provision an Aurora cluster with MySQL compatibility, as the name suggests, any MySQL-compatible application can function.</p> <p>When comparing the Amazon Aurora architecture to Oracle RAC, you can see major differences in how Amazon chooses to provide scalability and increased high availability in Aurora. These differences are due mainly to the existing capabilities of MySQL/PostgreSQL and the strengths that the AWS backend can provide in terms of networking and storage.</p> <p>Instead of having multiple read/write cluster nodes access a shared disk, an Aurora cluster has a single primary node (“master”) that is open for reads and writes and a set of replica nodes that are open for reads with automatic promotion to primary (“master”) in case of failures. Whereas Oracle RAC uses a set of background processes to coordinate writes across all cluster nodes, the Amazon Aurora Master writes a constant redo stream to six storage nodes, distributed across three Availability Zones in an AWS Region. The <em>only</em> writes that cross the network are redo log records, not pages.<br /> Let’s go deeper…</p> <p>Each Aurora cluster can have one or more instances which serve different purposes:</p> <ol> <li>At any given time, a single instance functions as the <strong>primary </strong>(“master”) that handles both writes and reads from your applications.</li> <li>In addition to the primary (“master”), up to <strong>15</strong> <strong>read replicas</strong> can be created, which are used for two purposes: <ul> <li><strong>For performance and read scalability:</strong> as read-only nodes for queries/report-type workloads.</li> <li><strong>For high availability: </strong>as failover nodes in case the master fails. Each read replica can be located in one of the three Availability Zones that hosts your Aurora cluster. A single Availability Zone can host more than one read replica.</li> </ul> </li> </ol> <p>The following is a high-level Aurora architecture diagram showing four cluster nodes: one primary (“master”) and three read replicas. The primary node is located in Availability Zone A, the first read replica in Availability Zone B, and the second and third read replicas in Availability Zone C.</p> <p><img class="aligncenter size-full wp-image-2118" src="" alt="" width="1684" height="968" /></p> <p>An Aurora Storage volume is made up of 10 GB segments of data with six copies spread across three AZs. Each Amazon Aurora read replica shares the same underlying volume as the master instance. Updates made by the master are visible to all read replicas through a combination of reading from the shared Aurora storage volume, and applying log updates in-memory when received from the primary instance After a master failure, promotion of a read replica to master is usually less than 30 seconds with no data loss.</p> <p>Drilling down, for a write to be considered durable in Aurora, the primary instance (“master”) sends a redo stream to six storage nodes, two in each availability zone for the storage volume, and waits until four of the six nodes have responded. <strong>No database pages are ever written from the database tier to the storage tier.</strong> The Aurora Storage volume asynchronously applies redo records to generate database pages in the background or on demand. Remember that Aurora hides the underlying complexity from the user.</p> <p><strong>Amazon Aurora cluster with 1 X Primary, 2 X Replicas (and shared distributed storage). </strong></p> <p><img class="aligncenter wp-image-2278 size-large" src="" alt="" width="1024" height="779" /></p> <p><strong>Primary writes REDO records to a quorum of 6 storage nodes spread across three AZs.</strong></p> <p><img class="aligncenter wp-image-2277 size-large" src="" alt="" width="1024" height="770" /></p> <p><strong>After 4 out of 6 storage nodes in the quorum confirm write, the ACK is sent back to the client.</strong></p> <p><img class="aligncenter wp-image-2280 size-large" src="" alt="" width="1024" height="783" /></p> <p><strong>Writes to 6 out of 6 quorum continues in the background.</strong></p> <p><img class="aligncenter wp-image-2281 size-large" src="" alt="" width="1024" height="771" /></p> <p><strong>In parallel, REDO vectors are sent over the network from the Primary to the Replicas to be applied in-memory.</strong></p> <p><img class="aligncenter wp-image-2282 size-large" src="" alt="" width="1024" height="771" /></p> <p><strong>High availability and scale-out in Aurora</strong><br /> Aurora provides two endpoints for cluster access. These endpoints provide both high availability capabilities and scale-out read processing for connecting applications</p> <ul> <li><strong>Cluster Endpoint:</strong> Connects you to the current primary instance for the Aurora cluster. You can perform both read and write operations using the cluster endpoint. Writes will always be directed to the “primary” instance when applications use the Cluster Endpoint. If the current primary instance fails, Aurora automatically fails over to a new primary instance. During a failover, the DB cluster continues to serve connection requests to the cluster endpoint from the new primary instance, with minimal interruption of service.</li> <li><strong>Reader Endpoint:</strong> Provides load-balancing capabilities (round-robin) across the replicas allowing applications to scale-out reads across the Aurora cluster . Using the Reader Endpoint provides better use of the resources available in the cluster. The reader endpoint also enhances high availability. If an AWS Availability Zone fails, the application’s use of the reader endpoint continues to send read traffic to the other replicas with minimal disruption.</li> </ul> <p><img class="aligncenter size-full wp-image-2119" src="" alt="" width="1722" height="1320" /></p> <p>While Amazon Aurora focuses on the scale-out of reads and Oracle RAC can scale-out both reads and writes, most OLTP applications are usually not limited by write scalability; From my own experience, many Oracle RAC customers use RAC firstly for high availability and to scale-out their reads. You can write to any node in an Oracle RAC cluster, but this capability is often a functional benefit for the application versus a method for achieving unlimited scalability for writes.</p> <p>Scaling reads is usually more important for the following reasons:</p> <ul> <li>Most RDBMS-centric applications are generally more read-heavy. A mix of 70 percent reads/30 percent writes is normal for most applications (for reference, the TPC-E benchmark has a 7:1 I/O read to write ratio).</li> <li>Scaling read performance is especially crucial when combining OLTP workloads (transactional) with analytical-type workloads, such as executing reports on operational data.</li> <li>Even with Oracle RAC, which can scale-out writes to some extent, because of concurrency issues that can occur when multiple sessions try to modify rows in the same database block(s), many customers choose to partition the read/write workload to specific nodes in their RAC cluster creating “RAC-Aware applications”.</li> </ul> <p><strong>Can Amazon Aurora Be Your Next Data Platform?</strong><br /> To summarize, Amazon Aurora provides the following benefits as a database platform:</p> <ul> <li>Multiple cluster database nodes provide increased high availability. There is no single point of failure from the database servers. In addition, since an <strong>Aurora cluster is <em>always</em> distributed across three availability zones</strong>, there is a huge benefit for high availability and durability of the database. These types of “stretch” database clusters are usually uncommon with other database architectures.</li> <li>AWS managed storage nodes also provide high availability for the storage tier. A <strong>zero-data loss architecture is employed</strong> in case a master node fails and a replica node is promoted to the new master. This failover can usually be done in under 30 seconds.</li> <li>Multiple cluster database nodes allow for scaling-out query read performance across multiple servers.</li> <li>Greatly reduced operational overhead using a cloud solution and reduced TCO by using AWS and open source database engines.</li> <li>Automatic management of storage. No need to pre-provision storage for a database. Storage is automatically added as needed, and you only pay for one copy of your data.</li> <li>With Amazon Aurora you can easily <strong>scale-out</strong> your reads (and<strong> scale-up</strong> your writes) which fits perfectly into the workload characteristics of many, if not most, OLTP applications. Remember that scaling out reads usually provides the most tangible performance benefit.</li> </ul> <p>When comparing Oracle RAC and Amazon Aurora side by side, you can see the architectural differences between the two database technologies. Both provide high availability and scalability but approach it with different architectures.</p> <p><img class="aligncenter size-full wp-image-2230" src="" alt="" width="1720" height="1390" /></p> <p>Stepping back and looking at the bigger picture, Amazon Aurora introduces a simplified solution that can function as an Oracle RAC alternative for many typical OLTP applications that need high performance writes, scalable reads, very high levels of high availability with lower operational overhead.</p> <table style="width: 100%" border="1px solid black"> <tbody> <tr> <td><strong>Feature</strong></td> <td><strong>Oracle RAC</strong></td> <td><strong>Amazon Aurora</strong></td> </tr> <tr> <td><strong>Storage</strong></td> <td>Usually enterprise-grade storage + ASM</td> <td>Aurora Storage Nodes: Distributed, Low Latency, Storage Engine Spanning Multiple AZs</td> </tr> <tr> <td><strong>Cluster type</strong></td> <td>Active/Active<br /> &middot;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; All nodes open for R/W</td> <td> <p>Active/Active</p> <p>&middot;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Primary node open for R/W<br /> &middot;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Replica nodes open for reads</p></td> </tr> <tr> <td><strong>Cluster virtual IPs</strong></td> <td>R/W load balancing: SCAN IP</td> <td>R/W: Cluster endpoint<br /> +<br /> Read load balancing: Reader endpoint</td> </tr> <tr> <td><strong>Internode coordination</strong></td> <td>Cache-fusion + GCS + GES</td> <td>–</td> </tr> <tr> <td><strong>Internode private network</strong></td> <td>Interconnect</td> <td>–</td> </tr> <tr> <td><strong>Transaction (write) TTR from node failure</strong></td> <td>Typically 0–30 seconds</td> <td>Typically &lt; 30 Seconds</td> </tr> <tr> <td><strong>Application (Read) TTR from node failure</strong></td> <td>Immediate</td> <td>Immediate</td> </tr> <tr> <td><strong>Max number of cluster nodes</strong></td> <td>Theoretical maximum is 100, but smaller clusters (2 – 10 nodes) are far more common</td> <td>15</td> </tr> <tr> <td><strong>Provides built-in read scaling</strong></td> <td>Yes</td> <td>Yes</td> </tr> <tr> <td><strong>Provides built-in write scaling</strong></td> <td>Yes (with limitations*)</td> <td>No</td> </tr> <tr> <td><strong>Data loss in case of node failure</strong></td> <td>No data loss</td> <td>No data loss</td> </tr> <tr> <td><strong>Replication latency</strong></td> <td>–</td> <td>Milliseconds</td> </tr> <tr> <td><strong>Operational complexity</strong></td> <td>Requires database, IT, network and storage expertise</td> <td>Provided as a cloud-solution</td> </tr> <tr> <td><strong>Scale-up nodes</strong></td> <td>Difficult with physical hardware, usually requires to replace servers</td> <td>Easy using the AWS UI/CLI</td> </tr> <tr> <td><strong>Scale-out cluster</strong></td> <td>Provision, deploy, and configure new servers, unless you pre-allocate a pool of idle servers to scale-out on</td> <td>Easy using the AWS UI/CLI</td> </tr> <tr> <td><strong>Database engine</strong></td> <td>Oracle</td> <td>MySQL or PostgreSQL</td> </tr> </tbody> </table> <p style="text-align: center"><em>* Under certain scenarios, write performance can be limited and affect scale-out capabilities. For example, when multiple sessions attempt to modify rows contained in the same database block(s).<br /> </em></p> <p><span style="text-decoration: underline"><strong>Summary</strong></span><br /> With the increasing availability of cloud-based solutions, many organizations are looking for a relational database engine that can provide very high levels of availability, great performance and easy read scalability—but in a cloud/PaaS model. Amazon Aurora can serve as a powerful alternative solution to commercial database cluster technologies, using AWS ecosystem components and open source database engines to greatly reduce complexity and operational database management overhead.</p>