AWS Security Blog https://aws.amazon.com/blogs/security/ Mon, 14 Aug 2017 20:47:07 +0000 en-US hourly 1 AWS Announces Amazon Macie https://aws.amazon.com/blogs/security/aws-announces-amazon-macie/ Mon, 14 Aug 2017 20:43:50 +0000 032e6f2688fb420fa97ae67efe9e9c575b919920 I’m pleased to announce that today we’ve launched a new security service,&nbsp;Amazon Macie. This service leverages machine learning to help customers prevent data loss by automatically discovering, classifying, and protecting sensitive data in AWS. Amazon Macie recognizes sensitive data such as personally identifiable information (PII) or intellectual property, providing customers with dashboards and alerts that […] <p><img class="size-medium wp-image-4607 aligncenter" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/08/14/macie_blog-300x157.jpg" alt="" width="300" height="157" /></p> <p>I’m pleased to announce that today we’ve launched a new security service,&nbsp;<a title="undefined" href="https://aws.amazon.com/macie/" target="null">Amazon Macie</a>.</p> <p>This service leverages machine learning to help customers prevent data loss by automatically discovering, classifying, and protecting sensitive data in AWS. Amazon Macie recognizes sensitive data such as personally identifiable information (PII) or intellectual property, providing customers with dashboards and alerts that give visibility into how data is being accessed or moved.&nbsp;This enables customers to apply machine learning to a wide&nbsp;array&nbsp;of security and compliance workloads, we think this will be a significant enabler for our customers.</p> <p>To learn more about the <a title="undefined" href="https://aws.amazon.com/blogs/aws/launch-amazon-macie-securing-your-s3-buckets/" target="null">see&nbsp;the&nbsp;full AWS&nbsp;Blog post</a>.</p> <p>– &nbsp;Steve</p> <p>&nbsp;</p> How to Establish Federated Access to Your AWS Resources by Using Active Directory User Attributes https://aws.amazon.com/blogs/security/how-to-establish-federated-access-to-your-aws-resources-by-using-active-directory-user-attributes/ Tue, 08 Aug 2017 15:09:24 +0000 f1f0a286433e4480a650fbe3b146038c1574a042 To govern federated access to your AWS resources, it’s a common practice to use Microsoft Active Directory (AD) groups. When using AD groups, establishing federation requires the number of AD groups to be equal to the number of your AWS accounts multiplied by the number of roles in each of your AWS accounts. As you […] <p>To govern <a href="https://aws.amazon.com/iam/details/manage-federation/" target="_blank" rel="noopener noreferrer">federated access</a> to your AWS resources, it’s a common practice to use <a href="https://technet.microsoft.com/en-us/library/dn579255(v=ws.11).aspx" target="_blank" rel="noopener noreferrer">Microsoft Active Directory (AD) groups</a>. When using AD groups, establishing federation requires the number of AD groups to be equal to the number of your AWS accounts multiplied by the number of roles in each of your AWS accounts. As you can imagine, this can result in the creation of a very large number of AD groups to manage federated access.</p> <p>However, some organizations have limits on how many AD groups they can create. For example, an organization might need to keep its AD group hierarchy reasonably flat and avoid a deep nesting of groups. Such a situation needs a solution that doesn’t require you to create exponentially more AD groups while still allowing you to use access control and automated user integration.</p> <p>In this blog post, I provide step-by-step instructions for integrating <a href="https://aws.amazon.com/iam/" target="_blank" rel="noopener noreferrer">AWS Identity and Access Management (IAM)</a> with <a href="https://msdn.microsoft.com/en-us/library/bb897402.aspx" target="_blank" rel="noopener noreferrer">Microsoft Active Directory Federation Services</a> (AD&nbsp;FS) by using <a href="https://msdn.microsoft.com/en-us/library/ms680541.aspx" target="_blank" rel="noopener noreferrer">AD user attributes</a>, allowing you to establish federated access without expanding your total number of AD groups. Your organization’s enterprise administrator probably has existing processes in place for managing AD group memberships and provisioning, and you can extend these processes to the management of AD user attributes and the reduction of your organization’s dependency on AD groups.<span id="more-4464"></span></p> <h3>Prerequisites</h3> <p>This post assumes:</p> <ul> <li><a href="https://aws.amazon.com/blogs/security/enabling-federation-to-aws-using-windows-active-directory-adfs-and-saml-2-0/" target="_blank" rel="noopener noreferrer">You have a working AD directory and AD&nbsp;FS server</a>.</li> <li>You have <a href="https://aws.amazon.com/blogs/security/enabling-federation-to-aws-using-windows-active-directory-adfs-and-saml-2-0/" target="_blank" rel="noopener noreferrer">created an identity provider</a> (IdP) in your AWS account using your XML file (<span style="font-family: courier">https://&lt;your-server-name-here&gt;/FederationMetadata/2007-06/FederationMetadata.xml</span>) from your AD&nbsp;FS server. Remember the name of your IdP because you will use it later in this solution.</li> <li>You have created the appropriate <a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html" target="_blank" rel="noopener noreferrer">IAM roles</a> in your AWS account, which will be used for federated access.</li> </ul> <p>After you satisfy these prerequisites, you can proceed to the next section of this post to configure your AD users and AD&nbsp;FS server.</p> <h3>Solution overview</h3> <p>To benefit fully from the solution in this post, your AD and AD&nbsp;FS environment should look similar to what is shown in the following diagram. I focus this post on AD users and <a href="https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/technical-reference/the-role-of-claim-rules" target="_blank" rel="noopener noreferrer">claim rules</a> in an AD&nbsp;FS server. AD&nbsp;FS claim rules provide the logic to identify who has been correctly set up in AD with the appropriate user attributes to sign in via AD&nbsp;FS to the AWS Management Console.</p> <p><img class="alignnone size-full wp-image-4468" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/PierreLiddle-Diagram-072917-a.jpg" alt="" width="900" height="498" /></p> <p>In the preceding diagram:</p> <ol> <li>An AD user (let’s call him <span style="font-family: courier">Bob</span>) browses to the AD&nbsp;FS sample site (<span style="font-family: courier">https://<span style="color: #ff0000"><strong>Fully.Qualified.Domain.Name.Here</strong></span>/adfs/ls/IdpInitiatedSignOn.aspx</span>) inside this domain.</li> <li>The sign-in page authenticates <span style="font-family: courier">Bob</span> against AD. If <span style="font-family: courier">Bob</span> is already authenticated or using a domain joined workstation, he also might be prompted for his AD user name and password.</li> <li><span style="font-family: courier">Bob</span>’s browser receives a SAML assertion in the form of an authentication response from AD&nbsp;FS. <span style="font-family: courier">Bob</span>’s access is authorized based on his AD group membership or on AD user attributes configured on his account.</li> <li><span style="font-family: courier">Bob</span>’s browser automatically posts the SAML assertion to the AWS sign-in endpoint for SAML (<span style="font-family: courier">https://signin.aws.amazon.com/saml</span>). The endpoint uses the&nbsp;<a href="http://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithSAML.html" target="_blank" rel="noopener noreferrer">AssumeRoleWithSAML&nbsp;API</a> to request temporary security credentials and then constructs a sign-in URL for the AWS Management Console using those credentials.</li> <li><span style="font-family: courier">Bob</span>’s browser receives the sign-in URL and redirects to the AWS Management Console.</li> </ol> <h2>Deploy the solution</h2> <h3>A. &nbsp;Configure an AD user’s account</h3> <p>Because the AD user attributes hold all the associated AWS account and role information when using this solution, you will start by configuring an AD user’s accounts.</p> <p>To edit the user attributes in an AD user’s account:</p> <ol> <li>On your AD server, in the Active Directory Users and &nbsp;Computers console, go to <strong>View</strong> &gt; <strong>Advanced Features</strong> in <strong>Active Directory Users and Computers</strong> to see the <strong>Attribute editor </strong>tab.<br /> <img class="alignnone wp-image-4586 size-full" title="Screenshot showing &quot;Advanced Features&quot; in the &quot;View&quot; menu" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/08/06/PL-080817-1.png" alt="Screenshot showing &quot;Advanced Features&quot; in the &quot;View&quot; menu" width="478" height="278" /></li> <li>For AD user <span style="font-family: courier">Bob</span>, edit one attribute using the built-in AD attribute editor. The attribute I am using is <span style="font-family: courier">url</span>, which is a multi-valued string. If you use another AD user attribute, consider how you will need to modify your AD&nbsp;FS claim rules later because different attributes may return the values differently back to the AD&nbsp;FS server. The name of the AD user attribute will be used in the AD&nbsp;FS claim rules later in this post.</li> <li>Bob has two AWS accounts: <span style="font-family: courier">111122223333</span> and <span style="font-family: courier">444455556666</span>. Each of <span style="font-family: courier">Bob</span>’s AWS accounts has two roles: <span style="font-family: courier">AWS-Dev</span> and <span style="font-family: courier">AWS-ReadOnly</span>. I have configured <span style="font-family: courier">Bob</span>’s <span style="font-family: courier">url</span> attribute with the corresponding values associated with his AWS accounts and roles. As part of the attribute entries, I prefixed each entry with <span style="font-family: courier">AWS-</span> to have a unique identifier. As shown in the following screenshot, I added the entries one at a time so that each value can be returned back to my AD&nbsp;FS server: <ul style="margin-left: 30px"> <li><span style="font-family: courier">AWS-111122223333-Dev</span></li> <li><span style="font-family: courier">AWS-111122223333-ReadOnly</span></li> <li><span style="font-family: courier">AWS-444455556666-Dev</span></li> <li><span style="font-family: courier">AWS-444455556666-ReadOnly</span></li> </ul> </li> </ol> <p style="padding-left: 60px"><img class="alignnone wp-image-4584 size-full" title="Screenshot of Bob’s &quot;url&quot; attribute with the corresponding values associated with his AWS accounts and roles" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/08/06/PL-080817-2.png" alt="Screenshot of Bob’s &quot;url&quot; attribute with the corresponding values associated with his AWS accounts and roles" width="832" height="577" /></p> <ol start="4"> <li><span style="font-family: courier">Bob</span> also requires an email address because that information will be used in the role session name when <span style="font-family: courier">Bob</span> signs in to the AWS Management Console via his chosen AWS account and associated role. We use <span style="font-family: courier">Bob</span>’s email address only because it’s a common user attribute most users have and is also unique across users. The unique identifier is then forwarded by AD&nbsp;FS to be used by AWS as a unique value for users. If you have enabled <a href="https://aws.amazon.com/cloudtrail/" target="_blank" rel="noopener noreferrer">AWS CloudTrail</a>, the role session name is captured in CloudTrail and allows for ease of identification of who assumed the role and subsequent API calls the user or role might have executed on the platform (for example, <span style="font-family: courier">ec2:terminateinstance</span>).<br /> <img class="alignnone wp-image-4520 size-full" title="Screenshot showing Bob's email address" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/31/Bob-Email-Address.png" alt="Screenshot showing Bob's email address" width="425" height="564" /></li> </ol> <p>Now that you have configured <span style="font-family: courier">Bob</span>’s account, you will configure the AD&nbsp;FS server claim rules.</p> <h3>B. &nbsp;Configure the AD&nbsp;FS server claim rules</h3> <p>Because this blog post assumes your environment is already up and running and to ensure that you can follow along, I am providing <a href="https://s3.amazonaws.com/awsiammedia/public/sample/ActiveDirectoryUserAttributes/AWS_ADFS_Blog_Sample-Code.zip" target="_blank" rel="noopener noreferrer">example Windows PowerShell code</a> that you can run on your AD&nbsp;FS server. This code allows you to choose a conventional approach by using AD groups in AD&nbsp;FS claim rules, or for the purposes of this post, to use AD&nbsp;FS claim rules with AD user attributes. If you use the AD group approach on your AD&nbsp;FS server with the example code, your AD group naming convention must be: <span style="font-family: courier">AWS-<strong>YourAccountNumber</strong>–<strong>YourRoleName</strong></span>. If you have already created claim rules for AWS on your AD&nbsp;FS server, I encourage you to run this code against a different AD&nbsp;FS server that has no existing AWS rules.</p> <p>To configure the AD&nbsp;FS claim rules:</p> <ol> <li>Open the AD&nbsp;FS console. You can find it by searching for <span style="font-family: courier">ad</span>, as shown in the following screenshot.<br /> <img class="alignnone wp-image-4587 size-full" title="Screenshot of searching for the AD FS console" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/08/06/PL-080817-4.png" alt="Screenshot of searching for the AD FS console" width="344" height="204" /></li> </ol> <ol start="2"> <li>Expand <strong>Trust Relationships</strong> and choose <strong>Relying Party Trusts.<br /> <img class="alignnone wp-image-4588 size-full" title="Screenshot of the &quot;Relying Party Trusts&quot; folder" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/08/06/PL-080817-5.png" alt="Screenshot of the &quot;Relying Party Trusts&quot; folder" width="950" height="216" /><br /> </strong></li> </ol> <ol start="3"> <li>Run the <a href="https://s3.amazonaws.com/awsiammedia/public/sample/ActiveDirectoryUserAttributes/AWS_ADFS_Blog_Sample-Code.zip" target="_blank" rel="noopener noreferrer">example Windows PowerShell code</a> from the command prompt in the same directory where you extracted the .zip file. The following screenshot shows a list of the example files from the .zip file.<br /> <img class="alignnone wp-image-4484 size-full" title="Screenshot of some example files from the supplied .zip file" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/PierreLiddle-Image5-072917-a.png" alt="Screenshot of some example files from the supplied .zip file" width="899" height="205" /></li> </ol> <ol start="4"> <li>Run the <strong>01-Configure-ADFS-AD-User-URL-mapping.ps1</strong> Windows PowerShell script to set up the AD&nbsp;FS claim rules. <strong>Note:</strong> Run this script with Administrative permissions. A&nbsp;log file is generated to which you can refer, as shown in the following screenshot.<br /> <img class="alignnone wp-image-4485 size-full" title="Screenshot showing the log file that is generated" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/PierreLiddle-Image6-072917.png" alt="Screenshot showing the log file that is generated" width="893" height="66" /></li> </ol> <ol start="5"> <li>After you run the Windows PowerShell script, you will see the new relying party trust that has been created in your AD&nbsp;FS configuration for Amazon Web Services, as shown in the following screenshot.<br /> <img class="alignnone wp-image-4589 size-full" title="Screenshot of the new relying party trust that was created by the script" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/08/06/PL-080817-8.png" alt="Screenshot of the new relying party trust that was created by the script" width="946" height="217" /></li> </ol> <ol start="6"> <li>Right-click <strong>Amazon Web Services &amp; AD User URL</strong> and choose <strong>Edit Claim Rules</strong>.<br /> <img class="alignnone wp-image-4590 size-full" title="Screenshot of choosing &quot;Edit Claim Rules&quot;" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/08/06/PL-080817-9.png" alt="Screenshot of choosing &quot;Edit Claim Rules&quot;" width="676" height="314" /></li> </ol> <ol start="7"> <li>The following screenshot shows what your AD&nbsp;FS server claim rules should look like now.<br /> <img class="alignnone wp-image-4591 size-full" title="Screenshot of how the cliam rules should look" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/08/06/PL-080817-10.png" alt="Screenshot of how the cliam rules should look" width="511" height="187" /></li> </ol> <p>About these four claim rules:</p> <ol> <li>Claim rule 1 captures the Windows account name of the current user whose attributes will then be queried further with claim rule 3.</li> <li>Claim rule 2 captures <span style="font-family: courier">Bob</span>’s email address for use in the role session name.</li> <li>Claim rule 3 queries the current user’s URL attributes to identify which account and role the user is authorized to assume access to. These URL attribute values are then stored in a variable (<span style="font-family: courier">http://temp/variable</span>) for use in claim rule 4.</li> <li>Claim rule 4 works by matching the first pattern match, <span style="font-family: courier">([^d]{12})</span>, to <span style="font-family: courier">$1</span> and the second pattern match, <span style="font-family: courier">(\w*)</span>, to <span style="font-family: courier">$2</span> for each entry in <span style="font-family: courier">http://temp/variable</span>. With this final rule, you define the resulting value for the AWS role attribute in a dynamic way, which allows the configuration to scale to support virtually&nbsp;any number of AWS accounts and IAM roles without further configuration within AD&nbsp;FS. By using these claim rules, you query, store, and then convert the values in the URL attributes to the <a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_saml_assertions.html" target="_blank" rel="noopener noreferrer">IAM role attributes</a> that AWS expects.<img class="alignnone wp-image-4592 size-full" title="Screenshot of editing claim rule 4" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/08/06/PL-080817-11.png" alt="Screenshot of editing claim rule 4" width="891" height="491" /></li> </ol> <p>At the beginning of this post, I mentioned that you need to remember the name of the IdP you created in your AWS account, and now is when you will use your IdP’s name. Replace <span style="font-family: courier">myADFS</span>, <mark>highlighted</mark> in the following code, with the name of your IdP. (When modifying the rules, be careful not to insert any additional spaces because they can cause claim rules to not work as designed.)</p> <div class="hide-language"> <pre><code class="lang-text"><strong> 4)RuleName &quot; Dynamic ARN - Adding Roles &quot;</strong> c:[Type == &quot;http://temp/variable&quot;, Value =~ &quot;(?i)^AWS-([^d]{12})-(\w*)&quot;] =&gt; issue(Type = &quot;https://aws.amazon.com/SAML/Attributes/Role&quot;, Value = RegExReplace(c.Value, &quot;AWS-([^d]{12})-(\w*)&quot;, &quot;arn:aws:iam::$1:saml-provider/<mark>myADFS</mark>,arn:aws:iam::$1:role/AWS-$2&quot;));</code></pre> </div> <h3>C. &nbsp;Test AD user Bob’s federated access</h3> <p>Go to the AD&nbsp;FS sign-in page (<span style="font-family: courier">https://<span style="color: #ff0000"><strong>Fully.Qualified.Domain.Name.Here</strong></span>/adfs/ls/IdpInitiatedSignOn.aspx</span>) to test <span style="font-family: courier">Bob</span>’s federated access. Note that you might see a certificate warning if the server uses a locally self-signed certificate from <a href="https://www.iis.net/" target="_blank" rel="noopener noreferrer">Internet Information Services</a>.</p> <p>To test <span style="font-family: courier">Bob</span>’s federated access:</p> <ol> <li>Choose&nbsp;<strong>Sign in to one of the following sites</strong>, choose&nbsp;<strong>Amazon Web Services</strong><strong>&amp; AD User URL</strong> from the list, and then choose&nbsp;<strong>Continue to Sign In</strong>.</li> <li>If prompted, type <span style="font-family: courier">Bob</span>’s user name and password. You will be redirected to sign in to the&nbsp;<strong>Amazon Web Services AD&nbsp;FS </strong>page previously defined when you set up the AD&nbsp;FS relying party trusts.<br /> <img class="alignnone wp-image-4496" title="Screenshot of the sign-in page" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/PierreLiddle-Image10-072917-a.jpg" alt="Screenshot of the sign-in page" width="400" height="411" /></li> </ol> <ol start="3"> <li>After you authenticate to the server as <span style="font-family: courier">Bob</span>, your browser is redirected to <span style="font-family: courier">https://signin.aws.amazon.com/saml</span>, and you can choose which of <span style="font-family: courier">Bob</span>’s accounts and roles to use from. Choose a role and then choose&nbsp;<strong>Sign In</strong>.<br /> <img class="alignnone wp-image-4497 size-full" title="Screenshot showing Bob's roles and accounts to choose from" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/PierreLiddle-Image11-072917-a.jpg" alt="Screenshot showing Bob's roles and accounts to choose from" width="500" height="409" /></li> </ol> <ol start="4"> <li>You have signed in as <span style="font-family: courier">Bob</span>, and his email address now appears as part of the role session name, as shown in the following screenshot.<br /> <img class="alignnone wp-image-4512 size-full" title="Screenshot showing Bob's email address as part of the role session name" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/31/2017-07-31_7-37-47-a.png" alt="Screenshot showing Bob's email address as part of the role session name" width="814" height="205" /></li> </ol> <p>You can now see <span style="font-family: courier">Bob</span>’s email address used in the role session name. If you have enabled CloudTrail, the role session name is captured in CloudTrail and allows you to easily identify who assumed the role. If <span style="font-family: courier">Bob</span> wants to switch to a different account or role, he can return to his AD&nbsp;FS sign-in page (<span style="font-family: courier">https://<span style="color: #ff0000"><strong>Fully.Qualified.Domain.Name.Here</strong></span>/adfs/ls/IdpInitiatedSignOn.aspx</span>) and choose an alternative account or role.</p> <h3>Summary</h3> <p>In this blog post, I demonstrated how to use dynamic resolution of federated access using AD user attributes to scale your configuration and support a large number&nbsp;of AWS accounts and associated IAM roles. This is a powerful technique for managing a large number of AWS accounts and the federated access of associated AD users. Even though I demonstrate the integration of IAM with AD&nbsp;FS and AD, you can replicate this solution across your choice of SAML federated access technology, such as <a href="https://en.wikipedia.org/wiki/Shibboleth_(Internet2)" target="_blank" rel="noopener noreferrer">Shibboleth</a> or <a href="https://en.wikipedia.org/wiki/OpenLDAP" target="_blank" rel="noopener noreferrer">OpenLDAP</a>.</p> <p>If you have comments about this blog post, submit them in the “Comments” section below. If you have implementation or troubleshooting questions, start a new thread on the&nbsp;<a href="https://forums.aws.amazon.com/forum.jspa?forumID=76" target="_blank" rel="noopener noreferrer">IAM forum</a>.</p> <p>– Pierre</p> AWS Encryption SDK: How to Decide if Data Key Caching Is Right for Your Application https://aws.amazon.com/blogs/security/aws-encryption-sdk-how-to-decide-if-data-key-caching-is-right-for-your-application/ Mon, 07 Aug 2017 14:44:33 +0000 306c6dd1cf66c5cc3bd1516e96016a7455bb3daf Today, the AWS Crypto Tools team introduced a new feature in the AWS Encryption SDK: data key caching. Data key caching lets you reuse the data keys that protect your data, instead of generating a new data key for each encryption operation. Data key caching can reduce latency, improve throughput, reduce cost, and help you […] <p style="text-align: center"><img class="alignnone wp-image-4380 size-full" title="AWS KMS image" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/26/KMS_feature.png" alt="AWS KMS image" width="250" height="250" /></p> <p>Today, the AWS Crypto Tools team introduced a new feature in the <a href="http://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/introduction.html" target="_blank" rel="noopener noreferrer">AWS Encryption SDK</a>: <em>data key caching</em>. <a href="https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/data-key-caching.html" target="_blank" rel="noopener noreferrer">Data key caching</a> lets you reuse the data keys that protect your data, instead of generating a new data key for each encryption operation.</p> <p>Data key caching can reduce latency, improve throughput, reduce cost, and help you stay within service limits as your application scales. In particular, caching might help if your application is hitting the <a href="https://aws.amazon.com/kms/" target="_blank" rel="noopener noreferrer">AWS Key Management Service</a> (KMS) <a href="http://docs.aws.amazon.com/kms/latest/developerguide/limits.html#requests-per-second" target="_blank" rel="noopener noreferrer">requests-per-second limit</a>&nbsp;and <a href="https://console.aws.amazon.com/support/home" target="_blank" rel="noopener noreferrer">raising the limit</a> does not solve the problem.</p> <p>However, these benefits come with some security tradeoffs. Encryption best practices generally discourage extensive reuse of data keys.</p> <p>In this blog post, I explore those tradeoffs and provide information that can help you decide whether data key caching is a good strategy for your application. I also explain how data key caching is implemented in the AWS Encryption SDK and describe the security thresholds that you can set to limit the reuse of data keys. Finally, I provide some practical examples of using the security thresholds to meet cost, performance, and security goals.</p> <h3><strong>Introducing data key caching</strong></h3> <p>The AWS Encryption SDK is a client-side encryption library that makes it easier for you to implement cryptography best practices in your application. It includes secure default behavior for developers who are not encryption experts, while being flexible enough to work for the most experienced users.<span id="more-3955"></span></p> <p>In the AWS Encryption SDK, by default, you generate a new data key for each&nbsp;encryption operation. This is the most secure practice. However, in some applications, the overhead of generating a new data key for each operation is not acceptable.</p> <p>Data key caching saves the plaintext and ciphertext of the data keys you use in a configurable cache. When you need a key to encrypt or decrypt data, you can reuse a data key from the cache instead of creating a new data key. You can create multiple data key caches and configure each one independently. Most importantly, the AWS Encryption SDK provides <a href="https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/thresholds.html" target="_blank" rel="noopener noreferrer">security thresholds</a> that you can set to determine how much data key reuse you will allow.</p> <p>To make data key caching easier to implement, the AWS Encryption SDK provides <a href="https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/data-key-caching.html#simplecache" target="_blank" rel="noopener noreferrer">LocalCryptoMaterialsCache</a>, an in-memory, least-recently-used&nbsp;cache with a configurable size. The SDK manages the cache for you, including adding store, search, and match logic to all encryption and decryption operations.</p> <p>We recommend that you use LocalCryptoMaterialsCache as it is, but you can customize it, or substitute a compatible cache. However, you should never store plaintext data keys on disk.</p> <p>The AWS Encryption SDK documentation includes <a href="https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/sample-cache-example.html" target="_blank" rel="noopener noreferrer">sample code</a> in Java and Python for an application that uses data key caching to encrypt data sent to and from <a href="https://aws.amazon.com/kinesis/streams/" target="_blank" rel="noopener noreferrer">Amazon Kinesis Streams</a>.</p> <h3><strong>Balance cost and security</strong></h3> <p>Your decision to use data key caching should balance cost—in time, money, and resources—against security. In every consideration, though, the balance should favor your security requirements. As a rule, use the minimal caching required to achieve your cost and performance goals.</p> <p>Before implementing data key caching, consider the details of your applications, your security requirements, and the cost and frequency of your encryption operations. In general, your application can benefit from data key caching if each operation is slow or expensive, or if you encrypt and decrypt data frequently. If the cost and speed of your encryption operations are already acceptable or can be improved by other means, do not use a data key cache.</p> <p>Data key caching can be the right choice for your application if you have high encryption and decryption traffic. For example, if you are hitting your KMS&nbsp;requests-per-second limit, caching can help because you get some of your data keys from the cache instead of calling KMS for every request.</p> <p>However, you can also create a case in the <a href="https://console.aws.amazon.com/support/home" target="_blank" rel="noopener noreferrer">AWS Support Center</a> to raise the KMS limit for your account.&nbsp;If raising the limit solves the problem, you do not need data key caching.</p> <h3><strong>Configure caching thresholds for cost and security</strong></h3> <p>In the AWS Encryption SDK, you can configure data key caching to allow just enough data key reuse to meet your cost and performance targets while conforming&nbsp;to the security requirements of your application. The SDK enforces the thresholds&nbsp;so that you can use them with any compatible cache.</p> <p>The data key caching security thresholds apply to each cache entry. The AWS Encryption SDK will not use the data key from a cache entry that exceeds any of the thresholds that you set.</p> <ul> <li><strong>Maximum age </strong>(required): Set the lifetime of each cached key to be long enough to get cache hits, but short enough to limit exposure of a plaintext data key in memory to a specific time period.</li> </ul> <p style="padding-left: 30px">You can use the maximum age threshold like a key rotation policy. Use it to limit the reuse of data keys and minimize exposure of cryptographic materials. You can also use it to evict data keys when the type or source of data that your application is processing changes.</p> <ul> <li><strong>Maximum messages encrypted</strong> (optional; default is 2<sup>32</sup> messages): Set the number of messages protected by each cached data key to be large enough to get value from reuse, but small enough to limit the number of messages that might potentially be exposed.</li> </ul> <p style="padding-left: 30px">The AWS Encryption SDK only caches data keys that use an <a href="https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/supported-algorithms.html" target="_blank" rel="noopener noreferrer">algorithm suite</a> with a <a href="https://en.wikipedia.org/wiki/Key_derivation_function" target="_blank" rel="noopener noreferrer">key derivation function</a>. This technique avoids the cryptographic limits on the number of bytes encrypted with a single key. However, the more data that a key encrypts, the more data that is exposed if the data key is compromised.</p> <p style="padding-left: 30px">Limiting the number of messages, rather than the number of bytes, is particularly useful if your application encrypts many messages of a similar size or when potential exposure must be limited to very few messages. This threshold is also useful when you want to reuse a data key for a particular type of message and know in advance how many messages of that type you have. You can also use an <a href="https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/data-key-caching.html#caching-encryption-context" target="_blank" rel="noopener noreferrer">encryption context</a> to select particular cached data keys for your encryption requests.</p> <ul> <li><strong>Maximum bytes encrypted</strong> (optional; default is 2<sup>63</sup> – 1): Set the bytes protected by each cached data key to be large enough to allow the reuse you need, but small enough to limit the amount of data encrypted under the same key.</li> </ul> <p style="padding-left: 30px">Limiting the number of bytes, rather than the number of messages, is preferable when your application encrypts messages of widely varying size or when possibly exposing large amounts of data is much more of a concern than exposing smaller amounts of data.</p> <p>In addition to these security thresholds, the LocalCryptoMaterialsCache in the AWS Encryption SDK lets you set its <em>capacity</em>, which is the maximum number of entries the cache can hold.</p> <p>Use the capacity value to tune the performance of your LocalCryptoMaterialsCache. In general, use the smallest value that will achieve the performance improvements that your application requires. You might want to test with a very small cache of 5–10 entries and expand if necessary. You will need a slightly larger cache if you are using the cache for both encryption and decryption requests, or if you are using encryption contexts to select particular cache entries.</p> <h3><strong>Consider these cache configuration examples</strong></h3> <p>After you determine the security and performance requirements of your application, consider the cache security thresholds carefully and adjust them to meet your needs. There are no magic numbers for these thresholds: the ideal settings are specific to each application, its security and performance requirements, and budget. Use the minimal amount of caching necessary to get acceptable performance and cost.</p> <p>The following examples show ways you can use the LocalCryptoMaterialsCache capacity setting and the security thresholds to help meet your security requirements:</p> <ul> <li><strong>Slow master key operations: </strong>If your master key processes only 100 transactions per second (TPS) but your application needs to process 1,000 TPS, you can meet your application requirements by allowing a maximum of 10 messages to be protected under each data key.</li> <li><strong>High frequency and volume:</strong> If your master key costs $0.01 per operation and you need to process a consistent 1,000 TPS while staying within a budget of $100,000 per month, allow a maximum of 275 messages for each cache entry.</li> <li><strong>Burst traffic: </strong>If your application’s processing bursts to 100 TPS for five seconds in each minute but is otherwise zero, and your master key costs $0.01 per operation, setting maximum messages to 3 can achieve significant savings. To prevent data keys from being reused across bursts (55 seconds), set the maximum age of each cached data key to 20 seconds.</li> <li><strong>Expensive master key operations:</strong> If your application uses a low-throughput encryption service that costs as much as $1.00 per operation, you might want to minimize the number of operations. To do so, create a cache that is large enough to contain the data keys you need. Then, set the byte and message limits high enough to allow reuse while conforming to your security requirements. For example, if your security requirements do not permit a data key to encrypt more than 10 GB of data, setting bytes processed to 10 GB still significantly minimizes operations and conforms to your security requirements.</li> </ul> <h3><strong>Learn more about data key caching</strong></h3> <p>To learn more about data key caching, including how to implement it, how to set the security thresholds, and details about the caching components, see <a href="https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/data-key-caching.html" target="_blank" rel="noopener noreferrer">Data Key Caching</a> in the <a href="https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/introduction.html" target="_blank" rel="noopener noreferrer">AWS Encryption SDK</a>. Also, see the AWS Encryption SDKs for&nbsp;<a href="https://github.com/awslabs/aws-encryption-sdk-java" target="_blank" rel="noopener noreferrer">Java</a> and <a href="https://github.com/awslabs/aws-encryption-sdk-python" target="_blank" rel="noopener noreferrer">Python</a> as well as the <a href="https://awslabs.github.io/aws-encryption-sdk-java/javadoc/" target="_blank" rel="noopener noreferrer">Javadoc</a> and <a href="http://aws-encryption-sdk-python.readthedocs.io/en/latest/" target="_blank" rel="noopener noreferrer">Python documentation</a>.</p> <p>If you have comments about this blog post, submit them in the “Comments” section below. If you have questions, file an issue in the GitHub repos for the Encryption SDK in <a href="https://github.com/awslabs/aws-encryption-sdk-java" target="_blank" rel="noopener noreferrer">Java</a> or <a href="https://github.com/awslabs/aws-encryption-sdk-python" target="_blank" rel="noopener noreferrer">Python</a>, or start a new thread on the <a href="https://forums.aws.amazon.com/forum.jspa?forumID=182" target="_blank" rel="noopener noreferrer">KMS forum</a>.</p> <p>– June</p> The First AWS Regional Financial Services Guide Focuses on Singapore https://aws.amazon.com/blogs/security/the-first-aws-regional-financial-services-guide-focuses-on-singapore/ Thu, 03 Aug 2017 23:00:05 +0000 13e7eba60f9e545493c5e347dccd69689686eec3 To help Financial Services clients address Singapore’s regulations on financial institutions in a shared responsibility environment, AWS has published the&nbsp;AWS User Guide to Financial Services Regulations and Guidelines in Singapore. This first-ever AWS Financial Services guide is the culmination of the work AWS has done in the last year to help customers navigate the Monetary […] <p style="text-align: center"><img class="alignnone wp-image-4389 size-full" title="Financial Services image" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/27/financial-services_banking_payments_large.png" alt="Financial Services image" width="601" height="301" /></p> <p>To help Financial Services clients address Singapore’s regulations on financial institutions in a <a href="https://aws.amazon.com/compliance/shared-responsibility-model/" target="_blank" rel="noopener noreferrer">shared responsibility environment</a>, AWS has published the&nbsp;<a href="https://d0.awsstatic.com/whitepapers/compliance/Financial_Services_Regulations_Guidelines_in_Singapore.pdf" target="_blank" rel="noopener noreferrer">AWS User Guide to Financial Services Regulations and Guidelines in Singapore</a>. This first-ever AWS Financial Services guide is the culmination of the work AWS has done in the last year to help customers navigate the Monetary Authority of Singapore’s 2016 updated guidelines about cloud services.</p> <p>This new guide examines Singaporean requirements and guidelines, providing information that will help you conduct due diligence on AWS with regard to IT security and risk management. The guide also shares leading practices to empower you to develop your own governance programs by using AWS.</p> <p>The guide focuses on three top considerations for financial institutions operating in Singapore:</p> <ul> <li><strong>Outsourcing guidelines</strong> – Conduct a self-assessment of AWS services and align your governance requirements within a shared responsibility model.</li> <li><strong>Technology risk management</strong> – Take a deeper look at where shared responsibility exists for technology implementation and perform a self-assessment of AWS service responsibilities.</li> <li><strong>Cloud computing implementation</strong> – Assess additional responsibilities to ensure security and compliance with local guidelines.</li> </ul> <p>We will release additional AWS Financial Services resource guides this year to help you understand the requirements in other markets around the globe. These guides will be posted on the <a href="https://aws.amazon.com/compliance/resources/" target="_blank" rel="noopener noreferrer">AWS Compliance Resources</a> page.</p> <p>If you have questions or comments about this new guide, submit them in the “Comments” section below.</p> <p>– Jodi</p> Announcing the New AWS Customer Compliance Center https://aws.amazon.com/blogs/security/announcing-the-new-customer-compliance-center/ Thu, 03 Aug 2017 01:57:20 +0000 33533000accce830fda1e255e45936bcfca79b50 AWS has the longest running, most effective, and most customer-obsessed compliance program in the cloud market. We have always centered our program around customers, obtaining the certifications needed to provide our customers with the proper level of validated transparency in order to enable them to certify their own AWS workloads [download .pdf of AWS certifications]. […] <p><img class="size-full wp-image-4569 aligncenter" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/08/02/AWSComplianceLogoBlue.png" alt="" width="190" height="150" /></p> <p>AWS has the longest running, most effective, and most customer-obsessed compliance program in the cloud market. We have always centered our program around customers, obtaining the certifications needed to provide our customers with the proper level of validated transparency in order to enable them to certify their own AWS workloads [<a href="https://d0.awsstatic.com/whitepapers/compliance/AWS_Certifications_Programs_Reports_Third-Party_Attestations.pdf">download .pdf of AWS certifications</a>]. We also offer a rich suite of embedded compliance tooling, enabling customers and partners to more effectively manage security controls and in turn provide evidence of effective control operation to their auditors. Along with our customers and partners, we have the largest, most diverse, and most comprehensive compliance footprint in the industry.</p> <p>Enabling customers is a core part of the AWS DNA. Today, in the spirit of that pedigree, I’m happy to announce we’ve launched a new<a href="https://aws.amazon.com/compliance/customer-center/"> AWS Customer Compliance Center</a>. This center is focused on the security and compliance of our customers on AWS. You can learn from other customer experiences and discover how your peers have solved the compliance, governance, and audit challenges present in today’s regulatory environment. You can also access our industry-first <a href="https://aws.amazon.com/compliance/auditor-learning-path/">cloud Auditor Learning Path</a> via the&nbsp;customer center. These online university learning resources are logical learning paths, specifically designed for security, compliance and audit professionals, allowing you to build on the IT skills you have to move your environment to the next generation of audit and security assurance. As we engage with our security and compliance customer colleagues on this topic, we will continue to update and improve upon the existing resource and publish new enablers in the coming months.</p> <p>We are excited to continue to work with our customers on moving from the old-guard manual audit world to the new cloud-enabled, automated, “secure and compliant by default” model we’ve been leading over the past few years.</p> <p>– Chad Woolf, AWS Security &amp; Compliance</p> Newly Updated: Example AWS IAM Policies for You to Use and Customize https://aws.amazon.com/blogs/security/newly-updated-example-policies-for-you-to-use-and-customize/ Wed, 02 Aug 2017 15:56:10 +0000 437c7b90078c4bc3a1aecae92009612628cb5303 To help you grant access to specific resources and conditions, the&nbsp;Example Policies&nbsp;page in the AWS Identity and Access Management (IAM) documentation now includes more than thirty policies for you to use or customize to meet your permissions requirements. The AWS Support team developed these policies from their experiences working with AWS customers over the years. […] <p>To help you grant access to specific resources and conditions, the&nbsp;<a href="https://docs.aws.amazon.com/console/iam/example-policies" target="_blank" rel="noopener noreferrer">Example Policies</a>&nbsp;page in the AWS Identity and Access Management (IAM) documentation now includes more than thirty policies for you to use or customize to meet your permissions requirements. The AWS Support team developed these policies from their experiences working with AWS customers over the years. The example policies cover common permissions use cases you might encounter across services such as <a href="https://aws.amazon.com/dynamodb/" target="_blank" rel="noopener noreferrer">Amazon DynamoDB</a>, <a href="https://aws.amazon.com/ec2/" target="_blank" rel="noopener noreferrer">Amazon EC2</a>, <a href="http://aws.amazon.com/elasticbeanstalk" target="_blank" rel="noopener noreferrer">AWS Elastic Beanstalk</a>, <a href="http://aws.amazon.com/rds/" target="_blank" rel="noopener noreferrer">Amazon RDS</a>, <a href="https://aws.amazon.com/s3/" target="_blank" rel="noopener noreferrer">Amazon S3</a>, and IAM.</p> <p>In this blog post, I introduce the updated Example Policies page and explain how to use and customize these policies for your needs.</p> <h3>The new Example Policies page</h3> <p>The Example Policies page in the IAM User Guide now provides an overview of the example policies and includes a link to view each policy on a separate page. Note that each of these policies has been reviewed and approved by AWS Support. If you would like to submit a policy that you have found to be particularly useful, post it on the <a href="https://forums.aws.amazon.com/forum.jspa?forumID=76" target="_blank" rel="noopener noreferrer">IAM forum</a>.<span id="more-4089"></span></p> <p>To give you an idea of the policies we have included on this page, the following are a few of the EC2 policies on the page:</p> <ul> <li><a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_ec2_ebs-owner.html" target="_blank" rel="noopener noreferrer">Attach or Detach Amazon EBS Volumes to EC2 Instances Based on Tags</a></li> <li><a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_ec2_region.html" target="_blank" rel="noopener noreferrer">Allows Full EC2 Access Within a Specific Region, Programmatically and in the Console</a></li> <li><a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_ec2_instances-subnet.html" target="_blank" rel="noopener noreferrer">Allows Launching EC2 Instances in a Specific Subnet, Programmatically and in the Console</a></li> </ul> <p>To see the full list of available policies, see the <a href="https://docs.aws.amazon.com/console/iam/example-policies" target="_blank" rel="noopener noreferrer">Example Polices</a> page.</p> <p>In the following section, I demonstrate how to use a policy from the Example Policies page and customize it for your needs.</p> <h3>How to customize an example policy for your needs</h3> <p>Suppose you want to allow an IAM user, <span style="font-family: courier">Bob</span>, to start and stop EC2 instances with a specific&nbsp;<a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html" target="_blank" rel="noopener noreferrer">resource tag</a>. After looking through the Example Policies page, you see the policy, <a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_ec2_tag-owner.html" target="_blank" rel="noopener noreferrer">Allows Starting or Stopping EC2 Instances a User Has Tagged, Programmatically and in the Console</a>.</p> <p>To apply this policy to your specific use case:</p> <ol> <li>Navigate to the <strong>Policies</strong> section of the <a href="https://console.aws.amazon.com/iam/home" target="_blank" rel="noopener noreferrer">IAM console</a>.</li> <li>Choose <strong>Create policy</strong>.<br /> <img class="alignnone wp-image-4092 size-full" title="Screenshot of choosing &quot;Create policy&quot;" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/06/29/createpolicy-DS-July2017.png" alt="Screenshot of choosing &quot;Create policy&quot;" width="384" height="262" /><strong><br /> </strong></li> <li>Choose the <strong>Select</strong> button next to <strong>Create Your Own Policy</strong>. You will see an empty policy document with boxes for <strong>Policy Name</strong>, <strong>Description</strong>, and <strong>Policy Document</strong>, as shown in the following screenshot.<br /> <img class="alignnone wp-image-4184 size-full" title="Screenshot of an empty policy document" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/07/reviewpolicy-DS-July2017-c.png" alt="" width="762" height="429" /></li> <li>Type a name for the policy, copy the policy from the&nbsp;<a href="https://docs.aws.amazon.com/console/iam/example-policies" target="_blank" rel="noopener noreferrer">Example Policies</a>&nbsp;page, and paste the policy in the <strong>Policy Document</strong> box. In this example, I use “start-stop-instances-for-owner-tag” as the policy name and “Allows users to start or stop instances if the instance tag Owner has the value of their user name” as the description.</li> <li>Update the <strong>placeholder text</strong> in the policy (see the full policy that follows this step). For example, replace <strong>&lt;REGION&gt;</strong> with a region from&nbsp;<a href="http://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region" target="_blank" rel="noopener noreferrer">AWS Regions and Endpoints</a> and <strong>&lt;ACCOUNTNUMBER&gt; </strong>with your <a href="http://docs.aws.amazon.com/general/latest/gr/acct-identifiers.html#FindingYourAccountIdentifiers" target="_blank" rel="noopener noreferrer">12-digit account number</a>. The IAM policy variable, <span style="font-family: courier">${aws:username}</span>, is a dynamic property in the policy that automatically applies to the user to which it is attached. For example, when the policy is attached to <span style="font-family: courier">Bob</span>, the policy replaces <span style="font-family: courier">${aws:username}</span> with <span style="font-family: courier">Bob</span>. If you do not want to use the key value pair of <span style="font-family: courier">Owner</span> and <span style="font-family: courier">${aws:username}</span>, you can edit the policy to include your desired key value pair. For example, if you want to use the key value pair, <span style="font-family: courier">CostCenter:1234</span>, you can modify <span style="font-family: courier">“ec2:ResourceTag/Owner”: “${aws:username}”</span> to <span style="font-family: courier">“ec2:ResourceTag/CostCenter”: “1234”</span>. <div class="hide-language"> <pre><code class="lang-text">{ &nbsp;&nbsp;&nbsp; &quot;Version&quot;: &quot;2012-10-17&quot;, &nbsp;&nbsp;&nbsp; &quot;Statement&quot;: [ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &quot;Effect&quot;: &quot;Allow&quot;, &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &quot;Action&quot;: [ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &quot;ec2:StartInstances&quot;, &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &quot;ec2:StopInstances&quot; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ], &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &quot;Resource&quot;: &quot;arn:aws:ec2:<strong>&lt;REGION&gt;</strong>:<strong>&lt;ACCOUNTNUMBER&gt;</strong>:instance/*&quot;, &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &quot;Condition&quot;: { &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &quot;StringEquals&quot;: { &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &quot;ec2:ResourceTag/Owner&quot;: &quot;${aws:username}&quot; &nbsp;&nbsp;&nbsp; &nbsp; &nbsp;&nbsp;&nbsp;&nbsp;} &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; } &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }, &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; { &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &quot;Effect&quot;: &quot;Allow&quot;, &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &quot;Action&quot;: &quot;ec2:DescribeInstances&quot;, &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &quot;Resource&quot;: &quot;*&quot; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; } &nbsp;&nbsp;&nbsp; ] }</code></pre> </div> </li> <li>After you have edited the policy, choose <strong>Create policy</strong>.</li> </ol> <p>You have created a policy that allows an IAM user to stop and start EC2 instances in your account, as long as these instances have the correct resource tag and the policy is attached to your IAM users. You also can <a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-using.html#attach-managed-policy-console" target="_blank" rel="noopener noreferrer">attach this policy</a> to an IAM group and apply the policy to users by adding them to that group.</p> <h3>Summary</h3> <p>We updated the <a href="https://docs.aws.amazon.com/console/iam/example-policies" target="_blank" rel="noopener noreferrer">Example Policies</a> page in the IAM User Guide so that you have a central location where you can find examples of the most commonly requested and used IAM policies. In addition to these example policies, we recommend that you review the list of <a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html#aws-managed-policies" target="_blank" rel="noopener noreferrer">AWS managed policies</a>, including the <a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.html" target="_blank" rel="noopener noreferrer">AWS managed policies for job functions</a>. You can choose these predefined policies from the IAM console and associate them with your IAM users, groups, and roles.</p> <p>We will add more IAM policies to the Example Policies page over time. If you have a useful policy you would like to share with others, post it on the <a href="https://forums.aws.amazon.com/forum.jspa?forumID=76" target="_blank" rel="noopener noreferrer">IAM forum</a>. If you have comments about this post, submit them in the “Comments” section below.</p> <p>– Deren</p> How to Monitor and Visualize Failed SSH Access Attempts to Amazon EC2 Linux Instances https://aws.amazon.com/blogs/security/how-to-monitor-and-visualize-failed-ssh-access-attempts-to-amazon-ec2-linux-instances/ Wed, 02 Aug 2017 12:51:56 +0000 745f68f97ed3a01f4e0c79f3c67a2f3d696f7527 As part of the AWS Shared Responsibility Model, you are responsible for monitoring and managing your resources at the operating system and application level. When you monitor your application servers, for example, you can measure, visualize, react to, and improve the security of those servers. You probably already do this on premises or in other […] <p>As part of the <a href="https://aws.amazon.com/compliance/shared-responsibility-model/" target="_blank" rel="noopener noreferrer">AWS Shared Responsibility Model</a>, you are responsible for monitoring and managing your resources at the operating system and application level. When you monitor your application servers, for example, you can measure, visualize, react to, and improve the security of those servers. You probably already do this on premises or in other environments,&nbsp;and you can adapt your existing processes, tools, and methodologies for use in the AWS Cloud. For more details about best practices for monitoring your AWS resources, see the “Manage Security Monitoring, Alerting, Audit Trail, and Incident Response” section in the <a href="https://d0.awsstatic.com/whitepapers/Security/AWS_Security_Best_Practices.pdf" target="_blank" rel="noopener noreferrer">AWS Security Best Practices whitepaper</a>.</p> <p>This blog post focuses on how to log and create alarms on invalid <a href="https://en.wikipedia.org/wiki/Secure_Shell" target="_blank" rel="noopener noreferrer">Secure Shell</a> (SSH) access attempts. Implementing live monitoring and session recording facilitates the identification of unauthorized activity and can help confirm that remote users access only those systems they are authorized to use. With SSH log information in hand (such as invalid access type, bad private keys, and remote IP addresses), you can take proactive actions to protect your servers. For example, you can use an <a href="https://aws.amazon.com/lambda/" target="_blank" rel="noopener noreferrer">AWS Lambda</a> function to adjust your server’s security rules when an alarm is triggered that indicates an invalid SSH access attempt.</p> <p>In this post, I demonstrate how to use <a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html" target="_blank" rel="noopener noreferrer">Amazon CloudWatch Logs</a> to monitor SSH access to your application servers (<a href="https://aws.amazon.com/ec2/" target="_blank" rel="noopener noreferrer">Amazon EC2</a> Linux instances) so that you can monitor rejected SSH connection requests and take action. I also show how to configure CloudWatch Logs to send SSH access logs from application servers that reside in a public subnet. Last, I demonstrate how to visualize how many attempts are made to SSH into your application servers with bad private keys and invalid user names. Using these techniques and tools can help you improve the security of your application servers.</p> <p><span id="more-4411"></span></p> <h3>AWS services and terminology I use in this post</h3> <p>In this post, I use the following AWS services and terminology:</p> <ul> <li><a href="https://aws.amazon.com/cloudwatch/" target="_blank" rel="noopener noreferrer">Amazon CloudWatch</a> – A&nbsp;monitoring service for the resources and applications you run on the AWS Cloud. You can use CloudWatch to collect and track metrics, collect and monitor log files, set alarms, and automatically react to changes in your AWS resources.</li> <li><a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/aws-namespaces.html">CloudWatch namespaces</a> – Containers for metrics. Metrics in different namespaces are isolated from each other so that metrics from different applications are not mistakenly aggregated into the same statistics. You also can create custom metrics for which you must specify namespaces as containers.</li> <li>CloudWatch Logs&nbsp;– A feature of CloudWatch that allows you to monitor, store, and access your log files from EC2 instances, <a href="https://aws.amazon.com/cloudtrail/" target="_blank" rel="noopener noreferrer">AWS CloudTrail</a>, and other sources. Additionally, you can use CloudWatch Logs to monitor applications and systems by using log data and create alarms. For example, you can choose to search for a phrase in logs and then create an alarm if the phrase you are looking for is found in the log more than 5 times in the last 10 minutes. You can then take action on these alarms, if necessary.</li> <li><a href="http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-logstream.html" target="_blank" rel="noopener noreferrer">Log stream</a> – A log stream represents the sequence of events coming from an application instance or resource that you are monitoring. In this post, I use the EC2 instance ID as the log stream identifier so that I can easily map log entries to the instances that produced the log entries</li> <li><a href="http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-loggroup.html" target="_blank" rel="noopener noreferrer">Log group</a> – In CloudWatch Logs, a group of log streams that share the same retention time, monitoring, and access control settings. Each log stream must belong to one log</li> <li>Metric – A specific term or value that you can monitor and extract from log events.</li> <li><a href="http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-metricfilter.html" target="_blank" rel="noopener noreferrer">Metric filter</a> – A metric filter describes how Amazon CloudWatch Logs extracts information from logs and transforms it into CloudWatch metrics.&nbsp;It defines the terms and patterns to look for in log data as the data is sent to CloudWatch Logs. Metric filters are assigned to log groups, and all metric filters assigned to a given log group are applied to their log stream—see the following diagram for more details.</li> <li>SSH logs – Reside on EC2 instances and capture all SSH activities. The logs include successful attempts as well as unsuccessful attempts. Debian Linux SSH logs reside in <span style="font-family: courier">/var/log/auth.log</span>, and stock CentOS SSH logs are written to <span style="font-family: courier">/var/log/secure</span>. This blog post uses an <a href="https://aws&lt;/span&gt;.amazon.com/amazon-linux-ami/" target="_blank" rel="noopener noreferrer">Amazon Linux AMI</a>, which also logs SSH sessions to <span style="font-family: courier">/var/log/secure</span>.</li> <li><a href="https://aws.amazon.com/iam/" target="_blank" rel="noopener noreferrer">AWS Identity and Access Management (IAM)</a> – IAM enables you to securely control access to AWS services and resources for your users. In the solution in this post, you create an IAM policy and configure an EC2 instance that <a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html" target="_blank" rel="noopener noreferrer">assumes a role</a>. The IAM policy allows the EC2 instance to create log events and save them in an <a href="https://aws.amazon.com/s3/" target="_blank" rel="noopener noreferrer">Amazon S3</a> bucket (in other words, CloudWatch Logs log files are saved in the S3 bucket).</li> <li><a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Dashboards.html" target="_blank" rel="noopener noreferrer">CloudWatch dashboards</a> – Amazon CloudWatch dashboards are customizable home pages in the CloudWatch console that you can use to monitor your resources in a single view, even those resources that are spread across different regions. You can use CloudWatch dashboards to create customized views of the metrics and alarms for your AWS resources.</li> </ul> <h3>Architectural overview</h3> <p>The following diagram depicts the services and flow of information between the different AWS services used in this post’s solution.</p> <p><img class="alignnone wp-image-4523 size-full" title="Solution diagram" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/08/01/AssafDiagram-AugustFINAL.png" alt="" width="937" height="588" /></p> <p>Here is how the process works, as illustrated and numbered in the preceding diagram:</p> <ol> <li>A CloudWatch Logs agent runs on each EC2 instance. The agents are configured to send SSH logs from the EC2 instance to a log stream identified by an instance ID.</li> <li>Log streams are aggregated into a log group. As a result, one log group contains all the logs you want to analyze from one or more instances.</li> <li>You apply metric filters to a log group in order to search for specific keywords. When the metric filter finds specific keywords, the filter counts the occurrences of the keywords in a time-based sliding window. If the occurrence of a keyword exceeds the CloudWatch alarm threshold, an alarm is triggered.</li> <li>An IAM policy defines a role that gives the EC2 servers permission to create logs in a log group and send log events (new log entries) from EC2 to log groups. This role is then assumed by the application servers.</li> <li>CloudWatch alarms notify users when a specified threshold has been crossed. For example, you can set an alarm to trigger when more than 2 failed SSH connections happen in a 5-minute period.</li> <li>The CloudWatch dashboard is used to visualize data and alarms from the monitoring process.</li> </ol> <h2>Deploy and test the solution</h2> <h3>1. &nbsp;Deploy the solution by using CloudFormation</h3> <p>Now that I have explained how the solution works, I will show how to use <a href="https://aws.amazon.com/cloudformation/" target="_blank" rel="noopener noreferrer">AWS CloudFormation</a> to create a stack with the desired solution configuration. CloudFormation allows you to create a stack of resources in your AWS account.</p> <ol> <li>Sign in to the <a href="https://console.aws.amazon.com/console/home" target="_blank" rel="noopener noreferrer">AWS Management Console</a>, choose <strong>CloudFormation</strong>, choose <strong>Create Stack</strong>, choose&nbsp;<strong>Specify an Amazon S3 template URL </strong>and paste the following link in the box:<span style="font-family: courier"> https://s3.amazonaws.com/awsiammedia/public/sample/MonitorSSHActivities/CloudWatchLogs_ssh.yaml</span><br /> <img class="alignnone wp-image-4526 size-full" title="Screenshot of creating the stack" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/08/01/CreateStack-a.png" alt="Screenshot of creating the stack" width="902" height="375" /></li> <li>Choose <strong>Launch</strong> to deploy the stack.</li> <li>On the <strong>Specify Details</strong> page, enter the <strong>Stack name</strong>. Then enter the <strong>KeyName</strong>, which is the SSH key pair for the region you use. I use this key-pair later in this post; if you don’t have a key pair for your current region, follow <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html" target="_blank" rel="noopener noreferrer">these instructions</a> to create one. The <strong>OperatorEmail</strong> is the CloudWatch alarm recipient email address (this field is mandatory to launch the stack), which is the email address to which SSH activity alarms will be sent. You can use the <strong>SSHLocation</strong> box to limit the IP address range that can access your instances; the default is <span style="font-family: courier">0.0.0/0</span>, which means that any IP can access the instance. After specifying these variables, click <strong>Next.<br /> <img class="alignnone wp-image-4529 size-full" title="Screenshot of the &quot;Specify Details&quot; page" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/08/01/SpecifyDetails-a.png" alt="Screenshot of the &quot;Specify Details&quot; page" width="900" height="380" /><br /> </strong></li> <li>On the <strong>Options </strong>page, tag your instance, and click <strong>Next</strong>. Tags allow you to assign metadata to AWS resources. For example, you can tag a project’s resources and then use the tag to manage, search for, and filter resources. For more information about tagging, see <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html" target="_blank" rel="noopener noreferrer">Tagging Your Amazon EC2 Resources</a>.<br /> <img class="alignnone wp-image-4531 size-full" title="Screenshot of the &quot;Options&quot; page" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/08/01/Options-a.png" alt="Screenshot of the &quot;Options&quot; page" width="900" height="219" /></li> <li>Wait until the CloudFormation template shows <strong>CREATE_COMPLETE</strong>, as shown in the following screenshot. This means your stack was created successfully.<br /> <img class="alignnone wp-image-4421 size-full" title="Screenshot showing the stack was created successfully" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/Screen-Shot-2017-04-30-at-1.34.10-AM-a.png" alt="Screenshot showing the stack was created successfully" width="900" height="145" /></li> </ol> <p>After the stack is created successfully, you have two distinct application servers running, each with a CloudWatch agent. These servers represent a fleet of servers in your infrastructure. Choose the <strong>Outputs</strong> tab to see more details about the resources, such as the public IP addresses of the servers. You will need to use these IP addresses later in this post in order to trigger log events and alarms.</p> <p><img class="alignnone wp-image-4534 size-full" title="Screenshot showing the public IP addresses of the servers" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/08/01/publicIPs.png" alt="Screenshot showing the public IP addresses of the servers" width="900" height="414" /></p> <p>The CloudWatch log agent on each server is installed at startup and configured to stream SSH log entries from <span style="font-family: courier">/var/log/secure</span> to CloudWatch via a log stream. CloudWatch aggregates the log streams (<span style="font-family: courier">ssh.log</span>) from the application servers and saves them in a CloudWatch Logs log group. Each log stream is identified by an <span style="font-family: courier">instance-ID</span>, as shown in the following screenshot.</p> <p><img class="alignnone wp-image-4533 size-full" title="Screenshot of two log streams' instance-IDs" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/08/01/instanceIDs.png" alt="Screenshot of two log streams' instance-IDs" width="900" height="233" /></p> <p>The application servers <a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html" target="_blank" rel="noopener noreferrer">assume a role</a> that gives them permissions to create CloudWatch Logs log files and events. CloudFormation also configures two metrics: <span style="font-family: courier">ssh/InvalidUser</span> and <span style="font-family: courier">ssh/Disconnect</span>. The <span style="font-family: courier">ssh/InvalidUser</span> metric sends an alarm when there are more than 2 SSH attempts into any server that include an invalid user name. Similarly, the <span style="font-family: courier">ssh/Disconnect</span> metric creates an alarm when more than 10 SSH disconnect requests come from users within 5 minutes.</p> <p>To review the metrics created by CloudFormation, choose <strong>Metrics</strong> in the CloudWatch console. A new <strong>SSH</strong> custom namespace has been created, which contains the two metrics described in the previous paragraph.</p> <p><img class="alignnone size-full wp-image-4425" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/2-metrics.png" alt="" width="900" height="545" /></p> <p>You should now have two application servers running and two custom CloudWatch metrics and alarms configured. Now, it’s time to generate log events, trigger alarms, and test the configurations.</p> <h3>2. &nbsp;Test SSH metrics and alarms</h3> <p>Now, let’s try to trigger an alarm by trying to SSH with an invalid user name into one of the servers. Use the key pair you specified when launching the stack and <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html" target="_blank" rel="noopener noreferrer">connect to one of the Linux instances</a> from a terminal window (replace the <strong>placeholder values</strong> in the following command).</p> <div class="hide-language"> <pre><code class="lang-text">ssh -i <strong>&lt;the-key-pair-you-specified-in-the-CloudFormation-template&gt;</strong> ec2-user@<strong>&lt;ec2 DNS or IP address&gt;</strong></code></pre> </div> <p>Now, exit the session and try to sign in as <span style="font-family: courier"><span style="color: #ff0000"><strong>bad-user</strong></span></span>, as shown in the following command.</p> <div class="hide-language"> <pre><code class="lang-text">ssh -i <strong>&lt;the-key-pair-you-specified-in-the-CloudFormation-template&gt;</strong> <span style="color: #ff0000"><strong>bad-user</strong></span>@<strong>&lt;ec2 DNS or IP address&gt;</strong></code></pre> </div> <p>The following command is the same as the previous command, but with the placeholder values replaced by actual values.</p> <div class="hide-language"> <pre><code class="lang-text">ssh -i &quot;my-keycap&quot; <span style="color: #ff0000"><strong>bad-user</strong></span>@ec2-XX-XXX-XXX.compute-1.amazonaws.com</code></pre> </div> <p>Because the alarm triggers after two or more unsuccessful SSH login attempts with an invalid user name in 1 minute, run the preceding command a few times. The server’s log captures the bad SSH login attempts, and after a minute, you should see <span style="font-family: courier">InvalidUserAlarm</span> in the CloudWatch console, as shown in the following screenshot. Choose <strong>Alarms</strong> to see more details. The alarm should disappear after another minute if there are no more SSH login attempts.</p> <p><img class="alignnone wp-image-4426" title="Screenshot showing InvalidUserAlarm in the CloudWatch console" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/Picture1.png" alt="Screenshot showing InvalidUserAlarm in the CloudWatch console" width="900" height="387" /></p> <p>You can also view the history of your alarms by choosing the <strong>History</strong> tab. CloudWatch metrics are saved for 15 months.</p> <p>When the CloudFormation stack launches, a topic-registration email is sent to the email address you specified in the template. After you accept the topic registration, you will receive an alarm email with details about the alarm. The email looks like what is shown in the following screenshot.</p> <p><img class="alignnone wp-image-4427 size-full" title="Screenshot of the alarm email" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/Screen-Shot-2017-05-04-at-4.21.10-PM.png" alt="Screenshot of the alarm email" width="900" height="492" /></p> <h3>3. &nbsp;Understanding CloudWatch metric filters and their transformation</h3> <p>The CloudFormation template includes two alarms, <span style="font-family: courier">InvalidUserAlarm</span> and <span style="font-family: courier">SSHReceiveddisconnectAlarm</span>, and two metric filters. As I mentioned previously, the metric filters define the pattern you want to match in a CloudWatch Logs log group. When a pattern is found, it transforms into an Amazon metric as defined in the <span style="font-family: courier">MetricTransformations</span> section of the metric filter.</p> <p>The following is a snippet of the <span style="font-family: courier">InvalidUser</span> metric filter. Each pattern match—denoted by <span style="font-family: courier">FilterPattern</span>—is counted as one metric value as specified in the <span style="font-family: courier">MetricValue</span> parameter in the <span style="font-family: courier">MetricTranformations</span> section. The CloudWatch alarm associated with this metric filter will be triggered when the metric value crosses a specified threshold.</p> <div class="hide-language"> <pre><code class="lang-text"> InvalidUser: Type: AWS::Logs::MetricFilter Properties: LogGroupName: Ref: WebServerLogGroup FilterPattern: &quot;[Mon, day, timestamp, ip, id, status = Invalid, ...]&quot; MetricTransformations: - MetricValue: '1' MetricNamespace: SSH MetricName: sshInvalidUser</code></pre> </div> <p>When a CloudWatch alarm is triggered, the service sends an email to an <a href="https://aws.amazon.com/sns/" target="_blank" rel="noopener noreferrer">Amazon SNS</a> topic with more details about the alarm type, trigger parameters, and status.</p> <h3>4. &nbsp;Create a CloudWatch metric filter to identify attempts to SSH into your servers with bad private keys</h3> <p>You can create additional metric filters in CloudWatch Logs to provide better visibility into the SSH activity on your servers. Let’s assume you want to know if there are too many attempts to SSH into your servers with bad private keys. If an attempt is made with a bad private key, a line like the following is logged in the SSH log file.</p> <div class="hide-language"> <pre><code class="lang-text">Apr 27 21:44:49 ip-172-31-38-5 sash[28515]: Connection closed by &lt;<strong>ip address</strong>&gt; [preauth]</code></pre> </div> <p>You can produce this log line by modifying the pem file you are using (a <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html" target="_blank" rel="noopener noreferrer">pem file</a> holds your private key). In a terminal window, modify your private key by copying and pasting the following lines in the same directory where your key resides.</p> <div class="hide-language"> <pre><code class="lang-text">$ cat &lt;valid-pem-file&gt; | sed s/./A/25 &gt; bad-keys.pem $ cat bad-keys.pem | sed s/./A/26 &gt; bad-keys.pem</code></pre> </div> <p>These lines simply change the characters at positions 25 and 26 from their current value to the character <span style="font-family: courier">A</span>, keeping the original pem file intact. Alternatively, you can use <span style="font-family: courier">nano &lt;valid-keys&gt;.pem</span> from the command line or any other editor, change a character, save the file as <span style="font-family: courier">bad-keys.pem</span>, and exit the file.</p> <p>Now, try to use <span style="font-family: courier">bad-keys.pem</span> to access one of the application servers.</p> <div class="hide-language"> <pre><code class="lang-text">ssh&nbsp; -i &quot;bad-keys.pem&quot; ec2-user@ec2-XX-XXX-XXX-85.us-west-2.compute.amazonaws.com</code></pre> </div> <p>The SSH attempt should fail because you are using a bad private key.</p> <div class="hide-language"> <pre><code class="lang-text">Permission denied (public key).</code></pre> </div> <p>Now, let’s look at the server’s <span style="font-family: courier">ssh.log</span> file from the CloudWatch Logs console and analyze the error log messages. I need to understand the log format in order to configure a new filter. To review the logs, choose <strong>Logs</strong> in the navigation pane, and select the log group that was created by CloudFormation (it starts with the name you specified when launching the CloudFormation template).</p> <p><img class="alignnone wp-image-4428 size-full" title="Screenshot of the ssh.log file" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/Screen-Shot-2017-04-26-at-5.34.54-PM-copy.png" alt="Screenshot of the ssh.log file" width="900" height="514" /></p> <p>In particular, notice the following line when you try to SSH with a bad private key.</p> <p><img class="alignnone wp-image-4429 size-full" title="Screenshot of one line of the ssh.log file" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/Screen-Shot-2017-04-26-at-5.34.54-PM.png" alt="Screenshot of one line of the ssh.log file" width="740" height="18" /></p> <p>Let’s add a metric filter to capture this line so that we can use this metric later when we build an SSH Dashboard. Copy the following line to the <strong>Filter events</strong> search box at the top of the console screen and press <strong>Enter</strong>.</p> <div class="hide-language"> <pre><code class="lang-text">[Mon, day, timestamp, ip, id, msg1= Connection, msg2 = closed, ...] <img class="alignnone wp-image-4430 size-full" title="Screenshot of pasting the line from the ssh.log file into the &quot;Filter events&quot; search box" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/Screen-Shot-2017-06-23-at-12.14.41-AM.png" alt="Screenshot of pasting the line from the ssh.log file into the &quot;Filter events&quot; search box" width="900" height="275" /> </code></pre> </div> <p>You can now see only the lines that match the pattern you specified. These are the lines you want to count and transform into metrics. Each string in the message is represented by a word in the filter. In our example, we are looking for a pattern where the sixth word is <span style="font-family: courier">Connection</span> and the seventh word is <span style="font-family: courier">closed</span>. Other words in the log line are not important in this context. The following image depicts the mapping between a string in a log file and a metric filter.</p> <p><img class="alignnone wp-image-4460 size-full" title="Screenshot of the mapping between a string in a log file and a metric filter" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/AssafLogLine-July2017-b.png" alt="" width="900" height="206" /></p> <p>To create the metric filter, choose <strong>Logs</strong> in the navigation pane of the CloudWatch console. Choose the log groups to which you want to apply the new metric filter and then choose <strong>Create Metric Filter</strong>. Choose <strong>Next</strong>.</p> <p><img class="alignnone wp-image-4441 size-full" title="Screenshot of creating a metric filter" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/Screen-Shot-2017-04-28-at-2.56.54-PM.png" alt="Screenshot of creating a metric filter" width="900" height="272" /></p> <p>Paste the filter pattern we used previously (the sixth word equals <span style="font-family: courier">Connection</span> and the seventh word equals <span style="font-family: courier">closed</span>) in the <strong>Filter Pattern</strong> box. Select the server you tried to sign in to with the bad private key to <strong>Select Log Data to Test</strong> and click <strong>Test Pattern</strong>. You should see the results that are shown in the following screenshot. When completed, click <strong>Assign Metric.</strong></p> <p><img class="alignnone wp-image-4442 size-full" title="Screenshot of applying the filter pattern" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/Screen-Shot-2017-04-28-at-3.06.19-PM.png" alt="Screenshot of applying the filter pattern" width="900" height="733" /></p> <p>Type <span style="font-family: courier">SSH</span> for the <strong>Metric Namespace</strong> and <span style="font-family: courier">sshClosedConnection-InvalidKeysFilter</span> for <strong>Filter Name</strong>. Choose <strong>Create Filter</strong> to see your new metric filter listed. You can use the newly created metric filter to graph the metrics and set alarms. The alarms can be used to inform your administrator via email of any event you specify. In addition, metrics can be used to generate SNS notification to trigger an <a href="https://aws.amazon.com/lambda/" target="_blank" rel="noopener noreferrer">AWS Lambda</a> function in order to take proactive actions, such as blocking suspicious IP addresses in a <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#vpc-security-groups" target="_blank" rel="noopener noreferrer">security group</a>.</p> <p><img class="alignnone wp-image-4443 size-full" title="Screenshot of creating a filter" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/Screen-Shot-2017-07-05-at-9.59.03-AM.png" alt="Screenshot of creating a filter" width="890" height="371" /></p> <p>Choose <strong>Create Alarm</strong> next to <strong>Filter Name</strong> and follow the instructions to create a <a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html" target="_blank" rel="noopener noreferrer">CloudWatch alarm</a>.</p> <p><img class="alignnone wp-image-4444 size-full" title="Screenshot of the &quot;Create Alarm&quot; option" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/Screen-Shot-2017-07-05-at-9.59.48-AM.png" alt="Screenshot of the &quot;Create Alarm&quot; option" width="783" height="549" /></p> <p>Back at the <strong>Metrics</strong> view, you should now have three SSH metric filters under <strong>Custom Namespaces</strong>. Note that it can take a few minutes for the number of SSH metrics to update from two to three.</p> <p><img class="alignnone wp-image-4445 size-full" title="Screenshot showing you now have three SSH metric filters" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/3-Metrics.png" alt="Screenshot showing you now have three SSH metric filters" width="900" height="521" /></p> <h3>5. &nbsp;Create a graph by using a CloudWatch dashboard</h3> <p>After you have configured the metrics, you can display SSH metrics in a graph. <a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Dashboards.html" target="_blank" rel="noopener noreferrer">CloudWatch dashboards</a> allow you to create reusable graphs of AWS resources and custom metrics so that you can quickly monitor your operational status and identify issues at a glance. Metrics data is kept for a period of two weeks.</p> <p>In the CloudWatch console, choose <strong>Dashboards</strong> in the navigation pane, and then choose <strong>Create dashboard</strong> to create a new graph in a dashboard. Name your dashboard <strong>SSH-Dashboard</strong> and choose <strong>Create dashboard</strong>. Choose <strong>Line Graph</strong> from the graph options and choose <strong>Configure</strong>.</p> <p><img class="alignnone wp-image-4457 size-full" title="Screenshot of creating a CloudWatch dashboard" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/Screen-Shot-2017-04-28-at-5.01.38-PM-a.png" alt="Screenshot of creating a CloudWatch dashboard" width="500" height="246" /></p> <p>In the <strong>Add metric graph</strong> window under <strong>Custom Namespace</strong>, choose <strong>SSH </strong>&gt;<strong> Metrics with no dimensions</strong>. Select all three metrics you have configured (the CloudFormation template configured two metrics and you manually added one more metric).</p> <p><img class="alignnone wp-image-4458 size-full" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/Screen-Shot-2017-04-28-at-5.05.36-PM-a.png" alt="" width="400" height="583" /></p> <p>By default, the metrics are displayed on the graph as an average. However, you configured metrics that are based on summary metrics (for example, the total number of alarms in two minutes). To change the default, choose the <strong>Graphed metrics</strong> tab, and change the statistic from <strong>Average</strong> to <strong>Sum</strong>, as shown in the following screenshot. Also, change the time period from 5 minutes to 1 minute.</p> <p><img class="alignnone wp-image-4448 size-full" title="Screenshot of changing the statistic from &quot;Average&quot; to &quot;Sum&quot;" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/Screen-Shot-2017-04-28-at-5.09.14-PM.png" alt="Screenshot of changing the statistic from &quot;Average&quot; to &quot;Sum&quot;" width="900" height="390" /></p> <p>Your graphed metrics should look like the following screenshot. When you have provided all the necessary information, choose <strong>Create Widget.</strong></p> <p><img class="alignnone wp-image-4449 size-full" title="Screenshot of thet graphed metrics" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/Screen-Shot-2017-07-11-at-9.58.06-AM.png" alt="Screenshot of thet graphed metrics" width="900" height="343" /></p> <p>You can rename the graph and <a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/add_remove_text_dashboard.html" target="_blank" rel="noopener noreferrer">add static text</a> to give the console more context. To add a text widget, choose <strong>Widget</strong> and select text. Then edit the widget with <a href="https://en.wikipedia.org/wiki/Markdown" target="_blank" rel="noopener noreferrer">markdown language</a>. Your dashboard may then look like the following screenshot.</p> <p><img class="alignnone size-full wp-image-4450" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/29/Screen-Shot-2017-07-11-at-1.58.22-PM.png" alt="" width="900" height="264" /></p> <p>The consolidated metrics graph displays the number of SSH attempts with bad private keys, invalid user names, and too many disconnects.</p> <h3>Conclusion</h3> <p>In this blog post, I demonstrated how to automate the deployment of the CloudWatch Logs agent, create filters and alarms, and write, test, and apply metrics on the fly from the AWS Management Console. You can then visualize the metrics with the AWS Management Console. The solution described in this post gives you monitoring and alarming capabilities that can help you understand the status of and potential risks to your instances and applications. You can easily aggregate logs from many servers and different applications, create alarms, and display logs’ metrics on a graph.</p> <p>If you have comments about this post, submit them in the “Comments” section below. If you have questions about the solution in this post, start a new thread on the <a href="https://forums.aws.amazon.com/forum.jspa?forumID=138" target="_blank" rel="noopener noreferrer">CloudWatch forum</a>.</p> <p>– Assaf</p> How to Use AWS Organizations to Automate End-to-End Account Creation https://aws.amazon.com/blogs/security/how-to-use-aws-organizations-to-automate-end-to-end-account-creation/ Mon, 24 Jul 2017 13:54:53 +0000 85bb9da727523e50e7ba3d43324523fa628413d0 AWS Organizations&nbsp;offers new capabilities for managing AWS accounts, including automated account creation via the Organizations API. For example, you can bring new development teams onboard by using the Organizations API to create an account, AWS CloudFormation templates to configure the account (such as for AWS Identity and Access Management [IAM] and networking), and service control […] <p><a href="https://aws.amazon.com/organizations/" target="_blank" rel="noopener noreferrer">AWS Organizations</a>&nbsp;offers new capabilities for managing AWS accounts, including automated account creation via the <a href="http://docs.aws.amazon.com/organizations/latest/APIReference/Welcome.html" target="_blank" rel="noopener noreferrer">Organizations API</a>. For example, you can bring new development teams onboard by using the Organizations API to create an account, <a href="https://aws.amazon.com/cloudformation/" target="_blank" rel="noopener noreferrer">AWS CloudFormation</a> templates to configure the account (such as for <a href="https://aws.amazon.com/iam/" target="_blank" rel="noopener noreferrer">AWS Identity and Access Management</a> [IAM] and networking), and <a href="https://aws.amazon.com/blogs/security/announcing-aws-organizations-centrally-manage-multiple-aws-accounts/" target="_blank" rel="noopener noreferrer">service control policies</a> (SCPs) to help enforce corporate policies.</p> <p>In this blog post, I demonstrate the step-by-step process for end-to-end account creation in Organizations as well as how to automate the entire process. I also show how to move a new account into an organizational unit (OU).</p> <h3>Process overview</h3> <p>The following process flow diagram illustrates the steps required to create an account, configure the account, and then move it into an OU so that the account can take advantage of the centralized <a href="http://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scp.html" target="_blank" rel="noopener noreferrer">SCP functionality</a> in Organizations. The tasks in the blue nodes occur in the master account in the organization in question, and the task in the orange node occurs in the new member account I create. In this post, I provide a script (in both Bash/CLI and Python) that you can use to automate this account creation process, and I walk through each step shown in the diagram to explain the process in detail. For the purposes of this post, I use the <a href="https://aws.amazon.com/cli/" target="_blank" rel="noopener noreferrer">AWS CLI</a> in combination with CloudFormation to create and configure an account.<span id="more-4071"></span></p> <p><img class="alignnone wp-image-4330 size-full" title="The process flow diagram of this post's solution" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/19/DavidSDiagram-071917.png" alt="The process flow diagram of this post's solution" width="898" height="545" /></p> <h2>The account creation process</h2> <p>Follow the steps in this section to create an account, configure it, and move it into an OU. I am also providing a&nbsp;<a href="https://s3.amazonaws.com/awsiammedia/public/sample/AWSOrganizationsAutomateEndtoEndAccountCreation/AWSOrganizations_AutomateEndtoEndAccountCreation_ScriptandTemplates.zip" target="_blank" rel="noopener noreferrer">script and CloudFormation templates</a>&nbsp;that you can use to automate the entire process.</p> <h3>1. Sign in to AWS Organizations</h3> <p>In order to create an account, you must <a href="https://console.aws.amazon.com/organizations/" target="_blank" rel="noopener noreferrer">sign in</a> to your organization’s master account with a minimum of the following permissions:</p> <ul> <li><span style="font-family: courier">organizations:DescribeOrganization</span></li> <li><span style="font-family: courier">organizations:CreateAccount</span></li> </ul> <h3>2. Create a new member account</h3> <p>After signing in to your organization’s master account, <a href="http://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_accounts_create.html" target="_blank" rel="noopener noreferrer">create a new member account</a>. Before you can create the member account, you need three pieces of information:</p> <ul> <li><strong>An account name</strong> – The friendly name of the member account, which you can find on the <strong>Accounts</strong> tab in the master account.</li> <li><strong>An email address</strong> – The email address of the owner of the new member account. This email address is used by AWS when we need to contact the account owner.</li> <li><strong>An IAM role name</strong> – The name of an IAM role that Organizations automatically preconfigures in the new member account. This role trusts the master account, allowing users in the master account to assume the role, as permitted by the master account administrator. The role also has administrator permissions in the new member account. If you do not change the role’s name, the name defaults to <span style="font-family: courier">OrganizationAccountAccessRole</span>.</li> </ul> <p>The following AWS CLI command creates a new member account.</p> <div class="hide-language"> <pre><code class="lang-text">aws organizations create-account --email <strong>&lt;newAccEmail&gt;</strong> --account-name &quot;<strong>&lt;newAccName&gt;</strong>&quot; --role-name <strong>&lt;roleName&gt;</strong></code></pre> </div> <p>To explain the <strong>placeholder values</strong>&nbsp;in the preceding command that you must update with your own values:</p> <ul> <li><strong>newAccEmail</strong> – The email address of the owner of the new member account. This email address must <em>not</em> already be associated with another AWS account.</li> <li><strong>newAccName</strong> – The friendly name of the new member account.</li> <li><strong>roleName</strong> – The name of an IAM role that Organizations automatically preconfigures in the new member account. The default name is <span style="font-family: courier">OrganizationAccountAccessRole</span>.</li> </ul> <p>This CLI command returns a <span style="font-family: courier">request_id</span> that uniquely identifies the request, a value that is required for in Step 3.</p> <p><strong>Important: </strong> When you create an account using Organizations, you currently cannot remove this account from your organization. This, in turn, can prevent you from later deleting the organization.</p> <h3>3. Verify account creation</h3> <p>Account creation may take a few seconds to complete, so before doing anything with the newly created account, you must first verify the account creation status. To check the status, you must have at least the following permission:</p> <ul> <li><span style="font-family: courier">organizations:DescribeCreateAccountStatus</span></li> </ul> <p>The following CLI command, with the <span style="font-family: courier">request_id</span> returned in the previous step as an input parameter, verifies that the account was created:</p> <div class="hide-language"> <pre><code class="lang-text">aws organizations describe-create-account-status --create-account-request-id <strong>&lt;request_id&gt;</strong></code></pre> </div> <p>The command returns the state of your account creation request and can have three different values: <span style="font-family: courier">IN_PROGRESS</span>, <span style="font-family: courier">SUCCEEDED</span>, and <span style="font-family: courier">FAILED</span>.</p> <h3>4. Assume a role</h3> <p>After you have verified that the new account has been created, configure the account. In order to configure the newly created account, you must sign in with a user who has permission to assume the role submitted in the <span style="font-family: courier">createAccount</span> API call. In the example in Step 1, I named the role <span style="font-family: courier">OrganizationAccountAccessRole</span>; however, if you revised the name of the role, you must use that revised name when assuming the role. Note that when an account is created from within an organization, cross-account trust between the master and programmatically created accounts is automatically established.</p> <p>The following CLI command assumes a role.</p> <div class="hide-language"> <pre><code class="lang-text">aws sts assume-role --role-arn &lt;<strong>role-arn</strong>&gt; --role-session-name &lt;&quot;<strong>role-session-name</strong>&quot;&gt;</code></pre> </div> <p>To explain the <strong>placeholder values</strong>&nbsp;in the preceding command that you must update with your own values:</p> <ul> <li><strong>role-arn&nbsp;– </strong>The <a href="http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html" target="_blank" rel="noopener noreferrer">Amazon Resource Name</a> (ARN) of the role to assume.</li> <li><strong>role-session-name&nbsp;– </strong>An identifier for the assumed role session.</li> </ul> <h3>5. Configure the new account</h3> <p>After you assume the role, build the new account’s networking, IAM, and governance resources as explained in this section. Again, to learn more about and download the account creation script and the templates that can automate this process, see “Automating the entire end-to-end process” later in this post.</p> <ol type="A"> <li>Networking – Amazon VPC, web access control lists (ACLs), and Internet gateway: <ol type="1"> <li><a href="http://docs.aws.amazon.com/AmazonVPC/latest/GettingStartedGuide/getting-started-ipv4.html" target="_blank" rel="noopener noreferrer">Create a new Amazon VPC</a>&nbsp;to enable you to launch AWS resources in a virtual network that you define.</li> <li>Run the script at the end of this post to create a VPC with two subnets (one public subnet and one private subnet) in each of two Availability Zones.</li> <li>Set up <a href="http://docs.aws.amazon.com/waf/latest/developerguide/web-acl.html" target="_blank" rel="noopener noreferrer">web ACLs</a>&nbsp;to control traffic in and out of the subnets. You can set up network ACLs with rules similar to your security groups in order to add an additional layer of security to your VPC.</li> <li>Connect your VPC to remote networks by using a <a href="http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpn-connections.html" target="_blank" rel="noopener noreferrer">VPN connection</a>.</li> <li>If the resources in the VPC require access to the Internet, create an <a href="http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Internet_Gateway.html" target="_blank" rel="noopener noreferrer">Internet gateway</a> to allow communication between instances in your VPC and the Internet.</li> </ol> </li> <li>IAM – Identity provider (IdP), IAM policies, and IAM roles: <ol type="1"> <li>Many customers use enterprise federated authentication (such as Active Directory) to manage users and permissions centrally outside AWS. If you use federated authentication, <a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create.html" target="_blank" rel="noopener noreferrer">set up an IdP</a>.</li> <li>After you set up the IdP, author the <a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html#customer-managed-policies" target="_blank" rel="noopener noreferrer">customer managed IAM policies</a> you will use.</li> <li>Use <a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html#aws-managed-policies" target="_blank" rel="noopener noreferrer">AWS managed policies</a> or your customer managed policies to manage access to your AWS resources.</li> </ol> </li> <li>Governance – AWS Config Rules: <ol type="1"> <li>Create <a href="http://docs.aws.amazon.com/config/latest/developerguide/evaluate-config.html" target="_blank" rel="noopener noreferrer">AWS Config rules </a>to help manage and enforce standards for resources deployed on AWS.</li> <li>Develop a tagging strategy that specifies a minimum set of tags required on every taggable resource. A tagging rule checks that all resources created or edited fulfill this requirement. A noncompliance report is created to document resources that do not meet the AWS Config rule. <a href="http://docs.aws.amazon.com/config/latest/developerguide/evaluate-config_develop-rules_nodejs-sample.html" target="_blank" rel="noopener noreferrer">AWS Lambda scripts</a> can also be launched as a result of AWS Config rules.</li> </ol> </li> </ol> <h3>6. Move the new account into an OU</h3> <p>Before allowing your development teams to access the new member account that you configured in the previous steps, <a href="http://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scp.html" target="_blank" rel="noopener noreferrer">apply an SCP</a> to the account to limit the API calls that can be made by all users. To do this, you must move the member account into an OU that has an SCP attached to it.</p> <p>An OU is a container for&nbsp;accounts. It can contain other OUs, allowing you to create a hierarchy that resembles an upside-down tree with a “root” at the top and OU “branches” that reach down, ending with accounts that are the “leaves” of the tree. When you attach a policy to one of the nodes in the hierarchy, it affects all the branches (OUs) and leaves (accounts) under it. An OU can have exactly one parent, and currently, each account can be a member of exactly one OU.</p> <p>The following CLI command moves an account into an OU.</p> <div class="hide-language"> <pre><code class="lang-text">aws organizations move-account --account-id &lt;<strong>account_id</strong>&gt; --source-parent-id &lt;<strong>source_parent_id</strong>&gt; --destination-parent-id &lt;<strong>destination_parent_id</strong>&gt;</code></pre> </div> <p>To explain the <strong>placeholder values</strong> in the preceding command that you must update with your own values:</p> <ul> <li><strong>account_id</strong> – The unique identifier (ID) of the account you want to move.</li> <li><strong>source_parent_id</strong> – The unique ID of the root or OU from which you want to move the account.</li> <li><strong>destination_parent_id</strong> – The unique ID of the root or OU to which you want to move the account.</li> </ul> <h3>7. Reduce the IAM role permissions</h3> <p>The <span style="font-family: courier">OrganizationAccountAccessRole</span> is created with full administrative permissions to enable the creation and development of the new member account. After you complete the development process and you have moved the member account into an OU, <a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_manage_modify.html#roles-managingrole-editing-cli" target="_blank" rel="noopener noreferrer">reduce the permissions</a> of <span style="font-family: courier">OrganizationAccountAccessRole</span> to match your anticipated use of this role going forward.</p> <h2>Automating the entire end-to-end process</h2> <p>To help you fully automate the process of creating new member accounts, setting up those accounts, and moving new member accounts into an OU, I am providing a script in both Bash/CLI and Python. You can modify or call additional CloudFormation templates as needed.</p> <h3>Download the script and CloudFormation templates</h3> <p>Download the <a href="https://s3.amazonaws.com/awsiammedia/public/sample/AWSOrganizationsAutomateEndtoEndAccountCreation/AWSOrganizations_AutomateEndtoEndAccountCreation_ScriptandTemplates.zip" target="_blank" rel="noopener noreferrer">script and CloudFormation templates</a>&nbsp;to help you automate this end-to-end process. The global variables in the script are set in the opening lines of code. Update these variables’ values, and they will flow as input parameters to the API commands when the script is executed. I have prepopulated the <span style="font-family: courier">roleName</span> by using AWS best practices nomenclature, but you can use a custom name.</p> <p>I am including the following descriptions of the elements of the script to give you a better idea of how the script works.</p> <p>Bash/CLI:</p> <ul> <li><strong><span style="font-family: courier">Organization-new-acc.sh</span></strong> – An example shell script that includes parameters, account creation, and a call to the JSON sample templates for each of three subtasks in Step 5 earlier in this post.</li> <li><strong><span style="font-family: courier">CF-VPC.json</span></strong>&nbsp;– An example Cloud Formation template that creates and configures a VPC in the new member account. Each AWS account must have at least one VPC as a networking construct where you can deploy customer resources. Though AWS does create a default VPC when an account is created, you must configure that VPC to meet your needs. This includes creating subnets with specific IP <a href="http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Subnets.html" target="_blank" rel="noopener noreferrer">Classless Inter-Domain Routing (CIDR) blocks</a>, creating gateways (including an <a href="http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Internet_Gateway.html" target="_blank" rel="noopener noreferrer">Internet gateway</a>, a <a href="http://docs.aws.amazon.com/AmazonVPC/latest/NetworkAdminGuide/Introduction.html#CustomerGateway" target="_blank" rel="noopener noreferrer">customer gateway</a>, <a href="http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_VPN.html#VPNTunnels" target="_blank" rel="noopener noreferrer">a VPN tunnel</a>, <a href="https://aws.amazon.com/storagegateway/" target="_blank" rel="noopener noreferrer">AWS Storage Gateway</a>, <a href="https://aws.amazon.com/api-gateway/" target="_blank" rel="noopener noreferrer">Amazon API Gateway</a>, and a <a href="http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-nat-gateway.html" target="_blank" rel="noopener noreferrer">NAT gateway</a>), and <a href="http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-peering.html" target="_blank" rel="noopener noreferrer">VPC peering connections</a>. Web ACLs are also part of this process to limit the source IP addresses and ports that can access the VPC. The VPC created by this script includes four <a href="http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Subnets.html" target="_blank" rel="noopener noreferrer">subnets</a> across two Availability Zones. Two of the subnets are public and two are private.</li> <li><strong><span style="font-family: courier">CF-IAM.json</span></strong>&nbsp;– An example Cloud Formation template that creates IAM roles in the new member account. As part of a security baseline, you should develop a standard set of IAM roles and related policies. Update this template with the IAM role definitions and policies you want to create in the member account to controls privilege and access.</li> <li><strong><span style="font-family: courier">CF-ConfigRules.json</span></strong>&nbsp;– An example Cloud Formation template that creates an AWS Config rule to enforce tagging standards on resources created in the new account.</li> <li><strong><span style="font-family: courier">Organization_Output.docx</span></strong>&nbsp;– Example output of the results from running <span style="font-family: courier">Organization-new-acc.sh</span>.</li> </ul> <p>Python:</p> <ul> <li><strong><span style="font-family: courier">Create_account_with_iam.py</span></strong><em> – </em>An example Python template that creates an account, moves it into an OU, applies an SCP, and then calls additional templates to deploy resources. <span style="font-family: courier">CF-VPC.JSON</span> can be called by this template if you first customize the <span style="font-family: courier">.json</span> file.</li> </ul> <ul> <li><strong><span style="font-family: courier">Baseline.yml</span></strong> – An example CloudFormation template for creating a new IAM administrative user, IAM user group, IAM role, and IAM policy in the account.</li> </ul> <h3>Summary</h3> <p>In this post, I have demonstrated the step-by-step process for end-to-end account creation in Organizations as well as how to automate the entire process. I also showed how to move a new account into an OU. This solution should save you some time and help you avoid common issues that tend to crop up in the manual account-creation process. To learn more about the features of Organizations, see the <a href="http://docs.aws.amazon.com/organizations/latest/userguide/orgs_introduction.html">AWS Organizations User Guide</a>. For more information about the APIs used in this post, see the <a href="http://docs.aws.amazon.com/organizations/latest/APIReference/Welcome.html">Organizations API Reference</a>.</p> <p>If you have comments about this blog post, submit them in the “Comments” section below. If you have implementation or troubleshooting questions, start a new thread on the&nbsp;<a href="https://forums.aws.amazon.com/forum.jspa?forumID=219">Organizations forum</a>.</p> <p>– David</p> How to Use Batch References in Amazon Cloud Directory to Refer to New Objects in a Batch Request https://aws.amazon.com/blogs/security/how-to-use-batch-references-in-amazon-cloud-directory-to-refer-to-new-objects-in-a-batch-request/ Wed, 19 Jul 2017 13:39:01 +0000 cc580dd7130b6d922b093e53cb1ca0ee970443a9 In Amazon Cloud Directory, it’s often necessary to add new objects or add relationships between new objects and existing objects to reflect changes in a real-world hierarchy. With Cloud Directory, you can make these changes efficiently by using batch references within batch operations. Let’s say I want to take an existing child object in a […] <p>In <a href="https://aws.amazon.com/clouddirectory/" target="_blank" rel="noopener noreferrer">Amazon Cloud Directory</a>, it’s often necessary to add new objects or add relationships between new objects and existing objects to reflect changes in a real-world hierarchy. With Cloud Directory, you can make these changes efficiently by using batch references within batch operations.</p> <p>Let’s say I want to take an existing child object in a hierarchy, detach it from its parent, and reattach it to another part of the hierarchy. A simple way to do this would be to make a call to get the object’s unique identifier, another call to detach the object from its parent using the unique identifier, and a third call to attach it to a new parent. However, if I use batch references within a batch write operation, I can perform all three of these actions in the same request, greatly simplifying my code and reducing the round trips required to make such changes.</p> <p>In this post, I demonstrate how to use batch references in a single write request to simplify adding and restructuring a Cloud Directory hierarchy. I have used the <a href="https://aws.amazon.com/sdk-for-java/" target="_blank" rel="noopener noreferrer">AWS SDK for Java</a> for all the sample code in this post, but you can use <a href="https://aws.amazon.com/tools/" target="_blank" rel="noopener noreferrer">other language SDKs</a> or the <a href="https://aws.amazon.com/cli/" target="_blank" rel="noopener noreferrer">AWS CLI</a> in a similar way.</p> <h3><strong>Using batch references</strong></h3> <p>In my <a href="https://aws.amazon.com/blogs/security/write-and-read-multiple-objects-in-amazon-cloud-directory-by-using-batch-operations/" target="_blank" rel="noopener noreferrer">previous post</a>, I demonstrated how to add AnyCompany’s North American warehouses to a global network of warehouses. As time passes and demand grows, AnyCompany launches multiple warehouses in North American cities to fulfill customer orders with continued efficiency. This requires the company to restructure the network to group warehouses in the same region so that the company can apply similar standards to them, such as delivery times, delivery areas, and types of products sold.<span id="more-4248"></span></p> <p>For instance, in the <span style="font-family: courier">NorthAmerica</span> object (see the following diagram), AnyCompany has launched two new warehouses in the Phoenix (<span style="font-family: courier">PHX</span>) area: <span style="font-family: courier">PHX_2</span> and <span style="font-family: courier">PHX_3</span>. AnyCompany wants to add these new warehouses to the network and regroup them with existing warehouse&nbsp;<span style="font-family: courier">PHX_1</span>&nbsp;under the new node, <span style="font-family: courier">PHX</span>.</p> <p>The state of the hierarchy before this regrouping is shown in the following diagram, where I added the <span style="font-family: courier">NorthAmerica</span> warehouses (also represented as <span style="font-family: courier">NA</span> in the diagram) to the larger network of AnyCompany’s warehouses.</p> <p><img class="alignnone wp-image-4246 size-full" title="Diagram showing the state of the hierarchy before this post's regrouping" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/11/Diagram2-Vineeth-July2017-b.png" alt="Diagram showing the state of the hierarchy before this post's regrouping" width="900" height="495" /></p> <h3>Adding and grouping new warehouses in the NorthAmerica network</h3> <p>I want to add and group the new warehouses with a single request, and using batch references in a batch write lets me do that. A batch reference is just another way of using&nbsp;object references that you are allowed to define arbitrarily. This allows you to chain operations, which means using the return value from one operation in a subsequent operation within the same batch write request</p> <p>Let’s say I have a batch write request&nbsp;with two batch operations: operation A&nbsp;and operation B. Both batch operations operate on the same object X. In operation A, I use the object X found at <span style="font-family: courier">/NorthAmerica/Phoenix</span>, and I assign it to a batch reference that I call <span style="font-family: courier">referencePhoenix</span>. In operation B, I want to modify the same object X, so I use <span style="font-family: courier">referencePhoenix</span> as the object reference that points to the same unique&nbsp;object X used in operation A. I also will use the same helper method implementation from my <a href="https://aws.amazon.com/blogs/security/write-and-read-multiple-objects-in-amazon-cloud-directory-by-using-batch-operations/" target="_blank" rel="noopener noreferrer">previous post</a> for <span style="font-family: courier">getBatchCreateOperation</span>. To learn more about batch references, see the <a href="http://docs.aws.amazon.com/directoryservice/latest/APIReference/API_ObjectReference.html" target="_blank" rel="noopener noreferrer">ObjectReference documentation</a>.</p> <p>To add and group the new warehouses, I will take advantage of batch references to sequentially:</p> <ol> <li>Detach <span style="font-family: courier">PHX_1</span> from the <span style="font-family: courier">NA</span> node and maintain a reference to <span style="font-family: courier">PHX_1</span>.</li> <li>Create a new child node, <span style="font-family: courier">PHX</span>, and attach it to the <span style="font-family: courier">NA</span> node.</li> <li>Create <span style="font-family: courier">PHX_2</span> and <span style="font-family: courier">PHX_3</span> nodes for the new warehouses.</li> <li>Link all three nodes—<span style="font-family: courier">PHX_1</span> (using the batch reference), <span style="font-family: courier">PHX_2</span>, and <span style="font-family: courier">PHX_3</span>—to the <span style="font-family: courier">PHX</span> node.</li> </ol> <p>The following code example achieves these changes in a single batch by using references. First, the code sets up a <span style="font-family: courier">createObjectPHX</span> operation to create the <span style="font-family: courier">PHX</span> parent object and attach it to the parent <span style="font-family: courier">NorthAmerica</span> object. It then sets up <span style="font-family: courier">createObjectPHX_2</span> and <span style="font-family: courier">createObjectPHX_3</span> and attaches these new objects to the new <span style="font-family: courier">PHX</span> object. The code then sets up a <span style="font-family: courier">detachObject</span> to detach the current <span style="font-family: courier">PHX_1</span> object from its parent and assign it to a batch reference. The last operation uses that same batch reference to attach the <span style="font-family: courier">PHX_1</span> object to the newly created <span style="font-family: courier">PHX</span> object. The code example orders these steps sequentially in a batch write operation.</p> <div class="hide-language"> <pre><code class="lang-text"> BatchWriteOperation createObjectPHX = getBatchCreateOperation( &quot;PHX&quot;, directorySchemaARN, &quot;/NorthAmerica&quot;, &quot;Phoenix&quot;); BatchWriteOperation createObjectPHX_2 = getBatchCreateOperation( &quot;PHX_2&quot;, directorySchemaARN, &quot;/NorthAmerica/Phoenix&quot;, &quot;PHX_2&quot;); BatchWriteOperation createObjectPHX_3 = getBatchCreateOperation( &quot;PHX_3&quot;, directorySchemaARN, &quot;/NorthAmerica/Phoenix&quot;, &quot;PHX_3&quot;); BatchDetachObject detachObject = new BatchDetachObject() .withBatchReferenceName(&quot;<strong>referenceToPHX_1</strong>&quot;) .withLinkName(&quot;Phoenix&quot;) .withParentReference(new ObjectReference() .withSelector(&quot;/NorthAmerica&quot;)); BatchAttachObject attachObject = new BatchAttachObject() .withChildReference(new ObjectReference().withSelector(&quot;#referenceToPHX_1&quot;)) .withLinkName(&quot;PHX_1&quot;) .withParentReference(new ObjectReference() .withSelector(&quot;/NorthAmerica/Phoenix&quot;)); BatchWriteOperation detachOperation = new BatchWriteOperation() .withDetachObject(detachObject); BatchWriteOperation attachOperation = new BatchWriteOperation() .withAttachObject(attachObject); BatchWriteRequest request = new BatchWriteRequest(); request.setDirectoryArn(directoryARN); request.setOperations(Lists.newArrayList( detachOperation, createObjectPHX, createObjectPHX_2, createObjectPHX_3, attachOperation)); client.batchWrite(request);</code></pre> </div> <p>In the preceding code example, I use the batch reference, <strong><span style="font-family: courier">referenceToPHX_1</span></strong>, in the same batch write operation because I do not have to know the object identifier of that object. If I couldn’t use such a batch reference, I would have to use separate requests to get the <span style="font-family: courier">PHX_1</span> identifier, detach it from the <span style="font-family: courier">NA</span> node, and then attach it to the new <span style="font-family: courier">PHX</span> node.</p> <p>I now have the network configuration I want, as shown in the following diagram. I have used a combination of batch operations with batch references to bring new warehouses into the network and regroup them within the same local group of warehouses.</p> <p><img class="aligncenter wp-image-4252 size-full" title="Diagram showing the desired network configuration" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/11/Diagram4-Vineeth-July2017.png" alt="Diagram showing the desired network configuration" width="497" height="529" /></p> <h3>Summary</h3> <p>In this post, I have shown how you can use batch references in a single batch write request to simplify adding and restructuring your existing hierarchies in Cloud Directory. You can use batch references in scenarios where you want to get an object identifier, but don’t want the overhead of using a read operation before a write operation. Instead, you can use a batch reference to refer to an object as part of the intermediate batch operation. To learn more about batch operations, see <a href="http://docs.aws.amazon.com/directoryservice/latest/admin-guide/batches.html" target="_blank" rel="noopener noreferrer">Batches</a>, <a href="http://docs.aws.amazon.com/directoryservice/latest/APIReference/API_BatchWrite.html" target="_blank" rel="noopener noreferrer">BatchWrite</a>, and <a href="http://docs.aws.amazon.com/directoryservice/latest/APIReference/API_BatchRead.html" target="_blank" rel="noopener noreferrer">BatchRead</a>.</p> <p>If you have comments about this post, submit them in the “Comments” section below. If you have implementation questions, start a new thread on the <a href="https://forums.aws.amazon.com/forum.jspa?forumID=180" target="_blank" rel="noopener noreferrer">Directory Service forum</a>.</p> <p>– Vineeth</p> Write and Read Multiple Objects in Amazon Cloud Directory by Using Batch Operations https://aws.amazon.com/blogs/security/write-and-read-multiple-objects-in-amazon-cloud-directory-by-using-batch-operations/ Tue, 18 Jul 2017 13:31:57 +0000 9e3f15b5ac4400c6e5d97d625a89bea9996696ef Amazon Cloud Directory is a hierarchical data store that enables you to build flexible, cloud-native directories for organizing hierarchies of data along multiple dimensions. For example, you can create an organizational structure that you can navigate through multiple hierarchies for reporting structure, location, and cost center. In this blog post, I demonstrate how you can […] <p><a href="https://aws.amazon.com/cloud-directory/" target="_blank" rel="noopener noreferrer">Amazon Cloud Directory</a> is a hierarchical data store that enables you to build flexible, cloud-native directories for organizing hierarchies of data along multiple dimensions. For example, you can create an organizational structure that you can navigate through multiple hierarchies for reporting structure, location, and cost center.</p> <p>In this blog post, I demonstrate how you can use <a href="http://docs.aws.amazon.com/directoryservice/latest/APIReference/welcome.html" target="_blank" rel="noopener noreferrer">Cloud Directory APIs</a> to write and read multiple objects by using batch operations. With <a href="http://docs.aws.amazon.com/directoryservice/latest/APIReference/API_BatchWrite.html" target="_blank" rel="noopener noreferrer">batch write operations</a>, you can execute a sequence of operations atomically—meaning that all of the write operations must occur, or none of them do. You also can make your application efficient by reducing the number of required round trips to read and write objects to your directory. I have used the <a href="https://aws.amazon.com/sdk-for-java/" target="_blank" rel="noopener noreferrer">AWS SDK for Java</a> for all the sample code in this blog post, but you can use <a href="https://aws.amazon.com/tools/" target="_blank" rel="noopener noreferrer">other language SDKs</a> or the <a href="https://aws.amazon.com/cli/" target="_blank" rel="noopener noreferrer">AWS CLI</a> in a similar way.</p> <h3>Using batch write operations</h3> <p>To demonstrate batch write operations, let’s say that AnyCompany’s warehouses are organized to determine the fastest methods to ship orders to its customers. In North America, AnyCompany plans to open new warehouses regularly so that the company can keep up with customer demand while continuing to meet the delivery times to which they are committed.</p> <p>The following diagram shows part of AnyCompany’s global network, including Asian and European warehouse networks.<span id="more-4233"></span></p> <p><img class="alignnone wp-image-4245 size-full" title="Diagram showing AnyCompany's global network" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/11/Diagram1-Vineeth-July2017-b.png" alt="" width="900" height="316" /></p> <p>Let’s take a look at how I can use batch write operations to add <span style="font-family: courier">NorthAmerica</span> to AnyCompany’s global network of warehouses, with the first three warehouses in New York City (<span style="font-family: courier">NYC</span>), Las Vegas (<span style="font-family: courier">LAS</span>), and Phoenix (<span style="font-family: courier">PHX</span>).</p> <h4>Adding NorthAmerica to the global network</h4> <p>To add <span style="font-family: courier">NorthAmerica</span> to the global network, I can use a batch write operation to create and link all the objects in the existing network.</p> <p>First, I set up a helper method, which performs repetitive tasks, for the <span style="font-family: courier">getBatchCreateOperation</span> object. The following lines of code help me create an <span style="font-family: courier">NA</span> object for <span style="font-family: courier">NorthAmerica</span> and then attach the three city-related nodes: <span style="font-family: courier">NYC</span>, <span style="font-family: courier">LAS</span>, and <span style="font-family: courier">PHX</span>. Because AnyCompany is planning to grow its network, I add a suffix of <span style="font-family: courier">_1</span> to each city code (such as <span style="font-family: courier">PHX_1</span>), which will be helpful hierarchically when the company adds more warehouses within a city.</p> <div class="hide-language"> <pre><code class="lang-text"> private BatchWriteOperation getBatchCreateOperation( String warehouseName, String directorySchemaARN, String parentReference, String linkName) { SchemaFacet warehouse_facet = new SchemaFacet() .withFacetName(&quot;warehouse&quot;) .withSchemaArn(directorySchemaARN); AttributeKeyAndValue kv = new AttributeKeyAndValue() .withKey(new AttributeKey() .withFacetName(&quot;warehouse&quot;) .withName(&quot;name&quot;) .withSchemaArn(directorySchemaARN)) .withValue(new TypedAttributeValue() .withStringValue(warehouseName); List&lt;SchemaFacet&gt; facets = Lists.newArrayList(warehouse_facet); List&lt;AttributeKeyAndValue&gt; kvs = Lists.newArrayList(kv); BatchCreateObject createObject = new BatchCreateObject(); createObject.withParentReference(new ObjectReference() .withSelector(parentReference)); createObject.withLinkName(linkName); createObject.withBatchReferenceName(UUID.randomUUID().toString()); createObject.withSchemaFacet(facets); createObject.withObjectAttributeList(kvs); return new BatchWriteOperation().withCreateObject (createObject); }</code></pre> </div> <p>The parameters of this helper method include:</p> <ul> <li><span style="font-family: courier">warehouseName</span> – The name of the warehouse to create in the <span style="font-family: courier">getBatchCreateOperation</span> object.</li> <li><span style="font-family: courier">directorySchemaARN</span> – The Amazon Resource Name (ARN) of the schema applied to the directory.</li> <li><span style="font-family: courier">parentReference</span> – The object reference of the parent object.</li> <li><span style="font-family: courier">linkName</span> – The unique child path from the parent reference where the object should be attached.</li> </ul> <p>I then use this helper method to set up multiple <span style="font-family: courier">create</span> operations for <span style="font-family: courier">NorthAmerica</span>, <span style="font-family: courier">NewYork</span>, <span style="font-family: courier">Phoenix</span>, and <span style="font-family: courier">LasVegas</span>. For the sake of simplicity, I use airport codes to stand for the cities (for example, <span style="font-family: courier">NYC</span> stands for <span style="font-family: courier">NewYork</span>).</p> <div class="hide-language"> <pre><code class="lang-text"> BatchWriteOperation createObjectNA = getBatchCreateOperation( &quot;NA&quot;, directorySchemaARN, &quot;/&quot;, &quot;NorthAmerica&quot;); BatchWriteOperation createObjectNYC = getBatchCreateOperation( &quot;NYC_1&quot;, directorySchemaARN, &quot;/NorthAmerica&quot;, &quot;NewYork&quot;); BatchWriteOperation createObjectPHX = getBatchCreateOperation( &quot;PHX_1&quot;, directorySchemaARN, &quot;/NorthAmerica&quot;, &quot;Phoenix&quot;); BatchWriteOperation createObjectLAS = getBatchCreateOperation( &quot;LAS_1&quot;, directorySchemaARN, &quot;/NorthAmerica&quot;, &quot;LasVegas&quot;); BatchWriteRequest request = new BatchWriteRequest(); request.setDirectoryArn(directoryARN); request.setOperations(Lists.newArrayList( createObjectNA, createObjectNYC, createObjectPHX, createObjectLAS)); client.batchWrite(request);</code></pre> </div> <p>Running the preceding code results in a hierarchy for the network with NA added to the network, as shown in the following diagram.</p> <p><img class="alignnone wp-image-4246 size-full" title="Diagram with NA added to the network" src="https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3e253a90eee5098477c95c23d/2017/07/11/Diagram2-Vineeth-July2017-b.png" alt="" width="900" height="495" /></p> <h3>Using batch read operations</h3> <p>Now, let’s say that after I add <span style="font-family: courier">NorthAmerica</span> to AnyCompany’s global network, an analyst wants to see the updated view of the <span style="font-family: courier">NorthAmerica</span> warehouse network as well as some information about the newly introduced warehouse configurations for the <span style="font-family: courier">Phoenix</span> warehouses. To do this, I can use <a href="http://docs.aws.amazon.com/directoryservice/latest/APIReference/API_BatchRead.html" target="_blank" rel="noopener noreferrer">batch read operations</a> to get the network of warehouses for <span style="font-family: courier">NorthAmerica</span> as well as specifically request the attributes and configurations of the <span style="font-family: courier">Phoenix</span> warehouses.</p> <p>To list the children of the <span style="font-family: courier">NorthAmerica</span> warehouses, I use the <span style="font-family: courier">BatchListObjectChildren</span> API to get all the children at the path, <span style="font-family: courier">/NorthAmerica</span>. Next, I want to view the attributes of the <span style="font-family: courier">Phoenix</span> object, so I use the <span style="font-family: courier">BatchListObjectAttributes</span> API to read all the attributes of the object at <span style="font-family: courier">/NorthAmerica/Phoenix</span>, as shown in the following code example.</p> <div class="hide-language"> <pre><code class="lang-text"> BatchListObjectChildren listObjectChildrenRequest = new BatchListObjectChildren() .withObjectReference(new ObjectReference().withSelector(&quot;/NorthAmerica&quot;)); BatchListObjectAttributes listObjectAttributesRequest = new BatchListObjectAttributes() .withObjectReference(new ObjectReference() .withSelector(&quot;/NorthAmerica/Phoenix&quot;)); BatchReadRequest batchRead = new BatchReadRequest() .withConsistencyLevel(ConsistencyLevel.EVENTUAL) .withDirectoryArn(directoryArn) .withOperations(Lists.newArrayList(listObjectChildrenRequest, listObjectAttributesRequest)); BatchReadResult result = client.batchRead(batchRead);</code></pre> </div> <h3>Exception handling</h3> <p>Batch operations in Cloud Directory might sometimes fail, and it is important to know how to handle such failures, which differ for write operations and read operations.</p> <h4>Batch write operation failures</h4> <p>If a batch write operation fails, Cloud Directory fails the entire batch operation and returns an exception. The exception contains the index of the operation that failed along with the exception type and message. If you see <span style="font-family: courier">RetryableConflictException</span>, you can try again with exponential backoff. A simple way to do this is to double the amount of time you wait each time you get an exception or failure. For example, if your first batch write operation fails, wait 100 milliseconds and try the request again. If the second request fails, wait 200 milliseconds and try again. If the third request fails, wait 400 milliseconds and try again.</p> <h4>Batch read operation failures</h4> <p>If a batch read operation fails, the response contains either a successful response or an exception response. Individual batch read operation failures do not cause the entire batch read operation to fail—Cloud Directory returns individual success or failure responses for each operation.</p> <h3>Limits of batch operations</h3> <p>Batch operations are still constrained by the same Cloud Directory limits as other Cloud Directory APIs. A single batch operation does not limit the number of operations, but the total number of nodes or objects being written or edited in a single batch operation have enforced limits. For example, a total of 20 objects can be written in a single batch operation request to Cloud Directory, regardless of how many individual operations there are within that batch. Similarly, a total of 200 objects can be read in a single batch operation request to Cloud Directory. For more information, see <a href="http://docs.aws.amazon.com/directoryservice/latest/admin-guide/limits.html#limits_cd" target="_blank" rel="noopener noreferrer">limits on batch operations</a>.</p> <h3>Summary</h3> <p>In this post, I have demonstrated how you can use batch operations to operate on multiple objects and simplify making complicated changes across hierarchies. In my next post, I will demonstrate how to use batch references within batch write operations. To learn more about batch operations, see <a href="http://docs.aws.amazon.com/directoryservice/latest/admin-guide/batches.html" target="_blank" rel="noopener noreferrer">Batches</a>, <a href="http://docs.aws.amazon.com/directoryservice/latest/APIReference/API_BatchWrite.html" target="_blank" rel="noopener noreferrer">BatchWrite</a>, and <a href="http://docs.aws.amazon.com/directoryservice/latest/APIReference/API_BatchRead.html" target="_blank" rel="noopener noreferrer">BatchRead</a>.</p> <p>If you have comments about this post, submit them in the “Comments” section below. If you have implementation questions, start a new thread on the <a href="https://forums.aws.amazon.com/forum.jspa?forumID=180" target="_blank" rel="noopener noreferrer">Directory Service forum</a>.</p> <p>– Vineeth</p>