How to Implement a General Solution for Federated API/CLI Access Using SAML 2.0

Important note from July 18, 2019: The original version of this blog uses Python2.x scripts. We now have Python3.x scripts that you can download here:

Form-based authentication version of the Python3.x script

AD FS 3.0-specific version of the Python3.x script

Note from May 24, 2019: The features and services described in this post have changed since the post was published and the procedures described might be out of date and no longer accurate. Specifically, the procedures have not been tested with ADFS 4.0 or Python 3.x. If we update this post or create a replacement, we’ll add a notification about it here.

Note: Active Directory Federation Services (AD FS) 3.0 uses form-based authentication by default. If you are using AD FS 3.0 in this configuration, use the solution presented in this post.

In my earlier post, How to Implement Federated API and CLI Access Using SAML 2.0 and AD FS, I walked through how to implement federated API and CLI access by using AD FS and some Python code. Since then, I’ve received a number of requests asking if the same approach could be used with other identity providers that support SAML (Security Assertion Markup Language) 2.0. I am now happy to answer that question with “most definitely!”

In this blog post, I’ll show you how to extend my previous implementation to use form-based authentication, which is supported by nearly all Identity Providers (IdPs).

Getting started

To follow along with this post, you must have:

Already integrated your IdP with your AWS account for single sign-on (SSO) console access. If you need help getting up to this point, see this documentation, which includes links to a number of installation guides for common IdPs. Note: If you don’t yet have a SAML IdP, check out Single Sign-On: Integrating AWS, OpenLDAP, and Shibboleth to get started.
Enabled form-based authentication within your IdP configuration. During form-based authentication, the IdP presents the user with an HTML login form in which they enter their user name and password.
Read my earlier post and completed all the “Getting started” steps. The majority of the integration is identical, and this post will focus only on the specific differences between NTLM-based authentication and form-based authentication.
Downloaded either the Python2.x form-based authentication version of the script, the Python3.x form-based authentication version, the Python2.x AD FS 3.0-specific version of the script, or the Python3.x AD FS 3.0-specific version.

Using the form-based authentication script

In order to get started, you’ll need to know the IdP-initiated login URL. This is simply the URL that you are currently using for SSO access to the AWS Management Console. As an example, I’m using Shibboleth 2.x where the IdP-initiated login URL takes the form of: https://<fqdn>:<port>/idp/profile/SAML2/Unsolicited/SSO?providerId=urn:amazon:webservices. If I place this URL in my browser’s address field, I’m presented with the IdP login page, as shown in the following image.

Image of the IdP login page

You will notice that before the login page is displayed, the IdP has issued a number of HTTP redirects in the background. After this process, the URL shown in the toolbar is different from the one that initiated the process. Make sure you have the original IdP-initiated login URL, and enter it in the new script as the idpentryurl.

idpentryurl = 'https://<fqdn>:<port>/idp/profile/SAML2/Unsolicited/SSO?providerId=urn:amazon:webservices'

In the new script, you will notice the following section, which is for the form-based authentication implementation.

# Programmatically get the SAML assertion
# Opens the initial IdP url and follows all of the HTTP302 redirects, and
# gets the resulting login page
formresponse = session.get(idpentryurl, verify=sslverification)
# Capture the idpauthformsubmiturl, which is the final url after all the 302s
idpauthformsubmiturl = formresponse.url
				 
# Parse the response and extract all the necessary values
# in order to build a dictionary of all of the form values the IdP expects
formsoup = BeautifulSoup(formresponse.text.decode('utf8'))
payload = {}
					 
for inputtag in formsoup.find_all(re.compile('(INPUT|input)')):
    name = inputtag.get('name','')
    value = inputtag.get('value','')
    if "user" in name.lower():
        #Make an educated guess that this is correct field for username
        payload[name] = username
    elif "email" in name.lower():
        #Some IdPs also label the username field as 'email'
        payload[name] = username
    elif "pass" in name.lower():
        #Make an educated guess that this is correct field for password
        payload[name] = password
    else:
        #Populate the parameter with existing value (picks up hidden fields in the login form)
        payload[name] = value
					 
# Debug the parameter payload if needed
# Use with caution since this will print sensitive output to the screen
#print payload
					 
# Some IdPs don't explicitly set a form action, but if one is set we should
# build the idpauthformsubmiturl by combining the scheme and hostname
# from the entry url with the form action target
# If the action tag doesn't exist, we just stick with the idpauthformsubmiturl above
for inputtag in formsoup.find_all(re.compile('(FORM|form)')):
    action = inputtag.get('action')
    if action:
        parsedurl = urlparse(idpentryurl)
        idpauthformsubmiturl = parsedurl.scheme + "://" + parsedurl.netloc + action
					 
# Performs the submission of the login form with the above post data
response = session.post(
    idpauthformsubmiturl, data=payload, verify=sslverification)
					 
# Debug the response if needed
#print (response.text)

This code uses the Python requests module to initiate the IdP connection and follow all of the HTTP redirects to the IdP login page. The resulting login page is then parsed using BeautifulSoup in two ways. First, all of the input fields are captured and populated with the values from the generated form. During this process, the code determines where to inject the username and password provided by the user. Second, if present, the form action is captured as the idpauthformsubmiturl so that the code knows where to submit the form. With the parameter payload assembled, the final step posts the parameter payload back to the IdP. If the authentication is successful, the response will include the SAML authentication response necessary to request temporary security credentials from the AWS Security Token Service (STS).

Initial error handling

You may have noticed that the code that I provide in this post and in my earlier post doesn’t include proper error handling. I did this intentionally for two reasons:

This enables the presentation of the “happy path” in a cleaner and more compact fashion that can be understood and explained with relative ease.
I want it to be very clear that this is proof-of-concept code that requires adaptation before real use.

However, upon further reflection, there is one check that I should have left in to handle the most basic error condition—incorrect credentials. Locate the following section of the new script.

# Better error handling is required for production use.
if (assertion == ''):
    #TODO: Insert valid error checking/handling
    print 'Response did not contain a valid SAML assertion'
    sys.exit(0)

This error-handling snippet highlights a challenge with this approach—the IdP returns a HTTP 200 success code regardless of whether or not the authentication was successful. Any descriptive error message coming back from the IdP is embedded in the HTML and will need to be parsed out.

Putting it all together

The new script produces the following output.

janedoe@Ubuntu64:/tmp$ ./samlapi_formauth.py
Username: janedoe
Password: ****************
 
Please choose the role you would like to assume:
[ 0 ]:  arn:aws:iam::012345678987:role/Shib-Administrators
[ 1 ]:  arn:aws:iam::012345678987:role/Shib-Operators
Selection:  1
 
 
---------------------------------------------------------------
Your new access key pair has been stored in the AWS configuration file /home/janedoe/.aws/credentials under the saml profile.
Note that it will expire at 2015-07-16T17:16:20Z.
After this time, you may safely rerun this script to refresh your access key pair.
To use this credential, call the AWS CLI with the --profile option (e.g., aws --profile saml ec2 describe-instances).
---------------------------------------------------------------
 
Simple API example listing all S3 buckets:
[<Bucket: mybucket1>, <Bucket: mybucket2>, <Bucket: mybucket3>, <Bucket: mybucket4>, <Bucket: mybucket5>]

As a refresher, here’s what you’re seeing in the output:

The script prompts a federated user to enter credentials (user name/password). These credentials are used to securely authenticate and authorize the user against the configured IdP.
The script inspects the returned SAML authentication response and determines the AWS Identity and Access Management (IAM) roles the user has been authorized to assume. After she selects the desired role, the script uses AWS STS to retrieve temporary security credentials.
The utility automatically writes these credentials to the user’s local AWS credentials file, and she can begin issuing AWS API or CLI calls.

If you need guidance about how to reference the SAML profile in the various AWS SDKs, I’d suggest A New and Standardized Way to Manage Credentials in the AWS SDKs.

If you are having trouble

If you are having trouble getting things working, there are a couple of areas to focus on first. Most notably, the new script makes presumptions about the names of the fields that store the username and password. If you inspect the HTML of the IdP login page from your browser and see that these fields are named otherwise, replace the string user or pass in the code to match. You can also uncomment the print payload line temporarily to test that things are populating correctly.

You might also find that your IdP tries different authentication mechanisms (e.g., Kerberos) first and then falls back to form-based authentication. Depending on the specific implementation of such a fallback, you might have to do some additional inspection and parsing of the flow to arrive at the proper IdP login page. If you suspect this might be the case, additional print statements, such as print formresponse.text.decode(‘utf8’) are a good simple debugging tool to figure out what’s going on.

Wrapping it up

The code presented in this post has been developed over time while working with a wide range of different AWS customers and IdPs. In most cases, while not as elegant as a tightly integrated API-based approach, the code’s beauty lies in the fact that it rarely needs further modification beyond setting the proper IdP-initiated login URL. Having said that, I haven’t had the opportunity to work with every IdP out there. If you have any issues with the implementation, please post a comment below with the relevant details, and I’ll make every possible effort to extend the solution appropriately.

I hope you enjoyed this post, and I look forward to your continued feedback and tales of success using this solution.

– Quint

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

AWS Security Blog