Migrating Users to Amazon Cognito User Pools

November 2, 2023: An update to this post was published on the AWS Security Blog. Please see this post for the most up-to-date info.

Many customers ask about the best way to migrate their existing users in to Amazon Cognito User Pools. In this blog post, we describe the options and provide step-by-step instructions on how to do it.

Amazon Cognito User Pools offer a fully managed user directory so you can easily add sign-up and sign-in to your mobile app or web application. User pools scale to hundreds of millions of users and provide simple, secure, and easy to use options for you as a developer. You can take advantage of enhanced security features, such as email and phone number verification, and multi-factor authentication. User Pools provide a customizable user interface for sign-up and sign-in, and built-in federation with Google, Facebook, Login with Amazon, and SAML-based identity providers. User pool sign-in is based on OpenID Connect and OAuth 2.0 standards.

The main consideration when migrating users across directories is handling existing user passwords. Ideally, users can continue to use their existing passwords so that their experience is seamless. However, security best practices dictate that passwords are never stored directly as clear text in a user store. Instead, they are used to compute cryptographic hashes and verifiers that can later be used to verify submitted passwords. This means that you can’t simply export passwords in clear text form from an existing user directory and import them into a user pool. You can ask your users to choose a new password during the migration. Or if you want to retain the existing passwords, you need to retain access to the existing hashes and verifiers, at least during the migration period. The following two methods define the two approaches for migrating existing users into a user pool:

Batch user import: This approach is relatively quick and easy, but it requires users to reset their passwords. With this approach, you export your existing users into a comma separated (.csv) file, and then upload this .csv file to import users into a user pool. All of the user attributes except passwords can be included and mapped to attributes in the target user pool. The imported user data must include an email address or phone number for each user so each user can receive the code required to reset the password. Batch user import can also be used for an incremental migration to import users into a user pool of existing users.

One-at-a-time user migration: This approach requires more setup, but it allows users to continue using their existing passwords. Users are migrated into a user pool each time they sign in. When a user signs in, you first try to sign the user in to the user pool. If that sign-in fails, because the user does not exist in the user pool, you sign the user in through the existing user directory, capture the user name and password, and then silently sign them up in the user pool.

In the following sections, we describe the batch user import method in more detail, and then walk through the steps of one-at-a-time user migrations.

Batch user import

Batch import of users into an Amazon Cognito User Pool is done by uploading a .csv file that contains user profile data, including user names, email addresses, phone numbers, and so on. You can download a template .csv file for your user pool from the Amazon Cognito console. Navigate to the Users and groups tab of an existing user pool, and click the Import users button. You can then export your existing user data from your existing user directory or store into the .csv file, matching the column headings in the template. Since the batch import feature does not import passwords, these users need to reset their passwords. Each user must have an email address or a phone number that is marked as verified in the .csv file. You need to create an AWS Identity and Access Management (IAM) role that grants Amazon Cognito permission to write to Amazon CloudWatch Logs in your account, so that Amazon Cognito can provide logs on successful imports and errors.

After you have your .csv file and IAM role created, you are ready to import users into the user pool. You can create, run, and monitor import jobs from the Amazon Cognito console or via the SDK or CLI. From the console, again navigate to the Users and groups tab of an existing user pool, click Import users, and then click Create import job to create the job. Depending on the size of the .csv file, the job can run for minutes or hours, and you can follow the status from that same page in the Amazon Cognito console.

Batch imported users need to reset their passwords. If they attempt to sign in, Amazon Cognito will return PasswordResetRequiredException from the sign-in API, and the app should direct the user into the ForgotPassword flow.

For more information, see Importing Users into User Pools in the Amazon Cognito Developer Guide.

One-at-a-time user migration

The one-at-a-time user migration method involves first attempting to sign in the user through the Amazon Cognito User Pool. Then, if that sign-in fails, you sign them in through the existing user directory and capture the user name and password to silently create the user in the user pool. You could build all that logic into the client app, but building the migration logic in a secure backend allows you to access admin APIs for a more robust implementation. It also keeps the client app simpler, which could be a big benefit if you have multiple client apps. In AWS, the combination of Amazon API Gateway and AWS Lambda is a good choice for building that secure backend as a microservice, so we use them for the example.

We show an approach where the app is updated at the start to migrate users and to use a user pool to sign in users who have migrated. An alternative approach is to start by updating only the existing identity system to create users in a user pool on the backend as users sign in, and then switch over the app to sign in with Amazon Cognito after a sufficient number of users have migrated. That alternative approach migrates users before the app is updated, but it also requires that you propagate new user signups and user updates from the existing identity system to a user pool.

In this post, we skip the steps to create an Amazon Cognito User Pool. You can find this information in the Amazon Cognito Developer Guide. We also do not cover the basics of how to use AWS Lambda or Amazon API Gateway. You can learn about these services in the AWS Lambda Developer Guide and the Amazon API Gateway Developer Guide. When you set up the Lambda function, give it permissions in the execution role to access your user pool in addition to permissions for CloudWatch Logs.

The diagram below illustrates the steps for the sign-in flow. At a high level is the mobile or web app, which first tries to sign in the user in the user pool. If that fails, you call a backend microservice to create the user in the user pool, and then return to the app to try signing in again. This approach keeps the logic in the app simple. It allows the app to use the Amazon Cognito SDK to sign in users in the standard way. It inserts an extra call to the backend microservice and a retry of signing in.

The flow starts in the mobile or web app, which attempts to sign in the user using the AWS Mobile SDK for Android, the AWS Mobile SDK for iOS or the Amazon Cognito Identity SDK for JavaScript. If that sign-in fails because the user’s credentials are not correct, the app calls the backend Lambda function over HTTPS with the user’s user name and password to attempt to migrate the user. The backend will first check to see if the user is in the user pool using the AdminGetUser() API. The following Node.js example shows how to get started:

'use strict';
var CLIENT_ID = '<YOUR USER POOL APP CLIENT ID>';
var USER_POOL_ID = '<YOUR USER POOL ID>';
var AWS = require('aws-sdk');
AWS.config.update({region: '<THE REGION OF YOUR USER POOL>'});

console.log('Loading event');
			
exports.handler = function(event, context, callback) {
	var username = event.username;
	var password = event.password;

	var cognitoidentityserviceprovider = new AWS.CognitoIdentityServiceProvider();
  
	//Check to see if the user exists in the User Pool using AdminGetUser()
	var params = {UserPoolId: USER_POOL_ID, Username: username};
	cognitoidentityserviceprovider.adminGetUser(params, function(lookup_err, data) {
		if (lookup_err && lookup_err.code === "UserNotFoundException") {
			// User does not exist in the User Pool, try to migrate
			console.log("User does not exist in User Pool, attempting migration: " + username);
			
			//***********************************************************************
			// Attempt to sign in the user or verify the password with existing system
			// (shown in the next section of this article)
			//***********************************************************************
				
		} else {
			//User exists in the User Pool, so tell the app not to retry sign-in
			console.log("User exists in User Pool so no migration: " + username);
			callback(null, "NO_RETRY");
			return;
		}
	});		

};

If the user is not already in the user pool, attempt to migrate the user. We describe that next. If the user is already in the user pool, then the user may have provided an incorrect password. In that case, send a NO_RETRY response back to the app because the app should present the original sign-in error to the user rather than retry sign-in with the user pool.

If the user does not already exist in the user pool, the migration attempt starts. Now attempt to sign in the user and verify the user’s password in the existing user directory. That may involve a sign-in attempt or a check of the password against a stored hash. You need to customize this step for your existing user directory. The following example shows a simple check for a user name and password hash.

			//Attempt to sign in the user or verify the password with existing system
			//For demonstration we are doing a simple check of the username and a password hash
			if (username === 'migratinguser@gmail.com' && simpleHash(password)== 1075820758) {
				console.log("Verified user with existing system: " + username);
				
				//Read any user attributes to be migrated from existing system
				var name = 'john'; //simple example of a profile attribute that could be migrated from the existing system

				//***********************************************************************
				// Create the user in the Amazon Cognito User Pool
				// (shown in the next section of this article)
				//***********************************************************************

			} else {
				//User does not exist in the existing system, so tell the app not to retry sign-in
				console.log("User does not exist in User Pool or existing system: " + username);
				callback(null, "NO_RETRY");
				return;
			}

If you can sign in the user or verify the password in the existing system, you also read in any user attributes that you want to migrate. The following example migrates the user’s name and creates a UUID that is used for the username. Email addresses are used as aliases to sign in. At that point, you can create the user in the user pool, which we describe next. If the user does not exist in the existing system, you again send a NO_RETRY response back to the app because the app should present the original sign-in error to the user rather than retry sign-in with the user pool.

Creating the user in the user pool is a two-step process. First, use the AdminCreateUser() API to create the user with all of the attributes and a temporary password. At the same time, also set the email address as verified so the user does not need to re-verify it. Second, sign in as the user so that you can set the user’s permanent password.

				//Create the user with AdminCreateUser()
				params = {
					UserPoolId: USER_POOL_ID,
					Username: username, 
					MessageAction: 'SUPPRESS', //suppress the sending of an invitation to the user
					TemporaryPassword: password,
					UserAttributes: [
						{Name: 'name', Value: name}, 
						{Name: 'email', Value: username}, //using sign-in with email, so username is email
						{Name: 'email_verified', Value: 'true'}]
				};
				cognitoidentityserviceprovider.adminCreateUser(params, function(err, data) {
					if (err) {
						console.log('Failed to Create migrating user in User Pool: ' + username);
						callback(err);
						return;								
					} else {
						//Successfully created the migrating user in the User Pool
						console.log("Successful AdminCreateUser for migrating user: " + username);

						//Now sign in the migrated user to set the permanent password and confirm the user
						params = {
							AuthFlow: 'ADMIN_NO_SRP_AUTH',
							ClientId: CLIENT_ID,
							UserPoolId: USER_POOL_ID,
							AuthParameters: {USERNAME: username, PASSWORD: password}
						};

						cognitoidentityserviceprovider.adminInitiateAuth(params, function(signin_err, data) {
							if (signin_err) {
								console.log('Failed to sign in migrated user: ' + username);
								console.log(signin_err, signin_err.stack);
								callback(signin_err);
							} else {
								//Handle the response to set the password

								//Confirm the challenge name is NEW_PASSWORD_REQUIRED
								if (data.ChallengeName !== "NEW_PASSWORD_REQUIRED") {
									// unexpected challenge name - log and exit
									console.log("Unexpected challenge name after adminInitiateAuth (" + data.ChallengeName + "), migrating user created, but password not set");
								 callback("Unexpected challenge name");
								}

								params = {
									ChallengeName: "NEW_PASSWORD_REQUIRED",
									ClientId: CLIENT_ID,
									UserPoolId: USER_POOL_ID,
									ChallengeResponses: {
										'NEW_PASSWORD': password, 'USERNAME': data.ChallengeParameters.USER_ID_FOR_SRP
									},
									Session: data.Session
								};
								cognitoidentityserviceprovider.adminRespondToAuthChallenge(params, function(err, data) {
									if (err) console.log(err, err.stack); // an error occurred
									else {	 // successful response
										console.log('Successful response from RespondToAuthChallenge: ' + username);
										callback(null, "RETRY");  // Tell client to retry sign-in
										return;
									}
								});
							}
						});					
					}
				});	

The user is now migrated from the existing user directory to the user pool. Send a RETRY response back to the app because the app should now retry sign-in with the user pool to sign in the migrated user. You can see all of the code together at the bottom of this post.

One-at-a-time user migration – forgot password

You might have thought we were done with the one-at-a-time migration. For a robust migration, you also need to consider users who have forgotten their passwords and clicked a link or button to start the ForgotPassword flow. These might be users who need to migrate, but these users cannot complete the migration flow because they cannot sign in or be verified with the existing user directory. The diagram below illustrates the steps for handling a forgotten password flow for migration:

When users have forgotten their passwords, do not try to migrate their existing passwords to the user pool. Instead, migrate other attributes in their user profile first. So before calling ForgotPassword() for the user pool, call a backend microservice to attempt to migrate the user profile. Then return to the app to initiate the ForgotPassword flow. These users need to reset their passwords, so they need to have a verified email address or phone number migrated from the existing user directory. There is sample code for this flow at the bottom of this post.

Summary and best practices

In this article, we described the two basic approaches for migrating users into an Amazon Cognito User Pool. The batch method is easier to implement, but it does not preserve user passwords like the one-at-a-time migration does. You can decide what’s best for you. The one-at-a-time user migration is a best practice. The one-at-a-time migration is transparent to users and avoids the potential drop off of users that can occur when users need to reset their passwords. We hope with these explanations and code samples, you can set up either approach quickly.