The experts behind AWS Certification exams

As of September 2019, there were more than 240,000 individuals with an AWS Certification. If you are one of them, you may have wondered how we create the AWS Certification exams and the questions therein. As one of the AWS Certification technical architects at AWS, I am going to share a behind-the-scenes look at our exam development process.

You may be surprised to know that no one on the AWS Certification team writes the exam questions or even decides what questions to ask. AWS Subject Matter Experts (SMEs)—includes AWS employees and customers with experience using our products and services—volunteer their time to write and review exam content and questions and help us set the bar for competency in a particular role or technical domain. We rely on solutions architects, developers, systems operators, database administrators, and others whose work aligns with the expertise validated by our exams.

The subject matter experts from the database standard setting workshop.

We now offer 12 different AWS Certifications to cover the varied job roles and technical expertise of individuals who use AWS services. So, before we can begin creating a new AWS exam we have to look for SMEs who have experience in a matching role.

For our new AWS Certified Database – Specialty certification, we recruited database engineers and programmers who have used DynamoDB, RDS, Aurora, Neptune, and Quantum Ledger. The SME teams engaged throughout the year-long process included a mix of customers, partners, and AWS employees to ensure the certification reflected diverse, relevant experience.

The Exam Development Process
At AWS, we follow a robust process of exam development (diagram immediately following). Several of the various steps of this process depend upon the participation of SMEs.

Job Task Analysis (JTA)
The JTA is the first step in the exam creation process. A select group of SMEs participate in a workshop where they discuss the core competencies that should be validated for the particular certification. By the end of the workshop they have created a content outline for the exam, which we refer to as the exam blueprint. The blueprint defines the content areas and includes their relative percentages. The exam questions will be written and placed on the exam following the blueprint guidelines. For the AWS Certified Database – Specialty exam, the SMEs established the following knowledge domains:

1. Workload-Specific Database Design (26%)
2. Deployment and Migration (20%)
3. Management and Operations (18%)
4. Monitoring and Troubleshooting (18%)
5. Database Security (18%)

Blue Print Survey
Following the JTA, we expand the group of SMEs by distributing the blueprint to hundreds of industry experts to validate the content areas and ensure broad applicability. Over a period of several weeks, the experts complete a survey and rate each blueprint objective for how critical the knowledge is and how frequently it is used. The completed exam blueprint is published as part of the exam guide available on our website.

Item Writing
Now that the content areas for the exam are defined, SMEs can begin to write the exam questions, which we also refer to as items, according to the defined blueprint. SMEs may write together in item writing workshops, or write items remotely depending upon their schedules. SMEs commonly draw upon their experience to create items, which are based upon actual customer use cases or real-world problems.

Item Review
Exam items must make it through several layers of review and approval before they are ready to be tested on a beta exam. During the item writing workshops, the SMEs review items together as a group and check each for technical accuracy, clarity, appropriateness, and fairness. Items are typically refined and reviewed several times by the author and group before they are approved.

Beta Exam
Exam items go through one final step before appearing on a live exam. For all new AWS Certification exams and versions, approved items must be tested on a beta exam. Exam candidates can choose to take the beta exam and, if they meet the passing score, become certified before the live exam is published.

The beta exams contain more questions than the live exams. The purpose is to see how candidates respond to each item before it can be included in the calculation of the passing score, and moved to the live exam. For example, items that nearly every candidate got wrong, or ones that nearly everyone answered correctly, would not be counted. We collect and analyze this data, and notify candidates who take the beta exam of their score at the close of the beta period.

Standard Setting
Before publishing the live exam, a group of SMEs reviews the items that tested well on the beta exam. Together they determine the items to be included on the live exam and the passing score. To make these determinations, SMEs attend a Standard Setting workshop where they take the exam and rate each item for difficulty. The passing score is determined by the combined ratings across all items.

Live Exam
Shortly after the Standard Setting, the live exam will be published. A candidate who meets or exceeds the passing score has met the minimum qualifications for certification.

Item Refresh
To keep our exams current, we continue the process of writing and reviewing items. To evaluate these newly created items, we place them on the live exam but don’t use them to judge candidate competency. If they statistically perform well, they may be used in the future to replace items in the same content area. On the AWS Certified Database – Specialty exam, for example, a subset of the 65 items on the exam are being tested and don’t count in the candidate’s score. However, the candidates are not told which items are unscored.

Back to the Beginning
Because AWS engineers continue to innovate and introduce new features and services, we never stop writing, testing and adding new items to the exam. However, as best practices evolve and expectations for professionals in particular roles change, we must periodically begin this whole process from the beginning. We start with a new JTA, which may create an entirely new blueprint and lead us to create a new version of the exam. We have recently done this for the AWS Certified Solutions Architect – Associate exam and released a new version.

The rigor of our exam development process ensures that AWS Certifications remain current and validate that recipients have met the high bar set by AWS SMEs. Every part of our process relies heavily on the participation of SMEs.

While participating in exam development, SMEs share use cases and lessons learned, gain insights into AWS services and feature sets, and make valuable connections with peers. If you are interested in becoming an AWS SME for our AWS Certification development process, please complete an application form and the AWS Certification team will reach out to you with the next steps.