NCFE Makes Award Decisions Easier and Faster with RM’s Judgment Tool on AWS

Every year, NCFE’s Assessment Innovation Fund (AIF) provides up to £125,000 to two large-scale projects aimed at helping adult learners in the UK use digital technology to improve their knowledge and skills for the workplace. It also presents annual Aspiration Awards in 6 categories to recognize outstanding success among UK learners, apprentices, educators, support staff, and educational organizations.

The number of applications for both the AIF and Aspiration Awards has broken records in recent years, putting a strain on NCFE employees who had to spend hours reviewing that documentation using a complex rubric—a set of criteria for scoring—to make their decisions. It’s a process called absolute judgment.

Gray Mytton, assessment innovation manager at NCFE, says the process had become overwhelming for the 35 in-house judges, each of whom had to spend at least a half-day reviewing applications and making decisions to create a shortlist of awards contenders. “It was quite a large amount of time for the organization,” he says. “We were seeing people not completing their judgment in the window that we needed because of the commitment required. We knew it was a lot of time already and we were expecting a growth in applications, so it would be even more time.”

While attending various conferences to identify innovation opportunities, Mytton met Mark House, senior product manager for RM Compare, and discussed the possibility of testing RM’s algorithmic tool for comparative judgment. With a new awards season approaching, Mytton decided to begin testing the RM tool to see if it would make the shortlisting process easier and fairer.

Several months before a new round of AIF applications was expected, NCFE began testing RM Compare’s approach to decision making. “In education, we tend to treasure what we can measure, and what people are telling us is that they want to measure what they treasure,” says RM’s House. “We don’t really want to have this idea that people are writing to the test.”

Rather than using a complex rubric, RM’s software-as-a-service (SaaS) tool presents decision makers with two choices at a time and prompts them to choose which of the two better meets a simple criterion such as, “Which one made the biggest difference for the community?” The tool presents multiple pairs of options for judgment, and judges can make many choices in just a few minutes, then take a break before returning for more decisions at another time. Using this approach, every application is assessed by every judge multiple times. RM Compare’s algorithm then ranks all the applications based on each judge’s choices.

The tool uses Amazon CloudFront to securely deliver content with low latency and high transfer speeds, and AWS Auto Scaling to optimize the application’s performance and costs. It also uses Amazon Elastic Compute Cloud (Amazon EC2) for secure and resizable compute capacity, Amazon Simple Storage Service (Amazon S3) for object storage, and Amazon Virtual Private Cloud (Amazon VPC) to define and launch AWS resources in a logically isolated virtual network. “Could we solve this challenge without AWS?” asks House. “No, we couldn’t. The only reason we didn’t do comparative judgment is because we didn’t have compute. Right now, we’ve got compute, we can do it. So why would we not do comparative judgment?”

To test the effectiveness of RM’s approach, Mytton included 4 applications from the previous year’s awards along with the new awards for judges to consider. “The judges didn’t know if they were last year’s or this year’s, but we obviously knew and we kept an eye on them and how they were ranked,” he says. “The results showed that judges ranked the previous year’s awards similarly to how they were ranked under the old rubric.”

Encouraged by that success, NCFE in 2024 used the RM tool to create a shortlist of contenders for both the AIF and Aspiration Awards. “It seemed to be even more accurate than what we would have previously done,” says Danielle McCullough, communications and events manager at NCFE. “We knew we wanted the nominations to be seen by judges a lot more times than before. Originally, we’d struggled to get enough colleagues involved due to their own high workloads. But all the judges who completed Gray’s shortlisting wanted to do the Aspiration Awards because they’d seen how easy it was. It was a lot quicker.”

After shortlisting both sets of applications, NCFE found that the RM tool had reduced the time that judges spent on AIF applications by 58 percent, and it also reduced the time for Aspiration Awards decisions by 43 percent. Using the tool also increased the number of times that each application was reviewed by a judge: from 8 reviews per application to 27 for the AIF, and from 2 reviews per application to 12 for the Aspiration Awards. The solution has saved 95 hours of judges’ time and resulted in savings on staff costs of £2,000. In addition to the increase in speed, up to 6 times as many applications were judged.

The RM tool also made it possible for many more judges to participate, increasing the fairness of the decisions made. NCFE is now planning to use the RM tool for future funding and awards decisions. It is also considering adopting the RM tool for decision making in other areas, such as shortlisting resumes for hiring decisions, choosing vendors for procurement decisions, and even prioritizing projects in the project management office, where it’s important to make the best use of limited resources.

Beyond enabling faster and easier decision making by the judges, the biggest benefit of using RM’s tool has been improved fairness, says Mytton. “Being an awarding organization and prioritizing fairness, I think this way of doing things is fairer,” he says. “The fact that it was faster is almost a nice-to-have in this case.”