Penn State University Helps Faculty Gain Near-Real-Time Learning Insights Using Instructure and AWS

Penn State University built a learning analytics tool using data from its learning management system (LMS), Canvas—developed by AWS Partner Instructure—to provide real-time alerts about student engagement in a course. The tool, named Course Insights, uses Amazon Simple Storage Service (Amazon S3) and Amazon Elastic Compute Cloud (Amazon EC2) to process nearly 2 TB of raw learning activity data every day. The data is made available through Course Insights as a learning tools interoperability (LTI) feature embedded directly in Canvas. It helps instructors track student engagement, provide differentiated student care, and understand how they might improve course delivery and learning outcomes.

Serving nearly 89,000 students through 24 campuses across the state of Pennsylvania, Penn State University (PSU) was looking for ways to use data to better serve its diverse student body. With the adoption of the Canvas learning management system (LMS) in 2016, this served as a critical system for the university heading into the COVID-19 pandemic. As a customer of both Amazon Web Services (AWS) and AWS Partner Instructure—which makes Canvas—the university quickly adapted its LMS and other services to support remote learning when the pandemic hit. However, the institution soon realized it needed a better way to identify students who might not have been able to make the transition to remote learning.

To solve this problem, the university’s Data Empowered Learning team created a tool that would give academic advisers the information they needed and present it in a way that it was easy to see where intervention could be helpful. Later, the Data Empowered Learning team turned its attention to Course Insights for instructors, built right into Canvas. A central feature of these tools enables academic advisers and instructors to view the aggregated Canvas activity (but not specific actions) charted against a seven-day average for the entire class. This provides some insight into whether a student is engaging in the course at the same times and rates as their peers.

The goal was to help academic advisers and instructors to easily see when a student on their roster wasn’t engaging with coursework, enabling interventions to happen before a student gets too far behind. “Our faculty is hungry for data to support students,” says Ben Hellar, manager of Data Empowered Learning at Penn State. “But most of the analytics that they had access to were post-course analytics. So how do we understand which students need help and intervention now? This motivation helped us design tools that deliver real-time data insight during the semester so that instructors and academic advisers know when interventions can be most impactful.”

Penn State’s Data Empowered Learning team spent a great deal of time working with faculty to understand their student support, course delivery, and pedagogical needs, seeking to understand what questions they ask themselves about the courses they teach. From these discussions, the team learned that instructors had a broad need to understand the demographics of the population of students enrolled in their courses—for example, data about students’ previously enrolled courses could help instructors adapt their teaching approach each semester based on students’ level of understanding going in.

Instructors also said it was important for the tool to be accessible through their existing Canvas dashboard. Hellar says, “We received feedback from instructors that told us, ‘I don’t want to use another tool—I live in Canvas—that is where I teach. So if you’re going to give me any new analytic data, put it in Canvas.’” Penn State was able to build Course Insights using Instructure’s detailed documentation for Canvas and its support for learning tools interoperability (LTI).

The Course Insights pilot rolled out to a small number of faculty members in the fall of 2021 and has grown steadily since then. Although Penn State built the tool in-house, the team meets regularly with Instructure and interacts with the company almost daily. “Penn State partners really closely with us,” says Melissa Loble, chief academic officer at Instructure. “Our open, standards-first approach, alongside our detailed documentation and very active community, helped to enable the work that Penn State has done with Canvas—and we’re there to support them whenever they need us.”

Since launching as a limited-access pilot, Course Insights has been rolled out to more than 250 instructors who teach over 4,000 courses. To provide instructors with data insights at scale for every canvas course and every enrolled student at Penn State—up to 89,000 students—the tool analyzes 2 TB of learning activity data every day from sources including Canvas, Kaltura, Packback, TopHat, and other learning tools, and also includes advising data from Starfish and student data from the university’s Student Information System (SIS).

The data is stored using Amazon Simple Storage Service (Amazon S3), object storage built to retrieve any amount of data from anywhere, and processed using Amazon Elastic Compute Cloud (Amazon EC2), which offers secure and resizable compute capacity for virtually any workload, with instances spun up and down according to need. Using Course Insights, instructors can see a comprehensive view of learning activity from the supported learning tools showing a rolling 7-day average of student activity in each of their courses. Anomalous activity is highlighted—orange for periods of zero activity, yellow for activity levels that have fallen below 50 percent of the course average activity of their peers—making it easy for instructors to identify students who need immediate support to reduce the chances of failing or dropping out of class.

Thanks to Course Insights, the Data Empowered Learning Team of Penn State Information Technology won a Platinum 2023 Learning Impact Award from the 1EdTech Consortium. “This award helps validate that our approach of delivering learning analytic data to instructors has value not only to the institution but for the industry at large,” says Hellar. And Instructure itself is building on the success that Penn State has had with Course Insights. “The incredible work they’ve done has influenced where we’ve taken Canvas data and how we think about our documentation, our standards, our APIs,” says Loble. “We’ve used this to continue to inform our roadmap.” Instructure also updated Canvas in late 2023 to reduce the application’s latency—previously, the data displayed could be more than 24 hours old, but the new version brings reporting closer to real time.

Penn State continues to refine Course Insights so that it can draw out even more data insights to improve education for its students. For example, the team is currently developing additional analytics to support faculty and administrators engaged in program and curriculum development at the department, college, and campus levels of the institution.

Although Course Insights has focused on differentiated student care and course delivery, the ability to better understand the structure of a course and how students engage in the course material at scale will be a powerful tool to help support strategic initiatives around retention and the closing of equity gaps.

Penn State communicates regularly with Instructure to share what it’s learned so that other schools that use Canvas can do the same. “We want all students to learn,” says Andy Fisher, manager of learning engagement at Penn State. “The nice thing about working with Instructure is we can share information and talk about what we’re building and why we’re building it. This is different with education. We need to invent and we need to build what’s next. Some of this is very custom. But we always have that conversation around, if we build something, can another university use it?”