UFV Builds Common Data Lake House Architecture on AWS, Enhancing Governance and Quality

UFV is able to control data access security in several ways. On one hand, it allows access to certain users through AWS Identity and Access Management (AWS IAM) policies, which lets UFV specify who or what can access services and resources in AWS, centrally manage fine-grained permissions, and analyze access to refine permissions across AWS. On the other hand, it has secured access to data on AWS Lake Formation, a service that is used to create secure data lakes, by using different data cataloging techniques to grant permissions through tags.

“At the university, we believe that the combination of both methods is the key to making sure that the data-access solution is reliable and robust,” says José Antonio Marcos, chief information officer of UFV. Now, the university enjoys unified access to data across its campuses and can generate reports in near real time.

Based in Madrid, Spain, UFV is a private Roman Catholic higher education institution with a mission to train professionals who seek to transform society. As a growing university, it continues to attract more students and introduce new technologies in its seven facilities, Clinical Simulating Center, Motor Sports Institute, UFV College, postgraduate, and doctoral schools. In its commitment to excellence, UFV is adopting new teaching and learning methodologies, resulting in new learning experiences. In this context, the university added five new graduate degrees, including an artificial intelligence systems engineering degree. Due to this growth, UFV’s data increases every day.

As its technology landscape became more complex, UFV encountered difficulties managing data and maintaining robust governance. Its applications did not have a defined architecture, nor did they integrate with other data sources. Seeking a solution, UFV turned to AWS Professional Services, a global team of experts that helps organizations realize their desired business outcomes when using AWS.

Using an AWS development environment, the UFV and AWS Professional Services teams built a data lake house architecture in 25 1.5-hour sessions. It took only 2 months for UFV to develop sufficient skills to build similar solutions in different environments.

“Guided by AWS Professional Services, we have delivered a cloud architecture that meets UFV’s data needs,” says Marcos. “We followed a flexible work methodology to provide deep value through the transfer of knowledge and good practices, which facilitates the empowerment of our future organizational growth.”

UFV has integrated four of its core data sources with the data lake: UXXI, a leading academic management system in Spain; Canvas, UFV’s learning management system; Salesforce customer relationship management and cloud computing software; and Gestión Academica, an on-premises data source. Using AWS Glue, a serverless data integration service, UFV can prepare, move, and integrate data from these sources into the data lake using a common workflow. This process is automated and requires little intervention from the UFV team. Data from multiple repositories are stored on Amazon Simple Storage Service (Amazon S3), an object storage service offering scalability, data availability, security, and performance. The solution is highly scalable, making it simple to quickly integrate new sources into the data lake as needed.

By making data available in a common location, UFV has saved both time and money. In one example, the university developed an automated workflow to extract data from its Canvas learning management system using multiple AWS services, including Amazon S3; AWS Lambda, a serverless, event-driven compute service; and Amazon EventBridge, a serverless event bus. By automating this process, UFV estimates that it will save €20,000 per year. The university can also generate data-management reports in near real time. “Before the implementation of AWS services, data-management reporting usually took 2–3 days,” says Sergio Martín Míguez, cloud systems developer at UFV. “Now, with our data lake on AWS, we have the data updated almost in real time.”

UFV’s data lake house architecture performs multiple extract, transform, load processes that follow AWS best practices for data security and governance. The architecture is governed by AWS Lake Formation permissions, which make data available for authorized users. With a unified data catalog in place, the university can control access to and the publication of its data while supporting the highest levels of trust, security, and quality. Because UFV maintains snapshots of its data reports in near real time, the university can monitor that its data meets quality standards, complies with regulatory requirements, and adheres to AWS best practices.

In the design of the data lake architecture, cybersecurity is a vital aspect. UFV is fully committed to the protection and confidentiality of its data and pursues best practices in cybersecurity. For all these reasons, UFV is adapting to Spain’s National Security Scheme by deploying a specific conformance pack on AWS Config, a service that continually assesses, audits, and evaluates the configurations and relationships of resources on AWS, on premises, and on other clouds.