Fujitsu-IDOM Builds Cloud Data Lake on AWS for Consorcio Regional de Transportes de Madrid, Improving Agility and Analytics

The Consorcio Regional de Transportes de Madrid (CRTM), which manages the city’s public transportation, built a scalable data lake and big data processing solution on AWS to generate insights and improve its decision-making. The consortium collects data about each line of transportation to understand customer needs and improve its services. But it couldn’t efficiently scale to support increased big data workloads with an on-premises data center. With support from AWS Partner Fujitsu and its partner, IDOM, the CRTM migrated to a cloud data lake and enhanced data processing using several AWS services. On AWS, the consortium increased the scalability of its infrastructure and improved its analytics.

The CRTM is responsible for regulating and improving the transportation of millions of citizens throughout Madrid every day. As a public sector organization, it aims to provide reliable service for more than 1.6 billion trips per year in Madrid, but its existing on-premises infrastructure couldn’t efficiently scale to support the huge amount of data the consortium needed to make intelligent decisions. And while the CRTM knew it needed more agile data storage and better big data analytics capabilities, it didn’t have sufficient information systems to build and manage this solution on premises in a productive way.

Instead, the CRTM requested proposals from several technology providers for a solution that would migrate its data storage to the cloud, make it scalable, and use machine learning (ML) models to achieve better insights about user behavior that would help it improve communication and services. In 2020, the consortium awarded a 2-year contract and 3-year contract to Amazon Web Services (AWS) Partner Fujitsu, a global information and communication technology company that proposed to create a native big data solution on AWS that would unlock new analytics and ML capabilities for the CRTM.

For years, the CRTM had been using individual magnetic tickets for riders on buses and trains, but these collected little information about each ride. The consortium needed detailed data about rider behavior and the relationship between ticket acquisition and point-of-sale systems so that it could understand and meet riders’ needs while enhancing customer loyalty. When it transitioned to cards that facilitated data collection in December 2017, the CRTM began collecting an enormous amount of useful information. But its infrastructure wasn’t ready to handle the increased amount of data. “We needed to manage all the information that we were accumulating in our on-premises storage system,” says Luis Criado, systems area coordinator at the CRTM.

CRTM engaged the Fujitsu-IDOM partnership to help with this challenge. Fujitsu-IDOM provide technology services and solutions to help businesses create value using digital innovation. Using AWS, Fujitsu employed a cloud-native solution to overcome the CRTM’s challenges. “We wanted to build a big data solution on AWS to provide the CRTM with all the data and views of that data that it needed to make intelligent business decisions,” says David Sanz, business development manager at Fujitsu. The company chose to build its solution in the cloud because it could scale with ease while upholding its users’ privacy.

As it migrated to AWS, the CRTM became one of the first public administrations in all of Spain to use a data lake in the cloud. It also gained a better picture of how riders used the city’s transportation network and what changes would improve their experiences. By increasing its understanding of users’ habits, the consortium can continually improve its services and adapt to near-real-time changes in demand. As that demand increased in recent years, it also became extremely variable due to evolving guidelines related to the COVID-19 pandemic.

Alongside the Fujitsu-IDOM partnership, the CRTM built its data lake on Amazon Simple Storage Service (Amazon S3), an object storage service that gives the CRTM the scalability that it needs and makes its data available online and accessible from nearly anywhere with ease. The CRTM used Amazon S3 to integrate all of the internal and external data sources into a single new data model. “Using Amazon S3, we can move and access massive amounts of data and flexible information in the data lake,” says Criado. The consortium also uses Amazon Redshift to analyze its information across data warehouses in a way that’s simple and cost-effective. With simpler data storage and increased availability, the CRTM has reduced its time to market, improved its service, and adapted to changing conditions in near real time. “Before building our big data solution on AWS, all of the information that we needed to make a decision had to be gathered and analyzed by hand,” says Criado. “Now, we have all the information accessible with ease online.”

The solution that the Fujitsu-IDOM partnership built also uses Amazon EMR to run the CRTM’s big data processing workloads in a way that’s flexible, secure, and reliable. Using Amazon EMR, the CRTM has increased its ability to understand regional transportation network usage and analyze its various sales channels. Previously, it could respond to certain incidents only after they had occurred and had been recorded, but now the CRTM can use big data processing to anticipate and prevent incidents. “We have the vision in near real time to do much better decision-making,” says Criado. With this improved agility and data accessibility, the CRTM can make more timely decisions and analyze sales channels to improve customer loyalty.

As part of its efforts to improve analytics and extract information efficiently, the CRTM is using Amazon SageMaker to build, train, and deploy ML models quickly based on data that’s accessible in Amazon S3. “We are using ML models on AWS to bring value to all the information from the on-premises data center,” Criado says. Once the data is in the cloud, the CRTM can use ML to analyze it and generate insights that drive decision-making.

Another key part of the Fujitsu-IDOM partnership’s mission is to make the world more sustainable through technological innovation. One of the most important expected benefits of this migration to AWS is that the CRTM can make much better use of each of its lines of transportation to reduce its CO2 footprint. In addition, the CRTM has always been careful to protect the privacy of its users’ data as it gathers information to improve its services. “Working in the cloud is a good option to be able to continue protecting data privacy,” says Criado. “We can achieve scalability and gain insights on AWS without affecting users’ privacy.”

The CRTM is still in the beginning stages of its engagement with Fujitsu-IDOM. The consortium has built about 25 percent of its intended use cases and plans to continue optimizing its data analytics and ML workloads on AWS.

“We increased our capacity for storing and analyzing information exponentially on AWS,” says Criado. “We have a better understanding of what our customers experience on every trip and can quickly make decisions to improve it.”