Executive Conversations: Kim Macaulay, Head of Data, Quality and Governance, IATA

This interview between Kim Macaulay, Head of Data, Quality and Governance, IATA and Sekhar Mallipeddi, WW Travel Tech Leader at AWS, is taken from the Travel and Hospitality issue of AWS Architecture Monthly. Be sure to check out the magazine for more customer stories, thought leadership, reference architecture and more.

—

Sekhar Mallipeddi: Please explain the current challenges with data products at IATA and the new products being developed

Kim Macaulay: At the International Air Transport Association (IATA), we are data custodians. We pride ourselves on serving our aviation members with evolving and best-of-breed data products that further their business goals.

As you might expect, it’s a challenging time to be in aviation. But in a way, it’s also very exciting! Because of the pandemic, there is more data available today than there was 5 years ago. Though the pandemic has presented many unexpected challenges, it has also created opportunities. For example, we have so much “non-traditional” data available that we can create new datasets. By mixing the new and existing data, we can deliver value-added data products and services to cater to the “new ways of working.”

One of our most significant challenges occurs when data products are developed in silos. When this happens, we sometimes cannot use the full benefits of all the data we have access to. A lot of organizations struggle with this—often data is created or used based on an operating or functional model. To avoid silos, we created a shared data architecture where data assets can be mined and modeled efficiently. This architecture is not tied to any operating model. It is driven through explicit organizational use cases. This strategy drives data centricity, which allows us to target new opportunities that would not have ordinarily been open to us. Additionally, we can mix datasets, which allows more power in analytics and we can act on the insights we derive from the data.

SM: What is the IATA Unified Infrastructure and its architecture?

KM: IATA is an integral part of the airline ecosystem. Therefore, our data processing environment needs to operate with flexibility and agility to grow beyond existing products and enhance them with other datasets. To ensure scalability for product enhancement and to generate new products and services, we need to include datasets currently not referenced in everyday activities.

The IATA Unified Infrastructure seeks to achieve an integrated data ecosystem. It aligns tools and technological building blocks to reduce cost, simplify setup, consolidate operational/support teams, and reduce risks usually created by using multiple IT-related tools and products. When we introduce new products to the infrastructure, we reuse and enhance common core components to minimize waste and optimize costs.

Our Unified Infrastructure is built on AWS cloud to expand our API, data lake, event-based data architecture, DevOps, security, front-end and backend components, and microservices. The decoupled nature of this infrastructure means that we can find the latest technologies and easily install/uninstall them based on new or added functionalities. This decoupling means that we can move quickly and be proactive when new requirements come our way. With the ever-changing technologies out there, we want to make sure we aren’t tied to a specific technology.

Our current set of data products under development will be used as the baseline for the future design of new products and services. This ensures we do not have multiple solutions doing similar things. It will also confirm we have a data platform that can be mined and modeled across all available datasets.

We have a product mindset for our Unified Infrastructure—not only do our customers use it, we also use it internally. Treating it as an internal and external product means we pay special attention to monitoring the value it adds and the overall benefits realized when we deliver products or roll out features.

By creating an open platform, we can reuse components we have built and expand where we need to. This single integrated platform encompasses a myriad of solutions and technologies, but it is simple enough to ensure we reduce legacy technical debt. In the end, this simplified IT landscape gives rise to faster digitization and automation. That is a key principle for us, and we never embark on any technology without understanding how it will benefit our customers.

SM: What are the guiding principles and core tenets of IATA Unified Infrastructure?

KM: The underlying principle of our Unified Infrastructure is agility, agility, and agility. We do not have time to roll out projects that span multiple years. Our customers are tech savvy and they need what they need now. We co-develop products and services with our customers, which adds a special something to the final product. This is key to us keeping our customers at the heart of everything we develop. Because we engage the customer in the development process, we earn their trust, which ultimately means the migration and roll out of new solutions is much more successful. Creating “Friends of the Product” ensures we react quickly to feedback and that can be flexible if requirements change. For example, requirements often change once you see it “in the flesh,” and being agile means that we are not only quick to deliver but also quick to react.

The other tenets underpinning the Unified Infrastructure include reducing costs of our IT infrastructure but also spending where it makes sense. With an ever-growing need for data products, there comes an even bigger need to secure our customers’ data and interactions with IATA. We therefore follow a zero-trust philosophy when implementing new products and services and are always vigilant about security.

Our Unified Infrastructure is guided by further principles of cloud-first and open-secure architecture, allowing us to be always on and always secure.

SM: How do you use data lake and lake house architecture patterns for the IATA Unified Infrastructure?

KM: Many organizations struggle with the complexity of maintaining both data lakes and data warehouses. This complexity often translates into complicated data pipelines, slow-to-market products, and reduced decision making speed.

IATA embarked on a lake house architecture a few years ago. We looked at implementing similar data structures and data management features to those in a data warehouse directly onto our AWS cloud data lake. This enhanced, simplified architecture allows for traditional business intelligence analytics and data science to co-exist in the same system. We are expanding approach as part of the Unified Infrastructure strategy by opening our data formats. When combined with unstructured data, this approach will expand our artificial intelligence (AI)/machine learning (ML) use case reach. We are also implementing a take-on-any-format approach to assist our customers because they are not always able to comply with certain data standards and formats.

We have so many use cases we need to provide solutions for, and just implementing a data lake was never going to be enough. In the early days of data, organizations were almost always data warehouse only or data lake only shops. IATA is proud to have the best of both worlds. We have worked closely with AWS to optimize our architecture, but given the size of the datasets in the aviation space, we still need to join cross-organizational data lakes. Thus, expanding the data lake within the Unified Infrastructure to cater to all nuances is critical. This includes all ancillary data services like data science, predictive analytics, and real-time business analytics. By reusing the data pipelines we build, the landscape is simpler, and we can focus on the new and not the already in place.

The lake house architecture has helped us establish the foundation of our Unified Infrastructure as we expand our event-based data architecture to cater for a microservices-based strategy. However, I see it as almost becoming a lake house-lake architecture in some respects. (Yes, this is not exactly the correct technical term, but I think you understand where I am going. 🙂 ) For example, aircraft operational data can run into the thousands of petabytes, and storing all the data in one data lake may not make much commercial sense. If we could have our data lake talk to an OEM’s (original equipment manufacturer) data lake, then the increased insights would give us opportunities to join datasets and potentially new/unseen additional analytics.

SM: What is your development and support model for the IATA Unified Infrastructure and how do you use the modern data mesh methodology?

KM: Interestingly, we started investigating data mesh architecture a year ago when we were looking at enhancing our current data lake into a “game changing” platform for new product development. Enabling rapid delivery of new capabilities is very close to our heart, and it is something we share with data mesh methodology. As a DevOps organization, we follow the premise of “you build it, you run it,” and this mentality flows into the data space. We have specific concepts we follow for data-as-a-service and infrastructure-as-a-service. Now we have complemented these with security-as-a-service and operations-as-a-service to ensure all facets are addressed in the Unified Infrastructure.

The Unified Infrastructure looks at simplifying and speeding up data acquisition, processing, storage, analysis, and exposure processes with a specific focus on agile delivery. Our prioritization process prioritizes shifts that will directly impact business goals and strategies. Prioritizing items like real-time messaging platforms like Apache Kafka as well as streaming processes and analytics solutions like Apache Kafka and Apache Spark will add competitive advantage in the data space.

We have already rolled out data science models that use predictive analytics. We did this in a scaled, agile way to ensure iterative enhancements and extensible data models. This allows us to compare patterns and signals in the data and evolve quickly. For data mesh, we have streamlined our data pipelines and API integration to get the most out of our architecture. Exposing data via APIs ensures we have direct access to view and modify the data while allowing faster access times to common datasets in the platform.

The support and development model looks at establishing a network-based group that incorporates all facets of IT (operations, data, architecture, integration, and testing) into a cross-functional team that focuses on testing and deployment. These network-based groups allow us to focus on DevSecDataOps to ensure our design deploys new components quickly, securely, and iteratively.

Our Unified Infrastructure expands past creating a data culture. We aim to create a unified culture in technology practices that support and work within our business and with customers. It is a great time to be in aviation IT!

—

Kim Macaulay is the Head of Data, Quality and Governance for Information Technology at IATA. She began her career in the Financial Services & Banking sector where she executed strategic IT initiatives around data and digitization by implementing technologies to support strategy. In 2014, she joined the Standard Bank of South Africa (SBSA) and was responsible for implementing the technical data capability for Global Markets and Transactional Products and Services. After 15 years in the banking world, she joined IATA in 2020 to head up the data capability in IT specifically relating to transforming the AWS Cloud environment for an ever-changing data landscape. Kim has an Honours Degree in Business Commerce from the University of Johannesburg in South Africa as well as further postgraduate studies in Data and Digitization.