How Matillion is digging diamonds from data with generative AI

If data isn’t in a simple numerical or tabular format, it has historically been tough to systematically process using AI. The alternative is manual processing, which is labor intensive, expensive, and slow. Through game changing advances with generative AI, Matillion is making it possible to ingest multiple different unstructured formats. By working closely with Amazon Web Services (AWS), the software company is solving the complex technical challenge of bridging the gap between qualitative and quantitative data to unearth new findings. As Wiffen says, “We sit at the intersection of text and data and bringing them together is where the real insight comes.”

One healthcare organization is using the platform to accelerate medical studies and leverage richer data. Patients record voice notes of their symptoms using Amazon Transcribe in an app, before the data is transformed and analyzed by large language models (LLMs) on Amazon Bedrock. The organization can then generate reports to uncover detailed insights on medication side effects, homing in on specific symptoms and commentary. It can also leverage historical data across different formats at any point—without having to restart studies.

Multiple other industries are fueling new use cases by unearthing and transforming siloed data for powerful productivity improvements. “With a combination of our tools and Amazon Bedrock, customer surveys that used to take 4,000 hours a year to process now take around an hour,” says Wiffen. “Organizations might have years of data that we can rapidly unlock with modern techniques,” he continues.

Matillion is putting the power of data into non-expert users’ hands. Traditionally, building sophisticated data pipelines requires deep data expertise to navigate complicated extract, transform, and load (ETL) tools—making it off limits to many key stakeholders within an organization. By building generative AI into the user interface, Matillion’s agentic AI system Maia enables anyone to easily get what they need from data. The process is simple—users input a natural language prompt that triggers Maia to search relevant data and components. The agent then configures, compiles, and shares a tailored response based on the user’s request.

Built using high-performing foundations models in Amazon Bedrock, Maia empowers business users to bring valuable domain knowledge into the heart of data pipelines. Wiffen explains, “These are the folks that know the real business questions that need to be answered.” Ed Thompson, Chief Technology Officer at Matillion, adds that this “creates a free flow of data across the organization to support good quality decision-making.”

Already using multiple services to power its platform and guarantee seamless experiences for users, Matillion chose to collaborate closely with AWS to bring its AI agent to life. “AWS has been instrumental in our agentic AI strategy,” says Thompson. The business gained hands-on training to keep its teams’ technical skills sharp, including advice on the best models and development approaches. A robust generative AI foundation and flexible technology is also helping Matillion future-proof its services. Wiffen notes, “AWS provides us with a rich toolbox and gives a strong access to everything that’s coming out in this cutting-edge and fast-moving field.”

Having rapidly built its generative AI services, Matillion is constantly looking to optimize these offerings to stay at the vanguard. This means pushing for even greater accuracy and speed of response with its agentic system, as well as expanding the breadth of what generative AI can do within the user interface. Amazon Bedrock will continue to play a core role in making this possible by providing quick and easy access to the latest foundation models. “We can easily test models to find the best one for each problem. We’ve swapped to new models in the space of a week of them being launched, whereas previously it might have taken months,” says Wiffen.

Since incorporating Amazon Bedrock models into its AI agents, Matillion has unlocked exciting new language capabilities. “We made a fascinating discovery that users can accurately work in French, German, and even Japanese. This has opened whole new markets for us,” says Wiffen. Despite the interface being purely in English, the business can now easily support multiple languages.

Thompson sees Matillion’s advances as just the beginning of their AI journey: “I’m confident AWS can help us sit at the vanguard of AI and keep growing our platform.” Wiffen shares that Matillion currently provides “2.8 times productivity gains” and the business expects to see that “ramp up to 10 times in the next few months” then to “100 times in a year or two”. “It will fundamentally change the way customers think about building a data engineering team,” says Thompson.