Headquartered in Exeter, England, the Met Office is a widely respected national weather service in the United Kingdom. A Trading Fund within the Department of Business, Energy & Industrial Strategy, the organization has been providing weather information for more than 160 years, including seven-day forecasts, surface pressure charts, pollen forecasts, long-range weather outlooks, and more. It delivers critical data to industries including defense, energy, transport, civil contingency, and aviation. The Met Office also provides world-class guidance on the science of climate change and is the primary focus in the UK for climate science.
The Met Office uses two operational supercomputers—the eleventh-largest such implementation on the planet—in an on-premises environment that features numerical weather models that apply the laws of physics to more than 300 million observations collected daily. Each day, the organization performs 20 quadrillion calculations and generates more than 4 million forecasts.
Meteorological data is stored and processed on-site, but the Met Office needed an agile and cost-effective way to make it possible for citizens, businesses, and governments to consume its data on demand. Its on-premises implementation is a particularly sophisticated environment that is a challenge to secure and accredit. “The hard part is turning that huge amount of data into usable and communicable information that can be transferred in a highly secure architecture,” says James Tomkins, head of enterprise IT architecture at Met Office.
To increase agility and flexibility in its on-premises infrastructure, the Met Office—which has an exabyte (10^18 bytes) of archive data on-premises—sought a way to send data loads to users who consume information by using devices such as tablets and smartphones. It needed a solution that could scale to meet high-traffic events such as Storm Katie in March 2016. That storm, which brought 60-mile-an-hour wind gusts to London and led to dozens of flight cancellations, the Met Office received a 200 percent increase in visitors.
The Met Office also needs to respond to third parties that request access to atmospheric and climate data. For example, the aviation industry is one of the largest consumers of meteorological data, and cannot operate without it. Alongside a growing demand from all manner of commercial and government enterprises, the Met Office also needs to exchange large amounts of data with the highly collaborative meteorological community.
“In the past decade, there’s been a revolution in the way people think about weather information,” says Tomkins. “People are beginning to understand the impact that climate change and weather events have on their businesses. New use cases can involve everything from retail operations to renewable energy companies—organizations that want to analyze how meteorological information affects behavior in their market space.”
In determining its technology strategy, the Met Office chose Amazon Web Services (AWS) to support the Met Office Weather App, available for iPhone and Android phones. “Choosing AWS was initially a tactical choice,” says Tomkins. “It was the only way we could see to deliver this service. All the data accessible from the app is hosted on AWS.”
The organization’s AWS architecture includes Amazon Elastic Compute Cloud (Amazon EC2), a web service that automatically scales and provides Met Office with complete control of its computing resources. The Met Office also uses AWS Lambda, which makes it possible to run backend code without provisioning servers. Lambda responds to events including object uploads to buckets in Amazon Simple Storage Service (Amazon S3), updates to tables in Amazon Relational Database Service (Amazon RDS), or in-app activity.
To respond to peak demands for read-heavy, weather-related queries, the Met Office deployed Amazon ElastiCache to retrieve data from fast, managed in-memory caches. This technology is used to build distributed data-collection systems and handle real-time statistics and metadata associated with mobile applications; it is ideal for solutions that involve complex mathematical calculations because it can cache computational results.
The organization recently began exploring Amazon API Gateway to create, publish, monitor, and secure a series of data APIs that facilitate the dynamic sharing of content ranging from site-specific seven-day forecasts for locations around the globe, syndicated content, and warnings of severe UK-based weather. Users can personalize forecast consumption.
Since the Met Office Weather App went live in January 2016, it has attracted more than half a million users. “We’re thinking deeply about how to use the cloud more strategically,” says Tomkins. “The APIs we are now building are going to underpin all our data delivery in a move away from the FTP-based delivery model we have traditionally used.”
The organization engaged Cloudreach, an AWS Premier Partner, to support operations for the Met Office Weather App. By using the Amazon Cloud Adoption Framework, the Met Office determined it had room to grow in terms of its cloud maturity, so it engaged Cloudreach consultants to optimize internal capabilities. “We used Cloudreach to help us deliver our services,” says Tomkins, “so we could accelerate maturity in various disciplines in the cloud space.”
The Met Office now has a cloud operations team with five specialists. The organization intends to optimize cost outlays by exploring reserved Amazon EC2 pricing models and serverless components in the coming months. It plans to adopt Amazon EC2 Container Service (Amazon ECS) to run applications on a managed cluster of Amazon EC2 instances. The Met Office’s next iteration cloud architecture will explore the use of Amazon Direct Connect to securely connect supercomputer outputs to the cloud.
Using AWS, the Met Office has been able to increase agility, speed, and scalability while reducing costs. Tomkins reports that infrastructure provisioning has gone from months to minutes and getting data to a customer now takes 10 seconds instead of 10 minutes. “Our data storage now scales almost linearly by 1,000 percent, whereas the previous architecture could not support a 50 percent increase,” says Tomkins. “In addition, iterating our solution architecture removed more than 50 percent of the cost of our initial design for our on-premises solution.”
The organization is confident in the security of its data and its accreditation team is enthusiastic about the monitoring and auditing capabilities provided by AWS tools, at a maturity level the Met Office could not deliver as cost-effectively. “A number of the best-practice security patterns are achievable with the call of an API,” says Tomkins. “We can script and automate for a high degree of consistency. We can isolate systems and tightly control access. These capabilities are harder to achieve in our internal environment but are available out of the box with AWS.”
The Met Office takes advantage of AWS services to provision infrastructure and deploy applications onto that infrastructure. The automation of traditionally time-consuming workloads enables the organization to iterate and release new applications 30 times more frequently—a game-changing opportunity. In addition, the Met Office can rapidly scale up when the workload grows and then shut down resources that are no longer required. “With AWS, we can scale to distribute and serve a large quantity of data to a large number of computers,” Tomkins says.
By adopting AWS, the Met Office has been able to innovate and experiment to a degree previously impossible. For example, in one day, the organization easily compared the performance and cost-effectiveness of three different AWS storage backend solutions. “We wouldn’t have been able to compare those options on-premises because of the lead times involved,” says Tomkins. “By using AWS, we could very easily provision those services, run some tests, and tear them down again. Without the cloud, it would have taken months (and actually, we wouldn’t have done it all). Now we can fail fast and fail cheaply in terms of trying different architectures.”
In a world of technology disruption where the landscape of new opportunities includes the Internet of Things, big data analytics, and machine learning, the field of meteorology is very interested in innovation. “We collect and generate huge volumes of data that is of limited use to people until we can deliver services that provide consumable information,” says Tomkins. “We need to be innovative if we are to keep up with the way people consume our data. We are using the AWS Cloud to drive the mass-market availability of customizable weather information.”
To learn more about how AWS can help you manage big data, visit our Big Data page.