Our data storage scales by 1,000 percent with AWS, whereas the legacy system couldn’t support a 50 percent increase.


James Tomkins Chief Architect, Met Office
  • About the Met Office

    The Met Office is a national weather service in the United Kingdom. A part of the government’s Department of Business, Energy & Industrial Strategy, the organization delivers critical meteorological data to industries including defense, energy, transport, civil contingency, and aviation.

     

  • AWS Services Used

  • Benefits Realized

    • Data delivery in seconds not minutes
    • Storage easily scales by 1,000 percent
    • Iterative processes cut costs 50 percent
    • Increases software releases by 30 times
    • Gains monitoring and auditing simplicity
    • Testing completed in 24 hours not months

Headquartered in Exeter, England, the Met Office is a widely respected national weather service in the United Kingdom. A Trading Fund within the Department of Business, Energy & Industrial Strategy, the organization has been providing weather information for more than 160 years.

The Met Office has two supercomputers in an on-premises environment that features numerical weather models that apply the laws of physics to more than 300 million daily observations. It provides meteorological data to citizens, governments, and industries such as aviation, defense, and transport. “In the past decade, there’s been a revolution in the way people think about weather information,” says James Tomkins, chief architect at the Met Office. “New use cases include analyzing how meteorological information affects behavior in the marketplace.”

The Met Office needed an agile and cost-effective way for consumers to access its data on demand. Its on-premises IT was a challenge to secure and accredit. “The hard part was turning that huge amount of data into usable and communicable information that could be transferred in a highly secure architecture,” says Tomkins.

The Met Office—which has provision for an exabyte (10^18 bytes) of archive data on-premises—wanted to make data easily accessible via a weather app on smartphones. A key aim was for a highly scalable solution to meet data traffic spikes during unusual meteorological events such as the cold wave that hit the United Kingdom in 2018 called the Beast from the East.

The Met Office selected Amazon Web Services (AWS) to support the Met Office Weather App, available for iPhone and Android. “Choosing AWS was initially a tactical choice,” says Tomkins. “It was the only way we could see to deliver this service. All the data accessible from the app is hosted on AWS.”

The organization’s AWS architecture includes Amazon Elastic Compute Cloud (Amazon EC2), a web service that automatically scales and provides Met Office with complete control of its computing resources. The Met Office also uses AWS Lambda, which makes it possible to run backend code without provisioning servers. Lambda responds to events including object uploads to buckets in Amazon Simple Storage Service (Amazon S3), updates to tables in Amazon Relational Database Service (Amazon RDS), or in-app activity.

To respond to peak demands for read-heavy, weather-related queries, the Met Office deployed Amazon ElastiCache to retrieve data from fast, managed in-memory caches. This technology is used to build distributed data collection systems and handle real-time statistics and metadata associated with mobile applications.

The organization recently began exploring Amazon API Gateway to create, publish, monitor, and secure a series of data APIs that facilitate the dynamic sharing of content ranging from site-specific seven-day forecasts for locations around the globe, syndicated content, and warnings of severe UK-based weather. Users can personalize forecast consumption.

Since the Met Office Weather App went live in January 2016, it has attracted 6.3 million users. “We are now using the cloud more strategically,” says Tomkins. “The APIs we are building underpin all our data delivery in a move away from the FTP-based delivery model we traditionally used.”

The organization engaged Cloudreach, an AWS Premier Partner, to support operations for the Met Office Weather App. By using the Amazon Cloud Adoption Framework, the Met Office determined it had room to grow in terms of its cloud maturity, so it engaged Cloudreach consultants to optimize internal capabilities. “We used Cloudreach to help us deliver our services,” says Tomkins, “so we could accelerate maturity in various disciplines in the cloud space.”

Today, the Met Office has a cloud operations team with many specialists. They are helping the Met Office adopt the Amazon EC2 Container Service (Amazon ECS) to run applications on a managed cluster of Amazon EC2 instances. The Met Office’s production cloud architecture uses Amazon Direct Connect to securely connect supercomputer outputs to the cloud, transferring 10 terabytes of operational data to the cloud every day.

Using AWS, the Met Office has been able to increase agility, speed, and scalability while reducing costs. Tomkins reports that infrastructure provisioning has gone from months to minutes and getting data to a customer now takes seconds instead of minutes. “Our site-specific data storage scales by 1,000 percent with AWS, whereas the legacy approach couldn’t support a 50 percent increase,” says Tomkins. “In addition, iterating our solution architecture removed more than 50 percent of the cost of our initial design for the on-premises solution.”

The organization is confident in the security of its data, and its accreditation team is enthusiastic about the monitoring and auditing capabilities provided by AWS tools. “A number of the best-practice security patterns are achievable with the call of an API,” says Tomkins. “We can script and automate for a high degree of consistency. We can isolate systems and tightly control access. These capabilities are harder to achieve in our internal environment but are available out of the box with AWS.”

The Met Office uses AWS services to provision infrastructure and deploy applications. The automation of traditionally time-consuming workloads enables the organization to iterate and release new applications 30 times more frequently. In addition, the Met Office can rapidly scale up when the workload grows and then shut down resources no longer required. “With AWS, we can scale to distribute and serve a huge quantity of data to many computers,” Tomkins says.

By adopting AWS, the Met Office has been able to innovate and experiment to a degree previously impossible. For example, the organization compared the performance and cost-effectiveness of three different AWS storage solutions in one day. Tomkins says, “By using AWS, we could provision those services, run some tests, and tear them down again. Without the cloud, it would have taken months.”

In a world where technology is enabling disruption, the field of meteorology is embracing change. Tomkins concludes, “We need to be innovative if we are to keep up with the way people consume our data. We are using the AWS Cloud to drive the mass-market availability of customizable weather information.”