AWS Government, Education, & Nonprofits Blog

EO/GIS Training Lab with On-demand Services from AWS

Q&A with Professor Albert Remke from the Institute for Geoinformatics, University of Münster, Germany

In 2017, a group of students at the Institute for Geoinformatics at the University of Münster embarked on an ambitious project: to build an Earth Observation Infrastructure on AWS that would support disaster preparedness for flooding situations in Europe. To do this, they needed to:

  • Manage and process Earth observation data from the European Union’s Copernicus programme, the world’s largest earth observation programme;
  • Use a number of data sources to perform flood risk analysis; and
  • Make the technology available via a Web GIS.

We asked Professor Remke about the project, how AWS was used, and his recommendations for similar projects in the following Q&A:

Q1. Can you describe the project use case?

The goal of the project was to give students the opportunity to gain a comprehensive understanding of the European Copernicus Earth observation infrastructure. Every day, Copernicus delivers massive amounts of sensor data from orbit (space component), which are combined with data from other sources (in situ component) and processed into information products (services component). This requires a scalable infrastructure that can dynamically respond to resource demands. Cloud technologies are a prerequisite for this type of infrastructure.

The automated detection of water surfaces from Sentinel 1 (S1) radar data served as a use case. The S1 radar signal is reflected by rough surfaces and measured by the sensor. Since there is hardly any backscatter of signals on smooth water surfaces, these areas are easy to detect with S1 data. In addition, the radar signal penetrates the clouds effortlessly. The availability of usable data is better than with optical sensors, which only provide suitable data during weather conditions with good visibility of the Earth’s surface.

The data had to be combined with hydrological and meteorological in-situ data in a web application, which supports disaster management in severe flooding situations. The project’s focus was less on the data analysis methodology, and more on the technical architecture for processing large amounts of data.

Q2. Speaking as an educator, why was AWS a good choice for this project and what problems were you trying to solve when you selected an AWS solution?

The development of technical skills in the field of cloud technologies was one of the key learning objectives of the project. The students had to cope with an extensive technology stack. For the students as well as for the teachers, the documentation and the support of the AWS infrastructure were helpful. AWS also provides a foundation for using Esri ArcGIS technology, which was used for data analysis and visualization.

Q3. Were your students already familiar with AWS technologies and what was the general approach of the team?

The 15 geoinformatics students had no experience with cloud technologies prior to the project. Mentoring was carried out by three qualified software engineers from the Institute of Geoinformatics, con terra GmbH and 52° North, who also had no practical experience with AWS.

Three teams were formed, each supported by two mentors. The first team dealt with all aspects of data management, as well as scaling and cost management in the cloud. The second team was responsible for implementing the process chain for pre-processing and analysis of Sentinel-1 data. The third group dealt with the visualization of results, the integration of in-situ data, and the implementation of the web application.

All teams worked according to an agile methodology and met weekly at JourFixe meetings. Each student had around 150 hours to work on the study project.

Q4. Can you describe the solution and the AWS technology used?

The participants successfully implemented the entire process chain for the automated provision of information products from Sentinel-1 data on AWS.

Fig 1. Diagram of AWS Implementation.

This included the automated replication of Sentinel-1 data from the Copernicus Open Access Hub into an S3 bucket. Lambda functions were used to determine the availability of new data and to generate batch jobs for further data pre-processing using Docker images with SNAP tools and Python scripts. ArcGIS and Python were used to detect the open water surfaces from the satellite imagery and to integrate them as a raster dataset into a Mosaic dataset. The orchestration of these processing steps was supported by Amazon Simple Queue Services (SQS). The ArcGIS Geoevent Server was used to obtain real-time data from open data servers (Pegel Online, OpenWeatherMap) and prepare them for further visualization. The Leaflet WebApplication was installed on a t2.micro instance.

Q5. What did you and your students learn from the project?

Students now have good insight into AWS and hands-on experience developing AWS-based Copernicus applications. These skills will be useful to them in the future. The study project demonstrated that data-intensive applications that make use of spatial data infrastructures (such as Copernicus) and integrate with parts of these infrastructures need to be based on cloud and web technologies.

Q6. Were there any highlights for you during this project?

In the beginning, I thought the technology stack might be too demanding for the study project. The mentors’ task was limited to answering questions and providing assistance in critical situations, but I was happy to see that the students were able to accomplish the task independently. The documentation and accessibility of the AWS services and the support we occasionally used allowed us to get started quickly and make good progress.

It also became clear that AWS provides more than infrastructure in terms of storage and virtual machines. Elements such as Lambda functions, AWS Batch, and SQS are helpful for integrating existing and heterogeneous software components into powerful systems.

Thank you for the insight, Professor Remke, and congratulations to University of Münster for winning the City on a Cloud Innovation Challenge award for their senseBox project. Learn more here.