Using big data to help governments make better policy decisions

In Europe, government agencies and policy makers see the value in using new technology to unlock digital transformation and deliver better, more innovative citizen services. Using data for statistics initiatives, including open data, can help researchers produce innovative products and tools, including visualisation, to inform government officials ahead of making policy decisions that impact their citizens.

Data has always been important, even before big data. The European Commission started with evidence-based policy making many years ago. Benchmarking against each other and against commonly agreed targets using statistical indicators has been a notable way for European Union (EU) member states to measure their progress. Data-driven policy is getting more and more sophisticated, as we move from benchmarking towards more artificial intelligence (AI)-based predictive and prescriptive recommendation.

Researchers provide evidence-based knowledge for policy makers

When it comes to big data, policy makers need to collaborate with researchers to address issues and challenges in using these new data sources. To work toward this goal, Eurostat, the statistical office of the European Union (EU), hosted its bi-annual European Big Data Hackathon. The goal of this year’s hackathon was to foster expertise in using big data while producing innovative ideas for products and tools, and building prototypes, in the domain of international trade networks in support of the European Commission policy Directorates Generale (DGs).

Eurostat is the statistical office of the European Union (EU), responsible for publishing high-quality Europe-wide statistics and indicators that enable comparisons between countries and regions. The European statistical office is driving innovation by using new data and the latest technology to help with finding solutions for the problems we are facing, such as challenges related to COVID-19, to make Europe more resilient.

Amazon Web Services (AWS) was an official sponsor of the event and provided participants with access to Service Workbench on AWS.

A hackathon to help provide better decision making ability to the policy makers

The hackathon brought together 22 teams from across Europe, nominated by the National Statistical Institutes and universities participating in the European Master in Official Statistics (EMOS) network. Each team was responsible for developing innovative approaches and solutions that leverage statistics and big data to foster better decision making for policy makers when faced with pressing policy questions.

This year’s hackathon was focused on international trade, with data analysis from the point of view of networks instead of looking at the trade balance of the countries to explore the resilience of the European economy impacted by the COVID-19 pandemic. The participants needed to leverage two of Eurostat’s datasets (balanced global trade in goods and Comext) and build sophisticated algorithms or machine learning (ML) models that analyse the datasets and provide better decision-making power to policy makers. The new datasets contained data of 200 countries with trade flows in 9,000 categories only for goods. Sample questions included, “Are we at risk in the current situation because raw materials for mask and vaccines are produced somewhere else outside of Europe?”

Eurostat is one of the driving forces behind the European Commission’s data platform from the European Commission Directorate General (DG) Informatics (DIGIT), which is a platform as a service offering for data scientists, engineers, and analysts in the European Commission. The platform allows users to combine different building blocks for a customer solution. More and more of these building blocks are relying on cloud-native services as they inherently provide high availability, security, performance and scalability.

In this year’s event, Eurostat directly collaborated with DIGIT and AWS to provide cloud-native data, analytics and machine learning services to the hackathon participants.

Eurostat used Service Workbench on AWS to host the hackathon environments. Service Workbench on AWS enables IT teams to provide secure, repeatable, and federated control of access to data, tooling, and compute power that researchers need.

Numbers, data, facts: Trade trends influenced by the COVID-19 pandemic

The evaluation panel (EP) selected six winning solutions. The top three winners includes teams from Italy, Poland, and Lithuania.

Istat, the Italian National Statistical Institute, created the app Cosmopolitics. The app provides a range of functionalities including interactive maps, graph analysis, mobility policy analysis, and, finally, analysis of trade. The focus is on trade trends under the COVID-19 pandemic, and it models scenarios answering different trade related questions, such as the effects of trade disruptions. The application integrates additional data and defines a mobility policy indicator based on Google mobility data representing the level of restriction imposed by a selected national government.

The team from Poland created the app Forecasting and simulating trading trends. The application covers a large amount of data and functions, which are easily replicable. It’s a framework of 14 pieces of software and nine data sources. The application is open source and shared on GitHub.

The team from Lithuania created the app Divergence. They estimated vulnerabilities of trading partners expressed as a Gini-type diversity-disparity index. The diversity-disparity index is visualized as a time-series with granular temporal resolution and also a map representation. The time-series approach enables an analyst to spot associations between geopolitical, economic, and natural events and corresponding changes in partner vulnerabilities.

The solutions created at the hackathon can contribute to the European COVID-19 recovery dashboard, and national statistical institutes can use them going forward. By June 2021, the used dataset “balanced global trade in goods” will be publicly available for the international research community.

The team at Eurostat wants to use modern technologies to provide new statistics to solve the problems of the European society. We were the first user of the BDTI platform provided by DIGIT. Today we are the biggest user of the EC data platform. With this hackathon we gained new, valuable insights that will help the European society to make it through the COVID-19 pandemic.

Learn more about Service Workbench on AWS and the cloud for research and technical computing, and check out tips on how to host a virtual hackathon.