NHS Digital and Tableau Support UK COVID- 19 Pandemic Response with Modern Cloud Analytics on AWS

The open data and data visualisation team at NHS Digital was tasked to quickly gather, analyse, and share data related to the COVID-19 pandemic. This was used to inform rapid decision making and policy creation, and inform the public. Public facing data was aggregate and anonymous, with private and secure platforms used to deliver more sensitive data to NHS and care users that needed access to more granular information. NHS Digital needed backend infrastructure that could be scaled up to store and process large amounts of data, combined with front-end technology to manipulate it into a form that could be easily shared and understood. NHS Digital—using Amazon Web Services (AWS), and working with data visualization provider Tableau—was able to deliver the first crucial data dashboards in just 9 days.

As the COVID-19 pandemic started to hit the UK, the open data and data visualization team at NHS Digital was tasked with quickly gathering and analysing the associated data and presenting it in a way that would help decision makers and assist with creating policy in response to the pandemic.

With the COVID-19 pandemic rapidly evolving and escalating, the NHS Digital team had to quickly prioritise the data that would be collected, analysed, and presented to different stakeholders.

It used AWS to store and processe the data, with AWS Partner Tableau providing Modern Cloud Analytics to analyse and visualise the data and insights in the form of dashboards for the relevant stakeholders.

The organization made this choice because it needed a backend infrastructure that could be scaled up to store and process the vast amount of data, integrated with front-end technology that could manipulate this data into a form that could be easily shared and understood.

This was the first time NHS Digital had been commissioned to create a mass public dashboard. By using multiple providers, including Tableau and AWS, NHS Digital delivered the first of the crucial new dashboards in a few days. The project, as a whole, completely changed how the NHS can visualise data.

Previously, visualizations were built using aggregate data. NHS Digital has now developed secure infrastructure that can host patient-level data in dashboards, enabling vaccine management dashboards and appointment-level information. This enables clinicians and general practitioners (GPs), who can now use smartcard authentication to access these dashboards, to make risk-based decisions, changing the way population health is managed.

As these dashboards deal with the most sensitive patient-level data, the use of smartcards means they are also highly secure. Where data is non-patient level, NHS Mail with 2 factor authentication is used. “This ability to go from aggregate data, all the way down to patient level data in a secure way, while challenging, has been immensely transformational,” says Alistair Bullward, head of analytical functions at NHS Digital.

The NHS Digital team started its work by looking at the data of people reporting COVID-19 symptoms via the 111 website and call center, or via the 999 emergency services line. With mass testing not yet available, the data produced by these interactions was crucial in helping public health organizations and the government understand the symptoms of COVID-19. They could also use data to visualise how it was spreading across the UK, in order to identify potential hotspots to prepare the public health services in those areas.

The NHS Digital team worked closely with Tableau and Goldman Sachs who provided the skills and resources to develop dashboards that analysed and visualised the data quickly and accurately. The rapid scaling of resources needed was supported using AWS. “We needed to get the data out for public access in a safe and anonymised form as soon as possible,” says Bullward. “We then needed to make that data consumable so stakeholders could get insight into the questions they needed to ask.”

Soon after, the team was asked to produce a dashboard for high-risk individuals—those whose medical conditions made them especially vulnerable to COVID-19—who were asked to shield at home. This shielded patient list dashboard was used for central planning, such as making social care provisions and organizing for food parcels to be offered to shielding households. The data used was aggregated and anonymous and was published at a Clinical Commissioning Group (CCG) and national level on an open dashboard hub. A private version was also created to pilot functionality with users within the NHS.

The Tableau team approached its senior management to secure the rapid provision of these services. “I think it was 9 days from getting the infrastructure stood up to having the first dashboard go live,” says Janet Broome, enterprise account manager for NHS and healthcare at Tableau Software. “So now, when anyone says to me something will take months and months, I say ‘no, you can do it in 9 days’.”

The cloud-based infrastructure supported rapid decision making as new servers could be added almost instantly, making it easier to iterate and experiment. “In weeks we were going from nothing to having an enterprise solution for thousands of people a week and scaled to support something much, much bigger,” says Bullward.

In parallel, the NHS Digital team developed a range of dashboards for different stakeholders and groups. For example, directors of public health who needed to view data across England and its regions. These included dashboards for the shielded patients list, the NHS Pathways coronavirus triages (111 data), NHS Digital services dashboard, and the GP COVID-19 vaccine dashboard.

Knowing the number of positive cases in different areas was important for public health operations, including the introduction of the tier system in autumn 2020, with legal restrictions around movement and social mixing varying depending on the level of risk in local areas.

It was at this stage that the public-facing “coronavirus-in-your-area” dashboard was launched, allowing users to type in their postcode to view local case rates. During the first 24 hours after going live, the online service received 120,000 visitors.

The NHS Digital team remained hard at work during this time, launching a dashboard identifying people suitable for clinical trials for vaccines by the end of 2020. A university dashboard followed in January 2021 as students prepared to return.

A key element to the project’s success was a team spirit that developed as the different organizations worked together. There was a “no egos here please” policy, with everyone focused on the public good, according to Broome. Tableau and AWS regularly stepped in to help out.

Established hierarchies across NHS Digital adapted to the situation. “Everyone had a sense of purpose, it didn’t matter who you worked for, it was about getting the job done,” says Bullward, who found himself doing manual testing with an NHS director on Sunday evenings. “We just got on with it.”

The combination of the people, skills, expertise, and tooling helped the NHS Digital team overcome several challenges, such as the amount of traffic hitting the COVID “in-your-area” dashboard when it went live. The team used Amazon CloudFront, a content delivery network (CDN) service built for high performance, security, and developer convenience, to stop any issues occurring.

The capabilities NHS Digital developed during the pandemic now provide the means for real population health management as never done before. As a result, NHS Digital aims to launch new capabilities more quickly by starting with a minimum viable product. “I think a lot of people who were involved in the project now look back and think ‘Oh, that was wonderful, we need to keep that up as much as we can’,” says Bullward.

The team has since supported the UK’s COVID-19 vaccine programme, including the vaccination roll out. Providing patient-level dashboards to people making decisions is a key focus. “For me, that’s the future. It’s about being able to go from high-level stuff and then going to that patient-level stuff so public health professionals can do what they need to do to improve lives,” says Bullward.

“It was a real challenge for us to deliver that patient-level data in a secure and governed way, and I feel that has really changed the healthcare landscape. If you think about what they’ve done, it’s incredible,” says Broome.