Fever: Building a VPN-less secure network for corporate access

Fever is the leading global live-entertainment discovery platform, and it has helped millions of people discover the best experiences in their cities since 2014. With a mission to democratize access to culture and entertainment in real life through its platform, Fever inspires users to enjoy unique local experiences and events, from immersive exhibitions, interactive theatrical experiences, and festivals to molecular cocktail pop-ups, while empowering creators with data and technology to create and expand experiences across the world.

At Fever, we have different cross-functional tools and services that we use on a daily basis. These services must remain private, but they should be accessible to our corporate teams working remotely, so those must have worldwide private access.

We were addressing this requirement using a custom virtual private network (VPN) solution based on OpenVPN software. As we grew, the management of users’ certificates had become complex and time-consuming. Hence, we decided to search for a scalable, secure, and easy-to-maintain solution to provide access to our users. Also, that solution should be compatible with our existing single sign-on experience, achieved through our AWS IAM Identity Center portal, which is integrated with Google Workspace, our identity provider (IdP).

We found that AWS Verified Access met all the requirements. Verified Access provides secure access to corporate applications without a VPN while reducing risk with remote connectivity. Built using AWS Zero Trust principles, it grants access to applications only when users and their devices meet the specified security requirements. We tested it on a small number of users, and after a successful experience, we decided to roll out the solution on our production environment for a wider range of applications. The deployment was successful, and the applications can benefit from a secure single sign-on integration.

In this post, we provide further details throughout this journey to adopt Verified Access and the value it provides to Fever.

Certificate and user management

We had been working with OpenVPN for many years. We had two OpenVPN servers, one for the majority of our users to access our internal tools and the other one for special cases where the public IP address of some of our services needed to be allow-listed for third-party services we partner with. The following diagram (Figure 1) shows the architecture.

Figure 1: Legacy OpenVPN architecture on AWS

This solution was working as we expected for a small number of users, but as we and our business grew:

• More and more users needed signed certificates.
• We needed to revoke – on an individual basis – certificates belonging to users that no longer needed access to these internal applications.
• All these users were not synced with single sign-on.
• Both servers and clients needed updates and security patches, with some of them causing unexpected downtime.
• There was no high availability because all connections were handled by a single Amazon Elastic Compute Cloud (Amazon EC2) instance.

All those tasks were time-consuming and distracted us from more important business-related activities. We were signing and revoking certificates for each user, helping them with connectivity issues, and troubleshooting client configuration in different operating systems. What’s more, mandatory updates like upgrading the operating system of these servers to newer versions had to be done in hours with low activity. This was difficult due to our global user base, which required our VPN to be always online.

Shifting to AWS Verified Access

To evaluate the adoption of Verified Access, we chose a small application in a development environment. The application has a user interface with no access control (anyone can do anything) and has an internal Application Load Balancer as the entry point (as shown in Figure 2). The only active permission scope was the OpenVPN server, which granted access to anyone who had a valid VPN certificate. In addition, this application had no support for any type of single sign-on access.

Figure 2: The application selected to test AWS Verified Access (legacy OpenVPN architecture)

First, we created a new client on our IdP that implements the credentials grant type from OAuth 2.0. This client can be identified by a client ID and a secret key. Then, on Verified Access trust providers, we created a new OpenID Connect (OIDC) provider. A trust provider maintains and manages user and device identity information that is sent alongside the requests that Verified Access evaluates before allowing or denying them. This information is called trust context. It can include attributes based on user identity, such as an email address or membership in the sales organization, or device information, such as installed security patches or antivirus software version.

The information you need to provide when creating the trust provider depends on your OIDC provider. In our case, all the relevant fields were filled with information from Google Workspace API and the client ID and secret we previously created.

After that, we created a Verified Access instance for our newly created Verified Access trust provider. Next, we created a Verified Access group that would be used for testing. A group is a collection of Verified Access endpoints and a group-level policy. An endpoint is a Regional resource that specifies the application that Verified Access will be providing access to.

Verified Access policies allow you to define rules for accessing your applications hosted in Amazon Web Services (AWS). They are written in Cedar, an open-source policy language developed by AWS. We created this policy:

permit(principal,action,resource)
when {
    [
        "example.email@example.com",
    ].contains(context.GoogleSSO.email) &&
    context.GoogleSSO.email_verified == true
};

This policy is testing two conditions:

1. The email must be verified within the organization.
2. The email must match example.email@example.com.

Then, we attached the user group to the previous Verified Access instance.

The last part to configure was the Verified Access endpoint. Each endpoint is associated with a Verified Access group and inherits the access policy for the group. This endpoint must be public, so it was using an open security group for port 443. We have different fields here that need attention:

• Verified Access instance ID – We use the instance created before.
• Verified Access group ID – We use the group that has the attached policy.
• Attachment type – VPC.
• Endpoint type – Load balancer. We will use the Application Load Balancer showed before in Figure 2 as the entry point for our application.
• Security group IDs – Our security group that allows access through the TCP 443 (HTTPS) port.
• Application domain – This is important because we will be using a friendly domain name for our internal users. For this post, we’ll use test-app.example.com.
• Domain certificate ARN – The AWS Certificate Manager (ACM) certificate Amazon Resource Name (ARN) used by test-app.example.com.

The following diagram (Figure 3) shows a summary of the new architecture:

Figure 3. Access to applications using AWS Verified Access

Results

When working with the OpenVPN solution, every time we received a request for a new VPN user, we needed to gather information from the user, sign the certificate, update the OpenVPN server certificate store, and test the connection with the user. We also needed to resolve any issue with the local client or operating system misconfiguration on the user side.

This whole process was taking from 15 minutes to 1 hour for each user. After adopting AWS Verified Access, we lowered this time to 5 minutes for existing applications. In fact, we can onboard big groups of users or even the whole company in minutes (allowing any authenticated user in our organization to access an application).

Users also reported faster access to our internal tools because we were using a single EC2 instance running OpenVPN for all users. Now, with Verified Access, we have dedicated load balancers for each application that scale automatically.

Furthermore, we also removed all the operating system patching, upgrades, and maintenance needed for the OpenVPN servers because Verified Access is a fully managed service that relieves us of all these undifferentiated operational tasks.

What’s next

We will log every activity in each AWS Verified Access endpoint to be able to perform security audits based on our internal policies. We will start delivering logs to Amazon Simple Storage Service (Amazon S3) and later perform queries using Amazon Athena, as we already do in other areas, such as AWS CloudTrail, to analyze user activity and API usage.

Our security team wants to know who is accessing the resources, on which endpoint they are accessing them, and the time spent on the activity. Verified Access logs provide all the requested information and more. Those logs will be aggregated to our existing Google Workspace access logs to have a full view of the operation.

Additionally, because the endpoints are public-facing, we are going to integrate AWS WAF in our Verified Access instances to reduce our attack surface. All our AWS WAF logs are being analyzed using Athena, and we already have custom queries for common attack patterns. With the help of these logs, we can identify if our endpoints are being targeted by external attackers or bots.

Summary

At Fever, AWS Verified Access has simplified how we expose and manage permissions in our corporate applications. The service allows us to provide organization-wide secure access in minutes for every application we deploy while still creating more fine-grained authorization rules when needed.

About the authors