Caching Hudl’s News Feed with Amazon ElastiCache for Redis

Darin Briskman (@briskmad) is a developer evangelist for Amazon Web Services.

We hope you enjoy this guest post from Joel Hensley, Engineering Director for the Community Tribe at Hudl, a leading software company that is revolutionizing the way coaches and athletes prepare for and stay ahead of the competition. You can reach Joel @fusbal.

Hudl offers coaches and athletes the tools to edit and share video, study associated play diagrams, and create quality highlight reels for entertainment and recruiting purposes. When football season starts in August, coaches and athletes from all over the country sign in to Hudl every day. After users sign in, they land on a tailored news feed like the one you see below. It includes team content and accounts the user is following. Because this page is our users’ first impression of Hudl, its performance is critical. That’s why we cache the news feed by using Amazon ElastiCache for Redis.

Before we discuss the details, it’s important to understand our data model. There are six main collections used by the feed:

Let’s consider two users: Sally and Pete. Sally decides to follow Pete. At Hudl, this means Sally is a follower of Pete and Pete is a friend of Sally. When Pete posts something, that content is added to his user timeline and to Sally’s home timeline. When Sally signs in to view her feed, the content on her home timeline is displayed in reverse chronological order. If she clicks on Pete, she sees his user timeline and can view all of his posts.
Each time a user hits his/her feed, we grab a batch of post IDs from the user’s home timeline, fetch those posts, and load all of the users referenced in those posts. Since the feed’s creation in April 2015, the database has grown quickly. The total size is now up to 120 GB. We currently have 18 million follower relationships and 30 million pieces of content. So where does caching come into play?

Redis
In the world of caching, there are two primary options: Memcached and Redis. For a long time at Hudl, our default option was Memcached, but with the introduction of our news feed, we decided to explore the Redis data structures. We’re really excited by what we found:

• Lists
  This alone would have been reason enough to use Redis! Timelines are stored as lists, and being able to represent them that way in cache is amazing. As posts are added to timelines, we simply do an LPUSH (add to the front) followed by an LTRIM (used to cap the list at a max size). The best part is that we don’t have to invalidate the cache as posts are added because it’s kept in sync with the database.

• Hashes
  Displaying the number of followers and friends for a user is a critical component of any feed. By storing these as fields in a hash for each user, we can quickly call HINCRBY to keep the values in sync with the database. No need to invalidate the cache whenever a follow or unfollow occurs.

• Sets
  We love to use RabbitMQ to retry failed operations. Sets are the perfect way to guarantee we won’t accidentally insert the same post on a user’s timeline more than once. There’s no need for an extra database call. We use the post ID as the cache key, each user ID as the member, and then call SISMEMBER and SADD.

Amazon ElastiCache
We needed to spin up and configure a server so we could start testing Redis. Because we love AWS, we decided to try Amazon ElastiCache for Redis. Within minutes, our first test node was up and running Redis. We connected to it through the StackExchange.Redis C# driver.
By using ElastiCache, we were able to easily configure our Redis deployment, node size, and security groups. We were also able to use Amazon CloudWatch to monitor key metrics. We were able to create separate test and production clusters without having to wait for an infrastructure engineer to manually set up and configure the servers.
Here are the details of our production cluster:

• Node type: cache.r3.4xlarge (118 GB)
• Replication enabled
• Multi-AZ
• 2 read replicas
• Launched in VPC

Our final deployment step was to configure alerts through Stackdriver, which integrates seamlessly with ElastiCache.
We were most interested in these three metrics:

• Current connections: If these drop to 0, our web servers can no longer access the cache and require immediate attention.
• Average bytes used for cache percentage: If this reaches 95% or more, we might need to move to a larger node type or lower our expiration times.
• Swap usage: If this reaches 1 GB or more, the Redis server requires immediate attention.

Results
Since its launch more than a year ago, we’ve been very happy with the performance of the news feed. This year, during the week of September 5, 1.2 million unique users accessed their feeds. The feed service averaged 300 requests per second, with a peak of 800.
Here are some quick ElastiCache stats from that week:

• Total cached items: 21 million
• Cache hits: 175K/min (average), 350K/min (peak)
• Network in: 43 MB/min (average), 101 MB/min (peak)
• Network out: 600 MB/min (average), 1.25 GB/min (peak)

Let’s take a look at two calls in the feed service:

• Get Timeline
  This operation just fetches the timeline list from Redis. There are no dependencies.

• Hydrate Timeline
  This call takes the post IDs in the timeline and loads all referenced users and posts. This includes time spent loading records from the MongoDB database if they are not cached and then caching them. This is the primary call used when loading the feed on the web and on our iOS and Android apps.

Based on the success of the news feed, ElastiCache for Redis is quickly becoming our default option for caching. In the last year, five other Hudl services have made the switch from Memcached to Redis. It’s easy to set up, offers blazing-fast performance, and gives users all of the benefits that Redis has to offer. If you haven’t already, we strongly recommend trying Amazon ElastiCache for Redis. Please let us know how it works for you.

For more information, see these sources:
Hudl: http://www.hudl.com/
Hudl Bits blog: http://www.hudl.com/bits
What is Redis? https://aws.amazon.com/redis/
Amazon ElastiCache for Redis: https://aws.amazon.com/elasticache/redis/