How Flickr modernized caching with Amazon ElastiCache for Valkey

This is a guest post by Wilson Lam and Ethan Smith at SmugMug and Flickr, in partnership with AWS.

Flickr is one of the world’s largest photo-sharing communities, where caching is fundamental to delivering the fast, responsive experiences our users expect. As part of SmugMug, our team of approximately 200 employees operates two major online photo platforms. These platforms serve more than 100 million photographers and photo enthusiasts with tens of billions of images each year.

For years, our caching layer built on Redis clusters behind Twemproxy served us reliably. As our platform evolved, we identified an opportunity to take the next step: adopt Amazon ElastiCache for Valkey for its open source governance and cost advantages. We also wanted to consolidate clusters for simpler operations, enable cluster mode for better scalability, and move to a modern client library with native connection pooling. The goal was a more resilient, efficient, and future-ready caching architecture without disrupting the platforms that millions of photographers rely on every day.

In this post, we walk through how we planned and executed this modernization across multiple production caching workloads, the practical challenges we navigated, and what we learned.

What we wanted to improve

Our modernization targeted four areas: the cache engine, the proxy layer, the client library, and the advanced ElastiCache capabilities we weren’t yet using. Here’s what we wanted to improve in each.

Moving from Redis to Valkey

We wanted to make the switch from Redis to Valkey as the underlying engine for a few reasons:

• Open source governance – Valkey is licensed under a permissive open source license (BSD 3-Clause) and stewarded by the Linux Foundation, with backing from multiple companies, including AWS. This backing gives us confidence in the project’s longevity and community-driven development.
• Cost efficiency – Amazon ElastiCache for Valkey is priced 33% lower for Serverless and 20% lower for node-based deployments than the other supported engines, including the Redis OSS engine. Customers can also purchase AWS Database Savings Plans for additional savings.
• Community momentum – Since its inception in March 2024, Valkey has seen rapid adoption, with ongoing performance improvements reported by the community over previous versions.

Streamlining the proxy layer

We had introduced Twemproxy in 2019 to handle connection pooling for our PHP workers, because our client library at the time didn’t support it natively. Over time, we recognized functional gaps with Twemproxy. Twemproxy connected only to writer endpoints, which meant reader replicas weren’t being used for read traffic. Every cache request also traveled through the proxy as an additional network hop. With a client library that supports native connection pooling and cluster-aware routing, we could take full advantage of our replica capacity and remove an entire layer from the architecture.

Upgrading the client library

Our PHP application used Predis for all Redis operations. Although it served us well, the C-extension-based PhpRedis offered advantages we wanted, such as native connection pooling (removing the need for Twemproxy), full cluster mode support, better performance from the C implementation, and broader community support for Valkey going forward.

Using advanced ElastiCache capabilities

With cluster mode disabled across our workloads and each running as individual standalone clusters, we weren’t using multi-Availability Zone (multi-AZ) automatic failover, auto scaling, or the operational benefits of consolidated cluster management. Modernizing would let us turn these on across the board.

The modernization approach

Our caching layer supported several distinct workloads, each with different data sensitivity, traffic patterns, and tolerance for data loss. Some handled ephemeral data we could re-create if lost. Others tracked user-visible metrics where accuracy was critical. The topology had grown over the years: more than 40 individual, non-clustered instances spread across multiple workloads, with varying replica configurations.

We developed a phased plan that followed a consistent pattern for each workload:

• Engine upgrade: Transition from Redis to Valkey across clusters.
• Right-size and consolidate: Replace individual clusters with a single cluster-mode-enabled deployment sized for the workload.
• Consider your existing cost structure: Sizing analysis might not always point to the most cost-effective choice. Factors beyond raw sizing can make a different instance type the better overall decision.
• Migrate the client library: Swap Predis for PhpRedis (C-extension, native connection pooling).
• Retire Twemproxy: No longer needed with PhpRedis and cluster mode.
• Migrate data: Use RIOT (Redis Input/Output Tools) where existing data needed to be preserved.
• Implement best practices: Take advantage of the advanced features ElastiCache provides, like Multi-AZ and auto-failover, encryption, and proper replica configuration (see ElastiCache best practices).

We started with the lowest-risk workload and worked our way up to the most critical ones. All were live production workloads serving Flickr users, and the progressive rollout kept end-user impact minimal throughout. All production workloads, spanning more than 40 clusters, were consolidated into cluster-mode-enabled deployments with one primary and one replica per shard, auto-failover, and multi-AZ enabled. Shard counts ranged from 4–8 depending on throughput and memory requirements, using cache.t4g.small for lighter workloads and cache.r6g.large for memory-intensive ones.

We approached the migration in three phases. In preparation, we upgraded the client library before touching any infrastructure. In execution, we migrated each workload’s engine and cluster topology. In cutover, we progressively rolled out traffic and validated it before decommissioning the old setup.

Preparation: Predis to PhpRedis

Before migrating any workload, we tackled the client library upgrade. The switch from Predis to PhpRedis was foundational because it provided the native connection pooling, full cluster mode support, and better raw performance from the C implementation that the target architecture required.

We built a compatibility wrapper that maintained the same application-level API while swapping the underlying client. The wrapper was written in PHP, implementing the same interface as Predis while routing all operations through PhpRedis underneath. This decoupled the client migration from infrastructure migration, which was key. We deployed the wrapper behind a feature flipper, a global toggle that routed PHP operations through PhpRedis for a configurable percentage of users.

This flipper-based approach became our standard rollout pattern for every workload: start at a small percentage of users, monitor, and ramp up progressively. It gave us confidence to move quickly while keeping risk low.

Executing the migration

With the client library in place, we moved to the infrastructure itself, one workload at a time. We started with the engine swap, then reshaped the cluster topology.

Migrating the engine: Redis to Valkey

Our first step was upgrading from Redis to Valkey as the underlying engine across all clusters. Valkey offers full API compatibility with Redis OSS up to version 7.2, the point at which Valkey forked. That made this the simplest part of the entire modernization effort: no application code changes, no data migration, only an in-place engine swap. It validated our confidence in Valkey’s compatibility and set the stage for the more involved infrastructure work that followed.

Migrating cluster topology: Starting with ephemeral data

With Valkey running across our clusters, we turned to the bigger structural changes. We deliberately chose our lowest-risk workload first, one where the data is ephemeral and can be regenerated at any time. This made it the ideal candidate to validate the full approach: new client library, new cluster topology, no proxy.

We analyzed the existing cluster utilization and found significant headroom: memory usage under 3 percent, CPU under 5 percent, with the heaviest cluster handling more than 20 million GETs per hour. The analysis confirmed that consolidation into a single cluster-mode deployment would comfortably handle the combined load.

Since the data was regenerable, we started the new cluster fresh, with no data migration needed. We deployed the updated code, turned on the PhpRedis flipper at 10 percent, validated, and quickly ramped to 100 percent. Everything went as planned. After confirming stability, we cleaned up the original clusters and retired the Twemproxy configuration. This first migration gave us confidence in the entire stack and set the pattern for everything that followed.

Migrating cluster topology: Data-sensitive workloads with RIOT

For workloads where data needed to be preserved, particularly those with user-visible metrics, we used RIOT (Redis Input/Output Tools) to migrate data to the new consolidated clusters.

We stood up a dedicated c5n.2xlarge EC2 instance (up to 25 Gbps network bandwidth) as a RIOT server, purpose-built for running replication. Having dedicated compute for RIOT avoided contention with application workloads and gave us the network throughput we needed for a network-intensive migration. Because data for each workload was spread across multiple source endpoints, we ran one RIOT replicate command per source. RIOT supports many-to-one migration, but each source requires its own command invocation.

For each workload, we first ran RIOT in scan mode to replicate the full keyspace from all sources to the new cluster. After the bulk copy was complete, we switched RIOT to live mode for continuous replication using keyspace notifications. This kept the source and target in sync while we validated and progressively rolled out traffic.

A practical note on live replication with ElastiCache: RIOT’s live replication mode works well with Amazon ElastiCache. The key is to enable keyspace notifications through the cluster parameter group (set notify-keyspace-events) rather than using the CONFIG command, which ElastiCache restricts. See the RIOT Keyspace Notifications guide and AWS Knowledge Center article.

Progressive rollouts and what they caught

For every workload, we followed the same flipper-based progressive rollout: 5 percent (staff accounts), then 20 percent, 50 percent, and 100 percent. This pattern turned out to be critical.

During the migration of one data-sensitive workload, after reaching 100 percent, we discovered temporarily outdated cached content. The application would briefly show older cached content before refreshing correctly. We traced this to a small subset of keys that hadn’t been fully synchronized during the RIOT live replication window. Rather than rolling back, we ran a targeted backfill task from the database to repair the affected data. The impact was minimal and resolved quickly.

This experience directly shaped our approach for subsequent workloads. We started running RIOT’s compare mode to validate replication completeness before cutting over, not after. For teams using RIOT, we’d strongly recommend making this a standard step in your migration runbook.

Tuning for scale

On our most fragmented workload, which had the highest number of source clusters, RIOT stalled during the initial replication runs. We resolved this by increasing the thread count and reducing the batch size, which got the replication flowing smoothly. It’s worth testing production-scale data in staging before running against your live clusters, so you can dial in these parameters without pressure.

To validate replication accuracy after tuning, we compared data between the source and target by querying specific sorted sets and confirming that the timestamps matched. Both endpoints returned identical values, confirming data parity before we began shifting traffic.

Retiring Twemproxy

With each workload migrated to cluster-mode-enabled Valkey clusters using PhpRedis, Twemproxy was no longer needed. We retired it progressively, decommissioning the proxy hosts and removing the configuration after confirming each cutover was stable.

The benefits were immediate: reduced latency from eliminating the extra network hop, better capacity usage with PhpRedis routing reads to replicas, simpler operations with fewer components to maintain, and a smaller infrastructure footprint overall.

Cross-AZ data transfer considerations

During planning, we evaluated the cost implications of Valkey’s gossip protocol for cluster topology discovery. Two key points worth sharing:

First, gossip protocol traffic doesn’t incur cross-AZ charges. Topology discovery only triggers when a client detects a MOVED error. It’s not continuous traffic. For PhpRedis, see the keyspace map documentation.

Second, standard application requests follow normal cross-AZ pricing: managed instance to managed instance across AZs is $0/GB, whereas managed instance to EC2 across AZs is $0.01/GB in each direction ($0.02/GB round-trip).

Results

Cost efficiency: Despite consolidating to larger, more capable cluster-mode deployments with consistent replica configurations and multi-AZ failover, our monthly ElastiCache spend decreased by approximately 5 percent while gaining significantly better operational characteristics. The 20 percent Valkey pricing advantage offset the cost of running properly replicated, highly available clusters. During the migration period over several months in 2025, we saw temporary cost increases while running parallel old and new clusters, but costs normalized quickly after consolidation completed.

Conclusion

Over the course of several months, we migrated from Redis to Valkey and consolidated more than 40 individual clusters into cluster-mode-enabled Amazon ElastiCache for Valkey deployments. We retired our Twemproxy proxy layer, migrated from Predis to PhpRedis, and enabled multi-AZ failover across all workloads. We did all of this while keeping Flickr running for the more than 100 million photographers who depend on it.

The engine switch to Valkey was the simplest step because full API compatibility meant zero application changes, but it was also the foundation that made everything else possible. The phased approach, starting with the safest workload and building up, kept risk manageable throughout. Investing early in the PhpRedis compatibility wrapper meant we could tackle client migration and infrastructure migration independently. And the progressive rollout pattern, which caught issues early and kept their impact contained, is something we’d recommend for any similar migration.

For teams looking to modernize their caching infrastructure, Valkey’s API compatibility means you can upgrade the engine independently from the rest of the stack. Combine that with cluster mode, a modern client library, and tools like RIOT for data migration, and you have a practical path to a more resilient and efficient architecture.

Finally, for teams considering a similar modernization, the path is more approachable than it might seem. Start with your lowest-risk workload, build confidence, and work your way up. The Valkey community and tooling, especially RIOT for data migration, have matured rapidly. And the operational benefits of cluster mode, automatic failover, and native connection pooling compound over time. Those gains are hard to quantify but clear to appreciate at 2 AM when you’re not getting paged. To learn more, visit the Amazon ElastiCache for Valkey detail page and the ElastiCache best practices guide.

About the authors