Adobe Firefly: Simplified observability with Amazon Managed Prometheus

Adobe has used Amazon Web Services (AWS) since 2008. Adobe Firefly powers creative features across applications including Photoshop and Illustrator.

Adobe operates a GPU-based training infrastructure built on Amazon Elastic Kubernetes Service (Amazon EKS) to support Firefly. The infrastructure enables teams to run model training jobs across thousands of compute nodes and GPUs, designed to scale with growing demand.

The team initially relied on a self-hosted Prometheus infrastructure, sending data to a remote endpoint for long-term retention. As Firefly’s adoption increased and training jobs scaled, Adobe needed an observability solution that could deliver fast query performance over large metric volumes, remain highly available and scalable, and give infrastructure users self-service access to the infrastructure metrics they need to monitor and troubleshoot training jobs independently.

This post describes how Adobe evolved its observability architecture — from a self-managed Prometheus deployment for in-cluster metrics to Amazon Managed Service for Prometheus for critical metrics — and the measurable improvements in query performance, infrastructure reliability, and scale.

The challenge: GPU observability at scale

Monitoring GPU-based training infrastructure presents unique challenges that differ from traditional application monitoring. GPU training clusters generate high-cardinality telemetry across multiple dimensions like GPU health and performance metrics, compute and memory metrics and more.

Unlike CPU workloads where a single utilization metric may suffice, GPU training jobs require engineers to observe the interplay between compute, memory, and network layers to identify bottlenecks. For example, training jobs running across 2,000 nodes with 16,000 GPUs, scraped every 30 seconds, can generate over 1 billion data points in a single query window.

Self-hosted monitoring infrastructure was not meeting the performance requirements for queries at this cardinality and volume.

From self-managed Prometheus to Amazon Managed Service for Prometheus

Adobe’s observability evolution was not a single migration. It was an iterative process, with each phase addressing a specific set of limitations and informed by direct feedback from infrastructure users on what mattered most to them. As metric volumes grew, the team evaluated Amazon Managed Service for Prometheus as a fully managed alternative that could handle their horizontal scale requirements without the operational overhead of maintaining their own deployment.

Infrastructure users shaped the critical metric set iteratively through direct input on what they needed to see to run their training jobs effectively. The critical metrics were curated to support:

• Job-level monitoring: GPU utilization, memory consumption, and network throughput per training job, enabling users to identify bottlenecks in distributed training.
• Pod and node health: Kubernetes pod status, node readiness, and resource allocation metrics feeding into scheduler decisions.
• GPU health: Metrics that determine whether a GPU is healthy or needs to be cordoned and replaced.

The team has already moved critical 2M time series metrics to Amazon Managed Service for Prometheus, targeting the specific problem of query performance at scale. Adobe used Amazon Managed Service for Prometheus collector (managed scrapers) to handle the collection of metrics from their Amazon EKS-based training clusters and forward them directly to Amazon Managed Service for Prometheus workspaces. Rather than replacing the self-managed Prometheus deployment entirely, the managed scrapers operated alongside it, taking over the scraping role for metrics destined for Amazon Managed Service for Prometheus while preserving Adobe’s existing Prometheus setup. This allowed the team to adopt Amazon Managed Service for Prometheus incrementally without disrupting their current monitoring workflows.

Why Amazon Managed Service for Prometheus

Amazon Managed Service for Prometheus provided the capabilities that addressed Adobe’s core requirements:

• Query performance at scale: Purpose-built for fast queries over high-cardinality, high-volume time series data.
• High availability: Built-in high availability without custom HA configurations, providing a reliable data source for downstream automated systems that depend on timely metric queries.
• Migration ease: No agents required. The migration path uses remote write configuration with minimal changes to existing workflows.
• Scalability: Each workspace supports up to 50 million active time series, providing headroom for growth as the infrastructure scales (up to 1 billion) [1].
• AWS integration: Native integration with AWS services including Amazon EKS and Amazon Managed Grafana, simplifying metric collection and reducing configuration complexity.
• Managed operations: Minimizes the operational burden of administering self-hosted monitoring infrastructure, freeing engineering resources for infrastructure development.

Note: Amazon Managed Service for Prometheus and Amazon Managed Grafana are billable services. Costs are based on metrics ingested, stored, and queried. Review the pricing pages for Amazon Managed Service for Prometheus and Amazon Managed Grafana to estimate costs for your workload before deployment.

Results

After migrating critical metrics to Amazon Managed Service for Prometheus, Adobe Firefly achieved the following measurable improvements.

Query performance: before and after

Figure 1: Query performance comparison for GPU utilization metrics

Conclusion

Adobe Firefly evolved its observability architecture from a self-managed Prometheus deployment to Amazon Managed Service for Prometheus, using Amazon Managed Service for Prometheus collector to handle metric collection alongside their existing Prometheus infrastructure. This approach preserves current workflows while adding managed collection.

• Query performance improvement of more than 28x: Queries that previously timed out at 60 seconds or returned partial results in 2 minutes now complete in approximately 10 seconds.
• Extended observability windows for training jobs: Infrastructure users now view metrics across 24-hour windows, compared to the previous practical limit of 6 hours. This is particularly impactful for large, long-running training jobs spanning 256 or more nodes, where the ability to see the full lifecycle of a job helps identify when performance degraded, correlate issues with infrastructure events, and make informed decisions.
• Reduced operational overhead: Amazon Managed Service for Prometheus requires no agents and no additional Prometheus-related configuration on your end. Both data and control components are fully managed, minimizing the burden of maintaining self-hosted Prometheus infrastructure.

To learn more about Amazon Managed Service for Prometheus, visit the Amazon Managed Service for Prometheus documentation. For guidance on implementing sharding strategies, see the Amazon Managed Service for Prometheus best practices guide.

Looking ahead

The performance improvements demonstrated with GPU utilization queries were consistent across other GPU metrics as well, including GPU memory usage, power consumption, and thermal monitoring. These results confirm that Amazon Managed Service for Prometheus benefits extend across the full breadth of GPU telemetry. Adobe and AWS are collaborating on the next phase of this observability architecture to extend managed Prometheus to the remaining metric tiers, enabling a multi-tenant, highly available observability stack that supports the full scale of telemetry at Adobe Firefly.

About the authors