Optimizing operations for ground-based, extremely large telescopes with AWS

Ground-based, extremely large telescopes (ELTs), such as the Giant Magellan Telescope (GMT), will play a crucial role in modern astronomy by providing observations of the universe with remarkable clarity and detail. They achieve this by deploying multiple mirrors, each one being many meters in diameter, to capture an unprecedented amount of light. However, managing the vast amount of data generated by these instruments and supporting optimal performance can be a challenging task.

Amazon Web Services (AWS) provides a suite of cloud-based solutions that can help address these challenges and streamline ELT operations. Learn how various AWS services can be used to optimize data storage, management, and processing, as well as advanced monitoring and remote continuity techniques, leading to improved overall performance and efficiency for ELTs.

Data storage and management

As ELTs generate vast amounts of data, often nearing the petabyte scale, efficient data storage and management is vital. AWS offers several services that help manage this data effectively. Amazon Simple Storage Service (Amazon S3) provides scalable and cost-effective storage for the raw and processed data generated by ELTs. Amazon S3 supports the storage of data in different classes, such as Amazon S3 Intelligent-Tiering and Amazon S3 Glacier, allowing organizations to optimize costs based on data access requirements. Lifecycle policies for Amazon S3 automatically migrate data between Amazon S3 classes based on tags, age, and other rules to lower management burden. To reliably move petabytes of data, AWS DataSync copies data into Amazon S3 storage with end-to-end security, including data encryption and integrity validation checks for certainty that every bit replicates properly.

Processing and analysis

The large volumes of data generated by ELTs require efficient processing and analysis. AWS provides various services that can help in this regard. Amazon Elastic Compute Cloud (Amazon EC2) offers scalable and customizable computing resources for processing and analyzing the data. By using Amazon EC2 instances with high-performance graphics processing units (GPUs), ELTs can benefit from accelerated data processing. Built on top of Amazon EC2, managed services for scientific computing, research, and data analysis enable researchers to avoid the complexity in configuring compute environments, and focus attention on what matters most: science. For example, Amazon LightSail for Research offers managed research desktop environments pre-provisioned with Scilab, RStudio, Jupyter, and more with just a few clicks.

The AWS high performance computing (HPC) portfolio more broadly tackles large-scale compute workloads that are tightly-coupled, loosely-coupled, or embarrassingly-parallel. Amazon EC2 currently offers over 600 instance types–combinations of central processing unit (CPU) architecture, including AMD, Intel, and AWS Graviton; memory sizes up to 24TB in a single instance; and accelerators, including GPUs, field-programmable gate arrays (FPGAs), and AWS-custom machine learning (ML) accelerators, Inferentia and Trainium. AWS ParallelCluster can deploy traditional HPC environments based on any instance type, with the same look and feel as an on-premises HPC system, with the advantages of the cloud like elastic scaling of compute pools and flexible configuration. In minutes, ELTs can launch, reconfigure, and resize clusters, compared to month(s)-long lead-time to acquire and on-ramp new hardware on-site.

Research never happens in a vacuum, and ELTs are cornerstone instruments serving large collaborative communities with a myriad of applications. Native integration across AWS allows AWS Batch, a cloud-native HPC service for container workloads, to automate processing of large datasets as part of a broader event-driven “HPC-plus” workflow, i.e., “HPC plus machine learning,” “HPC plus internet of things (IoT),” etc. With event-driven architectures, AWS Batch processing jobs are triggered by arriving data, and in turn feed output to other services or subsequent jobs that optimize for downstream tasks. Using cloud-native integrations also optimizes resource allocation and cost management. AWS Lambda, a serverless computing service that runs code in response to events, removes the burden of infrastructure management and supports massive parallelism in data processing pipelines. For example, read more about the Lambda-driven architectures by the Italian National Institute for Astrophysics (Instituto Nazionale di Astrofisica or INAF) for telescope data, and UK Met Office for 400 land-based instruments.

Machine learning and artificial intelligence

Machine learning (ML) and artificial intelligence (AI) services can enhance ELT operations. Amazon SageMaker is a fully managed service that helps organizations build, train, and deploy ML models at scale. By integrating SageMaker, astronomers can develop models to automate tasks such as object identification, noise reduction, and data calibration.

Collaboration and sharing

AWS provides tools that facilitate collaboration and data sharing among researchers and institutions. AWS Data Exchange helps organizations share their datasets with other users, making it simpler to access and collaborate on ELT-generated data. AWS also hosts a number of astronomy-focused datasets on the Registry of Open Data (RODA), along with more than 100 petabytes (PB) of high-value, cloud-optimized data published through the AWS Open Data Sponsorship Program for public use. With RODA, researchers and citizen astronomers alike can find, share, and use datasets staged publicly for analysis on AWS, including but not limited to the Hubble Space Telescope and the Kepler mission.

Monitoring and optimization

Efficient monitoring and optimization of ELT operations are integral for maximizing performance and minimizing expensive downtime. ELTs are massive in scale and have to orchestrate thousands of interconnected components. These components are often networked Internet of Things (IoT) devices. AWS offers services that aid in IoT data collection, such as AWS IoT SiteWise, and AWS IoT Core, to transmit that data to the cloud. Once this data is collected, it can be fed into predictive models representing the physical characteristics of the components via AWS IoT TwinMaker.

Using TwinMaker, ELT teams can build a comprehensive, spatial model of the telescope that will be updated with near real-time IoT telemetry (see e.g., an example twin of the International Space Station (ISS) based on publicly available telemetry). This approach will allow for future defects and normal wear and tear of the overall ELT system to be caught well before damage can be done to critical components. Early failure detection could even be integrated into the ELT team’s supply chain management system to allow for appropriate logistical lead time in ordering replacement components.

Continuation of operations in remote areas

Perhaps the most challenging task in streamlining ELT operations is bringing the power of the cloud into or near ELTs. ELTs are located in remote areas, such as the Atacama Desert in northern Chile, to take advantage of their unparalleled, pristine views of the night sky. These conditions introduce natural challenges, such as drastic temperature swings and even seismic activity. With AWS HPC, ELT teams can model these natural effects and develop reinforced structures to handle these perturbations (see e.g., Giant Magellan Telescope’s example from AWS re:Invent 2022).

ELTs do not always have steady network connectivity to connect to the cloud given their distance and remoteness, but they must still be able to operate. AWS edge computing services such as AWS Snowball Edge and AWS Outposts can help ELTs run many of the aforementioned AWS storage and compute services on physical hardware on-site, even in remote areas. Additionally, AWS Local Zones can help ELT teams deploy select services at the closest metro area, drastically reducing potential latency to reach a Region.

Conclusion

Using AWS services for HPC, AI and ML, and IoT, ELTs can offload the undifferentiated heavy lifting of many IT operations and decrease the time to scientific discoveries.

If you would like to learn more about how AWS is helping ELTs develop and build for the future of astronomy, please watch Dr. Robert Shelton’s talk at re:Invent 2022 on computational deep space exploration.

Learn more about AWS for aerospace and satellite.

Read related stories on the AWS Public Sector Blog:

Subscribe to the AWS Public Sector Blog newsletter to get the latest in AWS tools, solutions, and innovations from the public sector delivered to your inbox, or contact us.

Please take a few minutes to share insights regarding your experience with the AWS Public Sector Blog in this survey, and we’ll use feedback from the survey to create more content aligned with the preferences of our readers.