AWS Quantum Technologies Blog

Running scalable hybrid quantum-classical workflows by integrating Amazon Braket with AWS Batch

Practical quantum computing is hybrid: a quantum processing unit (QPU) executes a classically hard subroutine whose problem structure is amenable to quantum methods, while classical resources handle the surrounding computation. What varies across quantum algorithms is the workflow pattern: how the computation is distributed, what communication is needed, and how the data flows. Variational algorithms such as the Variational Quantum Eigensolver (VQE) and the Quantum Approximate Optimization Algorithm (QAOA) interleave quantum circuit evaluations with a classical optimizer, exchanging data back and forth in a closed loop. Canonical algorithms like Quantum Phase Estimation and Shor’s algorithm, by contrast, concentrate most of the work in a deeper quantum circuit with comparatively little classical processing.

Over the past few years, a third group of algorithms has been gaining popularity by decoupling the quantum and classical subroutines into sequential steps. Instead of an optimization loop, these algorithms collect a batch of measurement samples from a QPU and then feed them into classical post-processing that can be computationally demanding, often requiring HPC-scale resources. The quantum-classical auxiliary-field quantum Monte Carlo (QC-AFQMC) algorithm and the sample-based quantum diagonalization (SQD) algorithm are two prominent examples. Orchestrating such workflows can be challenging: a single run spans heterogeneous resources that scale independently and must execute in sequence, with each step consuming the output of the previous one. Coordinating these steps with custom scripts and manual polling is error-prone and difficult to operate and monitor, and few off-the-shelf tools manage an end-to-end lifecycle that spans quantum and classical services.

In this post, we show how to run hybrid quantum classical workflows by integrating Amazon Braket and AWS Batch into a scalable compute pipeline orchestrated with AWS Step Functions. To give a practical example, we apply it to QC-AFQMC. The solution can be adapted to other algorithms, such as SQD, that share a similar workflow pattern.

Example Workflow

The QC-AFQMC algorithm is a variant of the classical AFQMC algorithm, typically used to find the ground state of many-body Hamiltonians [1-5]. It uses a quantum computer to prepare a trial state |ΨT〉. The trial state serves as the initial guess and guides the subsequent Monte Carlo time evolution through overlap 〈ΨT|φi〉 and local energy 〈ΨT|H|φi〉 / 〈ΨT|φi〉 evaluations, where |φi〉 represents a sample in the Monte Carlo time evolution, typically referred to as a walker (Fig. 1 inset). Using matchgate shadows, the QC-AFQMC algorithm holds the potential to surpass its classical counterpart in accuracy while maintaining a polynomial scaling in runtime.

Figure 1 - Overview of the QC-AFQMC algorithm, grouped into the three stages used throughout this post (quantum sampling, classical post-processing, and energy averaging). A quantum-prepared trial state seeds a population of walkers that evolve independently through imaginary time, with each walker's overlap and local energy evaluated from the trial state via classical shadows, and the ground-state energy obtained as the weighted average of the walkers' local energies. The inset shows the walkers driven from the trial state to the true ground state.

Figure 1 – Overview of the QC-AFQMC algorithm, grouped into the three stages used throughout this post (quantum sampling, classical post-processing, and energy averaging). A quantum-prepared trial state seeds a population of walkers that evolve independently through imaginary time, with each walker’s overlap and local energy evaluated from the trial state via classical shadows, and the ground-state energy obtained as the weighted average of the walkers’ local energies. The inset shows the walkers driven from the trial state to the true ground state.

Multiple ways to implement the QC-AFQMC algorithm exist, including an iterative approach that evaluates the overlaps and local energies for the walkers by executing Hadamard-test-type quantum circuits [2]. We explored this approach in a previous blog post. Here we focus on an alternative approach using shadow tomography [3], which decouples the quantum sampling from the classical post-processing. Shadow tomography builds a compact classical description of the trial state by executing a batch of quantum circuits with randomized measurements, whose outcomes (shadows) are saved as classical data. These shadows are then reused throughout the subsequent AFQMC evolution to estimate the overlaps and local energies for each walker. A typical calculation propagates thousands of walkers, and since they evolve independently, this classically demanding stage parallelizes naturally across many processors. The calculation concludes by estimating the system’s energy from a weighted average of the walkers’ local energies.

Solution Architecture

The QC-AFQMC workflow summarized above consists of three steps: an initial quantum sampling step, a computationally expensive parallel classical post-processing step, and a final energy-averaging step. We map each step to an AWS service whose execution model fits naturally:

  • Quantum sampling: The collection of shadow data involves the evaluation of a high volume of quantum circuits. We run this workload as an Amazon Braket Hybrid Job which allocates QPU or circuit simulator resources alongside the classical resources that drive quantum program execution, and combines the shadow tomography measurements in a self-contained way. The job is created conveniently with a single API call, resources are automatically allocated and released, and the collected shadow data is stored durably in a bucket on Amazon S3.
  • Classical post-processing: We run the embarrassingly parallel evolution of thousands of independent Monte Carlo walkers as an array job on AWS Batch. The job is submitted with a single API call, the service launches one task per walker across many compute nodes and automatically scales the compute fleet on demand. Each task of the array job reads the shadow data from S3, evolves a single walker, and writes partial results (the walker’s local energy) back to S3.
  • Energy averaging: We use a lightweight AWS Lambda function to gather the per-walker local energies stored in S3, compute the final estimate of the ground-state energy, and store it in the S3 bucket.

With the above service choices, the three steps are effectively decoupled, can allocate and release the computing resources they require, and scale independently, each launched with a single service API call. Instead of tying the steps together with a custom script and manual polling, we use AWS Step Functions for automated end-to-end workflow orchestration. The QC-AFQMC workflow is modeled as a state machine (see Fig. 2), which invokes the steps in sequence, starting each step once the previous has completed and passing a reference to the intermediate results stored in S3. The service provides workflow tracking, monitoring, and error handling to catch and retry failed steps.

Compared with running these stages on a traditional shared HPC cluster often operated by a separate administrative team, this cloud-native pipeline is defined, deployed, and owned by the application user: each stage provisions its own resources on demand and releases them when done, and the whole workflow is version-controlled and deployed as application-level infrastructure. The administrative boundary doesn’t disappear: the account owner can still set the guardrails (e.g. service quotas, IAM boundaries, and cost controls) the application user works within.

Figure 2 - End-to-end architecture of the QC-AFQMC workflow on AWS. The user packages the algorithm code and circuit ansatz into a container image in Amazon ECR (1), deploys and executes the workflow (2), and retrieves the results (3). AWS Step Functions orchestrates the run as a state machine, invoking the Amazon Braket Hybrid Job (quantum sampling), the AWS Batch array job (classical post-processing), and the AWS Lambda function (energy averaging) in sequence. Amazon S3 stores the input and output data for each step and returns the final results to the user.

Figure 2 – End-to-end architecture of the QC-AFQMC workflow on AWS. The user packages the algorithm code and circuit ansatz into a container image in Amazon ECR (1), deploys and executes the workflow (2), and retrieves the results (3). AWS Step Functions orchestrates the run as a state machine, invoking the Amazon Braket Hybrid Job (quantum sampling), the AWS Batch array job (classical post-processing), and the AWS Lambda function (energy averaging) in sequence. Amazon S3 stores the input and output data for each step and returns the final results to the user.

Workflow Execution

To run the whole workflow, we start an execution of the state machine and follow its progress in the Step Functions console. Fig. 3 shows a run underway, with the Braket Hybrid Job (quantum sampling) running while the Batch post-processing and the Lambda averaging steps have not yet started.

Figure 3 - Monitoring a running workflow from the AWS Step Functions console. Each state is colored by status. Here the initial Pass - PassHyperparameters step has completed (Green) and the Amazon Braket Hybrid Job - QuantumSampling step is still in progress (Blue), while the AWS Batch post-processing - PostProcessing and the AWS Lambda averaging - DataCollection steps have not yet started. The right pane shows the selected state's inputs and outputs. The whole hybrid quantum-classical run is observable from a single view, making it easy to see which stage is active and to catch failures.

Figure 3 – Monitoring a running workflow from the AWS Step Functions console. Each state is colored by status. Here the initial Pass – PassHyperparameters step has completed (Green) and the Amazon Braket Hybrid Job – QuantumSampling step is still in progress (Blue), while the AWS Batch post-processing – PostProcessing and the AWS Lambda averaging – DataCollection steps have not yet started. The right pane shows the selected state’s inputs and outputs. The whole hybrid quantum-classical run is observable from a single view, making it easy to see which stage is active and to catch failures.

For this example, we estimate the ground-state energy of a hydrogen molecule. We perform the shadow tomography step with 4000 quantum circuits at 100 shots each, executed on the on-demand Braket state-vector simulator SV1. These circuits are submitted as a single batch of parallel quantum tasks. The quantum sampling step takes 10 minutes. The post-processing step evolves 1000 walkers for 600 time steps in parallel, with each Batch child job taking 20 minutes. When the Batch job completes, the Lambda function processes and returns the averaged ground-state energy in one minute, as shown in Fig. 4, where we compare the results from quantum-classical and classical AFQMC runs to validate that both converge to the exact diagonalization reference energy, confirming that the pipeline works end to end.

Figure 4 - Hydrogen-molecule ground-state energy estimation from classical and quantum-classical (shadow-based) AFQMC algorithms, both converging to the exact value over imaginary time.

Figure 4 – Hydrogen-molecule ground-state energy estimation from classical and quantum-classical (shadow-based) AFQMC algorithms, both converging to the exact value over imaginary time.

Conclusion

In this post, we showed how AWS Step Functions can orchestrate a prototypical hybrid quantum-classical workflow using the QC-AFQMC algorithm as a concrete example. By expressing the run as a Step Functions state machine over Amazon Braket Hybrid Jobs, AWS Batch, and AWS Lambda, we get a single managed workflow that handles the asynchronous quantum step, the parallel classical fan-out, and the final energy averaging; with built-in sequencing, error handling, and data hand-off. This solution can be readily adapted to other similar hybrid workflows. To get started, you can deploy the full architecture in your own AWS account from the templates at Sample Integration of Amazon Braket and AWS Batch for Hybrid Quantum-Classical Workflows. To explore the algorithm, see the QC-AFQMC algorithm example in the Amazon Braket Examples repository for details.

References

[1] Huggins, William J., et al. “Unbiasing Fermionic Quantum Monte Carlo with a Quantum Computer.” Nature 603.7901 (2022): 416-420.

[2] Xu, Xiaosi, and Ying Li. “Quantum-assisted Monte Carlo algorithms for fermions.” Quantum 7 (2023): 1072.

[3] Wan, Kianna, et al. “Matchgate shadows for fermionic quantum simulation.” Communications in Mathematical Physics 404.2 (2023): 629-700.

[4] Huang, Benchen, et al. “Evaluating a quantum-classical quantum Monte Carlo algorithm with Matchgate shadows.” Physical Review Research 6.4 (2024): 043063.

[5] Zhao, Luning, et al. “Quantum-classical auxiliary field quantum Monte Carlo with matchgate shadows on trapped ion quantum computers.” Physical Review Research 8.3 (2026): 033061.