AWS Nitro Enclaves for running Ethereum validators – Part 1

In this series of posts, we provide prescriptive guidance in secure operation of Ethereum validator keys using AWS Nitro Enclaves for node operators who provide staking pools and staking-as-a-service.

In this post (Part 1), we explain why AWS Nitro Enclaves are well suited to run Ethereum validators in a secure fashion and we provide a high-level architecture of such a Nitro Enclaves-based validator signing service.

In the code sample GitHub repository, we provide a step-by-step walkthrough of how to deploy a Nitro Enclaves-based validator signing service.

In Part 2, we conduct a technical deep dive into the Nitro Enclaves-based validator architecture.

Ethereum proof-of-stake and validators

Public blockchains differ in their protocol and consensus design, implementation language, and smart contract capabilities.

Determining how external participants can be incentivized to join a network and run their own node is key in economic design principles rooted in game theory, where you’re designing the network to be more decentralized while improving robustness.

Ethereum transitioned its consensus mechanism from proof-of-work to proof-of-stake in September 2022 to achieve better security, lower energy consumption requirements, and further enable scaling solutions built on top of it. In a proof-of-stake network, the network participants, called validators, are responsible for checking and attesting that new blocks proposed by other validators are valid, and occasionally creating and proposing (propagating) new blocks themselves. As an economic incentive, validators earn rewards in the form of Ether for performing these tasks.

To ensure that validators behave honestly, each validator must stake 32 Ether (ETH) as collateral into a smart contract. If the validator behaves dishonestly, such as attempting to propose an invalid block, their collateral will be slashed. There are also minor slashing penalties for being offline, but these are far less significant than penalties for dishonest behavior. Each validator has a public-private keypair associated with it, which we explore in more depth later in this post. For more information on proof-of-stake, refer to PROOF-OF-STAKE (POS).

There are multiple ways for users to participate in Ethereum staking:

• • Run their own validator nodes – Users run and maintain validator nodes and have 32 ETH per validator to deposit as collateral. The user has full control of the node and the validator key.
  • Use a staking-as-a-service provider – Users still provide 32 ETH per validator, but offload the operational aspects to a node provider who runs the validator nodes, and maintains the validator keys for the user.
  • Join a staking pool – Users can stake any amount of ETH. The ETH staked by the users are pooled and validator nodes are run by an entrusted third party. This option is attractive to users who don’t have 32 ETH or are not comfortable staking that amount.

In the first scenario, the user realizes all of the rewards, whereas in the other two scenarios, the providers may take a percentage of the rewards as payment for their services.

Importance of securing validator keys

Running Ethereum validator nodes is crucial for the health of the Ethereum network but poses a potential security risk. The validator keys (the public and private keys) are used to collect and withdraw the 32 ETH collateral and any rewards that have been collected. The withdrawal feature is available in Ethereum’s Shapella update. Generating and storing validator keys in an unsecure manner poses risks of leaked keys. Other than withdrawal of the collateral from malicious parties, leaked keys can also be used to perform long-range attacks, putting the entire Ethereum network at risk of a history rewrite. Long-range attacks work by creating and maintaining a parallel chain, thereby rewriting the entire history of the blockchain. To reconstruct a parallel chain, the keys of past validators are needed.

You can use services like AWS Key Management Service (AWS KMS) to mitigate some of the risks associated with the handling of private keys that need to be programmatically accessible (hot wallets or warm wallets). For additional information on how to use AWS KMS for regular Ethereum account keys, refer to How to sign Ethereum EIP-1559 transactions using AWS KMS.

Due to the BLS12-381 elliptic curve introduced for the Ethereum consensus layer, services like AWS KMS can’t be used to protect validator keys as of this writing. Therefore, there is a need for a secure run environment to use the validator keys.

The following solution is primarily aimed at node operators who provide staking pools and staking-as-a-service. Individual stakers can also adopt the solution, but might find the cost of running the solution prohibitive if they only stake the minimum amount of ETH.

Solution overview

The following diagram illustrates the architecture for our Nitro Enclaves-based validator signing service.

Nitro Enclaves use the Nitro Hypervisor to provide an isolated compute environment to protect and securely process highly sensitive data like personally identifiable information (PII) or private keys.

The hypervisor isolates the CPU and memory of the enclave from the parent instance and just exposes a secure local channel (vsock) for the communication between the parent instance and the enclave. That way, no user, application, or library of the parent instance is able to get access to critical information processed inside the enclave.

Every 12 seconds, a new block is proposed to the Ethereum network. This is referred to as a slot. Validator clients such as Lighthouse and Teku attest the proposed block. Occasionally, instead of attesting, a validator is randomly selected to propose a block. Both attestations and block proposals have to be signed by a validator private key. In our solution, the validator client has been configured to perform remote signing—in other words, offloading the signing process to Web3Signer, an open-source blockchain signing service. It does this by invoking Web3Signer’s HTTPS REST API.

Web3Signer will sign attestations and block proposals, using the plaintext validator private keys that are loaded into the enclave. The encrypted validator private keys are stored in Amazon DynamoDB. In a staking-as-a-service provider, there can be hundreds of validator private keys mapped to multiple Web3Signer instances, therefore it’s operationally simpler and more cost-efficient to use DynamoDB instead of AWS Secrets Manager. The validator private keys are encrypted using a KMS key.

In our architecture, Web3Signer runs in the Nitro Enclave. It’s important to point out that in terms of security, the Web3Signer runtime is provided by a Nitro Enclave and therefore is protected from attacks or risks like memory dumps or insufficient file system privileges. Nevertheless, by exposing Web3Signer’s HTTPS REST API via vsocket to the VPC, new potential attack vectors are being introduced associated with the application itself, which can’t be mitigated by a secure runtime environment like Nitro Enclaves. These attack vectors require a different set of measurements, like preventing public API access or advanced authentication mechanisms like client certificate based authentication.

The next section in this post provides details on how the private keys are decrypted and loaded into Web3Signer in Nitro Enclave.

To ensure high availability, the Web3Signer Amazon Elastic Compute Cloud (Amazon EC2) instances are deployed in an Auto Scaling group across multiple Availability Zones. These instances load the same validator keys from DynamoDB. A Network Load Balancer (NLB) forwards the signing request to one of the instances. Multiple validator clients can connect to the same NLB.

EC2 instances hosting Nitro Enclaves should not be publicly accessible and therefore are located in private subnets. EC2 instances have secure outbound internet access via a NAT gateway to download operating system updates and other packages. In a production environment, updates and other dependencies can be provided by custom Amazon Machine Images (AMIs). That way, no outbound internet connectivity is required.

Communication with AWS services like AWS KMS or DynamoDB should not traverse the internet and therefore is kept private via interface VPC endpoints that have been placed inside the subnets in each Availability Zone.

Also, the provided AWS Cloud Development Kit (AWS CDK) source just covers the area in the architecture diagram marked in red. Validator clients and beacon-chain or consensus layer clients have to be deployed separately. The high-availability design and slashing protection for validator and beacon-chain clients are outside the scope of this series of posts.

Bootstrapping and signing flow

The following diagram depicts the components deployed inside the EC2 instances and the general communication flow with external AWS services like DynamoDB and AWS KMS.

In general, there are two prerequisites:

• One or more Ethereum validator keys have been encrypted and stored inside DynamoDB. To securely generate and encrypt Ethereum validator keys, refer to Securely generate Ethereum validator keys at low cost using a serverless architecture on AWS. For testing purposes, a similar functionality is provided in the walkthrough section of the accompanying AWS CDK code.
• A TLS key-pair for Web3Signer has been encrypted and stored inside DynamoDB. Instructions on how to generate the TLS keypair are provided in the README section of the accompanying AWS CDK code.

The following steps explain at a high level how Ethereum attestations and block proposals are signed inside Nitro Enclaves. The numbers correspond with the numbers shown in the preceding diagram.

1. If all config artifacts are present, the enclave can start.
2. All artifacts are passed into the enclave in encrypted fashion.
3. kmstool-enclave-cli, a tool provided by the Nitro Enclave service team, is used from inside the enclave to decrypt the config artifacts using cryptographic attestation.
4. The decrypt request augmented with the cryptographic attestation document is routed to AWS KMS via vsock-proxy provided by AWS.
5. All decrypted config artifacts are stored on the enclaves in memory file system.
6. The Web3Signer process starts. After the enclave has successfully been bootstrapped with the provided TLS keypair and the Ethereum 2 private key, https_proxy will act as an HTTPS endpoint for Web3Signer running inside the enclave.
7. The Web3Signer API endpoint is now exposed to outside of the EC2 instance and able to process incoming signing requests, for example, originating from an Ethereum 2 validator node.
8. Signing requests are forwarded inside the enclave via the vsock connection, in an encrypted way. The TLS connection between the validation client and Web3Signer is stopped inside the enclave, resulting in an end-to-end encrypted transport.

Additional details about the HTTPS over vsocks tunnel and the bootstrapping process can be found in Part 2 of this series.

In terms of configuration, you shouldn’t configure more than one validator client to the same validator keys. Doing so poses a high risk of signing attestations for different block proposals, or proposing different blocks in the same slot. These activities will lead to slashing and result in losing some of the validator’s staked ETH. Web3Signer offers a slashing protection feature to protect against these scenarios. To simplify the deployment, Web3Signer in the accompanying AWS CDK code doesn’t have slashing protection. However, this could be included in future revisions of the code.

Conclusion

In this post, we provided a high-level introduction to Nitro Enclaves and explained why Nitro is well suited to run Ethereum validators. We also explained the high-level bootstrapping and signing process using Nitro Enclave.

You can deploy the solution by following the instructions in the end-to-end walkthrough located in the GitHub repository. You can also dive deep into the Web3Signer integration pattern, secure bootstrapping of processes inside Nitro Enclaves, and HTTPS over vsock tunnels in Part 2.

About the Authors

David-Paul Dornseifer is a Blockchain Architect with the AWS Worldwide Specialist Solutions Architect organization. He focuses on helping customers design, deploy, and scale end-to-end blockchain solutions.

Aldred Halim is a Solutions Architect with the AWS Worldwide Specialist Solutions Architect organization. He works closely with customers in designing architectures and building components to ensure success in running blockchain workloads on AWS.