How PingCAP transformed TiDB into a serverless DBaaS using Amazon S3 and Amazon EBS

PingCAP, an AWS Partner Network (APN) Partner, is the company behind TiDB, an advanced open-source, distributed SQL database for building modern applications. TiDB is widely used and trusted by technologists around the world.

In July 2023, PingCAP released TiDB Serverless, a fully managed, autonomous DBaaS offering of TiDB. However, based on TiDB’s existing architecture, PingCAP needed to address major challenges when building this new product, including achieving pay-as-you-go billing, reliable data durability, and no performance degradations relative to the original design.

In this post, we explore how PingCAP solved these challenges by using Amazon Elastic Block Store (Amazon EBS) and Amazon Simple Storage Service (Amazon S3) to create a multi-tiered storage architecture for low-latency performance, virtually unlimited scalability, and relentless data reliability. You will learn how Amazon EBS and Amazon S3 helped PingCAP optimize its serverless architecture for PingCap customers to achieve high reliability of applications with strong fault tolerance, disaster recovery capabilities, and fast scale-out and scale-in without compromising the stability of existing workloads. We demonstrate how Amazon EBS can function as a supplementary extension to local storage in serverless database architecture, with the final data storage residing in Amazon S3.

Designing TiDB Serverless

To create TiDB Serverless, we first need to examine the design of TiDB and the limitations that we must address in the existing TiDB architecture.

TiDB architecture overview

TiDB’s architecture contains four main components, as illustrated in the following diagram:

1. TiDB-Server: The stateless computing layer
2. TiKV: The row-oriented storage layer
3. TiFlash: The column-oriented storage layer that uses a Massively Parallel Processing (MPP) computing engine to generate real-time analytics on operational data
4. Placement Driver (PD): A metadata service and global transaction timestamp allocator

TiDB’s architecture separates compute from storage, allowing each component to scale independently. This provides the scalability and flexibility needed for a serverless architecture. For example, computing nodes can quickly be added from a resource pool to handle increased user workloads. However, when compared to scaling compute, scaling storage presents additional technical challenges.

As a distributed database, each TiDB storage node stores a portion of the global data in different shards. This means that when expanding storage resources, TiDB has to transfer data from existing nodes to the new ones. This process can be quite resource-demanding, as data transfer and replication consume a significant amount of CPU and I/O resources. To prevent any business repercussions, the actual production expansion requires a significant amount of pre-planning. This is because it cannot be done at full speed to avoid interfering with the application workload. This design can’t automatically handle unpredictable scaling requirements in a serverless environment.

As a result, achieving fast scalability in TiDB’s storage layer became a challenging design consideration when creating TiDB Serverless.

Meeting cost-efficiency requirements

On-demand billing is a hallmark of serverless offerings. Traditional databases are more often billed based on pre-allocated resources, requiring users to provision resources carefully. In contrast, serverless databases abstract away the hardware layer underneath, relieving users from managing fluctuations in user workloads.

An initial idea was to always reserve sufficient resources for each user. However, the idea was later deemed unfeasible because it is difficult to predict business growth and foresee how much of the resources would be sufficient. Moreover, if this approach were implemented, cost efficiency would be severely compromised. In the end, either the database vendor or its users would have to bear the additional cost.

A critical design expectation is that the database must be able to respond quickly to workload changes, rapidly scaling in or out to adapt to actual user usage patterns. This is relatively easier for stateless services. However, it becomes challenging for databases like TiDB, which are stateful and require data consistency.

Fulfilling data integrity and performance needs

Data integrity is an essential design requirement for any database widely used in online systems. Due to the high IOPS demands of online systems, data persistence and performance are two critical factors when selecting different cloud storage products.

Finally, an important implicit assumption with this new architecture is that there will be no performance degradation as compared to the old design. This imposes an additional constraint on the new design.

Amazon S3 and Amazon EBS: Two crucial ingredients

To address these issues, Amazon S3 and Amazon EBS became the key storage choices for their scalability and performance characteristics. Together, they form two critical components of the new serverless architecture, providing rapid scalability, excellent performance, and data durability.

Amazon S3 for enhanced data scalability

To achieve better scalability, PingCAP needed shared storage as an intermediary between working nodes to avoid performance interference caused by transferred data.

Amazon S3 is an object storage service that offers industry-leading scalability, data availability, security, and durability. It provides users with cost efficiency, making it ideal for optimizing the cost of a serverless architecture. Based on this, PingCAP first examined a storage layer design centered around Amazon S3: writing the storage layers of TiKV (TiDB’s key-value distributed storage database) and TiFlash (TiDB’s columnar analytics engine) directly to Amazon S3 and using it as the shared storage medium. This design allows for the easy launching of new nodes and connecting them to Amazon S3 when scaling out.

Amazon EBS for low-latency performance

At this point, PingCAP made an improvement to the design and introduced Amazon S3 as a separate storage medium layer. In the new design, TiDB stores most of its data on Amazon S3 while putting Write-Ahead Logs (WALs) on EC2 instance store which provides the lowest latency. However, PingCAP quickly ran into another challenge of addressing the need for data recovery when storage nodes were disrupted. TiKV uses the Raft consensus protocol, which can tolerate the loss of minority nodes. For example, in a three-replica scenario, TiDB can tolerate the loss of one replica. The database also intentionally distributes replicas across different Availability Zones (AZs) to enhance availability and durability. However, in extreme cases, such as the loss of an entire AZ followed by failures in the remaining nodes, it’s essential to have a comprehensive disaster recovery process in place due to the nature of Raft algorithms.

The idea is to have persistent SSD storage that is decoupled from compute infrastructure and has a low-latency performance profile. PingCAP saw that if two or more storage nodes were disrupted for any reason, the new serverless architecture could quickly reattach data volumes and resume production using Amazon EBS.

Amazon EBS is an easy-to-use and scalable high-performance block storage service designed for Amazon Elastic Compute Cloud (Amazon EC2). In previous tests, the team found that Amazon EBS volumes offered more stable performance in scenarios with high write traffic. PingCAP found EBS to be an effective complement to S3 by storing important latency-sensitive information, including metadata and write-ahead logs. It provides a high level of durability wherever the TiDB storage engine is deployed.

TiDB Serverless storage architecture

With the help of Amazon S3 and Amazon EBS, PingCAP was able to redesign the storage architecture, as illustrated in the following diagram:

In this new serverless storage architecture, TiDB Serverless uses Amazon EBS as a supplementary extension to local storage, with the final data storage residing in Amazon S3. This multi-tiered storage design complements each aspect as follows:

1. To offer better latency when accessing data, TiDB Serverless caches the actual data files (excluding WAL and metadata) on Amazon EC2 instance store for frequent access.
2. By storing latency-sensitive and critical information such as metadata and WALs on Amazon EBS, it offers excellent performance for the read and write operations of each transaction. It also provides persistence against instance failure.
3. After utilizing Amazon S3 as the primary layer for data persistence, PingCap benefits from S3’s 11 9s of durability and virtually unlimited scalability and read-write bandwidth.

To further explain data workflows in the new storage architecture, you can see in the following diagram that (1) when TiDB writes data, (2) it first writes the WAL and Raft Log to Amazon EBS on the node where the leader replica resides, (3) and then it propagates the Raft log to the remaining follower replicas. (4) After cacheing data to EC2 instance store, TiDB asynchronously writes the changes to Amazon S3. (5) Finally, TiDB sends a data loading signal to the other two follower replicas, and upon receiving the signal, the follower replicas load the data from Amazon S3 to complete multi-replica replication.

The benefits brought to TiDB Serverless

In this new architecture, TiDB Serverless uses different Amazon storage products for different I/O behaviors:

• Amazon EBS handles small but critical logs and metadata that can’t be lost.
• Amazon S3 stores large volumes of actual user data such as first name, last name, and address.
• Amazon EC2 instance store temporarily caches Amazon S3 data to provide fast read/write access performance.

Fast scaling

When scaling out, TiDB Serverless no longer relies on each storage node to write data to the new node. Instead, the DBaaS uses Amazon S3 for data loading. The virtually unlimited throughput of Amazon S3 allows us to scale quickly. Through actual testing, this DBaaS has significantly increased scalability by an order of magnitude. This makes it easy to achieve the fast scaling required by serverless architecture. TiDB Serverless can effortlessly add or remove nodes based on the real-time monitoring of user resource usage without affecting the stability of existing workloads.

Relentless reliability

Simultaneously, with the capabilities provided by Amazon EBS, PingCAP can maintain the capabilities of the existing architecture of TiDB. Moreover, the new architecture has stronger fault tolerance when compared to the local disk-based, on-premises deployment. When a node fails, TiDB Serverless can start a new node, reconnect the original EBS volume, and recover data from Amazon S3. Even if the majority of replicas fail, this DBaaS can easily recover data and continue production. This significantly reduces the chances of a painful disaster recovery scenario.

High-performance architecture

Finally, by adding Amazon S3, PingCAP has evolved the design of TiDB Serverless to be a fully shared storage architecture. This DBaaS can now access shared storage through additional Amazon EC2 nodes and offload heavy database operations such as Analyze Table, Compaction, and DDL. In a traditional database architecture, these jobs compete with actual user workloads for compute resources. Therefore, when performing tasks like analyzing large tables or adding indexes, users must carefully plan for monitoring resource utilization to avoid impacts to their business. Thanks to the new architecture of TiDB Serverless, this problem is resolved due to shared storage with Amazon S3. Looking ahead, PingCAP aims to leverage this shared storage architecture to optimize user data storage with AI in background jobs, making the database even more intelligent. These capabilities can’t be achieved with TiDB’s original architecture.

Conclusion

In this post, we discussed how PingCap leveraged Amazon EBS’s low-latency, high-performance characteristics on top of Amazon S3’s industry-leading scalability, durability, and cost efficiency for the design of its new DBaaS offering, TiDB Serverless. By using both Amazon S3 and Amazon EBS in TiDB Serverless’s architecture, PingCAP is able to build a highly cost-effective, performant, and elastic database service that generates unique benefits for its customers, including the convenience of a fully managed service and the flexibility of a pay-as-you-go model. If you run into similar challenges for performance and scalability needs when building a new database product or service, the combination of EBS and S3 can be a robust storage solution for you.

Currently, TiDB Serverless is generally available on AWS with over tens of thousands of clusters. It’s suitable for deploying small to medium-sized transactional/HTAP workloads, and it can also be used as a database for general development and testing environments. You can get started with TiDB Serverless today by spinning up a cluster instantly for free. If you are coming to AWS re:Invent 2023, meet an PingCap expert at booth 1032.