Advanced data modeling: Using user-defined types and Protocol Buffers for Amazon Keyspaces

Amazon Keyspaces (for Apache Cassandra) offers a fully managed, serverless database service that alleviates the operational overhead of managing Cassandra clusters while maintaining compatibility with the CQL API and common drivers.

When working with complex data in Amazon Keyspaces, organizing related attributes into logical groups improves data model clarity and application performance. Amazon Keyspaces supports two approaches for modeling complex data:

User-defined types (UDTs) – Native Cassandra data structures that group related fields into reusable custom types.
Protocol Buffers (Protobuf) – A language-neutral, platform-neutral serialization format for structured data.

In this post, we show how to create and manage UDTs in Amazon Keyspaces, implement Protobuf serialization for flexible data modeling, and choose between UDTs and Protobuf based on your application needs. Both UDTs and Protobuf help you model complex data efficiently, but they have different strengths and use cases, which we explore throughout this post. These two approaches can help you build more maintainable and efficient applications with structured data while using the benefits of a serverless database service.

Solution overview

For this post, we use a real estate data model to demonstrate using UDTs and Protobuf because it naturally contains nested data structures. Property listings include multiple related components, such as physical characteristics, location details, and listing information.

This example models a property management system for a fictional real estate company that needs to track thousands of properties across multiple markets. The data model includes basic property information (ID, multiple listing service [MLS] number), address details (street, city, state), location quality metrics (school ratings, walkability scores), and listing details (agent information, dates, status).

This approach helps the company maintain consistent data structures while supporting efficient queries for their most common operations. The real estate domain provides a realistic scenario in which both the structure and relationships between data elements are complex enough to demonstrate the benefits of advanced data modeling techniques.

Prerequisites

Before getting started, you must have:

An AWS account with permissions to create and manage Amazon Keyspaces resources.
AWS Identity and Access Management (IAM) roles with AmazonKeyspacesFullAccess permissions.
Knowledge of Apache Cassandra Query Language (CQL).
Java Development Kit (JDK) 8 or later for Protobuf examples.
Protocol Buffers compiler (protoc) version 3.0 or later.
The AWS Command Line Interface (AWS CLI) version 2.0 or later (optional).

Using UDTs with Amazon Keyspaces

In traditional table design, modeling complex entities such as addresses or contact information often requires multiple columns or separate tables. With UDTs, you can encapsulate these related fields into a single logical type that can be reused across multiple tables. This encapsulation improves schema clarity and simplifies application logic. You can use UDTs across tables in the same keyspace, simplifying application development and common business rules. The schema is validated on write, so applications can’t persist invalid data.

Before designing your schema around UDTs, note the following service quotas. Amazon Keyspaces supports a maximum of 256 UDTs per AWS Region and 50 UDTs per table. UDTs support up to eight levels of nesting, a limit that becomes relevant quickly when combining UDTs with nested collections. UDT names are capped at 48 characters and the total schema size can’t exceed 25 KB. Applications that require more than eight nesting levels or that anticipate frequent structural changes are better served by Protobuf. Protobuf has no server-enforced nesting ceiling and supports schema evolution without coordination on the server side.

When using UDTs in Amazon Keyspaces, the frozen keyword is required when a UDT is nested inside another UDT, or when a UDT is used inside a collection such as a list, set, or map. A top-level UDT column in a table does not require frozen. This keyword indicates that the UDT is treated as a serialized blob in storage. The implication is that the entire UDT must be read or written as a single unit. In standard Cassandra, you can’t update individual fields within a frozen UDT because the entire structure must be replaced.

UDTs are recommended when you need server-side schema validation to provide data consistency across your application. They work best when your data has a stable structure with well-defined fields that don’t change frequently. UDTs are particularly useful when you want to query individual fields within the complex type, which allows for more targeted data retrieval. They also provide significant benefits when you need to reuse the same structure across multiple tables, promoting consistency in your data model and reducing duplication in your schema definitions.

Amazon Keyspaces doesn’t support ALTER TYPE. After a UDT is created, its fields cannot be added, removed, or modified. Design your UDT schemas to be stable before deploying them. If your application must evolve the structure of a complex type over time, Protobuf is the more appropriate choice. For example, you might need to add new fields without redeploying schema changes. Protobuf supports this because schema evolution is managed entirely in the application layer.

An example UDT

Imagine you want to store customer address information. Instead of having separate street, city, state, and zip columns, you can define an address UDT. You can use this across multiple entities, such as customer, business, or location. The following code shows an example of an address_details UDT:

create type address_details (
    street_number text,
    street_name text,
    unit_number text,
    city text,
    state text,
    zip_code text,
    county text,
    country text,
    latitude decimal,
    longitude decimal,
    time_zone text,
    neighborhood text,
    subdivision text
);

create type listing_details (
    listing_agent_name text,
    listing_agent_phone text,
    listing_agent_email text,
    listing_brokerage text,
    listing_date date,
    listing_status text,
    listing_type text,
    showing_instructions text,
    commission_rate decimal,
    buyer_agent_commission decimal,
    listing_remarks text,
    private_remarks text,
    virtual_tour_url text,
    video_tour_url text,
    floor_plan_url text
);

This approach makes it straightforward to store and retrieve an entire address as a single unit, improving both data organization and query efficiency.

UDTs with nested collections

You can also nest or combine UDTs in Amazon Keyspaces with collections such as lists, sets, and maps. For instance, you might define a location_quality UDT that includes school_ratings, nearby_amenities, and walkable_destinations, and then embed the UDT inside a properties table:

CREATE TYPE IF NOT EXISTS location_quality (
    school_district TEXT,
    elementary_school TEXT,
    middle_school TEXT,
    high_school TEXT,
    school_ratings MAP<TEXT, INT>,
    crime_index INT,
    walkability_score INT,
    transit_score INT,
    bike_score INT,
    noise_level TEXT,
    air_quality_index INT,
    flood_zone TEXT,
    earthquake_zone TEXT,
    hurricane_zone TEXT,
    nearby_amenities MAP<TEXT, DECIMAL>,
    commute_times MAP<TEXT, INT>,
    walkable_destinations LIST<TEXT>
);

You can then use this in your table as shown in the following properties table example:

CREATE TABLE IF NOT EXISTS properties (
    property_id UUID PRIMARY KEY,
    mls_number TEXT,
    address frozen<address_details>,
    listing frozen<listing_details>,
    location frozen<location_quality>,
    created_at TIMESTAMP,
    updated_at TIMESTAMP,
    data_source TEXT,
    data_quality_score INT
);

This design supports properties with multiple location attributes such as commute time, walkable destination, and school rating, all within a single table and using strongly typed schema elements.

Query and modification

With Amazon Keyspaces, you can work with UDTs in flexible ways. The following subsections demonstrate how to query and modify UDT data.

Select a UDT column

You can query a UDT column:

Select address from properties
where property_id = 33333333-3333-3333-3333-333333333333;

This query returns the specified address UDT column for the specified row.

Insert a UDT value

When inserting UDT values, use CQL UDT literal syntax: a brace-delimited set of field name and value pairs. The following example demonstrates inserting a property with multiple UDT fields:

INSERT INTO properties (
    property_id,
    mls_number,
    address,
    location,
    listing,
    created_at,
    updated_at,
    data_source,
    data_quality_score
) VALUES (
    11111111-1111-1111-1111-111111111111,
    'MLS2024001',
    {
        street_number: '1234',
        street_name: 'Bellevue Avenue',
        city: 'Seattle',
        state: 'WA',
        zip_code: '98102',
        county: 'King County',
        country: 'USA',
        latitude: 47.6205,
        longitude: -122.3212,
        time_zone: 'PST',
        neighborhood: 'Capitol Hill',
        subdivision: 'Bellevue Heights'
    },
    {
        school_district: 'Seattle Public Schools',
        elementary_school: 'Stevens Elementary',
        middle_school: 'Meany Middle School',
        high_school: 'Garfield High School',
        school_ratings: {'Stevens Elementary': 8, 'Meany Middle School': 7, 'Garfield High School': 9},
        crime_index: 25,
        walkability_score: 88,
        transit_score: 75,
        bike_score: 82,
        noise_level: 'moderate',
        air_quality_index: 45,
        flood_zone: 'X',
        earthquake_zone: 'moderate',
        hurricane_zone: 'none',
        nearby_amenities: {'grocery_store': 0.3, 'restaurant': 0.1, 'park': 0.2, 'hospital': 1.2},
        commute_times: {'downtown_seattle': 15, 'bellevue': 25, 'airport': 35},
        walkable_destinations: ['coffee_shops', 'restaurants', 'retail', 'parks']
    },
    {
        listing_agent_name: ' Jane Doe ',
        listing_agent_phone: '206-555-0123',
        listing_agent_email: 'jane.doe@anycompany.com',
        listing_brokerage: 'Any Company',
        listing_date: '2024-06-01',
        listing_status: 'active',
        listing_type: 'exclusive',
        showing_instructions: 'Call listing agent 24 hours in advance',
        commission_rate: 5.5,
        buyer_agent_commission: 2.75,
        listing_remarks: 'Stunning contemporary home with panoramic city views. Recently renovated with high-end finishes throughout.',
        private_remarks: 'Motivated seller, will consider reasonable offers'
    },
    '2024-06-01 10:00:00',
    '2024-07-15 14:30:00',
    'mls',
    95
);

The CQL UDT literal syntax provides a readable representation of the nested data structure.

Update a frozen UDT

When updating a frozen UDT column, you must provide the entire UDT structure. The following example shows how to update an address:

update properties
set address = {
    street_number: '500',
    street_name: 'Pine Street',
    unit_number: '2801',
    city: 'Seattle',
    state: 'WA',
    zip_code: '98101',
    county: 'King County',
    country: 'USA',
    latitude: 47.6097,
    longitude: -122.3331,
    time_zone: 'PST',
    neighborhood: 'Downtown',
    subdivision: 'Metropolitan Tower'
}
where property_id = 33333333-3333-3333-3333-333333333333

When updating a frozen UDT column in Amazon Keyspaces, you must provide the entire UDT structure in your UPDATE statement. The new value replaces the existing one as a whole, and any fields omitted from the UDT literal are set to null rather than preserved from the previous row.

Lightweight transactions

Amazon Keyspaces supports lightweight transactions (LWTs), so you can perform conditional inserts and updates by using the IF clause. You can use UDTs within LWTs to make sure conditional logic applies to complex data types. The following code example demonstrates a conditional update based on a UDT field value:

update properties
set address = {
    street_number: '400',
    street_name: 'Pine Street',
    unit_number: '2801',
    city: 'Seattle',
    state: 'WA',
    zip_code: '98101',
    county: 'King County',
    country: 'USA',
    latitude: 47.6097,
    longitude: -122.3331,
    time_zone: 'PST',
    neighborhood: 'Downtown',
    subdivision: 'Metropolitan Tower'
}
where property_id = 33333333-3333-3333-3333-333333333333
if address.street_number = '500';

This code makes sure the update is applied only if the existing street number matches the specified value. LWTs provide strong consistency for critical updates involving UDTs and are particularly useful in applications that require compare and set operations.

Using Protocol Buffers with Amazon Keyspaces

As an alternative to using UDTs, you can use Protocol Buffers (Protobuf) in Amazon Keyspaces to store structured data in CQL BLOB fields. This approach helps you serialize complex objects into a compact binary format and store them in Amazon Keyspaces. Protobuf also preserves schema flexibility and cross-language compatibility.

Translate UDTs to Protobuf

To illustrate how to use Protobuf as an alternative to UDTs, we translate the following ContactInfo and customers UDT-based schema into Protobuf format. The ContactInfo UDT includes fields for phone, email, and a map of social media handles. The customers table references a list of contact_info entries and a list of addresses. The following code shows how you can represent these UDTs in Protobuf format:

message ContactInfo {
    string phone = 1;
    string email = 2;
    map<string, string> social_handles = 3;
}

message Address {
    string street = 1;
    string city = 2;
    string state = 3;
    string zip = 4;
}

message Customer {
    string customer_id = 1;
    repeated ContactInfo contacts = 2;
    repeated Address addresses = 3;
}

You can serialize these data types into binary format and store them in Amazon Keyspaces as BLOBs, providing a flexible alternative to UDTs.

Store Protobuf data in Amazon Keyspaces

To store Protobuf data in Amazon Keyspaces, create a table with a BLOB column:

CREATE TABLE IF NOT EXISTS customers_blob (
    customer_id text PRIMARY KEY,
    data blob
);

This approach decouples your application schema from the database schema, making it ideal for systems that require schema evolution and multilanguage support.

Insert and select Protobuf data in Java

After defining and compiling your schema into Java classes, you can use the generated classes to serialize and deserialize data. The following examples show how to work with Protobuf in Java:

Customer customer = Customer.newBuilder()
        .setCustomerId("cust_456")
        .addContacts(ContactInfo.newBuilder()
                .setPhone("555-1234")
                .setEmail("user@example.com")
                .putSocialHandles("twitter", "@user")
                .build())
        .addAddresses(Address.newBuilder()
                .setStreet("123 Main St")
                .setCity("Seattle")
                .setState("WA")
                .setZip("98101")
                .build())
        .build();

ByteBuffer protobufData = ByteBuffer.wrap(customer.toByteArray());

PreparedStatement insertStmt = session.prepare("INSERT INTO customers_blob (customer_id, data) VALUES (?, ?)");

session.execute(insertStmt.bind("cust_456", protobufData));

This code creates a Customer object with nested ContactInfo and Address objects, serializes it to binary format, and inserts it into the database.

Select and decode Protobuf data

The following code shows how to select and decode Protobuf data:

PreparedStatement selectStmt = session.prepare("SELECT data FROM customers_blob WHERE customer_id = ?");

ResultSet resultSet = session.execute(selectStmt.bind("cust_456"));

Row row = resultSet.one();

if (row != null) {
    ByteBuffer dataBuffer = row.getByteBuffer("data");
    try {
        Customer decodedCustomer = Customer.parseFrom(dataBuffer.array());
        System.out.println("Name: " + decodedCustomer.getCustomerId());
    } catch (InvalidProtocolBufferException e) {
        System.err.println("Failed to parse Protocol Buffer data: " + e.getMessage());
    }
}

This code retrieves the binary data from the database and deserializes it back into a Customer object. The try-catch block handles potential parsing errors that might occur if the data is corrupted or incompatible with the current schema.

Advantages and disadvantages of using Protocol Buffers over UDTs in Amazon Keyspaces

Although UDTs are natively supported and tightly integrated into the CQL data model, Protocol Buffers offer several advantages that might make them the preferred approach for certain workloads:

Binary portability – Protocol Buffers (Protobuf) are stored as blobs, so you can take Protobuf data across multiple different data stores. This is different from UDTs, which can be used only in Cassandra-compatible stores.
Schema evolution – Protobuf schemas are designed for forward and backward compatibility. You can add or remove fields in the application layer without coordinating on the server side. This helps application teams work in a more decentralized way.
Compact serialization – Protobuf binary encoding is space efficient and doesn’t store default values, which can reduce storage and transfer overhead.

However, Protobuf has the following limitations in relation to UDTs:

Protobuf stores data as an opaque blob. To change any field, the application must read the entire object, modify it in memory, and write the full blob back. Lightweight transactions reduce race conditions during this cycle but do not eliminate the full-object read requirement. UDTs, by contrast, are defined in the Amazon Keyspaces schema with explicit field names and data types, making the structure visible and enforced at the database level.
Protobuf also provides no server-side schema validation. Amazon Keyspaces has no knowledge of the internal structure of a Protobuf blob. As a result, any application can write malformed or structurally incorrect data into a Protobuf column without the database raising an error. All validation must be built and enforced on the client side. With UDTs, Amazon Keyspaces validates every write against the registered schema and rejects data that does not conform to the defined field types.

Clean up

To avoid ongoing charges for resources created in this post, delete the resources when they’re no longer needed. The following steps show how to clean up the resources:

Delete the tables created in the examples:

DROP TABLE IF EXISTS properties;
DROP TABLE IF EXISTS customers_blob;

Delete the UDTs:

DROP TYPE IF EXISTS location_quality;
DROP TYPE IF EXISTS listing_details;
DROP TYPE IF EXISTS address_details;

Using the AWS CLI, you can also delete resources:

aws keyspaces delete-table --keyspace-name your_keyspace --table-name properties
aws keyspaces delete-table --keyspace-name your_keyspace --table-name customers_blob

Conclusion

Amazon Keyspaces provides options for modeling complex data through both UDTs and Protobuf. UDTs offer native CQL integration with field-level querying capabilities, but Protobuf provides compact storage and built-in schema evolution support. By understanding the strengths and limitations of each approach, you can choose the right solution for your specific application needs while benefiting from the fully managed, serverless architecture of Amazon Keyspaces.

Try implementing these patterns with your own data models, and explore the complete example code on GitHub. For more information, see the Amazon Keyspaces Developer Guide and AWS Database Blog.

AWS Database Blog