Achieving 30-Millisecond Search Results Using Amazon OpenSearch Service with CoStar Group

Founded in 1987, CoStar conducts expansive, ongoing research to produce and maintain a comprehensive database of real estate information. It provides commercial real estate information, analytics, and news to professionals around the world. In 2014, CoStar acquired Apartments.com as a complement to its commercial real estate offerings. The company further expanded into residential real estate in 2021 with the acquisition of Homes.com.

CoStar’s commercial real estate data systems were powered by an on-premises data center with 6,000 servers; however, Homes.com needed to support more listings with increased data sources and faster load times than the on-premises solution could handle. For commercial real estate data, CoStar needed to support millions of properties and tens of thousands of users. Meanwhile, with residential data, CoStar needed to scale to support hundreds of millions of properties with millions of daily users. In August 2021, the company chose AWS because of the scalability, performance, and ease of use that CoStar found through comprehensive testing. CoStar launched Homes.com with its modernized data plane 1 year later, in August 2022.

CoStar uses the data plane, a system of AWS native services, to unify data and provide data as a service to its developers. CoStar uses Amazon Elastic Kubernetes Service (Amazon EKS)—a managed service to start, run, and scale Kubernetes on AWS and in on-premises data centers—for the company’s containerized applications to build a model topic and put data into a database type that is most relevant to the developers’ needs. To provide the most up-to-date data for developers, CoStar actively tracks and updates model topics that contain the data for the model using data replication chains. Developers query and consume domain data like listings, agents, and properties using representational state transfer (RESTful) interfaces. By using RESTful interfaces rather than manually checking multiple services for data updates, developers have faster access to data, which facilitates innovation and accelerates the company’s product delivery.

For Homes.com, speed and scalability are important, given the volume of listings and concurrent customer queries. The data plane needs to stay current with updates from the residential multiple-listing service systems, data provided by users of Homes.com, and CoStar’s own internal analysts’ data. When data enters the data plane and is in a model topic search, every millisecond counts because Homes.com users expect to see the most up-to-date information.

In most use cases, CoStar stores model topic data using Amazon DynamoDB, a fully managed, serverless, key-value NoSQL database service for single-digit millisecond performance at virtually any scale. “Compute is expensive, but storage is not. By keeping objects cached in services like Amazon DynamoDB, we can avoid actively building the object and processing the data each time when new data requests come in,” says Andy Ventura, vice president of software development engineering at CoStar. CoStar stores model topic data that doesn’t change frequently or doesn’t need rapid access using Amazon Simple Storage Service (Amazon S3), an object storage service offering industry-leading scalability, data availability, security, and performance. With this infrastructure, the company updates listings in 4 seconds on average, which is important for price and availability changes. It’s significantly faster than the on-premises solution, which took 6–30 minutes to update real estate listings. “When our competitors advertised that they could update listings in minutes, we challenged ourselves to do it in seconds,” says Ventura. “We’ve achieved that using AWS.”

To provide search results across multiple fields—such as number of bedrooms and bathrooms, price, and location—CoStar uses Amazon OpenSearch Service to securely unlock near-real-time search, monitoring, and analysis of business and operational data. Even with up to 50 million monthly active users on Homes.com, CoStar can return search results in an average of 30 milliseconds using Amazon OpenSearch Service. “In terms of search engine optimization, every millisecond counts,” says Ventura. Prior to using AWS, CoStar would have needed to build a system to handle peak traffic. Using the elastic nature of AWS services, the data plane can scale as needed, depending on the number of users accessing Homes.com or the number of data elements being updated in model topics.

CoStar also saves on infrastructure and developer costs by using managed AWS services. Instead of maintaining additional staff or acquiring the necessary equipment, CoStar reduces the complexity of its systems and gets data to applications that need it in near real time. The company estimates that an on-premises solution to build the data plane would have started at four times the cost. “We can’t do the types of configurations that we need to do on premises, and we would have needed more employees to manage the servers and racks,” says Ventura. “It was a no-brainer for us to use AWS.”