AppFolio Leverages Datadog LLM Observability and Amazon Bedrock to Boost Adoption by 300% and Maximize Efficiency

Supporting customers in the property management industry, AppFolio helps clients manage operations through the entire resident lifecycle. This includes marketing vacancies and handling leads to exceeding the expectations of existing occupants and managing move-outs. AppFolio’s one powerful platform and centralized data structure make it easier for customers to manage all aspects of their operations such as tracking maintenance operations, recording payments, and tying expenses to corresponding accounting and financial records.

Throughout the course of their day, property managers may spend up to 50 percent of their time communicating with residents. AppFolio recognized that users wanted to reduce time spent on communications to focus more on other responsibilities and priorities. So, it set out to improve its communication features to help users be more efficient and focused on leveraging AI to achieve that goal. AppFolio wanted to integrate AI functionality while ensuring transparency and accuracy, with key guardrails in place. It also wanted to maintain reliability and incorporate responsible AI practices while deploying and scaling AI-powered applications.

To do this, AppFolio sought a solution that could identify potential issues and drive improvements during product development. They needed to ensure maintenance of high quality standards as the company expanded its services and to provide continuous usage monitoring for insights into user behavior.

AppFolio wanted its users to have an unmatched customer experience. With comprehensive observability in place, the company could make improvements to its services faster and release new capabilities to more customers with confidence. This was important for integrating new AI-driven features, due to the unpredictable nature of LLMs and the difficulty identifying operational and functional issues in LLM-based application architectures.

An AWS Advanced Technology and DevOps Competency Partner, Datadog presented a strong case for its cutting-edge LLM observability features. Datadog LLM Observability would allow AppFolio to monitor the performance and quality of its LLM-powered generative AI applications. For example, it provided deep tracing for LLM chain steps to identify bottlenecks, optimize logic, and improve LLM response accuracy—a must for AppFolio.

AppFolio had identified Amazon Bedrock as its service of choice to incorporate AI, so the company needed something that would integrate easily with it. With access to foundation models from leading AI companies through a single API, Amazon Bedrock would enable AppFolio to build, customize, and deploy generative AI applications with its data. Datadog’s LLM Observability capabilities can track usage, performance, and error rates for foundation models provided by Amazon Bedrock while ensuring response quality and data security. Therefore, AppFolio was able to use Amazon Bedrock to experiment with and evaluate various models, test different LLMs, and select the ones best suited for its use cases.

Datadog’s unified dashboards provide insight into LLM latency and token usage over time, and real-time alerts detect anomalies, such as error spikes, to ensure optimal application performance. Out of the box LLM evaluations such as Toxicity and Failure-to-answer help prevent negative user experiences and prioritize new features. With hundreds of thousands of messages going through AppFolio’s systems each day, having a solution to track, monitor, and analyze each conversation at a high quality was key.

AppFolio leveraged Amazon Bedrock to experiment with and evaluate foundation models, customize them with data, and incorporate them into its applications. By integrating with Amazon Bedrock, Datadog LLM Observability provided a comprehensive observability solution that helped AppFolio monitor, optimize, and secure its AI and machine learning (ML) applications effectively. Datadog's LLM Observability helped the AppFolio team monitor and improve the performance and response quality of its new Realm-X application. This, combined with the ability to quickly troubleshoot issues, improved operational performance and optimized the quality of its responses, leading to better user experiences.

Early in the release of Realm-X Messages, AppFolio noticed a strong correlation between latency and adoption—as the service got faster, adoption increased. Using that as a target for improvement, Datadog LLM Observability ensured latency was being monitored and used as a feedback loop to improve Realm-X Messages. This resulted in reducing AppFolio’s latency by 80 to 90 percent. The massive reduction in latency saw a corresponding increase in adoption of AppFolio’s services of nearly 300 percent. “Datadog’s solution gave us confidence as we rolled out Realm-X Messages to more customers,” added Teddy Ho, principal product manager for AppFolio. “It helped us ensure high performance and quality and allowed us to expand functionality quickly and safely.”

Feedback from AppFolio Realm-X users has been overwhelmingly positive. The product saves customers an average of five hours per week on tasks related to communication, with 75 percent of users experiencing a reduction in busywork (June 2024 survey of early Realm-X users). “Our priority is to constantly increase productivity gains for our customers,“ concluded Ho. “We want to make their jobs easier, and with Datadog’s help, we can now leverage generative AI to make that possible.”

Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models (FMs) from leading AI companies through a single API, along with a broad set of capabilities you need to build generative AI applications with security, privacy, and responsible AI.