Pioneering Efficiencies in Crop Insurance Underwriting: An AI-Driven Solution using Amazon Bedrock and Amazon OpenSearch Service

By Claude Heidgerken, UX/Web Manager — Farmers Mutual Hail Insurance Company of Iowa
     Matt Vincent, Founder — Source Allies
     Grey Lovelace, Senior Software Consultant and GenAI Specialist — Source Allies
     Carla Lienhard, Strategic Enablement and Transformation Director — Source Allies
     Qiong Zhang, Sr. Partner Solutions Architect — AWS

In this post, Source Allies presents a generative AI solution for crop underwriting. The solution implements a custom Retrieval Augmented Generation (RAG) system using Amazon Bedrock and Amazon OpenSearch Service. They also showcase the decision-making process that led to success when implementing a generative AI application in a highly risk-averse industry.

Farmers Mutual Hail (FMH), a top 10 crop insurance company, needs the capability to find applicable regulations during the underwriting process. Continually changing regulations makes it challenging for FMH to diversify product offerings as new regulations are added to the landscape.

Source Allies worked with FMH to create a task-specific crop underwriting solution. This custom solution provides underwriters with fast and accurate information using RAG. RAG is the process of optimizing the output of a large language model (LLM) by augmenting prompts with additional contextual information.

RAG evaluation played an essential role for the successful production and implementation of this solution. Source Allies’s RAG evaluation enables a Metrics-Driven Development process that measures the real-time changes of the accuracy of RAG. With this process, developers can determine which customization techniques can further optimize their solution in a feedback loop.

Challenges — the case for generative AI

Many companies are eager to start implementing generative AI solutions, but not all problems are meant to be solved with the same tool set. After fact-finding sessions between Source Allies and FMH, they determined that crop underwriting regulations were a great fit due to a few key reasons.

• There exists “source of truth” data that is readily available.
• That data is primarily stored in an unstructured format (PDFs).
• The existing method to access this information has a high barrier to entry for new underwriters, requiring both long wait times and in-depth crop underwriting knowledge to find information.

Customer requirements — measuring success and determining build vs buy

Defining success was paramount in understanding how to deliver a project quickly to production, while ensuring scalability and reliability. In this case, the focus was on speed and the ability to answer questions as accurately as subject-matter experts (SMEs), effectively creating new SMEs in a fraction of the time needed before. Thus, Source Allies worked with existing SMEs to create questions and “ground truth” answers to serve as the definition of success. These questions and SME responses fed heavily into the Metrics-Driven Development process outlined later.

As more companies adopt turn-key generative AI solutions, it is crucial to determine when to rely on these off-the-shelf solutions, versus building a custom solution. Source Allies decided that a custom solution better fit the data of this use case and can create true differentiation. A few key principles were used when determining the architecture.

• Give high visibility into sources of answers.
• Use event-driven architecture to ensure high scalability.
• Use infrastructure as code to streamline the replication of future projects.
• Avoid model training if a RAG solution meets the success metric, as this can reduce costs.

Solution architecture

There are three major workflows in the solution.

• Document Ingestion — Listen for uploads and document lifecycle events in Amazon Simple Storage Service (Amazon S3), then prepare the documents and create embeddings for them using an embedding model on Amazon Bedrock. Finally, upload them into Amazon OpenSearch Service vector store for later use. Specific parsing techniques for different document formats and contents were needed to properly prepare them for being indexed into OpenSearch.
• Chat — This workflow includes sending a question to the chat endpoints, then streaming back a response that uses our RAG workflow in an AWS Lambda function, called a chat streaming function. The function injects relevant information retrieved from OpenSearch in a prompt and sends the prompt to a foundation model on Amazon Bedrock. There are other Lambda functions in the workflow for interacting with chat history.
• Evaluation – The evaluation workflow can be scheduled or triggered on an ad hoc basis. It compares the gathered SME knowledge defined earlier to the outputs of the solution, as well as stores this data and makes it available to track over time. This workflow was extremely important to the success of this project and will be explored thoroughly later.

Figures 1 illustrates the reference architectures for the three workflows in the AI-driven solution.

Figure 1 — AWS reference architecture for the AI-driven solution

RAG evaluations

RAG evaluation is a non-negotiable step in generative AI applications, as evaluations can build trust in a solution based on a technology that can be subject to hallucinations or misinformation. Underwriting SMEs at FMH provided baseline truths as well as standardized multiple-choice tests typically given to underwriters.

Source Allies focused on the metric “answer-correctness” for evaluating RAG. This metric breaks down ideas or statements represented in a question, generated answer, and predetermined correct answer (ground truth), and categorizes them based on where those ideas or statements are present. Based on this categorization, the system calculates an answer-correctness score between 0-100 that represents the similarity of the content generated versus the ground truth. There are many other metrics for evaluating RAG, such as faithfulness, context relevancy, and content-recall. While these metrics are valuable because of an accelerated timeline, and tooling for metrics still evolving when this project was implemented, the answer-correctness metric was the main focus.

Source Allies monitored this solution by plotting it on a time-series graph to track the answer correctness of different questions over time. Figure 2 shows the answer correctness scores over time for both open-ended and multiple-choice questions. “Milestones” indicated at specific times represent actions like underlying data changes and experiments with document indexing, retrieval, and inference.

Figure 2 — Measuring answer correctness for RAG

In addition to monitoring the performance of the solution, RAG evaluation also enabled developers to employ the Metric Driven Development (MDD) process during extensive experimentation. For example, on the “open-ended” green line, after the “Index Special Provisions Docs” milestone, adding the “source of truth” documentation did not help the scores, but in fact made them drastically worse. This indicated that a naive-RAG solution (only using vector search) would not yield the expected results with the data. While this outcome was expected, developers used these scores to communicate with stakeholders at FMH. The MDD process improves customer engagement not with a pre-defined solution, but with tools to measure a solution and select customization techniques.

After observing very low “answer correctness”, Source Allies started to experiment and define where the challenges were. The issue with naive vector search is that the regulation documentation being used was both poorly formatted and also contained very similar information from document to document. This caused incorrect information retrieval when trying to ask questions about a specific combination of items like crop, policy, state, and county. Source Allies addressed this by using:

• Robust document parsing
  • Clean and order information in our documents.
  • Create hierarchical structure used to maintain context.
  • Extract metadata used for pre-filtering content.
• Advanced retrieval techniques
  • Hybrid search (keyword search combined with vector search).
  • Pre-filtering based on extracted metadata.
  • Self-querying retrievers used to generate filter conditions automatically in certain situations.

Solution results

With this tool, new underwriters, and others with limited exposure to the field, can answer questions similarly to the most experienced underwriters at FMH, providing consistency and expanded expertise across the organization. In addition, this tool is specific to their industry and can act as a differentiator from other crop insurance companies that are not leveraging task-specific AI.

Since implementing, FMH has experienced:

• An 800% improvement over naive RAG solutions in answer accuracy.
• 95% percent reduced operating cost for their generative AI solution compared to multiple comparable off-the-shelf implementations by leveraging AWS Bedrock Serverless offerings.
• Immediate time savings for underwriters and shorter upskilling time for new hires.
• Gaining competitive advantage using generative AI technologies.

Figure 3 and Figure 4 illustrate with and without task-specific AI, respectively.

Figure 3 – Without task-specific AI

Figure 4 – With task-specific AI solution

Users can also see the sources of regulation documentation that contain relevant information, providing transparency and user trust. Figure 5 shows the list of sources.

Figure 5 – The list of sources of a retrieval

Conclusion

By leveraging generative AI application evaluations, Metric-Driven Development, and robust scalable cloud infrastructure, Source Allies successfully implemented a solution for Farmers Mutual Hail. This solution saved time and money, differentiated them in the market, and set them up for future success in generative AI endeavors.

To learn more about Source Allies, their generative AI expertise, and thought leadership, find them in the AWS Marketplace or via the Source Allies website.

To learn more about Farmers Mutual Hail, their AI implementation experience, and their knowledge in the crop underwriting industry, visit the Farmers Mutual Hail website.

.
.

Source Allies — AWS Partner Spotlight

Source Allies is an AWS Advanced Consulting Partner with multiple competencies, including the AWS ML Service Competency and the AWS Migration Service Competency. Source Allies empowers your organization to use Generative AI to leverage your industry-specific data in transformative, scalable, and most importantly, reliable ways.

Contact Source Allies | Partner Overview | AWS Marketplace