Advanced Clinical Enhanced Clinical Trial Document Processing with Veeva’s TMF Bot

As clinical trials become more complex, larger, and more geographically dispersed, research organizations require more digital tools to enable greater visibility, collaboration, and flexibility. Recognizing these needs, Veeva built TMF Bot, a feature within Veeva Vault eTMF, to help its customers alleviate document processing, collection, and classification challenges, enabling organizations to deliver faster and higher quality clinical trials.

Advanced Clinical, a global contract research organization that specializes in managing and supporting Phase I-IV clinical trials across all major therapeutic areas has benefited from TMF Bot. Since the implementation, Advanced Clinical has seen improvements in several key areas of its clinical trial process. Powered by AWS, TMF Bot leverages artificial intelligence (AI) supported by AWS services such as Amazon SageMaker to automate time-consuming tasks and improve accuracy.

Previously, Advanced Clinical’s trial master file (TMF) documents were manually classified and tagged with the appropriate metadata by users in Veeva Vault eTMF. This approach to classifying documentation hindered Advanced Clinical in several ways. This process was time-consuming and prone to errors, which could lead to document misclassification, and took users’ attention away from more important strategic tasks.

Advanced Clinical worked with Veeva on becoming an early adopter of TMF Bot, an application feature built into Veeva Vault eTMF. Using AI to streamline TMF management and trial execution, TMF Bot allows customers to automate document classification of uploaded and mobile scanned documents. Using a customer’s existing data, TMF Bot automates TMF management, reduces the administration burden of study teams, mitigates risk of misfiled and misclassified documents, and improves compliance and efficiency without sacrificing quality.

“Using TMF Bot in Vault eTMF allows us to process documents in greater volumes, in less time, and with a higher level of accuracy than what was previously possible,” said JP Miceli, Director of Document Management at Advanced Clinical. “Through meaningful metrics—produced via easy-to-use, intuitive dashboards and reporting—we can consistently offer our clients the transparency and visibility they want, while ensuring proper oversight of the TMF.”

"We know how integral TMF management is to clinical trials process, and customers tell us that using the automation enabled by TMF Bot has allowed them to achieve better outcomes faster and more easily. Reducing work for our end users is the priority," Shohei Etzel, product manager at Veeva Systems.

The AI built into TMF Bot makes automation of time-consuming tasks possible. It helps organizations ensure their TMFs are as complete and accurate as possible, rationalize the large volume of documents they possess, and extract meaningful, actionable insights from that data.

• Alleviating workloads with automation: TMF Bot automates repetitive trial processes, such as document classification. This allows users who support numerous clinical trial activities to reduce the resources needed to perform TMF completeness checks and reviews and focus more on strategic initiatives and tasks.

• Improving compliance readiness: A key aspect of clinical trials is maintaining compliance and being prepared for agency inspections. TMF Bot improved quality and minimized errors for improved compliance. Since implementing Vault eTMF and TMF Bot, Advanced Clinical has successfully conducted and supported client, mock, and internal audits in Vault as well as agency inspections by the Medicines and Healthcare products Regulatory Agency (MHRA) and the Food and Drug Administration (FDA).

• Personalizing for an optimized approach: Customers are able to customize TMF Bot through a configurable confidence threshold, allowing them to take a more aggressive or conservative approach to auto-classification. They are also able to select their Training Set to ensure they are creating a model based on the highest quality documents.

• Driving efficiency and accuracy: Utilizing the TMF Bot to auto-classify documents and populate metadata, users can save time managing documents in the system and drive significant improvements in TMF accuracy, timeliness, and completeness.
  

TMF Bot uses custom algorithms, developed by Veeva, for training of machine learning models, optical character reader (OCR)/text extraction, and prediction of document classifications. TMF Bot leverages Amazon SageMaker to run those algorithms at scale as containers. To train models, TMF Bot gives Veeva Vault customers the instrumentation to select documents in their Vault, create training sets, and run Amazon SageMaker training jobs to train models without having to write a single line of code. To classify documents, TMF Bot operates on documents uploaded in various formats by Veeva Vault customers. It uses Amazon SageMaker inference endpoints to extract text and metadata from those documents. Then it feeds the extracted information to the customerspecific model that is hosted on Amazon SageMaker multi-model inference endpoints to predict the classification.

“AWS, and specifically Amazon SageMaker, provide value to Veeva by allowing us to develop and offer AI-driven automation in a hyper-personalized manner for our customers,” said Etzel. “This expands Vault Clinical with features like document auto-classification with TMF Bot. The system performing tasks on behalf of users is a big step forward.”

In building TMF Bot, Veeva had requirements in the areas of performance, scalability, security, logging, monitoring, zero downtime deployment, and compliance, which were all satisfied out of the box by Amazon SageMaker.

By leveraging Amazon SageMaker's out-of-the box capabilities, Veeva saved many weeks of development time, allowing the team to deliver TMF Bot much faster to market. In addition, by using multi-model endpoints, Veeva scaled from ten models to hundreds of models using a much smaller number of real-time inference instances. That resulted in a reduction of operational costs due to the lower hosting cost and simpler endpoint management.

Veeva Vault also leverages Amazon Simple Storage Service (Amazon S3) to temporarily store training sets and permanently store trained models. Amazon Key Management Service (Amazon KMS) enables Vault to encrypt Amazon S3 objects and Amazon SageMaker storage. With Amazon Elastic Container Registry (Amazon ECR), Vault hosts images for training and inference pipelines for deployment anywhere and then utilizes Amazon CloudWatch to log and monitor those training and inference pipelines.

After implementing TMF Bot, Advanced Clinical recognized an immediate improvement in document processing by saving up to four hours per week compared to performing this manually prior. At the same time, the accuracy in classification and quality control (QC) of documents increased from 90 to 95 percent.

Documents are successfully extracted for auto-classification at a 94% rate, meaning those documents contained enough content for TMF Bot to auto-classify them. From those documents, TMF Bot successfully auto-classified them 98 percent of the time.

Because of the key performance metrics noted above, enabled by the use of TMF Bot, Advanced Clinical has improved the overall quality of its trials and demonstrated increased compliance and oversight.

“Vault eTMF gives us and our partners a strategic advantage. We can accelerate document collection and processing from start-up through trial close, while enabling anytime-anywhere access and visibility for study partners throughout the entire study,” said Miceli. “It has enabled us to conduct faster, higherquality trials and ultimately enabled us to drive an engaged culture of ongoing TMF inspection readiness.”