AWS Machine Learning Blog
Custom document annotation for extracting named entities in documents using Amazon Comprehend
This blog was last reviewed and updated in June, 2022 to include code updates and fixes.
Intelligent document processing (IDP), as defined by IDC, is an approach by which unstructured content and structured data is analyzed and extracted for use in downstream applications. IDP involves document reading, categorization, and data extraction, by using AI’s processes of computer vision (CV), Optical Character Recognition (OCR), and natural language processing (NLP) on provided texts.[1] Companies in financial services, healthcare, and manufacturing process millions of documents each year and find the process to be painstaking. Document processing is manual, slow, expensive, and error-prone, and data is often spread across disparate sources. As a result, creating and managing a document processing pipeline remains a challenge for many companies.
Amazon Comprehend is an NLP service that provides APIs to extract key phrases, contextual entities, events, and sentiment from documents. Entities refer to things in your document such as people, places, organizations, credit card numbers, and so on. But what if you want to identify entity types unique to your business, like proprietary product codes or industry-specific terms? Custom entity recognition in Amazon Comprehend enables you to train models to extract entities that are unique to your business in just a few easy steps. Historically, you could only use Amazon Comprehend APIs on plain text documents. If you wanted to process Word or PDF documents, you first needed to preprocess or flatten the documents into a plain text format, which often reduces the quality of the context within the document.
Today, Amazon Comprehend launched a feature to help organizations extract insights and automate processing documents of different formats (PDF, Word, plain text) and layouts (bullets, lists). You can now use custom entity recognition in Amazon Comprehend directly on more document types (scanned PDF, machine-readable PDF, and Word documents) without needing to convert your files to plain text. Custom entity recognition can now extract custom entities and process varying document layouts such as dense text and lists or bullets in PDF, Word, and plain text documents—at no additional cost.
To train a custom entity recognition model that you can use on your PDF, Word, and plain text documents, you need to first annotate PDF documents using a custom Amazon SageMaker Ground Truth annotation template provided by Amazon Comprehend. The custom template renders a PDF and allows you to annotate directly on the document. The custom model leverages both the natural language and structural or positional information (coordinates) of the text to make sure that we accurately extract custom entities that might have been previously impacted when flattening a document.
In this post, we walk through the steps of setting up the custom annotation template and show examples of how to annotate finance documents (SEC filings). After you annotate your data, you can use the generated manifest file to train a custom entity recognition model.
Feature overview
We cover the following steps:
- Install the annotation tool.
- Upload training documents.
- Create a private workforce to label the training documents.
- Set up the Ground Truth labeling job.
- Access the annotation login portal to start labeling documents.
- Label documents with the custom template.
Only PDF files are used for training a custom entity recognition model, but you can use that model on plain text, Word, and PDFs documents during inference (for now, asynchronous processing only).
The custom annotation template is available to download on GitHub. You need to Git clone or download this application and install it in local or in an AWS Cloud 9 integrated development environment (IDE). AWS Cloud9 is a cloud-based IDE that lets you write, run, and debug your code with just a browser. This annotation tool needs to be built and deployed using Python and AWS Serverless Application Model (AWS SAM) (the AWS SAM CLI lets you locally build, test, and debug serverless applications that are defined by AWS SAM templates).
Prerequisites
You need to install the following to build and deploy this solution:
- Install Python 3.8.x.
- Install jq.
- Install the AWS SAM CLI.
- Make sure you have pip installed.
- Install and configure the AWS CLI.
- Configure your AWS credentials.
If you’re setting up this tool in us-east-1, you can skip installing the AWS CLI and AWS SAM CLI, because it’s already installed with Python environment. You just need to create a virtual environment to use Python 3.8 in AWS Cloud9.
Install the annotation tool
Download the annotation tool from GitHub. You can either Git clone this repository or download the setup as a .zip file in your local. The .zip file contains all the installation parts required to build out the custom annotation workflow to annotate PDFs in Ground Truth.
First, you run and configure this tool from the AWS CLI. With AWS SAM, we build and deploy this tool to create an AWS CloudFormation template. The CloudFormation template configures resources such as an AWS Lambda function, AWS Identity and Access Management (IAM) roles and permissions to create an Amazon SageMaker labeling job, and your Amazon Simple Storage Service (Amazon S3) bucket.
Once you unzip the tool, you need to run commands to build and deploy from with-in the annotation tool. In the CLI, navigate to folder amazon-comprehend-semi-structured-documents-annotation-tools or run
cd amazon-comprehend-semi-structured-documents-annotation-tools to change your directory to the annotation tool folder.
- Run the following command to build and deploy the tool:
make ready-and-deploy-guided
- For Stack Name¸ enter
comprehend-semi-structured-annotation-tool. - For AWS Region, enter the Region you’re in.
- Leave the remaining parameters as default.
- For Confirm changes before deploy, enter
Y.
- For Allow SAM CLI IAM role creation, enter
Y.
After you complete these steps, the AWS CloudFormation resources are deployed in your AWS account, which you use to set up the annotation jobs. If deployment fails, delete the CloudFormation stack via the AWS CloudFormation console and try make deploy-guided again from your local or AWS Cloud9 IDE to reinstall this tool.
- When the status of the CloudFormation deployment changes from
In-ProgresstoComplete(which usually takes a few minutes), note the S3 bucket name created by the CloudFormation template.
You need this S3 bucket to upload training documents that are used for annotation in the next step.
Upload the training documents
You can run the following command in your local to copy local files to the S3 bucket created in the previous step. As a reminder, these training documents should be PDF documents only.
Or you can directly upload documents to this S3 bucket created by the CloudFormation template by creating a folder source-semi-structured-documents. Refer to this link to learn how to upload files in Amazon S3 and creating folders.
Create a private workforce
You need to create a private workforce in Ground Truth, which labels the training documents you uploaded in Amazon S3. You can view or create a private workforce using the Ground Truth console. For detailed instructions, see the section “Creating a private work team” in Developing NER models with Amazon SageMaker Ground Truth and Amazon Comprehend. To follow along with the next steps, we recommend adding yourself to the private workforce. All new workers added to your private workforce receive an enrollment email with the URL, user name, and a temporary password to log in to the labeling portal. Your workers use this login information after we create the labeling job.
After you create a private workforce, note the private workforce name, which we need when creating a Ground Truth labeling job using the AWS CLI.
Set up the Ground Truth labeling job
Ground Truth allows you to create custom labeling workflows, which you can use when the provided templates don’t suit the requirements for your labeling efforts.
To create a Ground Truth labeling job, you need to set up parameters such as the input Amazon S3 path, CloudFormation stack name, work team name, Region, job name prefix, entity types, and annotator metadata.
- input-s3-path: S3 Uri to the source documents you copied earlier in
Upload Source Semi-Structured Documents to S3 bucket - cfn-name: The name of the CloudFormation stack name entered in the Package and Deploy step:
comprehend-semi-structured-annotation-tool - work-team-name: The workforce name created from the previous step
- job-name-prefix: The prefix to have for the SageMaker Ground Truth labeling job (LIMIT: 29 characters). Extra text will be appended to job name prefix, ex.
-labeling-job-task-20210902T232116 - entity-types: The entities you would like to use during the labeling job (separated by commas)
Run the following AWS CLI command to trigger a Ground Truth labeling job:
You can view the labeling job on the SageMaker console. As shown in the following screenshot, our labeling job is created, in progress, and using a custom task type.
You have now created a custom labeling job for annotating your PDF documents.
Historical options for processing PDF documents required converting documents to raw text format before processing through Amazon Comprehend custom entity recognition models. However, the custom Ground Truth template you created allows you to interact with the document in its native PDF format. It does this by rendering a text layer on top of the PDF. This combination allows you to reference the entity within the table or structured format in a familiar, easier, and more efficient manner than using a document that has been converted to a plain text format. In addition to the user-friendly document rendering, the template also helps pick the right span for the text and provides the ability to wrap lines for entities that span several lines within the document.
Access the annotation login portal
You can access the annotation portal either by using the URL in your private workforce enrollment email or via the SageMaker console. To use the SageMaker console, complete the following steps:
- Choose Labeling workforces in the navigation pane.
- Choose Private.
- Choose the link under Labeling portal sign-in URL.
- Enter the email and password you received in your enrollment email.
- Choose Login to start labeling.
Label documents with the custom template
After you create the labeling job, the labeling workforce can access the documents and begin annotating the required entities. First, we walk you through the annotation UI template, then three examples of how to annotate documents with finance documents (Securities and Exchange Commission (SEC) filings).
Explanation of the annotation UI template
On the annotation user interface, you can see one of your PDF documents open (see the following screenshot). On the top left, you can see the custom entity types that you provided during setup. To the right of your entity types, you can see arrows to navigate through your training documents. The document in the example below is blurred out to focus on the annotation UI.
Additionally, you have options to remove, undo, or auto tag your annotations on each document. To use auto tag, simply annotate a word and associate it with one of your custom entity types. If you choose Auto Tag, all other instances of that entity type (for example, all addresses) are automatically annotated with that entity type. After you annotate each PDF, choose Submit to save annotations before moving to the next document.
The following screenshot shows the options within the right panel of the annotation interface. You can edit, copy, remove, and reset your annotations.
Now with this understanding of the annotation UI, we show end-to-end examples of how to annotate custom entities for finance documents (SEC filings).
Finance
Finance customers can now easily process bank statements and SEC filings to extract custom entities such as board of director names, acquisition price, earnings per share, and more. In the following example, we show how to extract the following entities from SEC form S3 (registration) and SEC form 424B5 (prospectus filing): OFFERING_PRICE and OFFERED_SHARES. More specifically, we demonstrate how to annotate these forms so you can extract custom entities that appear in both a table and paragraph and have multiple occurrences.
The SEC dataset is available for download here s3://aws-ml-blog/artifacts/custom-document-annotation-comprehend/sources/. You can use this dataset to train a custom entity recognition model after you finish annotating. Run the below command to copy the S3 data from s3://aws-ml-blog/artifacts/custom-document-annotation-comprehend/sources/ to the AWS CloudFormation created bucket s3://comprehend-semi-structured-docs-${AWS_REGION}-${AWS_ACCOUNT_ID}/source-semi-structured-documents/ by running following command:
Go to the command prompt where you set up annotation tool and run the following command to set up a training job for the entities OFFERING_PRICE and OFFERED_SHARES in the entity-types parameter:
Additional customizable options:
- Include –
-annotator-metadataparameter to reveal key-value information to annotators. Default metadata about the document is already revealed to the annotator within the UI side panel. - Specify –
-use-textract-onlyflag to instruct the annotation tool to only use Amazon Textract AnalyzeDocument API to parse the PDF document. By default, the tool tries to auto-detect what types of source PDF document format is, and use either PDFPlumber (native PDF) or Amazon Textract (scanned PDF) to parse the PDF Documents. When creating the labeling job, a customer has the option of only using Amazon Textract for both the scenarios, which may be more accurate for text extraction, but comes at an additional cost (see Textract Pricing).
This command triggers a Ground Truth labeling job for your private workforce. Now log in to your labeling portal. You’re redirected to the annotation UI to annotate entities we specified in the labeling job. In the following screenshot, we show how entities such as OFFERING_PRICE and OFFERED_SHARES are annotated. Your labeling workforce uses the pointer to draw bounding boxes around the appropriate entities and labels them with the appropriate custom entity label. You must annotate the OFFERING_PRICE entity in every occurrence, which in this case, is within the dense text paragraph and in semi-structured (columnar) format.
After you label all the pages, you can find annotations in JSON format in the Amazon S3 location s3://comprehend-semi-structured-docs-<AWS Region>-<AWS Account number>/output/<your labeling job>/annotations/consolidated-annotation/consolidation-response/iteration-1/annotations/
The user-labeled document information is also saved to an output manifest file in the Amazon S3 location provided during the setup of the custom labeling job. The output manifest file references all the annotations within your training documents. You can find your output manifest file in the Amazon S3 location s3://comprehend-semi-structured-docs-<AWS Region>-<AWS Account number>/output/<your labeling job>/manifests/output/. You use this manifest file to create an Amazon Comprehend custom entity recognition training job and train your custom model. For instructions, see Extract custom entities from more document types with Amazon Comprehend.
Review the output manifest file
When examining the output manifest file for the custom labeling training job on PDF documents, we can see the contextual information (natural language and positional or structural information) that is gathered and used to train the Amazon Comprehend custom entity recognition model.
First, let’s look at the output generated from the custom entity labeling job of a different dataset. In the following screenshot, the output manifest file for the labeling job has generated starting and ending offsets but also uses references to block IDs and child block IDs for each labeled entity to capture the associated 2-D information of the entity (what the entity is and where it’s located in the document). A block represents layout coordinate positional information of the tokens within the document. A child block ID identifies the word blocks within the referenced block ID. As shown in the highlighted code, the entity labeled contains information relating to where the entity is placed within the overall structure of the document.
To compare, let’s examine the output from a standard Ground Truth template for named entity recognition (NER). As shown in the following screenshot, the starting and ending offsets are given for each of the three entities labeled. However, no other contextual information, such as the coordinates or position of the text within the document, is provided. This is expected because the Ground Truth template for NER takes input files in as plain text format, which has removed all spatial information from within the PDF document in regards to positioning of words relative to each other and the document.
Labeling best practices
The following are best practices while annotating your data:
- Annotate your data with care and verify that you annotate every mention of the entity. You can use the Auto Tag feature and every reference of the entity will be annotated. Imprecise annotations can lead to poor results. In general, more annotations lead to better results.
- Input data should not contain duplicates, like a duplicate of a PDF you are going to annotate. Presence of a duplicate sample might result in test set contamination and could negatively affect the training process, model metrics, and model behavior.
- Make sure that all documents in the corpus are annotated, and that the documents without annotations are due to lack of legitimate entities, not due to negligence. For example, if you have a document containing “J Doe has been an engineer for 14 years,” you should provide an annotation for “J Doe” as well as “John Doe.” Failing to do so confuses the model and might lead to not recognizing “J Doe” as an
ENGINEER. This should be consistent within the same document and across documents. - Provide documents that resemble real use cases as closely as possible. Synthesized data with repetitive patterns should be avoided. The input data should be as diverse as possible to avoid overfitting and help the underlying model better generalize on real examples.
- It’s important that documents should be diverse in terms of word count. For example, if all documents in the training data are short, the resulting model has difficulty predicting entities in longer documents.
- Try to give the same data distribution for training as you expect to be using when you’re actually detecting your custom entities (inference time). For example, at inference time, if you expect to be sending documents that have no entities in them, this should also be part of your training document set.
- We recommend a minimum of 250 documents and 100 annotations per entity to ensure good quality predictions. With more training data, you’re more likely to produce a higher-quality model. If you want higher accuracy, we recommend increasing the volume of annotated data by 10% to further improve the accuracy. This improvement might be best visible to you by running inference on a held-out test set that remains unchanged and can be tested by different models. In this way, you can compare successive models.
For additional suggestions, see Improving Custom Entity Recognizer Performance
Conclusion
With this new contextual information included within labeled annotations, you can now train Amazon Comprehend custom entity recognition models with semi-structured information (bullets, lists, dense text). This additional structural context provides more information to the model to help identify the relevant entities within documents.
For more information about how to train your custom model and the impact this additional information can have on custom Amazon Comprehend model performance, see Extract custom entities from documents in their native format with Amazon Comprehend and our documentation.
References
[1] IDC Survey Spotlight, What Is the Landscape of the Emerging Document Artificial Intelligence Market?, Doc # US47701421, July 2021
About the Authors
Mona Mona is an AI/ML Specialist Solutions Architect based out of Arlington, VA. She helps customers adopt machine learning on a large scale. She is passionate about NLP and ML Explainability areas in AI/ML.
Anant Patel is a Sr. Product Manager-Tech on the Amazon Comprehend team within AWS AI/ML.
Andrea Morton-Youmans is a Product Marketing Manager on the AI Services team at AWS. Over the past 10 years she has worked in the technology and telecommunications industries, focused on developer storytelling and marketing campaigns. In her spare time, she enjoys heading to the lake with her husband and Aussie dog Oakley, tasting wine and enjoying a movie from time to time.