Why is the Deleted Documents metric so high in my Amazon OpenSearch Service cluster?

5 minute read
0

I've deleted documents in my Amazon OpenSearch Service cluster, but I don't see any disk space reclaimed.

Short description

In OpenSearch Service, the DeletedDocuments metric is a counter that shows the number of documents that are marked for deletion. The DeletedDocuments metric shows an increase after the delete requests are processed and decreases after the index segments are merged within your cluster.

OpenSearch Service automatically runs the merge API operation using the merge policy setting. During a merge, the smaller segments are merged into larger segments to maintain the index size. Documents marked for deletion are also expunged to free up additional disk space.

To reclaim disk space immediately, you can delete an index instead of deleting individual documents. Or, you can also use the force merge API along with the only_expunge_deletes parameter to clear up the deleted documents within an index.

To maintain the index metadata while reclaiming disk space in your cluster, consider the following approaches:

Resolution

Check the number of deleted documents

To check the number of deleted documents in your OpenSearch Service cluster, run the cluster stats API. The value obtained from the cluster stats API call appears in the DeletedDocuments metric for your cluster. The output returns a summation of deleted documents for all the indices present in your cluster. You can check this count using the "docs.deleted" field in the response output.

For example, if your cluster has three indices (index1, index2, and index3), then run the index stats API call:

GET index1/_stats
...
"docs": {
     "count": 100,
     "deleted": 1
}
... 
GET index2/_stats
...
"docs": {
     "count": 100,
     "deleted": 5
}
...
GET index3/_stats
...
"docs": {
     "count": 100,
     "deleted": 8
}
...

The cluster stats API call then adds the "docs.deleted” field for all indices that are present in your cluster:

...
"docs": {
     "count": 1227677521,
     "deleted": 14
}
...

If you delete index2, then the cluster stats API call calculates the values only for index1 and index3:

GET _cluster/stats
...
"docs": {
     "count": 1227677521,
     "deleted": 9
}

The segments are now merged and the index metadata for index2 is erased. As a result, the DeletedDocuments metric value decreased to 9.

Confirm the size of your documents

To check the document sizes and count for an index, use the cat indices API. Make sure that the new document is the same size as the existing document in your OpenSearch Service cluster. Using the same document size makes sure that deleted documents don't take up additional disk space.

Expunge the deleted documents

To manually reclaim disk space, run the force merge API along with the only_expunge_deletes parameter set as “true”:

POST /<index-name>/_forcemerge?only_expunge_deletes=true

Note: This operation expunges only the segments containing documents marked for deletion.

As a result, the force merge decreases the amount of disk space in use. After the new segments are created, old segments are removed and new segments no longer contain the documents marked for deletion. For more information about deleted documents, see Lucene's handling of deleted documents on the Elasticsearch website.

However, be aware of the following when performing the force merge operation:

  • Perform force merge on your cluster only when there is enough free storage space. This action is a resource-intensive operation.
  • The force merge operation triggers an I/O intensive process and blocks all new requests to your cluster until the merge is complete.
  • Only call the force merge operation against read-only indices, when no additional data is being written to the index. Calling force merge against a read/write index can cause very large segments to be produced (>5 GB per segment). When this happens, the automatic merge policy doesn't consider these very large segments for future merges until the segments contain mostly deleted documents. As a result, disk usage increases and search performance worsens.

You can also use the delete by query API or delete API to manually delete any documents in your cluster.

Reclaim disk space immediately

To reclaim disk space immediately, use the delete index API. Deleting an index doesn't create any delete markers. Instead, the delete index API clears the index metadata, and disk space is immediately reclaimed. The reclaimed disk space is reflected in the DeletedDocuments metric.

Note: It's a best practice to delete old indices that aren't in use. If you delete an active index, make sure to block the automatic creation of indices. For more information, see Automatically create data streams and indices on the Elasticsearch website.


Related information

How do I troubleshoot low storage space in my Amazon OpenSearch Service domain?

Monitoring OpenSearch cluster metrics with Amazon CloudWatch

AWS OFFICIAL
AWS OFFICIALUpdated a year ago