Posted On: Nov 22, 2023
OpenSearch Service 2.11 now comes with OpenSearch Neural Sparse Retrieval. Search practitioners now have an additional search method to use for their search applications with improved semantic understanding, while keeping computational cost and computational latency low, more in line with lexical search.
Neural Sparse Retrieval is a new kind of sparse embedding method, similar in many ways to classic term-based indexing, but with a better understanding of low-frequency words and phrases. Neural Sparse Retrieval uses transformer-based models (e.g. GPT or BERT) to build information-rich embeddings which solve for the lexical challenge with vocabulary mismatch in a scalable way. This new sparse retrieval functionality with OpenSearch Service offers document-only mode and bi-encoder mode, each with different advantages. Document-only mode can deliver low-latency performance more comparable to lexical search, with limitations for advanced syntax as compared to dense methods. Bi-encoder mode maximizes search relevance while performing at higher latency. With this update, users can now choose the method that works best for their performance, accuracy, and cost requirements.
Neural Sparse Retrieval is now available in all AWS regions where Amazon OpenSearch Service is available. For information on upgrading to OpenSearch Service 2.11, please see documentation.
To learn more about Amazon OpenSearch Service, please visit the product page.