How IGNYTE runs candidate search with Amazon S3 Vectors

Executive search starts with a large talent pool and a narrow brief. A recruiter filling a chief technology officer role needs people whose titles, work history, and location match the brief, even when their profiles use different words for the same job. IGNYTE, a multi-tenant recruiting platform, provides that information with semantic similarity search over a talent database of more than 1 billion contacts. More than 800 million of those contacts are embedded for semantic search today.

IGNYTE originally stored its vector embeddings in an Amazon OpenSearch Service domain alongside the full candidate profiles. OpenSearch Service answers vector queries with approximate nearest-neighbor search over a graph it holds in memory. As the database grew past 1 billion contacts, that graph no longer fit in the cluster’s memory. Queries slowed, and the team had to keep adding larger nodes to hold the graph. The team moved every embedding to Amazon S3 Vectors, a capability of Amazon Simple Storage Service (Amazon S3), and kept OpenSearch Service for structured filtering on skills, experience, and other profile attributes.

In this post, we describe how IGNYTE built its ingestion pipeline and two-hop candidate search on Amazon S3 Vectors and OpenSearch Service. We cover the per-continent index layout, the metadata design, and the batch pipeline that loads both stores from vendor data. If you run vector search over a large and growing dataset, you can apply these patterns to your own workloads.

About IGNYTE

IGNYTE is a recruiting platform that functions as an applicant tracking system (ATS) for multiple client firms. Its talent database combines contact data from a range of third-party data providers with each tenant’s proprietary contact records. IGNYTE ingests data from all of these sources and generates an embedding for every contact. Proprietary records stay isolated from other tenants.

On top of the talent database, IGNYTE runs IGNYTE AI, a system of seven AI agents built with the Strands Agents SDK and hosted on Amazon Bedrock AgentCore. A supervisor agent receives a position brief, a candidate pool, and client context. It then delegates research, screening, shortlist generation, and outreach drafting to specialist agents. When an agent needs candidates, it searches the Amazon S3 Vectors and OpenSearch Service stores described in this post. IGNYTE AI entered early-release beta in October 2025 and has run in production since June 2026.

Scaling vector search beyond 1 billion contacts

Recruiters search IGNYTE by describing a role, not by typing exact keywords. A query for “CTO” needs to return people whose profiles say Chief Technology Officer, along with related leadership titles such as Chief Data Officer. IGNYTE handles this with a proprietary embedding model. Every contact in the database carries a vector, and every search runs as a similarity query against those vectors.

Before the migration, the OpenSearch Service domain held the embeddings, the metadata, and the full candidate profiles. Approximate nearest-neighbor search kept its graph in memory. At more than 1 billion contacts, that graph set the size of the cluster, and growth in data and query volume meant scaling the whole cluster to hold it.

The team weighed two options: scale the cluster further or split the index. Both options kept vector capacity tied to cluster memory. Instead, the team moved the vectors to Amazon S3 Vectors, a managed vector store, and kept the structured profile data in OpenSearch Service. The solution uses these two stores and no custom-built vector store. Amazon S3 Vectors holds the vectors and runs similarity search at this scale. OpenSearch Service filters on the structured data it already holds.

Amazon S3 Vectors adds native vector storage and similarity search to Amazon Simple Storage Service (Amazon S3). You create a vector bucket, create one or more vector indexes inside it, and write vectors with a key and optional metadata. The QueryVectors API returns the vectors closest to a query vector and applies metadata filters as part of the query.

The following properties matched IGNYTE’s requirements:

• Capacity per index – A single vector index holds up to 2 billion vectors, and one vector bucket holds up to 10,000 indexes.
• No cluster to size – Amazon S3 Vectors is serverless. IGNYTE doesn’t provision nodes or memory for vector data.
• Usage-based pricing – Charges follow the data stored, the data written, and the queries run, not provisioned capacity. For rates, see Amazon S3 pricing.
• Strong consistency – A vector written with PutVectors is visible to the next query, with no index refresh interval.

Solution architecture

The architecture has two paths. The ingestion pipeline turns vendor data into candidate profiles and embeddings and loads each into its own store. The query path serves candidate searches from IGNYTE AI agents running on AgentCore. Both paths run in the same AWS Region. The split into two stores follows the work. Amazon S3 Vectors scales vector capacity for similarity search without resizing a cluster, and OpenSearch Service keeps the structured filtering it was already tuned for. The Design decisions section covers the main choices behind this split.

Ingestion pipeline

The ingestion pipeline separates each vendor feed into company records and contact records, then loads the company data and the contact data into their own indexes. The following diagram shows the pipeline, and the numbered steps after it match the callouts.

Figure 1: IGNYTE ingestion pipeline for Amazon S3 Vectors and OpenSearch Service

1. Vendor data arrives in Amazon S3. Two AWS Glue jobs split it into company records and contact records and write each set to Amazon S3 as Parquet files.
2. An Amazon SageMaker AI job processes the company records. An AWS Lambda function loads them into a company index in OpenSearch Service.
3. A second SageMaker AI job cleans and enriches the contact records, looking up each contact’s employer in the company index. The job generates an embedding for each contact with IGNYTE’s embedding model. It writes the clean profiles and the embeddings to Amazon S3 as separate sets of JSON files. AWS Systems Manager Parameter Store holds the pipeline configuration.
4. Another Lambda function loads the clean contact profiles into the OpenSearch Service contacts index.
5. The upsert_s3_vectors Lambda function writes the embeddings to S3 Vectors with PutVectors. Each contact goes to the index for its continent, named ign-contacts-{continent}.
6. IgnyteAI agents on Amazon Bedrock AgentCore read from both stores when they search for candidates.

IGNYTE’s first ingestion design reacted to Amazon S3 event notifications as vendor files arrived. People data and company data arrive as separate feeds, so an event-driven pipeline could process a contact before the company record it referenced. The team replaced the event triggers with a scheduled batch. Companies load first in each run (step 2), so contact enrichment (step 3) always looks up employers in a current company index.

Both stores are loaded in the same run. After each run completes, the contact IDs that S3 Vectors returns match the profiles stored in OpenSearch Service.

Query path

The following diagram shows how an IGNYTE AI agent runs a candidate search.

Figure 2: IGNYTE candidate search query path

1. A recruiter sends a search request to IGNYTE AI, and an agent running on AgentCore picks it up.
2. The agent sends the query text to the SageMaker AI endpoint that hosts IGNYTE’s embedding model and gets back a query vector.
3. The agent calls QueryVectors on the continent index that matches the search location, passing the query vector and other metadata filters. S3 Vectors returns the IDs of the 100 closest contacts, which form the candidate set for the OpenSearch Service hop.
4. The agent queries the OpenSearch Service contacts index with those IDs, applies structured filters such as skills and years of experience, and gets back full candidate profiles.
5. The agent returns the matching candidates to the recruiter.

The first hop finds people whose profiles are semantically close to the role. The second hop narrows that set to people who meet the hard requirements. OpenSearch Service also continues to serve fuzzy matching and autocomplete in the IGNYTE user interface, and neither of those features uses vectors.

Embedding model

IGNYTE trained its own embedding model for recruiting vocabulary built from over 15 years of work done in the executive search space. The team assembled training data from the profile data stored in Amazon S3, using Amazon Athena to query it. The model supports semantic expansion, so a query for “CTO” also matches Chief Technology Officer and Chief Data Officer. One shared SageMaker AI endpoint serves the model to both the contact enrichment job and the query path, so queries and stored contacts are embedded by the same model.

Design decisions

This section covers the choices that shaped how IGNYTE stores and queries more than 800 million vectors.

Per-continent indexes in one vector bucket

Instead of keeping all contacts in a single index, the team split the data into one index per continent inside one vector bucket. Across all continents, the indexes hold more than 800 million vectors, and the US index is the largest at roughly 300 to 400 million vectors.

The split gave IGNYTE two things. Each search now runs against only the contacts in the target continent. Query and write traffic from concurrent tenants is also spread across per-index limits, because Amazon S3 Vectors sets request limits per vector index.

Minimal vector metadata

The team reduced each vector’s metadata to the attributes the query path filters on, such as location. Full profile fields, including skills and work history, live only in OpenSearch Service. Amazon S3 Vectors allows up to 2 KB of filterable metadata and 40 KB of total metadata per vector, and a small payload keeps each vector well inside those limits. Less metadata per vector also reduces the data processed per query, which is one of the inputs to S3 Vectors query pricing.

OpenSearch Service after the migration

OpenSearch Service no longer stores any embeddings. It holds two indexes: the company index that contact enrichment reads during ingestion, and the contacts index that answers the structured part of each search.

The following table summarizes the key parameters of the solution.

Conclusion

IGNYTE moved its vector embeddings out of OpenSearch Service and into Amazon S3 Vectors. The team split the data into per-continent indexes in one vector bucket. It reduced vector metadata to the filters that the query path uses and kept OpenSearch Service for structured filtering on candidate profiles. A scheduled batch pipeline built on AWS Glue, SageMaker AI, and Lambda loads both stores from vendor data, and IGNYTE AI agents on AgentCore search them. After the migration, an end-to-end candidate matching run averages under 30 seconds, down from 3 to 4 minutes on the previous OpenSearch Service setup. That time covers the full agent workflow, including query embedding, both search hops, and the agent’s own steps, not a single vector query. Vector growth no longer drives cluster sizing, because vector capacity scales in Amazon S3 Vectors rather than in a cluster the team has to resize.

The same pattern applies to other workloads that pair semantic retrieval with structured filters over a large dataset. Store the vectors and a small filter payload in Amazon S3 Vectors. Partition indexes along a dimension your queries already filter on. Keep full records in the system that serves your structured queries.

About the authors