How NASWA scales real-time labor market insights with AWS generative AI

State workforce agencies across America face a persistent challenge: how to provide timely, accurate labor market data that can guide policy decisions, inform training programs, and help job seekers navigate rapidly changing career landscapes. The National Association of State Workforce Agencies (NASWA) recognized that the National Labor Exchange (NLx) Research Hub—containing over 185 million job postings—held the answers, but unlocking those insights at scale required a forward-thinking approach.

The National Association of State Workforce Agencies (NASWA) saw an opportunity to address this challenge through the National Labor Exchange (NLx) Research Hub, a resource containing almost 200 million job postings. While this data offers valuable insights into the labor market, extracting meaningful information at scale requires a forward-thinking approach.

In this post, we share how NASWA scaled real-time labor market insights using AWS generative AI.

Architecting for scale and efficiency

The project’s ambition matched its technical complexity: Processing millions of job posting records to extract meaningful data from unstructured job descriptions to create actionable insights and improve regional workforce development required careful architectural planning. The initial approach relied on Amazon Bedrock batch processing, but as the team moved toward launch, performance and cost challenges emerged.

The AWS team worked closely with the NASWA team to find a faster, more cost-effective way to process millions of job postings, and discovered that the answer wasn’t simply picking a different tool, but rethinking the approach entirely. By combining batch and real-time processing with Amazon Nova Lite and breaking complex AI tasks into smaller, more focused steps (a technique called prompt decomposition), the team redesigned the data pipeline from the ground up. The result was dramatically lower costs and faster time to insight for workforce agencies nationwide.

Smarter prompts, faster results: Redesigning the processing engine

The breakthrough came from pairing the right model with a smarter way of using it.

The team selected Amazon Nova Lite, a model well-suited for high-volume text tasks like classifying and extracting data from job postings. But the bigger innovation was rethinking how they asked the AI to do its work.

Rather than sending each job posting through a single, complex prompt that tried to extract everything at once, the team used prompt decomposition. This works similar to an assembly line: instead of one worker doing 10 jobs, each station handles one task and does it well. One prompt identifies educational requirements. Another extracts pay information. Others pull out benefits, remote work status, and full-time classification. Each step is simpler, faster, and more accurate because the model isn’t juggling competing instructions.

The redesigned pipeline processes job postings through the following components:

• Amazon Nova Lite for efficient, on-demand natural language processing of job descriptions
• Optimized prompts that extract structured information on educational requirements, pay information, benefits, remote status, and full-time status with minimal tokens
• Streamlined data pipelines that alleviate processing bottlenecks

The result is the same data quality workforce agencies depend on, with a more efficient and sustainable offering.

Keeping collaboration at the center

Throughout the architectural evolution, cross-team collaboration proved essential. The AWS account team maintained regular engagement with NASWA stakeholders, confirming technical decisions aligned with workforce development goals.

“This partnership demonstrated the value of bringing together different perspectives,” says the project team lead. “AWS provided the technical architecture expertise, while the NLx Research Hub team contributed essential domain knowledge about the data and how system stakeholders will leverage this type of labor market information.”

The collaborative approach extended beyond problem-solving. Weekly touchpoints provided opportunities to validate assumptions, test processing approaches, and confirm the solution would meet real-world needs of workforce professionals across all states and territories.

Enabling insights that drive workforce decisions

The architectural transformation delivered measurable improvements:

• Realized an 80 percent cost reduction compared to the original processing approach
• Resolved performance bottlenecks that had threatened launch timelines
• Maintained data quality while dramatically improving processing efficiency
• Enabled scalable analysis of millions of historical and new daily job posting records

More importantly, the redesigned system positioned the NLx Research Hub to deliver on its core mission: improving access to actionable labor demand data and insights across the workforce and education system to better connect talent and opportunity, address gaps in labor market information, and advocate for system-wide solutions. The insights generated from the parsed data can help policymakers understand emerging trends in labor demand, guide training programs toward high-opportunity careers, and help create dynamic career pathways to assist learners, up-skillers, and job seekers of different backgrounds.

NLx Research Hub users can now access data showing how educational requirements are trending in their regions, how job requirements are evolving across industries, and where opportunities exist for workers with specific backgrounds—all derived from historical and real-time job posting data.

The AWS team continues working with NASWA to evaluate tools and capabilities that could unlock even more value from structured job posting data for workforce and education professionals, including applying the AI extraction approach to additional fields like skills and job qualifications.

Lessons for scaling AI in the public sector

The project offers valuable guidance for organizations tackling similar challenges:

• Engage architectural expertise early – Involving cloud architecture specialists during the design phase—rather than after encountering difficulties—can prevent costly redesigns and accelerate time to value.
• Know your data – The quality of AI output depends as much on understanding your input data as it does on the model you choose. Take time to study the structure, inconsistencies, and edge cases in your data.
• Understand the difference between extraction and inference –Not all AI tasks are the same. Extracting a salary range that’s explicitly stated in a job posting is fundamentally different from inferring an occupation code based on a job description.
• Processing approach to workload characteristics – Batch processing, on-demand processing, and streaming each have optimal use cases. Understanding your data patterns and processing timelines helps select the right approach.
• Prioritize cross-functional collaboration – The combination of technical expertise, project management knowledge, and domain understanding creates solutions stronger than one single perspective.
• Plan for scale from the start – Architectural decisions that work for pilot projects might not scale to production volumes. Testing at realistic data volumes and under desired timeline constraints early prevents surprises.

Conclusion

The project exemplifies how generative AI, combined with scalable cloud architecture, can address complex societal challenges. By transforming millions of job postings into structured data that can drive workforce intelligence, the NLx Research Hub is equipping system stakeholders with information, resources, and insights to make informed decisions that impact millions of American workers.

“The goal of this project was to increase the accessibility of labor market information to enable better decisions,” says the project team lead. “Stakeholders throughout the workforce and education system rely on up-to-date, accurate data to serve their communities effectively. This project assists Research Hub efforts by improving that foundation.”

As workforce needs continue evolving, projects like this demonstrate the potential for technology partnerships to create meaningful public sector impact—helping workers find opportunities, guiding training investments, and strengthening America’s economic competitiveness.

The pay, benefits, degree requirements, remote status, and full/part time status parsed data will be available to Research Hub users starting in 2027. To stay updated on offerings from the Research Hub, check out the website and subscribe to alerts.