Sign-Speak builds with AI on AWS to create accessible experiences

If you were to encounter one of the 430 million Deaf and Hard of Hearing (D/HH) individuals in the world, there are two main ways you can communicate with them: interpreters or texting back and forth. Interpreters, while providing great services, often are cost prohibitive for certain situations and texting is not the most optimal. Sign-Speak is changing that paradigm.

Sign-Speak—a startup whose language software recognizes American Sign Language (ASL) and translates it into spoken words (and vice versa) with machine learning—is doing exactly that. With the World Health Organization (WHO) predicting that nearly 1 in 10 people will have disabling hearing loss by 2050, Sign-Speak is building a solution to improve lives today and well into the future.

Founded in 2020 by chief executive officer (CEO) Yami Payano, chief product officer (CPO) Nikolas Kelly, and chief technology officer (CTO) Nicholas Wilkins, Sign-Speak is proving what’s possible when accessibility and machine learning (ML) merge. All three founders have a foundation in ASL, with a passion for building with (not for) the Deaf community that comes from personal experience: Nikolas, a Deaf individual, has experienced the challenges that arise from accessibility barriers. Yami has a family friend (like family) who—as a result of being Deaf—was unable to attend school in the Dominican Republic because hiring an interpreter was more expensive than the entire cost of tuition.

“From an early age, I grew up understanding the lack of access,” explains Yami. “As the child of Dominican parents who cannot communicate in English, I empathize with the children of Deaf parents. Like me, they often have the role of translator. No individual should face that kind of responsibility.” 

To make the world a more accessible and inclusive place, the Sign-Speak team built a platform that offers real-time ASL recognition, avatar, and transcription to facilitate communication with D/HH individuals. They decided to build on Amazon Web Services (AWS) for three reasons: the ability to scale up and down with customer demand, rapid compute for their artificial intelligence (AI) and ML needs, and cost-effective solutions for their substantial data corpus. “Our tech is hosted on AWS because we find the speed fantastic, as well as the compute,” Nicholas explains. “Primarily, we really like using AWS Trainium and AWS Inferentia because those are fantastic for low-cost training and inference. Additionally, they provide some of the fastest training and inference on the market, and are cost effective at that. We also find the infrastructure tools extremely helpful. Using Amazon Elastic Container Service (Amazon ECS) and AWS Fargate means that we can focus on just writing code, and AWS Code Pipeline ensures that we don’t need to worry about jumping through an arduous deployment process.”

The result? A solution that leads to easier communication between organizations and their customers. With Sign-Speak, businesses provide a better customer experience, improve their brand image, and comply with accessibility regulations.

Co-founders Yami and Nikolas demonstrate how Sign-Speak translates spoken words into ASL and vice versa.

Building an AI solution with (not for) the Deaf and Hard of Hearing community

Sign-Speak believes the power of AI comes from its ability to help their customers. They use ML to translate the visual language of ASL into spoken languages such as English. Additionally, with generative AI, they’ve created an avatar that translates spoken languages such as English into ASL. “I hope that Sign-Speak shows that AI technology can help a lot of communities that haven’t had access,” says Yami. “And at the same time, it will also show that helping these communities is a real business—it’s profitable, while also being good.”

Training Sign-Speak’s ML models within the Deaf community has been crucial to the development of the product. “Getting real-time feedback from the community is something we’re really proud of,” explains Yami. As their community engagement and product grew, they began accumulating a large corpus of data—much of which is videos—gathered from product pilots and through gamification. They recognized a growing need for powerful and scalable compute to support the features in their product roadmap.

With these challenges in mind, Sign-Speak applied for and was accepted into the AWS Impact Accelerator Latino Cohort, a program that provides under-represented founders with equitable access to funds, training, mentorship, tools, and resources.

Accelerating success with the AWS Impact Accelerator Latino Cohort

The Impact Accelerator was the on-ramp to expertise and support that would help Yami and her co-founders to prove both the profitability and the importance of technology such as Sign-Speak.

“We were very excited when AWS offered the opportunity to join the Impact Accelerator. It was a pivotal moment because of the business, technical, and monetary support from AWS,” Yami explains. “These helped us to solve a widespread problem: one that ranges from interpreter shortages to giving D/HH people access at places like restaurants and doctors’ offices.”

Going into the program, Sign-Speak had three goals:

1. Technical support. “It was very attractive to get a technical mentor for Nicholas, because a lot of accelerators don’t have that,” says Yami.
2. Networking and business support. Yami explains, “Making connections within an organization as big as Amazon has amazing implications for my business. I wanted to learn how an enterprise does things and apply that to my business as well.”
3. Financial support. “The cash grant and credits were critical for us to keep building,” says Yami. “And it was extremely attractive that the Impact Accelerator takes no equity from the startups they support.”

The eight-week program was “like nothing I’ve ever seen before, and I’ve been through a lot of accelerators,” says Yami. The intentionality of every piece of the program is what most stood out: from how mentors were assigned, to the one-to-one feedback that participants received about pitching and using Amazonian processes to improve their workflows.

“Every workshop with the Impact Accelerator was perfectly thought out,” she explains. “I met all three of my goals, and I also came out with a plethora of practical knowledge that allowed us to refine ourselves into better leaders and a better company.”

Yami’s proudest moment from the program is using what she learned to land a product pilot for Sign-Speak within a powerful company in her industry. “To see the drastic change in how we presented—from the start of the program to the end of the program—was amazing. And now I carry that skillset around with me and I use it.”

Gaining technical and business skills was important, and so was the atmosphere in which the cohort came together to grow as founders. “I think I cried four times in the cohort, because we formed such a strong relationship with each other,” says Yami. “Being around people who shared the same culture and faced similar challenges as me took it to the next level. It felt like a team and like a home.”

Yami attributes her Impact Accelerator experience to the mindset of AWS as a whole: “All Amazon thinks about is customers. And they got me into that mindset, too,” she laughs. “They understand fast iteration cycles to provide something of value—just like a startup does—and I think that’s what made this such a successful experience.”

Envisioning the future of Sign-Speak and Latino-led startups

Today, Sign-Speak continues to enhance their product in collaboration with the D/HH community. Yami believes the future of the interpretation industry will include the ability to have conversations anywhere at any time, for everyone: “AI has been a topic in communities of privilege for a long time. I want to change that—one day, I hope to see our AI technology built into enterprise products to enhance communication for all communities.”

For other founders who want to grow their startup with the help of an accelerator, Yami has three pieces of advice:

1. “Don’t build your product to make it a fit for the accelerator criteria. Instead, build it with your customer in mind, and everything else will follow.”
2. “Apply to accelerators that don’t take equity. It’s important that if you believe in yourself and your startup that you accept deals intentionally, and not because they’re easy or tempting.”
3. “Give constructive feedback to any programs you join. Make it better for the next stage of founders who participate. Founders are a community and we need to help each other.”

When it comes to supporting startup success, Yami advises: “I hope more companies take the leadership that AWS has taken to foster innovation and see the potential in people for greatness. When you’re dealing with communities such as the Latino community or the Black community, we’re people first: Developing a personal connection to us is great for business and it’s great for the community.”

For other Latino founders who are proving what’s possible with technology, Yami has a message:

“Pa’lante! We are innovators and we have grit. It’s hard, and funding is not always there, but I hope whoever is reading this understands: You can do it. It’s gonna be hard, but you can do it.”

Ready to begin your startup journey? Join AWS Activate to build and scale your startup with the right resources at the right time.

AWS Activate updates program benefits regularly, and credit offerings and/or the offerings reflected in this blog post may differ from current Activate offers. For the most up to date information about Activate benefits, please visit https://aws.amazon.com/activate/