NTT Docomo Rebuilds Infrastructure for Lemino Streaming Service Launch, Facilitating Millions of Concurrent Live Streams

NTT Docomo Lemino is a video streaming service created as an updated version of dTV, which NTT had been providing since 2015. “The user interface has been drastically revamped, and it was reborn as a service with enhanced communication, including features like searching for content using emotion keywords such as ‘heart pounding’ and ‘excitement’ and searching for content evaluated by your followers,” says Keiichiro Miyasato, video service Senior Manager at the Product Design Department 1 Smart Life Business company at NTT Docomo.

When launching Lemino, the team faced an obstacle with the live streaming service infrastructure and the application backend system. Existing live streaming was operated in an on-premises environment, so the infrastructure had difficulty increasing the number of live streams. The backend of the existing application was already on AWS services such as Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Relational Database Service (Amazon RDS); however, the architecture faced challenges during large-scale streaming in terms of operation workload and cost due to scaling interruptions. Therefore, the company decided to rebuild a video streaming infrastructure that would be compatible with large-scale live streaming and achieved the service stability service. 

“While dTV had millions of users, to accommodate an influx of NTT Docomo’s d-account users to the new service, we aimed to build a streaming platform that could support tens of millions of users. That’s why we decided to rebuild using AWS managed services,” says Miyasato. 

NTT Docomo adopted AWS Media Services—which help to create digital content and build live and on-demand video workflows—as its live streaming service infrastructure. “For dTV, operations were complicated, and it was difficult to isolate issues whenever a failure occurred, but integration into AWS has helped us to achieve comprehensive operation and reduce the operational workload,” says Minenobu Ohashi, manager of Product Design Department 1 Smart Life Business company at NTT Docomo. “Also, we used the AWS APIs to develop a mechanism that lets the team in charge of the live program easily deploy AWS Media Services for building a streaming environment common to the programs. It freed the engineers from deployment, and multiple concurrent live streams are supported.”

The engineers are not always able to handle small-scale live streams at the venue. As a solution to this, NTT Docomo adopted AWS Elemental Link, streaming dedicated hardware which offers a configuration-free, cost-efficient way to stably transfer high quality video to AWS so that on-the-go shooting crew can control video encoding. This has optimized the cost of a single live performance, because the engineers don’t have to visit the live venue.

The solution uses AWS Elemental MediaConnect, AWS Elemental MediaLive, and AWS Elemental MediaPackage. For large-scale streaming, NTT Docomo employed a redundant design with multiple Availability Zones. Additionally, Lemino could switch the streaming function to multiple regions, such as overseas regions or the Osaka region, in case of failures in the Tokyo region, providing further redundancy. The service uses a multi-content delivery network (CDN) configuration, which facilitates the addition of other CDNs as needed.

This scalable architecture can efficiently handle large-scale streaming demands. For scalability and cost-efficiency, the service splits database functions across Amazon Aurora, which provides unparalleled high-performance and availability at global scale with full MySQL and PostgreSQL compatibility; Amazon DynamoDB, which is a serverless, NoSQL database service; Amazon DynamoDB Accelerator (DAX), which is s a fully managed, highly available caching service built for Amazon DynamoDB; and Amazon OpenSearch Service, which is an open-source, distributed search and analytics suite derived from Elasticsearch, instead of setting up all functionality on Amazon RDS. Amazon DynamoDB is used as the backend for application containers with tens of thousands of requests per second, while Amazon OpenSearch Service is used for content data and user communication data to provide search function. “In the past, connection limits became bottlenecks when experiencing a rapid increase in access,” says Akio Kudo, manager of Product Design Department 1 Smart Life Business company at NTT Docomo. “We adopted Amazon DynamoDB, which is fully managed, for processes that need to be robust for request spikes.”

With its completely revamped architecture, Lemino has live streamed an array of high-profile content and performed stably at programs with large-scale access. Praised for its rich content, Lemino reached 5 million monthly active users within 3 months of launching the service, a nearly sevenfold increase.

“The ability to provide stable video streaming services with ample resource capacity to handle millions of accesses is the greatest advantage. With our conventional architecture, scaling resources was a time-consuming and gradual process due to limitations. However, our new architecture allows for rapid resource expansion and contraction, resulting in improved cost efficiency as we can dynamically adapt to demand within a short time frame,” says Miyasato. 

The company succeeded in live streaming 30 channels at the maximum, and it has a plan to expand this to a maximum of 50 channels in 2024, expecting a larger-scale live streaming environment.

“We provide live streaming of multiple types of content every day, but thanks to AWS Media Services and fully managed services provided by AWS, the workload on the engineers is about one-tenth of what it used to be,” says Ohashi. “We are now able to focus on developing services that generate high added value. AWS is familiar with our streaming infrastructure after years of support, so we are grateful to be able to appropriate technical support from Solutions Architects (SAs), risk mitigation support and case support for large-scale events from their Technical Account Managers (TAMs).”

Because Lemino has drastically improved the availability of the backend, users are virtually unaffected by application errors. “In the new environment, even if some tasks go down, the overall service is no longer affected, and we’ve reduced the operational load,” says Kudo.

The next step for Lemino is to insert targeted ads in live streams to generate more revenue. With that in mind, the company is currently implementing an ad-insertion function using AWS Elemental MediaTailor, which is a channel assembly and personalized ad-insertion service for video providers.

“We provide a video service that pleases millions of users with improved architecture,” says Miyasato. “We look forward to continued technical support from AWS.”