As a startup company in the liveliest segment of the new retail industry, we initially focused heavily on the time factor, knowing that faster product and service launches would be critical to our early success. After our product architecture stabilized, we began to also take costs and efficiency into consideration. Our growth and innovation would not be possible without AWS, whose standards of service are higher than the competition. Whether we look at it in terms of stability, security, performance, efficiency, or cost, AWS has brought us a compelling value and user experience.
Beijing Chaomeng International Culture Co., Ltd. (“CM Data”) specializes in using big data to help drive the decision-making of retailers – such as convenience stores, supermarket chains, and community grocery stores – through data services and solutions including API data interfaces, BI-enhanced decision-making platform, and AI-based product selection platform. Its customers include such domestic retailers as Chaoshifa, Tiantian Bianli, Lecoo, Everyday, WeeCan, and Six Wishes. Used by over 10,000 outlets (including 5,000 mom-and-pop stores), or 18% of the franchised convenience stores in China, its BI-enhanced decision-making products currently process tens of millions of transactions each day.
CM Data’s business model is to provide a one-stop data management and analytic platform – encompassing data collection, analysis and mining and data-supported decision-making – to convenience store chains. This complete decision-making system, which is powered by big data, helps retailers build a data-driven management system and develop a unique marketing strategy that works best for them.
Convenience stores are one of CM Data’s main clients. Typically, the manager of a convenience store would need to look through a catalogue of thousands of SKUs to find the products to be replenished. This traditional, manual ordering process is both slow and error-prone, often resulting in the shortage of best-selling products and an overflow of slow-moving products. “Our enhanced decision-making service, which tallies and analyzes the profile of nearby potential customers through dozens of dimensions including age group, gender, wealth, and nearby housing price, is just the solution to this problem and can help store managers optimize their inventory to maximize profit,” says Li Jianhao, CTO of CM Data. “Therefore, how to acquire data in a rapid, reliable, and efficient manner; process and analyze them in real-time; and accurately and appropriately present relevant conclusions, will define our productivity and competitive edge.”
Due to the pace of its business growth, innovation, and iteration of products and services, CM Data’s IT infrastructure began to hit a bottleneck. “Products and services are the lifeblood of a startup like us,” says Li, “We need to concentrate all our efforts on improving our products and providing more valuable services to fuel our growth. Our data services are created by the cloud and made by the cloud. This means our IT system must be highly stable and available, and feature an expandable, flexible, and secure setup with a developer-friendly test environment. The key challenge faced by the system is to guarantee an efficient, stable, secure, and cost-efficient business operation while also reducing operations and maintenance burdens. A simplified, automated, and seamless integrated IT back-ends will allow our teams to focus more on business development and innovation.”
CM Data used to rely on the cloud services from other vendors, but switched to AWS when it launched the receipt scanning project which connected directly to its customer database. Now, all its core businesses are running entirely on the AWS cloud. “Previously, some of the services offered by other vendors were unstable, even resulting in loss of stored objects, for example,” Li explains, “for customers who use our service, their products may have an exceedingly long lifecycle, which requires greater data security and system stability on our part. Not only is AWS more stable than the other solutions, but it also provides a wide range of hosted services and powerful tools. In the long run, AWS can more readily adapt to our business development needs and may contribute significantly to our growth.”
CM Data offers data analysis and mining solutions to the retail industry. Each day, the company needs to process tens of billions of records, but its business team had no prior experience in building and using a Hadoop analytics platform. After Li joined the company, he helped it build the new environment in just over a week using AWS and research and deploy Amazon EMR in less than three weeks. “In terms of business, our priorities are stability and speed. The forward-looking cloud services of AWS can help us build an agile development team and bring unique products and services to the market faster and more efficiently,” says Li. “In terms of technology, we put heavy emphasis on the level of maturity of the platform, the range of services on offer, how extensively open-source frameworks are supported, and, of course, security.”
Amazon Simple Storage Service (Amazon S3), for example, can be used as a data lake. Other solutions would have a hard time to cope with use cases that require high storage capacity and the sharing of a rapidly increasing dataset. “Amazon S3 has helped us solve many data flow and processing challenges. Its high durability ensures that our customers will never need to worry about losing their invaluable historical data.” Li then adds, “Another reason for choosing AWS is its developer-friendliness. With AWS, we can complete product development and iteration more rapidly and easily to support business growth; our development time for each project has been reduced by 2-3 months on average. These were our key considerations when we chose AWS.”
Figure 1 illustrates CM Data’s big data analytics platform, achieved on AWS using Amazon Elastic Compute Cloud (Amazon EC2), Amazon S3, Amazon Relational Database Service (Amazon RDS), Amazon Redshift, Amazon DynamoDB, Amazon Elastic Container Service (Amazon ECS), Amazon EMR, and AWS Lambda, among others.
Figure 1: CM Data’s Big Data Analytics Platform on AWS
Since CM Data switched to AWS in June 2016 for the receipt scanning project, the efficiency of its IT infrastructure has risen steadily, contributing tremendously to cost reduction and business innovation.
AWS cloud allows CM Data to significantly lower its upkeep and maintenance cost and management workload, while enabling the automatic, flexible scaling of services. “These cost-savings come in two forms: time and manpower. Moreover, AWS has also made the initial fixed investment for IT hardware – traditionally a substantial figure – a thing of the past,” says Li. “Services like Auto Scaling and Lambda especially has helped adapt our system to fluctuating and highly concurrent workloads, which further reduces our AWS expenses. In addition, Security Group and other security services ensure our system on AWS is well-protected. AWS has done much of the heavy lifting required for running an IT system, allowing us to focus more on business innovation.”
With CM Data’s expanding customer base comes the need to process tens of billions of data points – up from hundreds of millions – and display results in milliseconds. As one of the first companies in China to use Lambda, CM Data is able to keep pace with this dramatic business growth with finesse. “Data flow and transformation was once a disjointed process in which data had to be fully extracted before being processed. Lambda enables us to analyze data as they are being extracted. All extractions are now done in parallel in Lambda, while the subsequent cleaning and analysis are handled by Amazon EMR. This setup has reduced the time to process a task from 6 hours to 15 minutes. We are now restructuring our data flow so that post-extraction data processing can also leverage the power of Lambda,” says Li excitedly, “speed is very important to us, for it determines whether we can display analysis reports in virtually real-time. We process all our data at night so that store managers, supervisors, stockers, and chain managers can readily access our visualized analyses and assisted decision-making services the first thing in the morning. AWS has allowed our customers to enjoy a one-of-a-kind user experience.”
“Our current strategy is to go ‘all in’ on AWS, because this will give us ever-increasing efficiency at an ever-lowering cost,” says Li. “For example, Lambda has naturally reduced our use of Amazon EC2, and its pay-by-request fee structure has brought us significant cost savings. In addition, because codes only need to be written once, with subsequent re-scheduling and triggers being executed automatically, our development team can save time and effort from the reduced code complexity and re-assign our resources to more productive areas. We believe we can make the best products on the market in terms of ease-of-use and feature set, which is only made possible by the strength of AWS cloud services.”
Learn more about AWS big data solutions.