Cloudlytics :  Drawing Meaning out of Your Data Chaos

Cloudlytics is a new big-data service that assists business in analyzing and processing their AWS Cloud Logs. Cloudlytics currently supports Amazon Simple Storage Service (S3), Amazon CloudFront, AWS Elastic Load Balancer, CloudTrail log files & provides billing analytics.

Every startup has an interesting story. Tell us about how Cloudlytics got started?

When the solutions architects at BlazeClan (Cloudlytics’s parent company) were rattling their brains for a cloud migration solution for a U.S. multibillion-dollar company’s product, we noticed that there was a large untapped market in log analytics, specifically, in-depth AWS log analytics. We discovered that transforming such log data into useful information and giving it a proper business analytics context could help ISVs, media, broadcasting, gaming companies, and even e-learning make business decisions that were not possible before. The use cases keep getting more innovative and interesting.

What is the most powerful business lesson you’ve learned as you’ve brought your company to market?

An entrepreneur’s journey is full of unforeseen and untrodden paths. It’s important for founders to try out those paths aggressively to figure out which one works best for them. Here are a few lessons we learned on our more-than-three-year journey:

First, never build a product based on your whims and fancies. Always address your customers’ requirements and validate your product at every stage of construction. The base pillars of the product must be built on genuine customer needs and be able to evolve in a sustainable manner.

Second, when building a web-based product, build the basics of marketing (such as referrals) into your product from day one. There’s nothing better than having your customers refer other customers. Allow your product to do that seamlessly. Adding such things later in the lifecycle is usually an expensive proposition.

Finally, always focus on your core strengths. For us it was technology. For someone else it could be a function or a domain. Find your niche, build a minimal viable product (MVP) around it, and get help to complement your weaknesses.

What is log analysis and why is it important to startups?

Logs are simply raw data about your systems components recorded in log tables. Logs are the best way to monitor the effectiveness of your systems, be they operational logs, web logs, application logs, or even database and content delivery network (CDN) logs. The rich data present in these logs, if analyzed properly, can tell you a lot about your business challenges and opportunities. Effective reporting of your logs can provide precious information about your operating system, apps, and databases.

The ability to be agile is one of the strengths of a startup. Analyzing your log data will help your business react and make quick adjustments to align yourself to your user benefits faster. Here just a couple of the numerous examples available:

Video streaming and gaming companies use log analysis to get answers to those never-ending crucial questions about customer interaction related to popularity and engagement with content. One of the startups I know used inputs from Amazon CloudFront logs very innovatively and started publishing an “engagement factor” for most of their content, which helped them deliver value to their customer.

Another startup I know serves the equity markets, which inherently face unexpected traffic variations. By analyzing their Elastic Load Balancing logs, they were able to reduce the bug-fix cycle, getting to the root cause of a problem and avoiding a delayed launch.

And Cloudlytics is built to help create meaningful data from logs? Can you tell us a little about how it works?

Cloudlytics takes your AWS logs, analyzes them, and, based on our continuous end user interactions, churns out interactive reports. Cloudlytics is 100% powered by AWS, and this big-data analytics tool runs on top of Amazon Elastic MapReduce powered by On-Demand and Spot Instances with a Redis server. This, along with Amazon Redshift, AWS’s data warehouse, handles query processing and helps transform logs into contextual analytics. Running on top of this are business intelligence tools that give out interactive graphs and charts.

Reports contain a suite of all AWS logs like those from Amazon S3, CloudFront, Elastic Load Balancing, and (for all API access logs) AWS CloudTrail. These tell you how your web apps are being accessed across the world through devices and platforms, how your content is faring, and even how to keep your AWS resources healthy and secure.

What is the process is like to implement Cloudlytics?

A pre-requisite for log analysis is that the logs should be enabled for those services. Once this is in place, getting started with Cloudlytics is simple. Once the user registers, we help with a system-generated IAM policy for each service. This IAM policy is then required to be copied to the customer’s AWS account for granting appropriate access. At a click of a button a customer can then select the logs which they want to process.

Once logs are processed, various reports are available under the reporting dashboard. Customers also get the flexibility of inviting their colleagues to view these reports and setting up notifications & email alerts.

Can you share three best practices small businesses should keep in mind when analyzing and storing log data?

Without the proper tools, analyzing logs and finding these insights can be like searching for a needle in a haystack. But with the right practices in place, your logs can become a top source of both operational and business information. Further, holding onto and making log information available to others inside your organization can be incredibly valuable. With that in mind, here are three key points to consider:

• Never lose your log data. Store it in a reliable, safe, and secure manner. Storing logs gives you the ability to do research, look for trends, and make proactive decisions.
• Get data in the best form for easy processing.
• The right transformation tools (for example, IP address transformation) are the key!

What piece of advice would you give to a startup that is trying to get off the ground using AWS?

Startups need to get off the ground quickly and be flexible to adjust to market needs in a very agile way. AWS offers the flexibility to roll out those MVPs in shorter sprints and allows startups to iterate faster. AWS has a plethora of services to help reduce the time to market; using them the right way will also help cut your spending. So before launching on the AWS cloud, make sure you have the right guidance to help you create an optimized cloud road map, both in terms of costs and efforts.

What’s next for Cloudlytics?

We have a lot of things on our plate, and our users keep giving us more fodder brainstorming! We plan to expand offerings to all AWS services and keep up with the innovative launches in the Amazon cloud. We are also actively working on customer feedback for better and more intuitive reporting, so stay tuned! Finally, we have a few unique features for video analytics in the pipeline, but that’s all I can say at this time.