Data Lake POC

2.5 quintillion bytes of data are created each day. As the amount of data being generated continues to grow, the need to store, process, and analyze this data is also increasing. Without an effective process for managing large data sets, organizations cannot accurately identify business value from their data.

Fortunately, data lakes provide the technology that modern businesses need to store data at scale and run in-depth analytics on that data. However, building a data lake is not easy. It can involve numerous manual steps, making the process complex and time-consuming.

With ClearScale’s Data Lake POC, you can quickly create a simple, extensible, data lake using serverless managed services on AWS capable of storing both semi-structured and unstructured data.

Key Activities

-Using managed services on Amazon Web Services (AWS) and best-of-breed third-party operational insight tooling, ClearScale will deliver a data lake POC consisting of one data source for ingestion

-The POC will provide a simple, extensible data lake to unify, inventory, and catalog company data and increase data insight for data democratization, business analytics, and data monetization initiatives

Deliverables

-Data Lake POC deployed to a non-production environment in AWS (single account, single data source)

-Data ingestion once per hour for each source (through Amazon Glue JDBC or other custom ingestion flow)

-10 data transformations

-Data schema definitions

-Data catalog

-Documented architecture

-Working session to demonstrate the end-to-end workflow on the data lake (from data ingestion to data catalog)