Let’s Architect! Architecting for big data workloads

Big data is often defined by 3 Vs: greater variety, volumes, and velocity. Because of the three Vs, big data poses data management challenges that cannot be solved with traditional databases. Not only that, but trying to overcome these issues can lead to scaling problems, bottlenecks, and spiraling costs.

To help with this, you need to look at the whole data management pipeline. Don’t worry, AWS offer many tools and best practices to help you architect better for these challenges. In this post, we share insights into how to build and manage big data pipelines in your architecture.

Everything You Need to Know About Big Data: From Architectural Principles to Best Practices

There are so many tools, frameworks, and services for big data. It can be overwhelming to know where to start what best practices to apply.

This video distills down good practice and good architecture and principles for big data systems into easy topics and guidance.

Manos Samatas presenting the mental models for big data architectures

AWS workshops for big data

This hands-on practice will show you what’s possible for big data services on AWS.

If you are a builder, this AWS workshop catalog includes  several courses on big data and analytics. These resources provide new ideas and how to realize them in practice.

AWS workshops can help you learn the cloud services and recommended architectural patterns

Securely share your data across AWS accounts using AWS Lake Formation

It’s very common to share data stored across organizations or business units, but sharing data often comes with security risks.

This blog post explains how to share data across accounts via AWS Lake Formation in a secure and controlled manner, so your data is never exposed to the wrong people.

This diagram illustrates the architecture for cross-account access

How Amazon leverages AWS to deliver analytics at enterprise scale

Amazon.com is a customer of AWS like any other customer. But, as you can imagine, Amazon.com has very large and very complex datasets with tens of thousands of transactions at any one time.

This video goes through how Amazon.com uses AWS technologies to run their business successfully, and how you can add the same architectures and principles for yours.

Data warehouse architectures can be used to run queries on large amounts of data from different data sources

See you next time!

Thanks for joining our discussion on big data architecture! See you in two weeks for more architecture best practices and guidance.

Other posts in this series

• Let’s Architect! Architecting for Sustainability
• Let’s Architect! Architecting for Machine Learning
• Let’s Architect! Architecting for Security
• Let’s Architect! Tools for Cloud Architects
• Let’s Architect! Architecting for Blockchain
• Let’s Architect! Architecting microservices with containers
• Let’s Architect! Using open-source technologies on AWS
• Let’s Architect! Serverless architecture on AWS
• Let’s Architect! Creating resilient architecture
• Let’s Architect! Architecting for governance and management
• Let’s Architect! Architecting for front end
• Let’s Architect! Understanding the build versus buy dilemma
• Let’s Architect! Architecting for DevOps
• Let’s Architect! Designing Well-Architected systems

Looking for more architecture content?

AWS Architecture Center provides reference architecture diagrams, vetted architecture solutions, Well-Architected best practices, patterns, icons, and more!