A guide to data, analytics, and machine learning (ML) tools for AWS

In terms of sovereignty, ensuring the geographical residency of your data is enforced and automated are key features required.

Key tools for the job

Try Databricks in AWS Marketplace 

Get started with Amazon S3 

Data residency in Amazon S3

• Amazon S3 allows for region bounded buckets, which ensure all data stored in that bucket will reside in a specific geographical location. The cool thing about using Amazon S3 as a landing location for the bulk of your data is its flexibility towards transforming and pushing that data out to other systems. You can use Amazon S3 as the landing zone while leveraging data residency capabilities of other service such as Amazon Relational Database Service (Amazon RDS) and Amazon Elastic Compute Cloud (Amazon EC2), which lie at the core of most applications today, including those running in container orchestrated environments. If you add AWS Local Zones to the mix, you have even more control over the location of your data and its various projections.

Security and Compliance Enhanced Delta Lake with Databricks

• With Databricks you can build a Delta Lake on top of your S3 stored data, and using the Enhanced Security and Compliance Add On you get improved security and controls that help you guarantee your compliance needs are met. This Add On includes hardened host OS, inter-node encryption, and enforced updates, together with compliance security profiles available for HIPAA, IRAP, PCI-DSS, and FedRAMP High and Moderate out of the box.

Amazon Macie for automated sensitive data discovery on S3

• Having your data on S3 also allows you to easily enable Amazon Macie, a service that will continually evaluate, discover, and scan your data on Amazon S3, and gives you a very quick path to automating remediation using Amazon EventBridge and AWS Security Hub.

Start by creating your S3 Buckets in the appropriate regions and setting up your Databricks Workspace to use the buckets with the desired data residency and enabling the Compliance Security Profile. Then configure Amazon Macie to handle automated discovery of sensitive data in the applicable S3 buckets.

Durability and availability are different but usually interrelated concerns. Durability focuses on ensuring data remains intact, complete, and uncorrupted over time, while availability refers to the ability to access data at any point in time. Achieving durability and availability is relatively straightforward when dealing with object storage and using services like Amazon S3, but it becomes particularly challenging when you are working with block data that requires direct and fast access from your workloads, and you need to keep this data replicated across regions for durability as well as available for direct access.

Key tools for the job

Try NetApp Blue XP in AWS Marketplace 

Get started with Amazon FSx for ONTAP

Amazon FSx for ONTAP

• Fast performance, regional block storage for direct access by your compute nodes running Linux, Windows, and MacOS can be achieved using Amazon FSx. With Amazon FSx you can share file storage across Amazon EC2 instances, Amazon EKS nodes and even AWS Lambda functions, with very low latency and incredibly fast performance. Amazon FSx provides out-of-the-box support for multi-Availability Zone (AZ) deployment.

NetApp BlueXP

• With Amazon FSx for NetApp ONTAP and NetApp BlueXP, you get access to effortless cross region replication and data transport capabilities. Backup and cross-region disaster recovery are also readily available, which allows you to centrally manage your entire data estate across AWS regions, including automating through the criteria that you choose to apply to data being moved around or recovered in case of disaster using rules and policies.

You can start by creating your Amazon FSx ONTAP file systems directly in the AWS Console, which will allow you to discover them using NetApp BlueXP or directly create the environment from within BlueXP directly. In both scenarios you will need to grant the correct IAM permissions for the AWS resources to be accessible and managed through BlueXP.

From here, you can configure your high availability architecture, automatic capacity and throughput handling as well as all the other powerful capabilities available with NetApp BlueXP working with FSx volumes.

There are many challenges to storing and using data at scale, particularly when it comes to structured data:

• Infrastructure scale: Getting more storage and compute as your data volume and data access patterns become larger and more demanding without impacting users and ensuring continuous reliability and performance.

• Data import and integration: The data that lives in data warehouses comes from many sources, and changes over time. Reducing the effort in importing data as well as integrating different data sources directly for consolidate query access is a usual requirement of data at scale.

• Access, Analysis and Query Processing: Providing data engineers with the right tools and intuitive interfaces to work with the data towards gaining insights from it is a definite requirement to truly extract the value of data at scale.

Key tools for the job

Try Snowflake in AWS Marketplace 

Get started with Amazon S3

Snowflake

• Providing instantaneous capacity to handle any scale and making features available such as data stages and external tables, Snowflake provides out of the box capabilities to solve for all the challenges of working with data at scale we’ve discussed. Snowflake also provides an intuitive and familiar interface by supporting standard ANSI SQL to perform most operations. Additionally, Snowflake enables multi-region architectures off the shelf, by supporting multi-region deployment, data sharing and replication.

Amazon S3

• S3 Buckets can be directly used within Snowflake for data staging, which allows to load and unload data into your Snowflake deployment without having to build and maintain any complex data pipeline. You can also directly integrate buckets as external sources which enable you to query data stored in S3 without having to import it into the Snowflake data catalog at all.

Start by creating and loading data in your S3 buckets. Once you have your buckets created, you can use Snowflake storage integrations to allow it to read and write data from them. Storage integrations allow for authentication and authorization using AWS native IAM, and without using less secure secret keys and access tokens.

Data can be generated from many different sources, including edge devices, IoT clients, change data capture, and event driven systems architectures. Working with these data streams requires high-performance processing capabilities as well as dynamic and flexible mechanisms for ordering, deduplicating, and modifying that data in flight.

Data stream throughput can change very rapidly, and due to network and other conditions result in spiky behavior that requires the right mechanisms to buffer, throttle, and retry messages in those streams as necessary.

There will be an impact in data storage as well, as data will require to be persisted somewhere both while in transit as well as when directed to its final location. Processing the data stream must account for the eventual impact of scaling activities in target data storage systems, as well as be capable of handling varying volumes of data persisted temporarily during processing, staging, and prior to final storage.
 

Key tools for the job

Try Confluent Cloud in AWS Marketplace 

Get started with Amazon EventBridge 

Confluent Kafka

• Offers incredibly powerful and unique capabilities built on top of the battle-proven Apache Kafka. Addressing the challenges outlined above, Confluent offers automated cluster scaling, dramatically reducing the complexity of handling data streams at scale, as well as features towards automated data deduplication. Relying on Confluent Kafka you get various message delivery, receipt, and ordering guarantees out of the box, which gives you a wide array of solutions when building your cloud-native architecture. To handle challenges around scale, Confluent Kafka offers infinite storage and infinite retention capabilities, which makes storage concerns a thing of the past.

Amazon EventBridge

• Confluent Kafka has native integrations with Amazon EventBridge, which allows you to send messages from one or more Kafka topics, including customized mapping and other handy capabilities to a specified event bus. Amazon EventBridge can handle infinite load with fully serverless scaling capabilities, and offers native integrations to most of AWS services, as well as custom capabilities to integrate with third-party services.

Start by creating a Confluent Kafka cluster, and configuring all your data sources to stream data to it, using Kafka client libraries. Once your data is flowing into Confluent, use the EventBridge Sink connector to send your records to one or more EventBridge buses, which will allow you to configure any downstream processing or handling using the huge array of AWS services supported natively or through SDK by EventBridge.

Data teams need to empower non-technical users to create and manage data pipelines without requiring coding skills. Organizations require the ability to rapidly prototype ETL solutions for quick validation before investing in full-scale implementations. Businesses struggle to maintain and update data pipelines efficiently as data schemas and requirements evolve over time.

Key tools for the job

Try Matillion in AWS Marketplace 

Get started with Amazon Redshift 

• Matillion Data Productivity Cloud

  • Building data pipelines with no code is one of the most powerful capabilities of Matillion, providing access to move and work with data to more diverse users, as well as reducing the effort in maintaining those pipelines over time. Matillion is powered by PipelineOS, which provides a cloud-native approach to scaling the underling infrastructure that runs the pipelines by using stateless agents capable of scaling to work in parallel and process massive amounts of data in a short period of time. Matillion also offers native and seamless integration with Amazon Redshift, dramatically simplifying storing processed and optimized data in a flexible and highly scalable repository.

  
  Amazon Redshift

  • Offering both configurable as well as machine learning driven fully automated performance tuning, Amazon Redshift serves as an ideal candidate to store voluminous data generated by your data pipelines. The capacity of Amazon Redshift to scale up to meet demands can easily adjust to the requirements of a dynamic and elastic range of data pipelines and serves as a flexible landing location where you can analyze, visualize and, if necessary, project that data to other target systems.
    
    Start by creating your Redshift cluster in AWS, and create a user for Matillion to have the necessary privileges to create databases and manage their data. Also, get the Redshift endpoint and port, which you will need when creating your Matillion pipelines.

  Now you can start using Matillion’s wide range of capabilities to consume, extract or transform data from many different sources, including RDS data querying, which automatically stages data to S3, or any of the other data transfer sources including HDFS, SFTP, and many external third party services.

Companies need to create and manage complex data pipelines that integrate processes beyond simple data movement and transformation.

Bringing pipelines to production requires promotion across environments, data quality testing and many other processes that make teams require a solution that allows them to manage and develop data pipelines as code for better version control and collaboration.

Maintaining enterprise grade security and meeting demanding reliability and availability requirements are common challenges faced by organizations as they deliver pipelines into production and orchestrate data workflows across diverse cloud environments.

Key tools for the job

Try Astro by Astronomer 

Get started with Amazon S3 

• Astro by Astronomer

  • As an enterprise-grade platform built on Apache Airflow®, Astro by Astronomer provides a unified solution for orchestrating, monitoring, and scaling data workflows across diverse cloud environments. It tackles the difficulties of maintaining reliability and efficiency in data operations by offering elastic scalability, high availability, and advanced security features. You can work with Astro using the Astro UI, Astro CLI or Astro API, to accelerate pipeline development and deployment capabilities (1500+ connectors) to ensure seamless connectivity across various data sources and services.

  Start by signing up for Astro by Astronomer in AWS Marketplace. Create your first Deployment using either a pre-built template or your own custom project. From there, you can immediately begin developing, deploying, and running your data pipelines in a production-ready Airflow environment integrated with your AWS data storage services.

Displaying insights on data that are intuitive, easy to understand, and clearly communicate the desired information to the viewer requires a combination of data as well as presentation and user experience skills, both indispensable to allowing consumers of those insights to get true value from the data.

Allowing data scientists and business intelligence engineers to deliver insights to end users from data, requires tooling that provide a wide and dynamic range of ways to present and visualize those insights , as well as seamless query and integration capabilities with the various systems in which the data that you want to present to users can live.

Consumers of these dashboards need to have the flexibility to customize and navigate them intuitively, while ensuring that access to data is only available to the right people, thus requiring strong role-based access control and granular authorization controls to data attributes.
 

Key tools for the job

Try Grafana Cloud in AWS Marketplace 

Try AWS Identity and Access Management 

Grafana

• With over 26 different ways to present data, Grafana provides a powerful framework for business intelligence engineers, data scientists, and other engineers working to extract and share insights on data to build intuitive, simple yet comprehensive dashboards, while allowing for integration of many data sources and providing a common interface to query and inspect the data as the dashboard is being built. Grafana Cloud also provides enterprise-level capabilities, such as SSO and IAM integrations, providing robust ways to build security and access controls between your dashboard consumers and data.

AWS Identity and Access Management

• With AWS IAM you get access to incredibly granular controls and multiple ways to group, assign and managed those permissions to individuals as well as groups of users. These access controls interface directly with all AWS services and can be used to control access to dashboards and specific components within them.

Once you have subscribed to your Grafana Cloud account, simply use Grafana Data Sources to connect and consume data from your AWS resources, including Amazon CloudWatch, Amazon RDS. Amazon ElasticSearch Services and many others. Then use any of the built in Grafana Dashboards to visualize the health and status of each service, or build your own using Grafana’s query, exploration, visualization and alerting capabilities.

Traditional SQL-based systems struggle with representing and querying intricate data relationships. As data structures become more interconnected, SQL queries become increasingly complex, with multiple joins and subqueries, making them difficult to write, maintain, and optimize. This complexity often leads to reduced performance and increased potential for errors.

Traditional relational database schemas are often too rigid to efficiently handle complex, highly connected data structures. This inflexibility can limit the ability to uncover hidden patterns, relationships, and dependencies in large datasets, potentially constraining the depth and accuracy of data analytics insights.

Organizations face challenges in providing a unified data platform that caters to different roles such as developers and data scientists. Developers require a user-interactive, low-latency solution that easily integrates with applications, while data scientists need a platform for large-scale data analysis and interpretation. This challenge is compounded when dealing with complex data relationships and the need for real-time analytics, making it difficult to meet diverse needs without compromising on performance or analytical capabilities.

Key tools for the job

Try Neo4j AuraDB in AWS Marketplace 

Try Amazon SageMaker 

Neo4j AuraDB Professional

• Neo4j offers a Neo4j AuraDB, a fully managed graph database platform, that enables developers and data scientists to build scalable, AI-driven applications and perform complex data analysis. It allows for rapid development of intelligent, context-driven applications with real-time insights and built-in developer tools. For data scientists, Neo4j AuraDB provides a comprehensive data science as a service solution, integrating graph database capabilities with machine learning workflows. It offers a library of 65+ pre-tuned algorithms, data preparation techniques, and graph-native ML, enabling the analysis of relationships in big data and the creation of predictive models at scale.

• Neo4j Graph Database enables developers and data scientists to uncover hidden patterns and relationships in large datasets, addressing data quality issues through robust preparation and governance processes. Neo4j's unique property graph model facilitates real-time analytics on highly connected data, empowering businesses to make timely decisions in dynamic environments. For those seeking a cloud-native solution, Neo4j AuraDB offers the same capabilities in a fully managed format, allowing for rapid development of intelligent, context-driven applications with built-in developer tools.

To get started with Neo4j, sign up for Neo4j AuraDB Professional Pay-as-you-go in AWS Marketplace. Next, model your data as a graph and import your existing data using Neo4j's import tools. Then, start querying your graph using Cypher in the Neo4j Browser or your preferred development environment. Finally, integrate Neo4j with your chosen programming language to build graph-powered applications, leveraging Neo4j's driver libraries and APIs for seamless connectivity and data manipulation.

A lot has been learned from DevOps about accelerating the delivery of software products. Given the rapid rise of new tools, frameworks, and methodologies to build and deploy machine learning models, it becomes a requirement to apply many of those learnings to the workflows associated to developing AI capabilities.

Data lies at the center of machine learning development. It is therefore indispensable to integrate data into the development workflow, something particularly challenging due to the nature of data: voluminous, constantly changing, and with a limited shelf life.

Data is used for training, fine tuning, and enriching the context that machine learning models have access to. Many of these activities are time consuming and require considerable compute capacity . Optimizing workflows and automation to reduce the friction and time required to perform these processes is critical to successful machine learning development projects.

Versioning machine learning model artifacts and providing a testing framework to validate their performance and precision is also critical.
 

Key tools for the job

Try Databricks in AWS Marketplace 

Get started with Amazon SageMaker 

Databricks Data Intelligence Platform

• Databricks offers a strong set of capabilities in their MLFlow offering that manage the entire lifecycle of machine learning development workflows, from data preparation to training and notebook development to model deployment across a varied range of integrated model runtime environments.
  
  The Databricks Unity Catalog feature allows for storage of ML models, while enabling access controls, auditing, model discovery, and many other powerful capabilities when working with evolving versions of machine- learning projects.

Amazon SageMaker

• Databricks offers direct integration with Amazon SageMaker, allowing for direct and seamless deployment of models developed and managed using Databricks set of capabilities, while taking advantage of the powerful features built into Amazon SageMaker, including customizable scalability, API endpoint deployment and lifecycle management, and many other features required to put models into production environments (for example, tight IAM controls and direct access to data stored and managed within other AWS services).

You can use Databricks for the development and exploration of your model, and SageMaker to deploy it and make it available to users in a production ready capacity. Start by using Databricks by creating a workspace to work on the development of your model. Once you have a model ready to deploy, use the MLFlow API to deploy it to SageMaker, it is as simple as that.

Prototyping and developing RAG-based large language model (LLM) implementations can be a complex task that requires skills around a wide range of new tools and frameworks. Providing tooling to developers that simplify their development processes and exploration without adding new layers of complexity is a key concern to teams starting in their RAG development journey.

Familiarity with the environment, data structures, and interfaces are relevant requirements towards accelerating the skilling up of developers to explore machine learning model development.

Integrating existing data sets and making them available as vector embeddings for LLMs through RAG in a seamless and intuitive way provides and accelerated foundation for developers to explore these new development paradigms.

Key tools for the job

Try MongoDB Atlas in AWS Marketplace 

Get started with Amazon Bedrock 

Try Datasaur in AWS Marketplace 

Get started with Amazon Textract 

MongoDB Atlas

• Developers are very familiar with MongoDB as a data repository for JSON documents, arguably one of the most common NoSQL data formats in use today, which also means a lot of data is already stored in these databases. MongoDB can be directly integrated as an Amazon Bedrock knowledge base , and it can serve both as document storage as well as storage for all the corresponding vector embeddings that will be used by the RAG implementation on top of your foundational LLM of choice. MongoDB provides an intuitive and familiar interface for developers to work with document and vector data, while enabling them to deploy RAG-based solutions with little effort by leveraging Amazon Bedrock’s fully managed and serverless access to foundational LLMs.

Amazon Bedrock

• Amazon Bedrock is a fully managed serverless offering that enables seamless access to a wide variety of high-performing foundational models. It also offers powerful and intuitive integration capabilities using Amazon Bedrock Agents and Amazon Bedrock Knowledge Bases to enrich and extend the context available to LLMs when generating responses to users.

Start by creating your MongoDB Atlas cluster and storing the credentials to access it in AWS Secrets Manager. Store the data you want to use for your Knowledge base in an S3 bucket, this can be anything from PDF files to other documents. Now, create an Atlas Vector Search Index in your Atlas Cluster. Now you are ready to configure your Knowledge Base in Amazon Bedrock, adding the Atlas vector store you created and the S3 bucket holding your data in the Knowledge Base configuration. Finally create and Agent that uses this knowledge base and you’ll be ready to use this for as RAG source for your LLM.

• Datasaur LLM Labs and NLP Labeling

  • Datasaur LLM Labs leverages Amazon SageMaker JumpStart to provide access to 80+ popular foundational models. It features a sandbox environment for LLM experimentation, allowing users to customize prompt templates, adjust hyperparameters, and evaluate multiple model responses. Datasaur's NLP Labeling solution provides a comprehensive interface for text, document, and audio data annotation. It supports various NLP tasks from entity recognition to advanced techniques like coreference resolution and dependency parsing. The platform offers automation features such as integration with popular NLP libraries, weakly supervised learning capabilities, and connections to AWS services like Amazon S3 and Amazon Textract. Both solutions work synergistically, allowing users to leverage LLMs for initial data labeling, incorporate reinforcement learning with human feedback (RLHF), and even set up LLM distillation workflows for more efficient, task-specific models.

  Amazon Textract

  • Amazon Textract is a machine learning service that automatically extracts text, handwriting, layout elements, and data from scanned documents. It goes beyond simple optical character recognition (OCR) to identify, understand, and extract specific data from various document types, enabling businesses to quickly automate document processing and extract valuable information from forms, invoices, and receipts.

  Start by signing up for Datasaur LLM Labs in AWS Marketplace. Next, create your first project by importing your prepared dataset into the LLM Labs interface. Then, begin fine-tuning your language models using LLM Labs' comprehensive features, including prompt engineering tools and team collaboration capabilities. Finally, deploy your fine-tuned models through the platform's deployment options and integrate them into your applications using LLM Labs' APIs and SDKs.

Connecting diverse applications and data sources across cloud and on-premises environments can be complex and time-consuming, often requiring specialized coding skills.

Ensuring real-time data synchronization and maintaining data integrity across multiple systems as business processes evolve is challenging.

Managing and securing APIs for internal and external consumption while maintaining scalability and performance can be difficult, especially as the number of integrations grows.

Key tools for the job

Try Boomi in AWS Marketplace 

Get started with Amazon RDS

Boomi AtomSphere Platform

• Boomi simplifies the process of creating and managing integrations across diverse systems with its low-code development environment and extensive library of pre-built connectors. The platform's intuitive drag-and-drop interface enables both technical and non-technical users to build complex integrations quickly. Boomi's Master Data Hub ensures data consistency and integrity across integrated systems, while its API Management capabilities allow for secure and scalable API creation and governance. The platform's cloud-native architecture provides the flexibility to run integrations anywhere - in the cloud, on-premises, or at the edge - meeting various deployment needs and compliance requirements.

Amazon service integration

• Boomi seamlessly integrates with various AWS services, including Amazon S3, Amazon RDS, and Amazon Redshift, allowing for efficient data movement and transformation between AWS and other environments. This integration capability enables organizations to leverage the power of AWS alongside their existing on-premises or other cloud-based systems, facilitating hybrid and multi-cloud architectures.

Sign up for Boomi Pay-As-You-Go in AWS Marketplace to access a flexible, low-cost integration platform. Quickly connect your applications, data, and APIs using pre-built connectors, visual workflows, and advanced features like Master Data Hub and API management. Scale up or down as needed with no long-term commitments, and leverage Boomi's extensive support resources to drive your digital transformation initiatives.