What is Data Ingestion?

Data ingestion and approaches vary depending on the data's volume, velocity, and use case.

Batch data ingestion

Batch ingestion tools collect data over a designated period, ingesting a group of multiple data entries all at once. They are typically set up to retrieve data at scheduled intervals, such as end-of-day, weekend, or end of month. For example, image editing software could automatically upload all edited images to the cloud at the end of the day.

Processing data in large batches can be a fast process or a slow process if it involves large amounts of data. If it’s a slow transfer and there are errors, restarting the batch can be expensive and complex. Engineers who use batch processing create fault-tolerant pipelines that allow them to begin from where the batch was last interrupted.

This approach works best when you want to analyze historical data or when timing is not relevant. For ingesting near-real-time or real-time data, one of the following methods will often be preferable.

Streaming data ingestion

Streaming data ingestion tools collect data as soon as it is generated, such as when ingesting data from IoT sensors that take continuous readings. While streaming ensures access to the most recent data, it can be resource-intensive. Data engineers must handle system or network errors and network lag, which can cause data loss and create gaps in the data stream.

There are two approaches to streaming data ingestion.

Pull-based ingestion

The ingestion tool queries sources and performs data extraction. It may do this continuously or at preset intervals.

Push-based ingestion

The data source pushes the data to the ingestion tool as soon as it generates new information.

Micro-batch ingestion

Micro-batch data ingestion divides continuous data streams into smaller, more manageable chunks called discretized streams. This approach balances the advantages of batch and streaming ingestion. It is ideal for scenarios where real-time processing is desired, but full streaming is too resource-intensive. However, micro-batching still introduces some delay compared to pure streaming ingestion.

Micro batch processing is a cost-effective way to gain near real-time data ingestion without paying the higher costs associated with streaming.

Event-driven ingestion

This is a specialized form of push-based ingestion. Event-driven systems ingest data when a specific event or trigger occurs rather than continuously or at set intervals. This approach is commonly used for applications like order processing, customer notifications, and system monitoring. This method reduces unnecessary data movement and optimizes resource usage by only ingesting data when required. However, effective functioning relies on well-defined event triggers and event-handling mechanisms.

Change data capture

Change data capture (CDC) systems are a type of event-based ingestion commonly used for database replication, incremental data warehousing, and synchronization between distributed systems. The data ingestion tool ingests only the changes made to a database rather than transferring entire datasets. By monitoring transaction log events, CDC identifies inserts, updates, and deletes, propagating them to other systems in near real time. CDC minimizes data transfer costs and improves efficiency, but requires support from the underlying database system and may introduce some processing overhead.

Here are some challenges that organizations should consider when ingesting data.

Scale

Scaling data ingestion systems is challenging for organizations due to the volume of data, and when data velocity increases over time.

Horizontal and vertical scaling

Organizations use two main scaling strategies. Horizontal scaling involves distributing ingestion workloads across multiple nodes. It requires efficient load balancing and coordination to prevent bottlenecks. Vertical scaling relies on increasing processing power within a single node, which can be easier to engineer, but is limited by the processing power of the node. A key challenge here is ensuring that the ingestion pipeline can handle an increasing volume of data without causing delays or system failures.

To overcome scaling challenges, you can use Amazon Kinesis Data Streams for real-time data ingestion with horizontal scaling. Alternatively, Amazon EMR allows users to easily run and scale Apache Spark, Trino, and other big data workloads.

Serverless architectures

Serverless pipelines are on-demand data ingestion architectures that do not require instance configuration and deployment. Serverless architectures are best suited to variable data ingestion patterns or event-driven ingestion.

For example, serverless ingestion pipelines on AWS can be built with Amazon Data Firehose and AWS Lambda.

Security

Security and compliance are critical concerns during data ingestion, especially when dealing with sensitive information. Organizations must comply with data privacy regulations that impose strict requirements on collecting, transmitting, and storing data.

Some best practices for data security during ingestion include:

• Data encryption in transit and at rest
• Access controls and authentication mechanisms
• Data masking and anonymization techniques to protect personally identifiable information (PII)

To help protect data security during ingestion on AWS, you can use services such as:

• Amazon Macie to discover sensitive data using machine learning and pattern matching
• AWS Key Management Service to encrypt data across your AWS workloads
• AWS PrivateLink for connectivity between Amazon Virtual Private Clouds (VPCs) and AWS services without exposing data to the internet.

Network reliability

Network disruptions, API failures, and inconsistent data availability can disrupt the data ingestion process. These events create challenges like data corruption. Data overloading from any one source can result in potential data loss or temporarily slow down systems like your data warehouses. Adaptive throttling may be necessary to manage spikes in data flow. Backpressure management allows the data ingestion tool to handle incoming data at a rate matching its processing capacity.

Retrying or reattempting to process failed data is another error-handling strategy. The data ingestion tool sends resend requests to the source when it identifies corrupt or missing data. Retrying increases accuracy but may impact anticipated throughput and latency.

To implement automated retries on AWS, you can create your own workflows using AWS Step Functions, whereas Amazon Kinesis offers configurable policies and processes for managing inbound data flow.

Data quality

When data arrives in the data ingestion pipeline from various sources, there is no guarantee that it will be in a consistent format applicable to the organization. Raw data sources may contain missing values, incorrect data formats, and schema mismatches. This is especially the case when working with unstructured data, as the lack of uniformity adds layers of additional interaction and cleaning.

Data ingestion tools typically include data quality checks and implement methods to validate, clean, and standardize the data. Automated deduplication, schema enforcement, and AI-driven anomaly detection can help identify and correct errors before they propagate further into the data pipeline.

Data quality tools on AWS include AWS Glue Data Quality for quality rules and automation, and Amazon DataZone for data cataloging and governance.

AWS provides services and capabilities to ingest different data types into AWS cloud databases or other analytics services. For example:

• Amazon Data Firehose is part of the Kinesis family of services that automatically scales to match the volume and throughput of streaming data and requires no ongoing administration.
• AWS Glue is a fully managed serverless ETL service that categorizes, cleans, transforms, and reliably transfers data between different data stores simply and cost-effectively.
• AWS Transfer Family is a fully managed, secure transfer service for moving files into and out of AWS storage services.
• AWS Databases and AWS Database Migration Service (DMS) provide mechanisms for capturing and streaming changes from all AWS database services. You can use native CDC from Amazon DynamoDB or Amazon Neptune, which allows you to reduce the complexity of your data integration pipelines. Another option is to use CDC in AWS Database Migration Service (DMS), which extracts changes from the transaction log of the source. DMS is a highly available service, with resiliency for such long-running replication tasks. Your data streams can then be optionally transformed and distributed using Amazon MSK, Amazon Kinesis, or AWS Glue.
• Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that makes it easy to build and run applications that use the open-source Apache Kafka for stream ingestion.

You can also install custom data ingestion platforms on Amazon EC2 and Amazon EMR and build your own stream storage and processing layers. That way, you avoid the friction of infrastructure provisioning and gain access to various stream storage and processing frameworks.

Get started with data ingestion on AWS by creating a free account today.