What is Data Deduplication?

Data deduplication is the process of finding and removing copies of the same data to avoid redundancy and errors. After data deduplication, only one instance of the data exists. Data deduplication can refer to data at the file, block, object, or record level, and the process to find and remove each type of data differs. Deduping saves space and costs when referring to data in storage and improves accuracy when referring to data in analytics systems.

What is data duplication? Before and after deduplication

What are the types of data deduplication?

There are two different types of data deduplication, each of which has a distinct method and problem they tackle.

How does data deduplication work at the storage level?

Storage-level deduplication is a process that looks to remove redundant data at the infrastructure layer. Storage-level deduplication finds identical data segments and makes sure they're only stored once. This usually involves removing duplicates and replacing them with references. A storage-level approach can occur at the file level (by comparing whole-file hashes), at the block level (by breaking files into smaller segments and removing duplicates), or in object storage systems.

Data deduplication at the storage level begins by examining the data. If you are performing block-level deduplication, you break data into small segments, known as chunks. Every chunk (which can be fixed or variable in size) is then fingerprinted by hashing, giving it a unique reference. If you are performing file-level deduplication, each file is hashed.

The system then compares the active fingerprint of that chunk or file against your hash index of all previously stored data fingerprints. If there is a match, the system doesn't store the data again. It simply creates a reference that points to where to find that data within the existing records. You can then run background garbage data collection to reclaim space beyond the reference points.

However, this system is less effective when trying to store encrypted or compressed data. The specific compression technique and level of encryption can mean that two records, despite holding the same data, do not have a matching hash.

How does data deduplication work at the record level?

Record-level deduplication acts on structured or semi-structured data that has already been normalized. This method finds exact matches, plus looks for logically equivalent records that are in a different format. The main way of doing this is with rules-based and fuzzy matching, helping to detect similarities. Machine learning models can also compare how closely records resemble one another, with records that meet a certain threshold being flagged as duplicates. You can either merge these similar records or link them, depending on the context.

Record-level deduplication is most common in highly structured environments where consistency and accuracy are important, such as in customer relationship management systems or sales systems.

Why is data deduplication important? What are the benefits of data deduplication?

There are several reasons for a business to use data deduplication.

Enhance data integrity and quality

When your business is working with duplicated records, you might take the same data into account twice, thinking they're two different records. In cases where one record is updated, but not its duplicate, this leads to inconsistencies and out-of-date data. In analytics-heavy workloads, duplicated information can skew your data integrity and reduce the accuracy of reporting systems. In ML workloads, duplicate files can skew the training process and reduce training efficiency.

Reduce storage costs

Redundant data backups, multiple versions of the same file, and VM images can all accumulate in a business system over time, taking up space and increasing the total cost of storage. Data deduplication helps to locate redundant duplicates, removing them and reducing the total volume of data stored in a business. With less stored data, businesses have less to pay in storage fees.

Consume less bandwidth

By finding and removing duplicate data from your systems, you make sure that any data transfers only include the very minimal volume of necessary information. Transferring data with duplicate data blocks consumes more bandwidth during transmission. The deduplication process is an effective way of consuming less bandwidth, which is useful in cloud environments where migration costs can be high.

Meet compliance objectives

Regulations around the handling of sensitive data, especially personally identifiable information (PII), require you to manage data consistently and securely. Having a singular, authoritative record of each piece of information makes it much easier to ensure you're applying the correct data protection measures to that information.

What are the types of data deduplication schedules?

Data duplication doesn't necessarily occur at a fixed point in the data lifecycle. Depending on your storage requirements and performance needs, there are different data deduplication schedules you can use.

Inline deduplication occurs during the write process, checking whether the data is a duplicate before it's written to storage. While this minimizes how much storage space you use, it creates a higher write latency, as checking for duplicate data has to occur before writing to disk.

Post-process deduplication allows your system to write data immediately. After the write process finishes, your system scans all stored data, identifies any duplicates, and then removes them. Doing this solves the write latency problem, but it needs a higher temporary storage overhead to compensate for any duplicate data that makes it into your storage space. You can set schedules for post-process deduplication to run daily, weekly, or at any chosen interval.

What are the use cases for data deduplication?

Businesses use data deduplication solutions for a range of reasons.

Backup data management and disaster recovery

Backup systems generate large volumes of redundant backup data across multiple snapshots over time. Data deduplication reduces the data redundancy of these systems, identifying and removing duplicate data. Getting rid of redundant records helps to improve disaster recovery efficiency while also lowering storage costs.

File shares and home directories

Especially in collaborative corporate environments, businesses can store several sets of identical files for different teams or departments. Centralizing data and then eliminating duplicate data helps improve storage utilization, freeing up storage space while setting correct access permissions.

Software development environments

Including a deduplication process in the software development lifecycle can make sure that you only store modified data segments. Instead of holding several versions of the software artifacts with very minimal changes, you can create a change-only copy, improving storage efficiency.

VM storage

It's common in virtual environments to have multiple instances that all rely on the same virtual image. Data deduplication makes sure that imaging repositories only store unique VM images, helping to reduce the data storage requirements of running these systems at scale.

Customer data platforms

Customers often interact with your business across many different systems, generating data as they do. Instead of holding different records for these customers across several channels, you can combine datasets to improve data management and remove any duplicate data.

Healthcare and financial data

Highly regulated fields like finance and healthcare often use data duplication to remove any redundancies in their data. Doing so can reduce inaccuracies in data management while also reducing the total overhead in compliance with data privacy systems. Singular records are also easier to trace for auditing if you need to submit records for compliance reasons.

What are some data deduplication best practices?

Launching an effective data deduplication strategy relies on careful planning and clear direction. Here are some best practices you can follow.

Assess outcomes in advance

Improve the effectiveness of your strategy by clearly documenting the outcomes you want to achieve with deduplication ahead of time. Assess your systems ahead of time and accommodate when planning. For example, if you know you'll be working with encrypted or compressed data in storage, then a typical deduplication strategy won't have much of an impact.

Schedule jobs during off-peak hours

The actual data deduplication process can put strain on your system and consume resources, introducing latency and taking up bandwidth. To get around any performance impacts, you can schedule deduplication jobs for off-peak hours.

Use fuzzy matching in record deduplication

Fuzzy matching is a strategy you can use to greatly improve accuracy when removing duplicate data from inconsistent datasets. While direct matching can work, fuzzy matching is much more consistent and flexible at identifying similar data sets.

Plan deletion schedules

Removing data also requires you to remove any unused data blocks. Properly handling any references within your data sets requires careful planning. Be sure to audit your environment and have a clear deletion plan to reclaim as much storage space as possible.

How can AWS help with your data deduplication requirements?

AWS supports data deduplication for your storage and records, with the following services offering native deduplication:

Amazon FSx makes it easy and cost-effective to launch, run, and scale feature-rich, high-performance file systems in the cloud. It supports a wide range of workloads with its reliability, security, scalability, and broad set of capabilities for NetApp ONTAP, OpenZFS, and Windows File Server. You can enable data deduplication to automatically reduce costs associated with redundant data by storing duplicated portions of your dataset only once.
AWS Glue helps clean and prepare your data for analysis without you having to become an ML expert. Its FindMatches feature deduplicates and finds records that are imperfect matches of each other. The system then learns your criteria for calling a pair of records a "match" and builds an ETL job that you can use to find duplicate records within a database or matching records across two databases.

Get started with data deduplication on AWS by creating a free account today.

What is Data Deduplication?