Companies handle a significant amount of data, and today anyone with a digital device is constantly generating more of it.
With RAID and NAS, a rebuild or an initialization without a diagnosis can reduce the chances of recovery. Before touching anything, see Bot’s RAID recovery, which starts with a free diagnosis.
In other words, new data is being generated every second, and storing this data is a challenge.
And that data has to be captured and organized before it can reveal patterns your company can use.
But storage capacity is limited. Adding more storage drives up costs, and yet you still need all that data.
So what’s the solution?
Data deduplication, which is neither a synonym for single-instance storage nor the same thing as compression.
In this article, you’ll learn how deduplication differs from those two techniques, how it works and when to use it.
What does data deduplication mean?
Data deduplication is a process that eliminates redundant copies of data and reduces storage overhead.
Deduplication techniques ensure that only one unique instance of each piece of data is kept on the storage media, whether that’s a hard drive, flash storage or something else.
Redundant data blocks are replaced by a pointer to the unique data copy.
In this respect, deduplication works well alongside incremental backups, which only copy data that has changed since the previous backup.
For example, a typical email system might contain 100 instances of the same 1 MB file attachment.
If the email platform is backed up or archived, all 100 instances are saved, taking up 100 MB of storage.
With data deduplication, the attachment is stored only once, and every other instance simply points back to that saved copy.
In this example, 100 MB of storage drops to just 1 MB.
How deduplication handles redundant data
As we’ve seen, deduplication is a method of eliminating redundant data from a data set.
During deduplication, a tool identifies the extra copies of data and removes them, so that only a single instance is stored.
In practice, deduplication lets you get rid of redundant data and run backups more efficiently, using far less storage.
What is the difference between deduplication and single-instance storage?
Single-instance storage works at the file level: identical files in a file system are replaced with references to one stored copy. Deduplication goes further, analyzing the content of the data (often in blocks or segments) and replacing any repeated pieces with references to a single copy.
What is the difference between deduplication and compression?
It’s important to understand what sets the two apart, because that tells you which one works best in each case.
Here are the main differences between deduplication and compression:
Process: deduplication looks for blocks of data that are repeated. A single copy of each block is kept, and every other occurrence is replaced with a pointer to it. Compression, on the other hand, encodes the data more efficiently, removing repetition and patterns within each file or data stream to make it smaller.
Reduction ratio: compression typically reduces data at a ratio of about 2:1 to 2.5:1, according to some vendors, depending on the type of data. Deduplication ratios can range from 4:1 to 20:1, and certain highly repetitive data can reach 200:1. Results depend heavily on the type of data, so the same deduplication software can achieve very different ratios on different data sets.
Data loss: deduplication is lossless. Redundant copies are removed, but a single copy of every block is always kept, so the original data can be fully rebuilt. Compression used for storage and backup (such as ZIP or LZ4) is also lossless; only lossy formats, such as JPEG or MP3, permanently discard part of the information to save space.
Data format: compressed data can be restored with any compatible decompression tool, because the file is self-contained. Deduplicated data, on the other hand, is broken into blocks linked by hashes and pointers, so it can only be read back through the deduplication software or system that created it. If that system’s index or metadata is damaged, the data becomes unreadable.
When should you use data deduplication (dedup)?
Deduplication is ideal for highly redundant workloads, such as backups, where the same set of data is copied and stored again and again for recovery purposes.
It isn’t a one-off task: deduplication usually runs continuously as data is written (inline) or as scheduled jobs after it is stored (post-process), for example after each backup.
How does data deduplication work?
Deduplication splits the incoming data stream into segments, gives each segment a unique identifier and compares it with the data already stored.
If a segment is unique, it is written to disk. If it duplicates something already stored, the system just records a reference to the existing copy and doesn’t store the segment again.
Take, for example, a file or volume that is backed up every week, generating a large amount of duplicate data.
In this case, deduplication algorithms analyze the data and store only the unique segments of the file, often in compressed form.
With typical backup retention policies on ordinary business data, this process can reduce storage capacity requirements by 10 to 30 times on average.
In other words, companies can store 10 TB to 30 TB of backup data on 1 TB of disk, which brings significant savings.
File-level deduplication
In this mode, duplicate copies of files are not stored again; they are replaced with a link to the original file.
The system uses each file’s “fingerprint” (a unique string of characters that identifies its content) to check whether it has already been stored.
Fingerprints are usually generated with hashing methods or from file attributes, depending on the deduplication solution.
This method is easier to implement, because its indexes are smaller and quicker to compute.
On the other hand, it saves less space than block-level deduplication, with savings of up to around 80%.
That’s because, at the file level, even the smallest change makes the system treat the file as a new one.
The biggest savings are found in shared storage (such as NAS systems and shared folders), since these usually hold several copies of the same files.
Deduplication efficiency also depends on the file type. Images and audio files, for example, tend to be unique and gain little from the process, whereas templates and system files usually deduplicate very well.
Block-level deduplication
A more granular approach, block-level deduplication checks whether each block within every file is unique.
When a file is modified, the system only stores the parts that have changed (the blocks), not the whole file again.
Each block has its own identifier (usually generated with a hash algorithm), which the system compares with the metadata already stored.
This saves more space, as block-level deduplication can cut storage needs by up to 95%.
The trade-off is that it requires more processing power, since there are far more objects (blocks) to process.
Cloud backup storage
One way to further reduce the space used and cut storage costs is to use a backup system with a cloud back end.
The problem is that most storage providers don’t offer native deduplication, and those that do usually charge extra for it.
The alternative is to use separate deduplication software so that only deduplicated data is uploaded to the cloud.
Need to recover data from a drive that uses deduplication?
A drive with deduplicated data can still suffer physical damage or logical failures that lead to data loss.
When that happens, the best option is a specialized data recovery service, which can recover your files and documents safely, without risking further damage to the drive.
At Bot, for example, we work in a Class 100 (ISO 5) cleanroom, a controlled environment where airborne particles are kept to a minimum so that drives can be opened safely.
With more than two decades of experience in data recovery and more than 200,000 cases, we know how to handle complex storage systems, deduplicated data included.
Read next
Conclusion
Deduplication (often shortened to dedup) is a process that eliminates redundant copies of data and significantly reduces storage capacity requirements.
Contrary to what some people believe, deduplication is neither compression nor single-instance storage, and it comes in two main types: file-level and block-level.
Deduplication is recommended for highly redundant workloads (such as backups), runs continuously or on a regular schedule, and can be combined with cloud storage to keep your backups safe.
Finally, keep in mind that a drive with deduplicated data can also fail, causing you to lose your data.
If that happens, turn to a professional data recovery service, like the one we offer at Bot.
As well as handling your drive with care, we offer free collection of your device (islands not included) and a free standard diagnosis, with a quote within 48 hours.
So if you want to recover your files and documents quickly and safely, start your data recovery with us now!
