At the Storage Networking World conference this fall, Computerworld reported deduplication was high on the shopping list of IT managers. Not surprising: Deduplication promises to revolutionize disaster recovery.

Deduplication isn’t exactly new. I have been writing about it for years: here at Business Management, and at Tech Target, and again at Storage Magazine.

The reason is simple: Deduplication makes it practical to back up data to disk. Organizations are being swamped with a flood of data while their own practices cause them to repeatedly store and back up multiple copies of the exact same data. This is supremely wasteful. Deduplication eliminates much of this waste.

Say 500 people receive a companywide e-mail with a 1 MB attachment, maybe a budget with charts. If each recipient saves that attachment, it is stored 500 times. That’s wasteful, but it’s only the beginning. During the nightly backup, the system would back up that one attachment 500 times as it backs up each person’s personal storage area – consuming 499 MB more backup space than necessary.

Then, let’s say, the next day one person makes one change to that file and sends it to all the initial recipients who save it again. Now the organization will be backing up 499 more copies of a file that is exactly the same as the previously 500 copies with the exception of that one change. Did I say wasteful?

And you know it is not going to be one person making the change but six or eight or ten making and circulating changes. OK, you can do the math. This gets very wasteful very fast.

Data deduplication is a method of eliminating redundant data from storage, especially from backups. A deduplication tool would store only one copy of that 1 MB attachment. Everything else would be stored as a pointer to the stored original. For the changed versions, it would save only the original change and create and log a pointer to that, too. When you wanted the file, it would bring back the original and the changes. Different tools work a little differently, but the process is essentially the same.

If we extrapolate the example above beyond e-mail to the thousands of gigabytes of data stored and backed up every week or month or year, the numbers get astronomical. Inevitably, redundant data gets backed up many, many times.

By removing the redundant data, deduplication makes large-scale backup and recovery to disk feasible. Backing up data to disk is easier, faster, and more reliable than to tape. And the backed up data on disk is immediately accessible online before eventually being put on tape.

The revolutionary part: Deduplication allows weeks or months of backups in the same amount of disk space because each piece of data is backed up only once. Suddenly disk, which is more expensive than tape, becomes cost-effective for primary, large-scale backup. Recovery via disk is almost assured; far less so with tape. ###