The Next Generation in Data Deduplication - Deduplication Across Encrypted Data

White Paper - June 2009
The Next Generation in Data DeduplicationDeduplication Across Encrypted Data
By Gary Sumner, Founder & CTO Datacastle
ContentsIntroduction 1Problem Statement 1Previous Options 1-2Datacastle Solution 2-4Bene!ts 4-6Summary 5-6The Next Generation in Data Deduplication
WWhhiittee PPaappeerr -- JJuunnee 22000099
IntroductionAs storage requirements continue to grow, new techniques in data deduplication promise to curb exponential storage growth. But not all deduplication techniques are the same.
Current data deduplication techniques fall into two broad categories depending on where the process takes place. Client-side deduplication occurs at the source (where the data is created and stored); target-side deduplication takes places on the server (after the data has already been transported to its archival storage location).
While both forms of deduplication generally provide the same level of storage savings, client-side deduplication provides additional e"ciencies through reductions in network bandwidth consumption. Datacastle's client-side data deduplication process goes one step further and provides additional bene!ts with an enhanced security model that keeps data safe.
Problem StatementData deduplication is implemented through the use of a data block analysis algorithm that seeks to eliminate duplicate data blocks (also referred to as "chunks") across an entire data store. Before data is archived, the process identi!es chunks that already appear in the target archive. Once duplicates have been eliminated, the resulting data stream is reduced in size, resulting in reduced storage requirements. The data archive footprint can be further reduced by implementing the data deduplication process across multiple data stores (or "sources") depending upon security requirements. However, encrypted data can present a problem.
Due to its very nature, data that is encrypted using di#erent encryption keys results in data that cannot be deduplicated as the resultant encrypted data is di#erent. Consequently, to bene!t from deduplication data must be processed in an unencrypted format.
Previous OptionsLimited by their inability to cope with encrypted data, traditional data deduplication techniques have forced IT departments to pit security needs against storage budgets. An enterprise that wished to take advantage of the bene!ts o#ered by data deduplication would have had to forego the level of security and privacy o#ered by data encryption.
One solution to the problem is to deploy transient data encryption or channel encryption.
Channel encryption protects all communication between the client and server by encrypting the data path. At a minimum, this is required if no source data encryption is performed, or if any "secrets" such as data encryption keys are passed between the client and the server.
© Copyright 2010 Datacastle. All Rights Reserved. 1The Next Generation in Data Deduplication
WWhhiittee PPaappeerr -- JJuunnee 22000099
With transient data encryption, the original data on the client can be encrypted using symmetric or asymmetric methods. Encrypted data will then be decrypted on the server before it is deduplicated and subsequently archived.
Data could be deduplicated client side before being encrypted and uploaded, but in order to deduplicate data across multiple clients the server would need to be able to decrypt the data for processing.
Transient data encryption alleviates the need for an encrypted channel architecture, but the technique includes a potential pitfall. To implement a transient data encryption scheme the server needs to have access to and control of the encryption key and the process for storing encrypted data on disk. This means that if the server's security is breached, the security and privacy of data could also be compromised. An unlikely scenario? Maybe not.
This is the exact type of vulnerability that was recently exploited in the Heartland Payment Systems 1data breach. Data was sent encrypted to the server for processing and was momentarily decrypted on the server exposing the credit card and debit card details of over 100 million people to malicious network sni"ng software.
Datacastle SolutionAs one of the core pillars ... [download for more]