Hard Hat Area: Myths and Pitfalls of Data Mining.

Executive brief
Hard Hat Area: Myths and
Pitfalls of Data Mining
By Tom Khabaza
Director of Data Mining, SPSS
Table of contentsIntroduction........................................................................................................................... 2Myths and misconceptions about data mining....................................................................... 2Pitfalls of data mining and how to avoid them....................................................................... 4Conclusion............................................................................................................................ 8
SPSS is a registered trademark and the other SPSS products named are trademarks of SPSS Inc. All other names are trademarks of their respective owners. © 2007 SPSS Inc. All rights reserved. HHAEB-1207IntroductionThe intrepid data miner runs many risks, including being buried under mountains of data or disappearing along with the "mysterious disappearing terabyte." Myths and misconceptions create their own risks and need to be debunked. This article outlines some risks, debunks some myths, and attempts to provide some protective "hard hats" for data miners.
It's critical to understand that data mining is a business process-a way of finding patterns in your data that provide insight you can use to conduct your business more effectively. Data mining also makes predictions to guide customer interactions and other business decisions.
Myths and misconceptions about data miningMyth #1: Data mining is done in the lab, by a technology expertData mining uses advanced technology, and its workings, particularly those of modeling techniques, are unlikely to be understood by the wider IT community. Does this mean that data mining should take place in the laboratory and be conducted only by those who understand every nuance of the technology that is involved?
Quite the opposite is true, because data mining is a business process in which business knowledge is of paramount importance: the value of data mining is realized only when the results are put to use in business operations.
When performed without business knowledge, data mining can produce nonsensical or useless results (see pitfall #4, below), so it is essential that data mining be performed by someone with extensive knowledge of the business problem. Very seldom is this the same person who has extensive knowledge of the data mining technology. It is the responsibility of data mining tool providers to ensure that tools are accessible to business users.
Of equal importance is the need to deploy results into the business, to put them to use. Data miners should plan at the start of a project how their results will fit into business operational processes. Organizations should acquire infrastructure which enables them to deploy data mining results efficiently across the organization, and tool providers should ensure that their tools fit easily into this infrastructure.
"Data mining is a business process in which business knowledge is of paramount importance: the value of data mining is realized only when the results are put to use in business operations."
2 Hard Hat Area: Myths and Pitfalls of Data MiningMyth #2: Data mining is all about algorithmsA businessperson attending a typical data mining conference or reading its proceedings might form the impression that data mining is all about advanced data analysis algorithms. This misconception might be summarized as follows: "All you need for data mining is good algorithms. The better your algorithms, the better your data mining. Advancing the effectiveness of data mining means advancing our knowledge of algorithms."
To hold this view is to misunderstand the data mining process. Data mining is a process consisting of many elements, such as formulating business goals, mapping business goals to data mining goals, acquiring, understanding, and pre-processing the data, evaluating and presenting the results of analysis, and deploying these results to achieve business benefits.
This is not to minimize the importance of new or improved data mining algorithms. The problem occurs when data miners focus too much on the algorithms and ignore the other 90-95 percent of the data mining process.
The consequences of this misconception can be disastrous for a data mining project, possibly res... [download for more]