Most people assume that the starting point for any business intelligence (BI) project is a data warehouse. In reality, while data warehouses are important for many types of analytical systems, they aren’t always necessary. Building a data warehouse can dramatically increase the cost of a BI project. It can also reduce the value of the information by taking timely operational data and making it dated or even irrelevant. In our experience, many BI projects can realize benefits from alternate data integration scenarios.
The intended audience for this paper is project, business, and IT managers who have responsibility for BI activities. If you hold one of these positions, we’d like to expand your understanding of BI projects by describing seven proven techniques for accessing BI data. We’ll use real-world examples from Information Builders’ customer base to demonstrate the high-value, high-return data access options that are available to you.
Data warehouses themselves are not the problem. The problem arises when a data warehouse is viewed as a solution to all BI deployments, or there is an expectation that simply building a data warehouse will drive users to information. Data warehouses should not be implemented without a clear understanding of the business challenges they will solve. Before building a data warehouse, you should also carefully research potential data-access architectures to make sure you have devised the best method for connecting your BI tools with your data.
This paper describes data warehouses along with many other options for placing relevant, timely information in the hands of business users. As you’ll see, while a data warehouse is a good solution in some instances, many BI applications are better served with integration and portal technologies that simply pull data into reports on an as-needed basis.
In the pages that follow, we'll review seven basic ways to integrate and access data to solve various business problems:
1. A traditional data warehouse, periodically refreshed from production data sources
2. A real-time data warehouse, constantly updated by trickle-feeding data from production data sources
3. Operational data access, in which users obtain a real-time view of business activity from operational data and applications
4. Enterprise information integration (EII), in which BI users benefit from the real-time aggregation of corporate data across multiple data sources
5. Process integration, which involves delivering real-time information based on a business event or as part of a business process
6. Search technology that can rapidly scan indexed content to create Google-style results from data sources throughout the enterprise
7. Web services, which can expose or extract data from multiple sources of information, irrespective of underlying operating systems, applications, or databases
Seven Styles of Data Integration
Traditional Data Warehouse
Data warehouses are important for many BI projects, particularly when analytic systems are involved. Generally they involve gathering data from multiple sources to create an aggregated source of information for reporting. A data warehouse is a consolidated view of enterprise data, optimized for reporting and analysis. Data and information are extracted from production data sources as they are generated (real-time information), or in periodic stages (latent information), making it simpler and more efficient to run queries against that data, rather than to separately access each data source.
There are many valid reasons for building a data warehouse, including the following:
- To reduce overhead on a transaction-processing system or production application by staging data to a reporting database
- To reduce the complexity of the data and put it in a form that is suitable for reporting
- To maintain and analyze historical data that is no longer accessible in operational applications
For example, Moneris Solutions, Canada’s leading technology merchant of credit card processing, created a data warehouse to allow merchants to view daily and historical sales data. Developers used data-integration technology from iWay Software, an Information Builders company, to extract data from point-of-sale and transactional card systems in three data centers, and load it into a Microsoft SQL Server-based data warehouse.
Moneris maintains three months worth of daily transactions and 24 months of summary data in its data warehouse, which is dimensionally modeled to speed up reporting and analysis activities. The company downloads about five million rows of new transactional information into the warehouse each day to support a merchant base of more than 300,000 customers.
