When properly architected and executed, Data Services can provide the missing link that unifies conventional data systems with the emerging SOA paradigm. By offering a decoupled data facade and an easily virtualized API, Data Services can give SOA the opportunity to establish system control without always forcing enterprise data through an inefficient XML layer.
without always forcing enterprise data through an inefficient XML layer.
2
It's always been about the data. Decades of punditry about EAI, ETL, MDM and SOA still lead us to the same conclusions - data matters. If content is king in the consumer Web, then data is king in Enterprise Software.
Sometimes the Enterprise Software sector loses sight of that simple reality. In the past fifteen years, with the rise of Java, the hype surrounding EII, EAI and SOA, and the rapid rise of XML, and quietly, the billions spent in ETL projects - it's all too easy to forget why we build and buy all that infrastructure. We do it for the data.
Without the data, there would be no need for process orchestration. There wouldn't be any purpose to all those SOAP envelopes, all those service bus' wouldn't have anything to publish and application servers wouldn't serve anything. Data is king.
But data presents huge, looming, non-trivial problems. First, businesses have figured out how to collect more of it but still can't effectively understand it all. Second, with more of it around, the infrastructure and tooling is bursting at the seams to manage it effectively. Third, the approaches used to define it in small architectures simply won't scale out to large business sized problems. Finally, enterprise architects too often get caught up in the buzz of new technology and forget that thirty-years of hard-fought lessons about data management still apply.
There has been more data created since 2000 than in all of human history preceding then.
3
For our businesses and governments, the rise of sensors means that we can monitor anything in realtime: from where your shipments are, to the temperature of your factory, or your very own heart rate. All that data ends up somewhere. It is stored indefinitely, used for realtime dashboards, historical analytics, or put somewhere just in case. But we can now collect more data, at faster rates, than we can successfully interpret. And the rate of data collections, the use of sensors like RFID and other monitors, is growing exponentially. In other words, the data problem is getting worse, not better.
But enterprise infrastructure is surprisingly unchanged since the early 1990's. Back then, Message-Queues (MQ), Transaction Processing Systems (TPS), and ETL tools were really the backbone of enterprise software. Guess what? They still are. Despite the growing adoption rates of BPM, SOA, ESB, and EII - the MQ, TPS, and ETL backbones are still there. The strain of all that new data and the demand for mature tooling has paradoxically made the existing, proven software infrastructure look pretty attractive. Many new systems will try to put all the data in XML, or perhaps try to use Java Entity Beans as the data management tier. While these are acceptable for smaller applications or for specific use cases, neither of these approaches scales to the mult-terabyte sized problem that is typical of a Global 2000 business. Thus, a knowledgeable architect will revert to the proven patterns of RDBMS as the backbone of a data architecture using MQ, TPS, and ETL interfaces as the pipes for pushing all that data around. But the buzz of SOA is deadening. Why not SOA for data-centric architectures?
When the Service-Oriented Architecture craze started somewhere back in 2001, we thought it was magic. Remember the promises of dynamic discovery? Human readable messaging? Simple XML data objects? But soon enough, the problems started: competing vendor specs, security loopholes, performance problems.and so on.
Here in 2008, the good news is that SOA has finally matured into an Enterprise class infrastructure. Far from the original hope of solving all integration problems, the main tooling for SOA (Enterprise Service Bus and Business Process Engine) is almost at a level to realistically supplant the long-held dominance of MQ and TPS systems. Both the reliability and performance of basic SOA is strong enough for all but the most demanding problems. However, SOA is still not best for ETL and data integration.
Data integration use cases span from the simple to the impossible. On the simple side of things, transforming some small amount of data and putting it somewhere, a regular SOA with XS... [download for more]
