Choosing middleware to knit together disparate systems can be one of the biggest headaches in IT. But getting it right can also be one of the biggest opportunities to reap competitive advantage.
If you think your operating system constitutes the most critical software in your enterprise IT, think again. Middleware looks set to become the IT battleground of the future, the facet of your infrastructure from which you will need to thrash out business advantage. While the operating system is fast becoming a commodity item, the middleware tying applications and processes together is still an ill-defined range of software components, requiring considerable customisation and having no commonly agreed definition. Unfortunately, it is also far harder to get right.
The major analyst firms are predicting a rosy future for middleware. Gartner forecasts that middleware software will account for 28% of IT expenditure by 2015, compared with just 10% in 2002. This latter figure of 1% might seem puzzling given that integration has been a significant overhead in large IT projects for some time, but it only includes the software, not the internal IT effort of making it work. Indeed according to Ed Wrazen, marketing director of middleware consultancy CommerceQuest, integration already accounts for 30% to 40% of the cost of a typical IT project, and often as much as 70%.
Middleware software itself is supposed to ease the integration burden, by providing at least some of the functionality off-the-shelf to link applications together and provide more consistent access to information. In practice though middleware can create as many integration problems as it solves, if care is not taken to adopt a coherent strategy for enterprise application integration (EAI). For just as most large enterprises have a variety of different ERP systems, perhaps because of acquisitions or devolution of IT to local departments, so they have middleware from more than one supplier.
And yet, as Hanafi Houbart, senior consultant in e-commerce at Unipower Systems, points out, changing middleware on any scale is usually difficult and expensive. "In practice no one ever changes established middleware unless either the underlying operating system [bottomware], or overlying applications [topware] are also changed en masse, or the middleware itself has failed to establish well," says Houbart. "That's why middleware products tend to have longer product histories than front-line applications - they just keep getting upgraded."
If there is a time to change your principal middleware supplier though, for whatever reason, it is now. Within a few years middleware will be even more inextricably linked with key platforms and applications, making change yet harder. In theory it should be possible to change horses if the middleware adheres to common standards, but in truth the relevant standards do not yet exist and in any case are hard to crystallise in a continuously changing field. A major problem, according to Kay Hammer, chief executive officer of integration software firm ETI, is that applications are tied in with the middleware via application programming interfaces (APIs) in a way that is hard to standardise and needs completely reworking when you change supplier. This can be a complex process with the danger of introducing errors or unanticipated changes to processes.
"So while over time there will be some standard XML message format that would facilitate moving from one middleware supplier to another, to some extent while you're putting calls into your programs that directly reference their APIs, then you're really putting yourself at some risk," says Hammer.
To avoid becoming effectively locked in to their primary middleware, enterprises must build their own software shell between their applications and the back end or "bottomware" systems such as the database. The most enlightened organisations have already done this, says Hammer, who cites the case of one customer implementing a business-to-business (B2B) infrastructure based on BEA Systems' Web Logic middleware. "They're putting their own shell in because although happy with BEA Systems now, they recognise over time they may need to change their middleware supplier," says Hammer.
The shell handles all the calls into the underlying databases and provides a consistent layer connecting these to the topware applications. The middleware then plugs into this shell, which provides the link to the back-end systems. It is then easier, although still by no means trivial, to change the middleware, because now it just involves re-implementing the shell without changing the calls into backend systems.
Until recently enterprises needing to develop such a middleware shell would have had to do the job themselves, but now tools are emerging to help generate the interface layer. This layer needs to comprise not just hooks, but also information about what is being connected to, and what is being done. In effect, therefore, it should comprise metadata, providing a map to the back-end databases as well as information about how the front-end applications are accessing this. The big weakness of existing middleware suppliers is their lack of a metadata strategy, according to Hammer.
This applies even to the mature core application messaging products such as IBM's MQ Series and Microsoft's Message Queue. The latter is used mainly within the Microsoft environment, while MQ Series dominates the enterprise messaging environment with 70% of the market for software communication across platforms, as opposed to within a single operating system environment. MQ Series is almost synonymous with middleware in some minds, but while it has a good reputation it does not by itself provide the data and process management necessary to keep track of changes and ease the integration burden as new applications are developed.
MQ Series allows messages to be routed within an application framework to the correct system and then cause a process there to execute, triggering as it is called. What it does not do is provide a record of the routes and triggers, which would make it easier to manage the environment and effect changes.
There are some tools now that help with this, one being ETI's Extract Accelerator for MQ Series, which is a development aid for generating the calls and providing an audit trail of all the links to back-end systems and databases. Although currently only available for MQ Series, such a product would ease the pain of switching to another middleware supplier, because of the information it provides on all the inter-process links.
Middleware though is increasing its scope beyond application messaging. Products like MQ Series are asynchronous, providing a loose coupling between processes, so that one might send a message to another but not expect any response. There is also a need for protocols that support direct interaction, integrating applications more tightly together, and there are various products that allow this, including some of the tools within the object oriented Common Object Request Broker Architecture.
Because of the expanding scope of middleware, IBM has just gone through a re-branding exercise, bringing all the products together under the WebSphere logo. Not for the first time in IBM's case, the re-branding exercise is more a statement of intent than a fait accompli - ie, being a collection of often admirable but not particularly compatible products. According to Geoff Rayner, financial director in charge of IT at eHospital, an IBM middleware site which specialises in B2B trading in healthcare, WebSphere is not yet the answer to an enterprise's middleware integration problems that IBM would like its customers to believe. "WebSphere is an amalgamation of 1,000 things at present. People are confused by what's in there, and we've been through that learning curve ourselves just to find out what's in it."
Yet IBM's competitors can take little heart from these comments, for Rayner reckoned that they are all less well prepared for the challenges to come. "There's a big effort internally in IBM to bring it all together. The good thing for us is we don't have to pay for all that work."
In the hospital field, which Rayner describes as an IT nightmare with its huge collection of incompatible systems from a wide variety of suppliers, IBM's MQ Series is the best technology available for providing at least a basic level of integration between systems. This is providing real benefits for doctors and other workers within healthcare, Rayner insists. For example MQ is shortly to be used to integrate materials management with procurement and the supply chain within some hospitals, which will enable consultants, nurses and others to re-order surgical consumables by scanning as they leave or enter a ward rather than having to sit down in front of a terminal, or worse fill in a form. The key to MQ Series' success, says Rayner, is the number of different APIs it would communicate with, making it the best technology to form the core of a middleware strategy for a multi-supplier environment.
Not surprisingly, other suppliers dispute this relatively rosy view of IBM's position, arguing that it ignores the expanding role of middleware, beyond mere application integration, into unified content and document management. According to Tom Bird, UK head of collaborative working software supplier iManage, the missing piece of many middleware solutions is the support for common access to documents. "Over 80% of a corporation's knowledge exists in its documents," notes Bird, who adds that middleware did nothing on its own to open up documents and make them available to collaborative working communities, both within organisations and in B2B applications. The need is to elevate middleware into the spheres of content, document and knowledge management, Bird asserts.
In fact this point has not been lost on traditional middleware suppliers, and is one motive behind some of the recent rebranding exercises. Indeed BEA Systems, which is the main challenger to IBM in the middleware arena, is moving higher up the software food chain, according to its senior architect for UK and northern Europe, Mark Prichard. Particular targets for BEA, he says, are the content management suppliers such as Broadvision and Vignette. "Our differentiator compared with Broadvision is that we can provide almost as much functionality, but based entirely on the Enterprise Java Beans standards, with no proprietary kernel." This, says Prichard, made it easier to integrate content management with other key applications such as personalisation. "Enterprises prefer to go with a single supplier, but also want an open standards-based solution, even if the out-of-the-box functionality in a particular vertical market such as content management is not quite as deep."
With this comment, Prichard identifies the old dilemma for enterprises, which is whether to go with a perfect set of point solutions to individual problems, or compromise with a single supplier framework that will have weaknesses in some areas but eases the integration burden. In the case of middleware there is no sign of this dilemma being resolved.
Key middleware players
Middleware is dominated by five big suppliers, in this case IBM, BEA Systems, Oracle, Sun Microsystems, and Microsoft. According to the Giga Group, IBM will take over the lead in the market for multi-supplier middleware, with its share growing from 16% to 24% during this year. BEA will be number two. Microsoft will remain a big force for inter-application middleware within the Windows environment with its COM and Active/X technologies, while Oracle is strong among its own database customers. Both BEA and Sun have their main base among Unix sites, with the latter more predominant among users of its own hardware.
BEA is unique among the big five in being independent, or at least perceived to be, from any particular hardware or software platform. Yet it is IBM with MQ Series that is strongest in the market for middleware linking Unix and non-Unix systems, via its MQ Series messaging software.
NOTE: BEA Systems + Sun Microsystems are now merged with Oracle, making Oracle Weblogic and Fusion middleware as strong contenders for IBM's Websphere range of products. JBOSS needs a lot of corrections in its enterprise version (v6) before companies can transform. However IBM provides the worlds best after sales services including customer care. Oracle stands just next to it. JBOSS in not in this race.
(Not inclined to any vendor - it is just my perspective)
Tips for changing middleware
- Understand why you are changing middleware, and work out what is wanted, for example greater ability to integrate with other systems you have
- Consider the change in the light of overall IT strategy. For example you may be implementing mobile applications in which case the new middleware must be capable of supporting pervasive computing and Wap devices
Assess the features required. Typical features include asynchronous message queuing, message broking which facilitates synchronous and more closely integrated processes, support for distributed transactions, security, access to legacy systems, and support for small mobile devices
Look at the products. Many suppliers call their products middleware, but with wide differences in functionality, capability and all round coverage. A key point to consider is what level of integration is needed
Run a pilot project, especially if a major change of middleware right across an enterprise is being considered
Start by migrating just one application. If the new middleware cannot cope, perhaps lacking sufficient support for key emerging standards such as Compact XML for small devices, then it is less costly to withdraw at this stage.
Review the initial migration thoroughly before committing the whole enterprise. Determine the roadmap for future platform evolution, skills required, and define the enterprise's own middleware architecture accordingly - do not rely entirely on the supplier for this
If all appears well, plan the full roll-out
Execute the plan
