Category: services

I was going to just link to Mike Kavis’ post on the Top 10 Reasons Why People Are Making SOA Fail, but I wanted to added some of my own comments. By the way, he’s talking primarily about IT people, not business people, in the fail part of the equation.

Number 1 reason: they fail to explain SOA’s business value. Kavis recommends (and I completely agree) starting with business problems first, specifically using BPM as the “killer app” to justify the existence of SOA.

He continues with a number of cultural and organizational issues, such as change management and executive sponsorship, then discusses a few of the flat-out IT failure points: not having the skills to actually do SOA (and not getting the outside help required), trying to do it on the cheap, thinking of SOA as a one-time implementation project rather than an ongoing architecture, and neglecting SOA governance.

His final reason for failure is allowing the vendors to drive the architecture:

[T]he vendors promise flawless integration if you purchase all of your tools within their stack. The reality is, they have purchased so many products from other companies that their stacks do not deliver any better integration than if you bought the tools from a variety of vendors.

In the face of recent acquisitions, this could not be more accurate.

To finish up my trilogy of posts on legacy integration, I had a look at the Metastorm Integration Manager (MIM), which takes a very different approach from that of OpenSpan. This is based on a look at the product that I did a couple of months ago, and now that Metastorm’s back in the news with a planned IPO, it seemed like the right time.

A lot of what’s in MIM is rooted in the CommerceQuest acquisition of a few years back. In a nutshell, it leverages IBM WebSphere MQ (what we old timers refer to as “MQSeries”) to provide a process-centric, services-oriented ESB-style architecture for integrating systems, both for batch/file transfer and real-time integration, on multiple platforms.

Like OpenSpan, the idea is to create web services interfaces or non-SOAP APIs around legacy applications, but instead of wrapping the user interface and running on the client, MIM wraps the apps on the server/host side and stores the resultant services in a registry. If you have CICS applications, MIM for CICS runs natively in the CICS environment and service-enables those apps, as well as allowing access to DB2 databases, VSAM and other application types. The real focus is to allow the creation of web services for custom legacy systems; packaged enterprise applications (e.g., SAP) already have their own web services interface or there’s a well-developed market of vendors already providing them.

Although Metastorm’s BPM has some native integration capability, MIM is there to go beyond the usual email, web services and database integration, especially for mainframe integration. Metastorm BPM can call the message-driven micro-flows or web services created by MIM in order to invoke functionality on the legacy systems and return the results to the BPM process.

I saw a demo of how to create a service to access a VSAM data set, which took no more than 5 minutes: through the MIM Eclipse-based IDE, you access CICS registry, create a new VSAM service, import the record definition from the COBOL copybook for the particular VSAM file, and optionally create metadata definitions to rename ugly field names. Saving the definition generates the WSDL and makes it immediately available, with standard methods for VSAM access created by default.

They also showed how to create a service orchestration process flow — an orchestrated set of service calls that could call any of the services in the MIM registry, including invoking batch jobs and managing FTP. With MIM for CICS, everything in a micro-flow is tracked through its auditing subsystem in CICS, even if it calls services that are not in CICS; the process auditing is very detailed, allowing drilldowns into each step to show what was called when, and what data was passed.

Once created, service definitions can be deployed on any platform that MIM supports (Windows, Z-series, i-series, UNIX), and moved between platforms transparently.

We spent a bit of time looking at file transfer, which is still a big part of the integration market and isn’t addressed by messaging. MIM provides a way to control the file transfer in an auditable way, using MQ as the backbone and breaking the file into messages. This actually outperforms FTP, allows for many-to-many transfers more effectively due to the inherent overhead (chattiness) in each FTP transfer, and allows for file-to-message transfers and vice versa, e.g., file creation from message driven by SQL statement.

A directory monitor watches for inbound files and triggers actions based on file names, extensions and/or contents. A translator such as Mercator or TIBCO might be called to transform data, and the output written to multiple systems in different formats, e.g., XML, text, messages, SQL to database, files.

MIM for CICS can also drive 3270 green screens in order to extract data, using tools in the design environment to build screen navigation. This runs natively inside CICS, not on a client workstation, so is more efficient and secure than the usual screen-scraping applications.

In addition to all this, MIM can invoke any program on any platform that it supports, feed information to stdin, and capture output from stdout and stderr in the MIM auditing subsystem.

On its own, this is a pretty powerful set of integration tools for service-enabling legacy applications, both batch and real-time, but they’ve also integrated this into Metastorm BPM. Of course, any service that you define in MIM can be called from any system that can invoke a web service — which includes all BPM systems — but MIM can launch a Metastorm human-facing process for exception handling from within one of its service orchestration processes by passing a MIM exception to the BPM server. The BPM process can pass it back to MIM at a point in the process if that’s allowed, for example if the user corrects data that will allow the orchestration to proceed, and the BPM user may be given the option to select the step in the MIM process at which to reinject the process.

What happens in Metastorm BPM when it is invoked as an exception handler is not tracked in the MIM process auditor; instead, it captures the before and after of the data that’s passed to BPM, and this would need to be reconciled in some way with the analytics in BPM. This separation of processes — into those managed and audited by BPM and those managed and audited by MIM — is an area where some customers are likely to want more integration between the two products in the future. However, if you consider the services created by MIM as true black boxes from the BPM viewpoint, then there’s nothing wrong with separation at this level. It’s my understanding that MIM calls BPM using a web service call, so really any system that can be called as a web service, including most other BPMS, could be called from MIM for exception handling instead.

Want to mashup your legacy applications? I recently had a chance for some background and a demo of OpenSpan, which is one of the tools that you can consider for bringing those legacy apps into the modern age of composite applications.

A big problem with the existing user environment is that it has multiple disparate applications — Windows, legacy, web, whatever — that operate as non-integrated functional silos. This requires re-keying of data between applications, or copy and paste if the user is particularly sophisticated. I see this all the time with my clients, and this is one of the areas that I’m constantly working with them to find improvements through reducing double keying.

In OpenSpan Studio, the visual design environment, you add a Windows (including terminal emulator or even DOS window) or web application that you want to integrate, then use the interrogation tool to individually interrogate each individual Windows object (e.g., button) or web page object (e.g., text box, hyperlink) to automatically create interfaces to those objects: a very sophisticated form of screen-scraping, if you will. However, you can also capture the events that occur with those objects, such as a button being clicked, and cause that to invoke actions or transfer data to other objects in other applications. Even bookmarks in MS-Word documents show up as entry/access points in the Studio environment.

In a couple of minutes, they built and executed a simple example using the Windows calculator and the Google home page: whenever you hit the M+ button in Calculator, it transferred the contents of the calculator to Google and execute a search on it. This is more than the simple pushing of buttons that you typically find in old-style screen-scraping however; it actually hooks the message queue of the application to allow it to intercept any UI event, which means that other events in other applications can be triggered based on any detected event in the interrogated application: effectively allowing you to extend the functionality of an existing application without changing it.

A more complex example showed interrogating the FedEx website rate to bring results back to Excel, or to bring them back and compare with UPS rates in a custom UI form that was built right in the Studio environment as well. You don’t have to build a UI in their environment: you can use Visual Studio or some other application builder instead, or build .Net controls in another environment and consume them in the OpenSpan Studio (which is a .Net environment).

As you would expect from an integration environment, it can also introspect and integrate any web service based on the WSDL, but it can also encapsulate any solution that you create within OpenSpan and expose it as a service, albeit running on the local desktop: a very cool way to service-enable legacy applications. That means that you can make “web” service wrappers around green-screen or legacy Windows applications, exposing them through a SOAP interface, allowing them to be called from any development/mashup environment.

The integrations and applications defined in Studio are saved to an XML file, which is consumed by the lightweight runtime environment, OpenSpan Integrator; it can launch any applications as required or on startup, and manage the integration between them (which was created in Studio). You can use this to do single sign-on — although not in as secure a fashion as dedicated SSO applications — and can even call services between native desktop applications and virtual environments (e.g., a client service within Citrix). Although the Integrator needs to be running in order to access the integrations via SOAP calls, it can run completely in stealth mode, allowing other applications to call it as if it were a true web service.

You can also integrate databases, which allows you to make database calls based on what’s happening on the screen in order to create analytics that never existed before. As with all other aspects of the integration, these are based only on events that happen in the user interface, but that still has the potential to give you a lot more than what you might have now in your legacy applications.

Back in the old days (by this, I mean the 1990’s), when we wanted to integrate BPM with a legacy mainframe system, it was messy. In the worst cases, we wrote explicit screen-scraping code to interact with green screens; sometimes we were lucky enough to be able to hook a message queue or interact directly with the underlying mainframe database. Much of the development was tedious and time-consuming, as well as requiring a lot of manual maintenance when anything changed on the legacy side: sometimes I used to think that the mainframe developers intentionally changed things just to mess up our code. Don’t get me wrong, I’m all for keeping the mainframes in there as super-charged application and database servers, but don’t let the users anywhere near them.

These days, there’s a number of tools around to make integration with legacy mainframe systems easier, and although I don’t write code any more, I have a passing interest since most of my customers are financial services organizations that still have a lot of that legacy stuff lying around.

Strangely enough, the impetus for finally writing about the legacy integration tools that I’ve looked at came because of a conversation that I had recently with Sandra Wade, a senior director of product marketing at Software AG, even though we didn’t talk much about their product’s technical capabilities and I didn’t get a demo. Their announcement back in March was a repackaging of mostly existing applications into their Application Modernization Suite, which has three basic flavors:

• The web edition is webMethods ApplinX, which allows you to web-enable green-screen applications.
• The SQL edition uses the webMethods ConnecX adapters to provide unified, SQL-based access across heterogeneous data sources, including non-relational sources.
• The SOA edition bundles ApplinX, webMethods EntireX, webMethods ESB and CentraSite to provide everything required to service-enable legacy applications, including governance.

Although I saw some of the underlying applications when I attended IntegrationWorld last November, I tend to focus more on briefings and sessions when I’m at a conference, so don’t have a really good feeling for the functionality of ApplinX and EntireX, and how they help to web- and service-enable mainframe applications.

I was going to include all three vendors in a single post, but will follow this one with separate posts about OpenSpan and Metastorm Integration Manager so that it doesn’t get too unwieldy.