Category: BPM history

Continued from Part 6.

Part 7: The New Arrivals. In the years following the dot-com bust in 2000, a number of new BPM vendors came into being, mostly in the coveted pure-play space. (Funnily enough, “pure-play BPM” is now not the desirable place to be, having been replaced by the “BPM suites” space that, according to some large analysts’ research, seems to have nearly identical functionality to pure-play BPM.) In many cases, these were started by those who were bounced out of their previous positions during the bursting of the bubble, so there was a lot of experience being put to the task of starting this new generation of BPM vendors.

The big advantage that a new vendor has in any industry is the lack of baggage, and nowhere was this truer than in BPM: they could start designing the next generation of BPM without having to reuse their existing technology or support their installed base, because they had neither. The BPM market needed to be reinvented, and these upstart young companies were the only ones who could shake things up enough to do it. I’m not going to credit the new arrivals with all the innovation in BPM during that time, but they certainly lit a fire under the old guard. Suddenly, we had BPM calling web services, or being called as a web service, in order to speed integration (and eventually become part of the SOA ecosystem). We had BAM, or at least some half-decent process monitoring and analytics for a change. We had simulation and optimization. We had integration with third-party modelling tools. We had business rule integration.

The startup environment during that time wasn’t the best — not a lot of venture funding around for technology, the perception that this was just a rehashing of the well-established workflow market — but a few of the vendors have become successful and many others are still straggling in their wake.

Meanwhile, the established BPM vendors had a big challenge on their hands: although few of the upstarts were challenging them directly for sales, the new guys were changing the perception of what the BPM market should be, forcing the big guys to follow suit as Gartner and the other large analysts published lists of must-have features that included this new functionality. Many of the larger vendors lagged badly in implementing new features, and look a little tired these days in comparison to the shiny bright newcomers. In some very conservative industries, such as financial services and insurance where most of my clients are, this hasn’t been a problem because they’d rather pick a vendor with a longer track record and the proven ability to process hundreds of thousands of items per day. However, this is where many of the dinosaur-like CIOs are fighting the losing battle against the push for emerging technology, and eventually the new kids will prove themselves scalable and stable enough for even the most conservative industries.

Even if none of the post-2000 vendors survive the upcoming bout of acquisitions, they have to be credited with not just injecting new life into BPM, but helping to reinvent it.

Next: The Current State

Continued from Part 5.

Part 6: The Tower of Babel I’m going to skip forward a few chapters in Genesis to describe the utter confusion that happened next when the combined space was relabelled, primarly by the big analysts, as business process management. Now, instead of being confused by what’s a workflow product and what’s an EAI product, customers were confused by which product serves which part of this still-fragmented market. To make the confusion even worse, the same analysts who lumped all of this together as BPM created divisions within the BPM space based on functionality. For example, here’s my previous diagram overlaid with Gartner’s BPM taxonomy from 2003:

They called it all “BPM suites”, but broke it down into five categories, with the key part of the market falling into the “pure-play” and “integration-focussed” categories:

• Integration-focussed BPM manages system-to-system flows for process integration across multiple applications. At the time, products in this category were slowly emerging into the human-to-human flow applications but lagged in user sophistication and user-friendliness.
• Pure-play BPM provides overarching design and connection between application-specific and integration-focussed workflows, for example, by triggering a subflow in another system and carrying the result back to the master process. These were the über-BPM products, modeling processes across the enterprise, not just subprocesses within the enterprise, and including modeling and analysis tools, business rules engines, and process simulation and optimization.

Basically, Gartner (and other analysts, quickly and slavishly followed by the vendors) put the words “workflow” and “EAI” out of fashion, and ushered in the era of referring to anything even remotely related to process as BPM. They then split the BPM space (mainly) into pure-play and integration-focussed, which corresponded roughly to the old divisions of workflow and EAI vendors. Along the way, I’m sure that they sold a boatload of analysis and consulting.

Keith Swenson has a post this week about how the term “workflow” fell out of favour during this period due to “marketing fluff”, a point with which I completely agree — after about 2001, you couldn’t use the word workflow in a conversation with an IT group without someone sneering at you for being old school. His point is that BPM has now come to be synonymous with EAI; I’m not sure that I agree with that, but I do think that BPM is one of the most misused and misunderstood terms around today.

Next: The new arrivals

Continued from Part 4.

Part 5: Creation finishes, but what’s left?

Now, the workflow and EAI markets started to converge. Both workflow and EAI products extended to include functionality that was useful in both spheres, such as business rules engines and more advanced process modelling tools. Business rules engines were typically integrated through OEM agreements, while process modelling was usually built as an integral part of the workflow or EAI product. Workflow products beefed up their EAI capabilities, and EAI improved their workflow tools.

Customers started to get confused: where did workflow end and EAI begin? When should you choose one product over another?

Next: the Tower of Babel

Continued from Part 3.

Part 4: Creating the Light, Stars, Moon and Sun

(Okay, the Genesis analogy is getting a bit old, but this really is a tale of creation.)

Organizations that had implemented workflow quickly realized that once the process became electronic instead of paper-driven, it wasn’t possible to monitor the process just by walking around the shop floor. Workflow monitoring and reporting were born from the need to understand where the process was — and wasn’t — working, and this was often a highly customized capability. After years of customers building their own custom reporting and monitoring tools, or using third-party analytics, the workflow products started to extend their native capabilities into this area. The early real-time monitoring grew to become a part of what we now know as business activity monitoring, or BAM. Process management and governance, both through this real-time monitoring and historical reporting and analysis, became a critical part of the process to customers, especially those implementing quality programs such as Six Sigma. The need to optimize processes pushed beyond analytics to process simulation and optimization tools.

EAI products grew in a different direction altogether. Most of the products provided some degree of reporting and analytics, and didn’t need the process governance associated with human-facing workflow. Instead, EAI vendors started to look outside the organization, and extended EAI to business-to-business integration, or B2Bi. This process collaboration allowed their customers to implement processes — still primarily system-to-system — that loosely coupled their business processes with those of their customers and other trading partners, not just flows between internal systems.

This would become one of the most significant advances for the type of BPM that we have today, allowing organizations to include both human and system participants in processes that span multiple organizations. The ROI on B2Bi can be enormous, and allows for the creation of a sort of configurable “business process firewall” as a standard process interface between an organization and its trading partners so that either can change their internal processes and data without disturbing the other.

Next: convergence and confusion.

Continued from Part 2.

Part 3: Creating the Firmament

In the late 1990’s, as workflow vendors saw the benefits of EAI and understood how it could replace the more heavily customized integration that was typical for workflow solutions, they began adding EAI capabilities to their workflow products. Although there were exceptions, this was typically done by the workflow vendor striking an OEM agreement with an EAI vendor to allow the EAI product to be embedded seamlessly into the workflow product.

Not surprisingly, EAI vendors saw the benefits of having some human-facing steps in the system-to-system processes, and began adding rudimentary human-facing capabilities. These functions, usually built by the EAI vendor directly rather than partnering with a workflow vendor, were fairly rudimentary since they were intended just for the repair of processes that had an exception condition that could not be handled automatically. You could say that these processes were “human interrupted” rather than “human facing”.

Although we now had systems that (theoretically) spanned the full range of the integration space, these were really just workflow systems with a bit of EAI, or EAI systems with a bit of workflow, and the vendors still sold to their strong suit.

Next: sideways diversification.

Continued from Part 1.

Part 2: Creating Night and Day

While workflow was getting its start in the 1980’s, enterprise application integration, or EAI, emerged independently for system-to-system integration.

A key player in the early days (and still today) was IBM with their MQ Series messaging, which became a de facto standard for system-to-system communication in many large organizations. Many other vendors entered the market, and a variety of technical architectures for message brokers emerged, but all had the same basic goal: to automate the near-real-time exchange of data between systems, typically mainframe-based transaction processing systems or server-based relational databases.

Even the simple functionality provided by early versions of EAI could reap huge benefits in two areas:

• It was no longer necessary to re-enter data in order to transfer it between systems, reducing data entry efforts and error rates.
• Information could be exchanged between systems in near-real-time, rather than relying on overnight batch updates.

These early EAI systems had no human-facing functionality, since they assumed that the applications being integrated provided all required user interface. In other words, if an error occurred sending data from system A to system B, the data or other problem would be fixed in one of those two systems, not in the EAI system.

Workflow and EAI lived very separate lives during that time, although they are now seen as two ends of the same integration spectrum. Not only were they created by two different camps of vendors (with a few not-very-succesful exceptions), and based on different technologies and standards, but they were sold to two different parts of the customer organization: workflow was typically sold to business units as a departmental solution, where as EAI was sold to IT as an infrastructure component.

Next: workflow meets EAI.

Since I am fast approaching the 19th anniversary of starting my first company, which provided integration services for imaging, document management, workflow, e-commerce and eventually BPM, I have a bit of an historical perspective on the the field and often end up explaining to customers how BPM got to where it is today: sometimes as part of my Making BPM Mean Business course, and sometimes just as a standalone presentation. Although a lot of readers of this blog are BPM professionals of some sort, I’m going to reproduce this history lesson here in several parts. Click on the category BPMhistory in the sidebar to see the complete set to date.

I welcome comments from any of you other “old-timers” out there, and will incorporate relevant corrections and comments into the body of the post.

Part 1: In The Beginning

In the beginning there was workflow. To be more precise, there was person-to-person routing of scanned documents through a pre-determined process map.

As far as I know, FileNet was the first to use the term “workflow” in this context, back in the early 1980’s when they started up, although they were quickly joined in the marketplace by IBM and other product vendors. Most of these early workflow systems were document-focused, that is, the only purpose of the workflow was to move a scanned image of a paper document from one person to another so that they could perform some action on a different system, such as transaction data entry. This was a logical step after organizations started scanning their documents in order to preserve and share them: why not scan them before working on them, then pass them around electronically to try and improve efficiencies? Unfortunately, any direct integration between these other systems and the workflow processes was custom-built, expensive, and not very flexible.

From these early beginnings, workflow systems evolved fairly slowly during the 90’s into products that were much more functional and flexible, primarily in the area of better integration capabilities, more diverse server and client platforms, and some basic process modelling and process monitoring tools. In many cases, however, the workflow products were still very document-centric: the document scanning/management business was such a cash cow for many of these vendors that they didn’t show a lot of vision when it came to finding uses for workflow concepts outside of routing documents around an office. That set the stage for two things to happen:

• Third-party add-ons to popular workflow products would proliferate, to do all the things that the workflow vendors didn’t deem important.
• More nimble “pure” workflow startups (the early BPM products) would ignore the document-centric side and build highly-functional workflow products that were unable to handle an enterprise workload, but would push the envelope in terms of functionality.

Before we get to that, however, we need to talk a bit about EAI.