BPM and automation

A couple of weeks ago, I posted a letter from a reader who asked the question “Is Automation BPM?” Peter Dawson and David Chassels both chimed in with their comments, and I want to add my thoughts. Excepts from the original letter:

I have a hard time referring to it as BPM when a person simply automates a business process. To me that is workflow. Granted, we have powerful stand-alone workflow engines now that allow a workflow to route work between more people and systems than ever before, but it is still workflow.

If you have a workflow process that is less focused on automating work and more focused on gathering data about a process for later analysis, then you have entered the realm of BPM.

I disagree. I started in the old workflow world: for the past 18 years, I’ve been focussed on processes that usually have a large human-facing component. I started with traditional workflow products that only provided person-to-person routing of scanned documents, which is about as old fashioned as it gets in the workflow world. Since I worked for most of that time providing services to the customers of these systems, I did a lot of automation and integration work from scratch because that functionality just didn’t exist in the workflow systems. Want an automated function in your 1993 IBM ImagePlus system to escalate items that have been sitting in a queue too long? No problem, we’d write a daemon that would cycle through the work in a queue and push specific items to an exception queue. Want to integrate your 1991 Keyfile system with an Oracle database application? Give us a few weeks, we’d write a front-end application that integrated the two systems at the presentation layer by making API calls to each system. Want to have some process reports from your 1994 FileNet Visual WorkFlo system? Okay, we’d write a batch process to interrogate the database history logs each night, extract the process-related data and print some reports.

The point is, the systems as provided by the vendors had no automation and no integration: they were concerned only with the flow of work from one queue to another. Work. Flow. Workflow. Everything else — integration, automation, process monitoring and governance — was custom built. I could, in fact, make the argument that what we did as services firms was to turn vendors’ workflow products into business process management systems.

From a semantic standpoint, the term “workflow” has been absorbed into most definitions of “BPM” (except, perhaps, those definitions created by ESB vendors who prefer an integration-focussed definition). Even the old-style workflow that I describe above — just routing from one person to another — is a part of BPM, even though a system that only provides that level of functionality would not compete very well in the broad BPM marketplace. The fact is, however, that almost every product in the BPM space has a set of functions that doesn’t quite overlap with anyone others, and doesn’t overlap at all with some other products that are also called BPM. Gartner published a report on BPMS selection criteria a few months back that listed 10 major areas of functionality, but there are many products in the BPM space that don’t have one or more of those functions covered. The problem, as I’ve written about before, is that the term is too broad, and was a bit of an artificial creation on the part of the analysts, who took workflow, EAI, and anything else remotely related to process and called it all BPM.

To get back to my rebuttal, then, I believe that as soon as you add integration and automation capabilities to human-facing workflow, you’ve got BPM (as opposed to just being able to call it BPM because workflow is now defined as a subset of BPM). From the other side of the house, the same can be said about adding human-facing functionality to EAI to create BPM.

SOA webinars this week

Two worthwhile SOA webinars from ebizQ in the past two days. Yesterday, it was Developing an SOA Ecosystem with Steve Craggs from the Integration Consortium. Craggs, who also runs a U.K.-based consulting company, Saint Consulting, spoke for a solid 40 minutes, starting with a quick but thorough definition of SOA, moving on to the concept and components of an SOA ecosystem as it should exist in your organization, and finishing up with some tips for success for building an ecosystem that can support 100’s of services. And, I loved two of his quotes near the end: “SOA is not just about technology — it’s a transformation of the business” and “SOA ecosystems ensure sustainable advantage”, very business-oriented views of SOA benefits that are often lost in the more-frequent technical discussions about SOA.

He talked about how SOA has changed from a focus purely on web services (the little blue bit in the diagram below) to a focus on all the other things that need to exist in order to make SOA successful:

My only argument with his taxonomy of the ecosystem is that he puts orchestration as part of the mediation services, and BPM floating somewhere out there as a consumer of the SOA ecosystem, almost as part of the application layer. However, I consider orchestration to be part of the larger definition of BPM, hence believe that BPM belongs as part of the SOA ecosystem itself and in fact includes most of the mediation services that he lists.

My favourite bit was where he referred to web services as “an answer looking for a problem”, and went on to list why just using web services over HTTP is problematic in building an SOA, and why it led to the development of the SOA ecosystem concept. He also gave some compelling reasons why the “best of breed” approach is still the best route in assembling your SOA ecosystem today, and the value of an enterprise service bus and the service registry. None of this is rocket science — in fact, his bit on the importance of correct granularity for services is really just an SOA twist on the age-old programmers’ dilemma of granularity — but it is a nicely packaged talk, delivered in a way that’s understandable to an SOA newbie. Definitely worth a listen.

Today, Frank Kenney from Gartner gave a short but impassioned talk on Policy-Driven SOA, followed by a fairly informative session from Sean Fitts, chief architect at Amberpoint, the webinar sponsor.

Under the heading “SOA does not come for free”, Kenney talked about everything that’s required to get SOA in place: infrastructure, methodology, services creation, testing, etc., but states that it’s worth the price — an argument that I seem to be making to customers over and over again.

He feels that you need to have a certain type of infrastructure that allows you to govern what’s going on in SOA. It’s a different view than Cragg’s SOA ecosystem shown above — kind of a slice across the lower half of Cragg’s diagram, or maybe a third dimension.

Kenney talks about services, processes and policies as assets that can be used for competitive differentiation, which really lines up with Cragg’s statement about ensuring sustainable advantage. What they’re both saying is that the old cowboy web services methods aren’t enough any more: if you really expect to reap the potential benefits of SOA, you need to start putting some discipline in place.

He finished up by stating that governance is a business issue, and that no one vendor can sell governance, but they can help you to achieve governance.

Replays are available at the links. I’d love to see these available as downloadable video for my iPod, by the way.

AIIM Expo registration now open

Registration for the AIIM Expo 2006, to be held May 16-18 in Philadelphia, is now available online. I haven’t attended AIIM in a few years, but I’m thinking about attending the exhibits. So far, however, the list of BPM vendors and BI vendors look pretty sparse — I think that most of these vendors tend to hit industry-related shows rather than general (and somewhat outdated) shows like AIIM, or hold public webinars. The advantage of attending a trade show, in theory, is that you see the new/little vendors/products that might not otherwise hit your radar screen, but I’m not sure that many of them are coming to AIIM.

And, if I want a really good product demo, there’s better ways to do that than at a trade show: I’ve had two private demos directly from vendors via Webex in the past two weeks, with another coming up soon. Not that I don’t want to visit Philly, but Webex is definitely the way to go for demos.

Gartner on BPA for 2006

Gartner has released their Magic Quadrant for business process analysis, and the first big news is that they’ve updated the style of the graphics: rounded corners, one-sided arrows and –wait for it — orange dots!

Seriously, though, this is a good report and comparision of the primary tools used for modelling and analyzing your business processes, whether or not you’re going to automate some of those processes using BPM. However, it’s the linkage between BPA and BPM that is really driving the BPA marketplace, including the whole round-tripping process that allows BAM results to be fed back into the BPA tool for further analysis and optimization.

Many of the BPM vendors have BPA included in their product; in fact, Gartner makes a note that they dropped FileNet from this report since their BPA is so deeply embedded that it’s not sold as a separate product, but they’ve added other BPM vendors such as Fuego (now BEA) and Savvion, although they languish in the lower left corner. In other words, you’d use their BPA if you were using their BPM, but you’re not going to buy it as a standalone BPA tool, as I’ve discussed previously. For that, you look to the big guns in the top right of the quadrant, such as IDS Scheer and Proforma. Although IDS Scheer focusses purely on process modelling, Proforma, Telelogic (formerly Popkin) and a few others go far beyond that to full enterprise architecture modelling, of which process modelling is an important but small part. I haven’t written much about EA lately, but it’s definitely a topic very near and dear to my heart.

I find it interesting that Gartner has chosen the vendors for this MQ in such a way that they are only in the “leaders” or “niche players” quadrants: not a one in the “visionaries” or “challengers” quadrants. They give an explanation for why this happened in the full report, but I feel that the comparison chart is less useful for tracking future trends without the visionaries and challengers. Personally, I would have put Microsoft (Visio in the BPA product under review) a bit more to the left into the challengers’ quadrant, since it’s not clear to me that their vision for making Visio a full BPA offering is particularly complete.

Last year, I posted about BPTrends‘ report on enterprise architecture, process modelling and simulation tools, which includes many of those in Gartner’s upper right quadrant. The vendors paid to be part of the BPTrends report, so it’s not exactly indepedent analysis, but it includes some good background material on the market (and it’s free).

Read my blog

Several months ago, I ordered some of Hugh MacLeod‘s Blogcards, which I hand out regularly in place of my regular corporate-looking business card to people who I meet in more casual situations. In particular, I ordered the “Read My Blog” card, because I often find myself referring people to my blog for more information on something that we’re discussing.

What I really need to do, however, is start handing them to people that I have known for years.

Yesterday, I had the occasion to meet or telephone three friends. All of them work in technology. At one point, we used to all work for the same BPM technology company (during my brief hiatus in corporate-land). Two of them (both very technical) still work there, and the third works for a company that provides aggregated business analysis content through RSS feeds, among other methods.

None of them read my blog, although the two who still work for the BPM technology company could undoubtedly learn something about their market, their customers, or even their own company by reading it. In fact, all three of them don’t read any blogs on a regular basis, and don’t know how to use an RSS reader — even the one who now sells content via RSS to businesses.

I only have one thing to say: read my blog.

Passion in entrepreneurship

I went to a literary reading last night that paid tribute to author Barbara Gowdy, who read from her novel in progress after we heard readings from five other great authors about their views and experiences of Gowdy. Catherine Graham, one of the other authors who spoke, made the most amazing statement in her short piece “It Chooses You”: she said that if you write, you should write about what obsesses you. I’m a huge believer in being passionate about my work, and definitely do my best work when it’s something that I just can’t put down, so that really resonated with me.

This morning, I saw these entrepreneurial proverbs (for geeks) by Marc Hedlund on O’Reilly Radar (via Boing Boing), which included basically the same advice as Graham offers: “pay attention to the idea that won’t leave you alone”.

James Taylor reporting from Gartner BI

James Taylor‘s been at the Gartner Business Intelligence Summit this week. On Monday, he posted some great thoughts on process, rules, BI and agility:

You can use business rules to automate decisions in business processes and then use analytics to optimize these decisions and hence the processes…

You must be able to change a process that you are monitoring when your monitoring tells you that something is wrong. Real-time measurement should not be combined with systems that take weeks or months to change.

Although there are caveats to that last sentence — for example, some real-time measurement is intended to allow the human elements in a process to change rather than the system, such as work re-allocation — I’d still like to have it tattooed on my forehead for every client to read. Making measurements with the intention of enabling agility is useless in many of the BPM installations today, not because the underlying BPMS isn’t agile, but because the customer chooses (or is coerced) to undertake a huge degree of customization that effectively pours concrete over the system.

Then later that day, he posts more on how BPM, BRE and analytics go together like chocolate and peanut butter (that’s my characterization, but I’m sure James would agree) — that seems to be a popular theme at the summit. He also posts about the Tuesday and Wednesday sessions, although less BPM-related than the Monday sessions.

Maybe because I come from the BPM side of the house, I don’t really see why the big fuss to rename parts of the BI space: BI seems to be an outdated term now, referring only to reporting on historical information from a data warehouse or operational data store. Other terms like CPM (corporate performance management), BAM (business activity monitoring), CEP (complex event processing) and EDM (enterprise decision management, which also involves BRE) have sprung up to cover the near-real-time space that I still think of as BI — after all, there’s much of the data aggregation, analytics and other common technology at the core. Many of these newer terms are touted as “[something more fabulous] BI”, such as James’ reference to EDM as “deployable BI”, but it feels a bit like the emperor’s new clothes. Maybe they’re all just BI 2.0.

Eventful mashup hits Boing Boing

Before I went to Mashup Camp, I exchanged emails with Chris Radcliff of EVDB/Eventful, and it was great to meet him face-to-face at camp. EVDB makes an API for managing event, venue, and calendar data, and Eventful uses that API in an events/calendaring/social networking mashup of events submitted directly to Eventful plus those grabbed from other event sites.

Today, I see that Eventful was covered on Boing Boing, which should bring it a huge amount of well-deserved attention. Congrats!

Is Automation BPM?

I received the following opinion by email from a Column 2 reader who makes a distinction between automation/workflow and BPM. I invite discussion, and I’ll be adding in my $0.02.

I have a hard time referring to it as BPM when a person simply automates a business process. To me that is workflow. Granted, we have powerful stand-alone workflow engines now that allow a workflow to route work between more people and systems than ever before, but it is still workflow. Just because the workflow engines now call themselves BPM does not mean that a person is doing BPM when they use them.

When a website or an article about BPM touts the benefits of BPM as being the ability to speed up or automate a business process, reduce the time work is in transit, pinpoint where work is real-time, gather all relevant data on one screen for the worker, increase accountability or allow for the automation of exceptions, I feel that that author just does not ‘get it’. Getting the right thing to the right person (or system) at the right time is workflow; its benefits have long been established (i.e. http://e-workflow.org).

Workflow is powerful and generally has very high payback. One of the weaknesses of workflow is that the process you just automated may not be the best or most efficient process. That is where BPM comes in. If you have a workflow process that is less focused on automating work and more focused on gathering data about a process for later analysis, then you have entered the realm of BPM. If there are places in the BPM process that can be automated while still keeping the focus on gathering the metrics, that is and added bonus. In my mind that constitutes the incorporation of workflow capabilities and benefits into your BPM effort.

I envision a BPM flow initially being only human steps where a person reports back that they have done certain things. In the beginning it would just be additional overhead. As more information is gathered and analyzed this human process is rearranged quickly to remove bottlenecks and inefficiencies. Over time automation of redundant or time consuming steps are added. When automation is added we must be very careful not to make the flow inflexible since the primary goal is to gather data and be flexible, not to automate. If a point of automation is identified but it will add inflexibility, then maybe a tactical workflow application is created and the BPM flow just consumes it as a service. You get the automation but keep the BPM flow as flexible as possible.

Workflow is workflow and BPM is BPM. The trend is to use the workflow engines to do BPM, but they are still distinct entities and are implemented in distinctly different ways. I think this distinction is important because you are not going to get all the benefits promised by the BPM crowed if you spend all your time automating. Automation is good, but BPM + Strategic Automation is better.

Toronto bloggers dinner wrapup

I attended the bloggers’ dinner in Toronto on Monday (photographic evidence here), and had the chance to meet Shel Israel, Alec Saunders (who helped to arrange the event and blogged about it here) and his business partner Howard Thaw, and a lot of other bloggers who I have previously met only via RSS. Something that I found interesting: every one of the people with whom I exchanged cards also uses LinkedIn (the professional networking site), although a couple of them have a dismally small number of connections. My usual practice is to check for each new contact on LinkedIn and invite them to connect immediately, before we both forget why our conversation was important.

I had a lengthy chat with Peter Flaschner and Lucia Mancuso from The Blog Studio about how they design and consult on corporate blogging for small businesses, which I found interesting considering all of my friends with small businesses who I’ve been nagging to start blogging. I use my blog as my primary marketing vehicle, and Peter and Lucia are trying to bring that same sensibility to other small businesses.

Also had interesting conversations with Mike Bowler from Gargoyle Software, who helps companies to improve their (web) software development procesess through the introduction of XP/Agile concepts; Shelley McKay and Michael Bodalski from Cricket Marketing and PR, whose home page of their corporate site is actually in blog format (!); and Peter Dawson, with whom I discussed the relative merits of using the term “landscaping” versus “architecture” as applied to business, IT and enterprise architecture.

I also met Timothy Li, an eager young engineering student at University of Waterloo who, when I told him that I graduated Waterloo engineering in 1984, rather untactfully pointed out that that was the year that he was born (he also said that he didn’t realize that the Systems Design Engineering program was “that old”). Tim followed up our exchange by posting a comment to my blog, so I checked out his blog and almost fell off my chair laughing at his retelling of a conversation where I was present. I passed it on to Rick Segal (coincidentally, the only person at the event who I actually knew face-to-face), although I’m not sure he laughed as much of the description of him as “mid aged” and obviously striking terror in the heart of younger men. 🙂

Although dinner never materialized, it was a great get-together for Toronto bloggers. Coming from a closet introvert who cringes at the thought of business networking events, that’s high praise.