sandy

BPMN survey results

I really didn’t sit down this afternoon to write that last enormous post on the Great BPMN Debate, I remembered that Jan Recker (co-author on the research paper that sparked the debate, although not a participant in the debate) had sent me a pre-release copy of a paper that he authored, “BPMN Modeling — Who, Where, How and Why”, which summarizes the results of the survey that he conducted last year. One thought led to another, and before you know it, I’d written an essay on the most exciting thing to happen in BPM standards in ages.

Back to Jan’s paper however, which will be published this month on BPTrends. He surveyed 590 process modelers using BPMN from over 30 countries, and found some interesting results:

BPMN usage is split approximately in half over business and IT, which is a much higher percentage of IT users that I would have guessed. Business people are using it for process documentation, improvement, business analysis and stakeholder communications, whereas IT people are using it for process simulation, service analysis and workflow engineering.
As you might expect given that result, there’s a wide variation in the amount of BPMN used, ranging from just the core set for basic process models, to an extended set, to the full BPMN set. It would be interesting to see a correlation between this self-assessment and usage statistics based on the actual BPMN diagrams created, although as far as I know, the survey respondents didn’t submit any examples of their diagrams.
Not surprisingly, only 13.6% received any formal BPMN training, and I believe that this is the primary reason that most people are still using only a tiny subset of the BPMN constructs in order to create what are effectively old-fashioned flowcharts rather than full BPMN diagrams.

He finished with a list of the major obstacles that the respondents reported in using BPMN, or places that they would like to see improvement:

Support for specifying business rules, which echoes many of the other discussions that I’ve seen around having some standardization between process and rule vocabularies and modeling languages.
Support for process decomposition, although I really didn’t follow his argument on what this means.
Support for organizational modeling, particularly as that relates to the use of pools and lanes: sometimes, for examples, a lane indicates a role; other times, a department. There are some things happening at OMG with the Business Motivation Metamodel and Organizational Structure Metamodel that may help here.
There are some BPMN constructs that are less often used, although it’s not clear that anyone recommended getting rid of them.
The large number of different event types is problematic: “ease of use of process modeling is sacrificed for sheer expressive power”. This is a variation on the previous point (and on the crux of the Great BPMN Debate), indicating that actual BPMN users are a bit overwhelmed by the number of symbols.

I’ll publish a link to the paper when it appears on BPTrends; it’s fairly short and worth the read.

The Great BPMN Debate

If you have even a passing interest in BPMN, you’re probably aware of the great debate happening amongst a few of the BPM bloggers in the past week:

Michael zur Muehlen and Jan Recker publish an academic research paper on BPMN usage, “How Much Language is Enough? Theoretical and Practical Use of the Business Process Modeling Notation”, to be presented at an upcoming conference. To quote the introduction in the paper, its aim is “to examine, using statistical techniques, which elements of BPMN are used in practice”, and they laid out their methods for gathering the underlying data. They used some standard cluster analysis techniques to identify clusters of BPMN objects based on usage, and determined that the practical complexity (what’s really used) was significantly different from the theoretical complexity (the total set) of BPMN. Michael teaches in the BPM program at Stevens Institute of Technology, so I wasn’t surprised to see a stated objective related to BPMN training: “BPMN training programs could benefit from a structure that introduces students to the most commonly used subset first before moving on to advanced modeling concepts.” Note that he says “before moving on to”, not “while completely disregarding”.

Michael then blogged about the paper but went further by listing three implications that were not expressed in the paper:

Practitioners should start with the more commonly-used BPMN elements, and leave the more specialized constructs for analysts who will presumably be doing more complex modeling.
Vendors that support BPMN can make a realistic determination of what percentage of BPMN diagrams can be represented in their tool based on today’s usage of BPMN.
Standards bodies should consider if they should be creating additional complexity if no one is using it.

It was these implications that sparked the arguments that followed, starting with Bruce Silver’s post directly challenging much of what Michael said in his post. It appeared to me that Bruce hadn’t read the full research paper, but was commenting only on Michael’s blog post, hence didn’t fully appreciate that the paper was really just analyzing what people are doing now, not making any value judgements about it. Bruce was a bit harsh, especially where he suggests that Michael’s “BPMN Overhead” label on the least-used objects was “clearly meant to mean ‘useless appendages’.” Bruce had some valid rebuttals to Michael’s three implications, and although I disagree somewhat with Bruce’s first two points (as I commented on his post, and was rewarding by Bruce telling me that I was stating the bloody obvious), I agree that the standard makers have not included unnecessary complexity, but that they have created a standard that the market still needs to grow into. However, I find the BPMN specification to be overly verbose, creating a greater degree of perceived complexity than may actually exist.

Michael responded to Bruce’s post by pointing out that the aim of their research was to find out how people actually use BPMN, not how vendors, consultants and standards bodies think that they use it (or think that they should use it). Michael restates his three implications in greater detail, the first two of which seem to align with what I thought that he said (and stated in my comment on Bruce’s original post). His clarification on his third point was interesting:

We actually like BPMN’s advanced vocabulary. But have you asked end users what they think? Well, we did. Not only in this study but also in Jan’s large-scale BPMN usability studies we did find that users are in fact very troubled by the sheer number of, for example, event constructs. Are they used at a large scale? No. Do users understand their full capacity? Typically not. Why is this not at all reflected in BPMN development? That is exactly our point. Sure, our argument is a somewhat provocative statement. But if it helps to channel some attention to end usage, that’s fair by our standards.

Bruce responds in turn, saying that if Michael had presented this as “statistical correlations between diagram elements in a sample of BPMN collected in the wild”, then it would have been okay, but that the conclusions drawn from the data are flawed. In other words, he’s saying that the research paper is valid and interesting, but the post that Michael wrote promoting the paper (and including those unintentionally provocative implications) is problematic. As it turns out, in terms of Michael’s group of the 17 least-used BPMN constructs, Bruce could live without 15 of them, but will fight to the end for the other two: exception flow and intermediate error event. However, Michael doesn’t say that these are useless — that’s Bruce’s paraphrasing — just that they’re not used.

There’s a bit of chicken-and-egg going on here, since I believe that business analysts aren’t using these constructs because they don’t know that they exist, not because they’re useless. Many analysts don’t receive any sort of formal training in BPMN, but are given a BPMN-compliant tool and just use the things that they know from their swimlane flowcharting experience.

Anyway, Bruce finishes up by misinterpreting (I believe) the conclusion of Michael’s post:

Michael ends his piece by asserting that the real BPMN is not what vendors, consultants, and trainers like me say it is, but the way untrained practitioners are using it today.

What Michael actually said was:

[O]ur own experiences with BPMN and with those organizations using it gave us this hunch that the theoretical usage (what vendors and consultants and trainers tell us) often has little to do with what the end users think or do (the practical usage). And why is it important to know what the end users think and do? Because it can help the researchers, vendors, consultants and trainers of this world to channel their attention and efforts to those problems real users face. Instead of the problems we think exist in practice.

Although it’s not completely clear, I believe that Michael is saying that we need to understand what people are doing with BPMN now in order to design both training and systems.

This was an interesting debate to watch, since I know and respect both Michael and Bruce, and I found merit in the arguments on both sides although I don’t fully agree with either.

There was an interesting coda on the validity of BPMN for model-driven development with Tom Baeyens weighing in on the debate and stating that BPMN should stick to being a modeling notation and not be concerned with the underlying execution details: “[t]he properties can be removed and the mapping approach to concrete executable process languages should be left up to the vendors.” Bruce responded with some thoughts on model portability, and how that drives his categorization of BPMN constructs.

If you’re at all interested in BPMN, it’s worth taking the time to work your way through this debate, and keep and eye on Bruce and Michael’s blogs for any further commentary.

What’s the Difference?

I’ve been heads down in work for the past couple of weeks since I last attended a conference, and although I seem to find time to read and comment on a number of blogs, I haven’t written much.

I did find time, however, to be involved in a small video project with my talented photographer friend Rannie for SXSW called 20 x 2. I just sat on the couch, he did all the work.

20 x 2 : What's The Difference? from photojunkie on Vimeo.

You guys at Redmonk can’t say that I don’t push your brand, even if accidentally.

BPMN and the Business Process Expert

There’s something funny about chatting via IM with someone as you’re listening to them give a public webinar, even when you do know that the presentation is pre-recorded — I was on Skype with Bruce Silver today during his webinar The Business Process Expert and the Future of BPM on ebizQ, where he was speaking with Marco ten Vaanholt of SAP’s BPX community.

Except for one “happy smiling faces” graphic worthy only of Jim Sinur’s blog ~~pimping~~ marketing team, I really enjoyed Bruce’s presentation, although I’ve heard at least parts of it before. He started with a comprehensive description of BPM and why model-driven design is so critical to process agility, which he segued into a description of BPMN and its importance in making process models executable: the heart of model-driven design. He feels that it’s necessary to define the role of Business Process Expert (BPX): someone that bridges between business and IT, creating executable requirements for BPM solutions. Obviously, BPMN is a critical skill for the BPX, and Bruce offers a number of resources including a free series of articles and e-learning modules that he’s done on the SAP BPX community and the longer paid courses that he offers online and public classes through BPM Institute. No wonder he hasn’t blogged for months: he’s been too busy creating all this.

Marco ten Vaanholt talked about the importance of BPM and SOA — fairly motherhood sort of stuff — then dug into some details of the SAP BPX community, which is an incredibly well-developed resource for anyone involved in BPM, whether you’re an SAP customer or not. The core of the BPX community is collaboration and collective learning on business scenarios, process lifecycles, change leadership, social responsibility, horizontal and vertical practices, modeling tools, methodologies and a variety of other topics. It’s not just a discussion forum, however: there’s a lot of really valuable content, such as Bruce’s articles and e-learning, from both SAP and the community in general.

Marilyn Pratt, the BPX community evangelist, has been keeping me up to date on what’s happening on BPX and the worldwide community events in which she’s been involved, and I’m looking forward to catching up with her and seeing more of BPX in action when I attend SAPPHIRE in May.

There was some good Q&A at the end about process modeling and the BPX community. Definitely worth watching the replay, which should be available online at the original webinar link above.

Blogging conferences

It seems that some conferences still aren’t plugged into the blogosphere as a PR engine, including some from surprising quarters. I applied for a press pass to next week’s O’Reilly’s ETech back in January via the press link on their site, and after a couple of weeks received the following reply from a Maureen Jennings, their conference publicist:

Press credentials for our conferences are limited and intended for journalists from established technical publications with significant readership. Therefore, I’m unable to issue you media credentials for the conference. Thanks for your interest in ETech.

Ouch! I understand that they have a limited number of press passes, but I’ve been writing this technical blog (that would be a “publication”, Maureen) for three years, and see a readership of around 1,200 unique visitors per day on my own site, plus I’m syndicated on Intelligent Enterprise and the FASTforward blog. I know, hardly Scoble numbers, but it’s not insignificant.

Regular readers also know that I have a long track record of prolific live-blogging coverage of more than 20 technology conferences dating back to 2005; at the recent FASTforward conference, I wrote over 10,000 words about the conference in two days.

I responded to Maureen:

Just to clarify, do you mean that you don’t consider a technically-focused blog about of 3 years to be an “established technical publication”?

After a pause of several days, she came back with:

As I’m sure you understand, we have a limited number of passes that we can issue for each conference. So we sometimes have to make hard calls, based on getting the news the widest possible readership. Sorry about that, and thanks for your understanding.

Yes, I understand that there are limited press passes, and that I might not get one. What I don’t understand is why your initial email to me would suggest that my blog is not an established technical publication, and doesn’t have significant readership. Someone needs to attend a remedial class in blogger relations.

State of the BPM Market white paper

I’ve been working on a white paper with BEA for the last couple of months, and it’s finally been released for free download.

We dipped into research from the big analyst firms as well as the extensive survey data collected by BEA directly. The result: a comprehensive 36-page white paper covering how and why BPM has taken a position of importance within organizations, the market size and segmentation, and market trends such as SOA and Enterprise 2.0.

Making travel civilized (almost)

I travel a lot these days, and couldn’t do it without all of the great tools available:

As I’ve written previously, if you’re applying for a Canadian passport, do yourself a huge favour and fill out the form online; that reduces the actual visit to the passport office from about 1.5 hours to about 15 minutes.
If you cross the Canadian-US border frequently (or enter either country from elsewhere), get a NEXUS card so that you can use the automated kiosks instead of standing in immigration/customs lines; this reduces your time from 15-45 minutes standing in line to about a minute. The first time that I used the NEXUS kiosk, bypassing a line of about 1000 people at Toronto airport on a Monday morning, I almost wept with joy. Works at airports and land crossings.
Use TripIt to organize and share your travel plans. TripIt is definitely the most useful online service that I’ve found in last year: you forward your air and hotel itineraries to it, and it auto-parses them into an online itinerary. I can share my trips with my other half, so that he knows where I’m staying and when to pick me up at the airport. There’s also a mobile retrieval to get any particular part of your itinerary emailed to you in short plain text in case you forget to print out the itinerary. They’ve also just started accepting itineraries from corporate booking services such as Orbitz.
Dopplr is more of a travel social network, where you indicate when you’ll be where, and can see if your friends overlap in the same locations. TripIt is trying to do something similar, but since that’s not their main focus, they aren’t quite doing it so well, and don’t have the mindshare. Personally, I’d rather have all of this information in one place (TripIt) than use two services, but I’d need to have more of my social network using TripIt.
SeatGuru and their related mobile site lets you get a good seat on any airplane, or at least avoid the ones that don’t recline and are beside the lavatory. Pick by carrier and craft, and it shows you seating plans of the plane with the good, bad and cautioned seats marked.
FlightStats and their related mobile site lets you track any actual flight, sometimes more accurately than airline sites. However, it doesn’t always get updated in the case of cancellations.
FlightAware is quite similar to FlightStats, and provides a map for a specific flight to tell you exactly where it is and when it will land. Great for checking on the inbound flight when you’re waiting to take the same plane outbound.
Google maps on your mobile device can now use cell tower triangulation to give you an approximate real ground location even if your mobile doesn’t have onboard GPS: it sucks down your battery fast, but works as a low-res GPS in a pinch.
Mobile airline sites — I’ve used Air Canada, United, US Air, Delta, WestJet, Northwest and American — allow you to check flight status and set up alerts for changes in status; some even allow you to check in for your flight via your mobile.

The best thing that you can do for yourself if you fly frequently is to use one airline (and its partners) in order to accumulate status, and get yourself to gold level status if you can. This may (depending on the airline) give you lounge access to get you out of the madding crowds in airports — a sanity-saver when there’s a massive weather delay — and get at least a desk and a plug, and sometimes free wifi, food and drink. Status sometimes gets you free upgrades to business class, as could a full-fare economy ticket if your company springs for the fully-flexible alternative. Gold status also allows you to board the aircraft during pre-boarding, which means that you can carry on the maximum allowable bag size, avoiding checking any bags. Don’t feel guilty when you cram that suitcase into the overhead bin: if you don’t use the space, someone else will.

I read a newspaper column last week (not surprisingly, on an airplane) in which the author was complaining about things that have been a fact of life for a while: taking shoes off at security, no liquids through security, no free food onboard. My advice: you know about these things in advance, so suck it up and learn to compensate. Wear slip-ons. Budget ahead to buy an overpriced bottle of water after security. Pack a lunch. If you must whine, at least whine about unpredictable events, not the ones that you know are going to happen.

Update: one thing that I forgot, is that if something goes seriously and unexpectedly wrong, don’t be afraid to complain, although try to do it nicely. A few weeks ago in Vegas, I spent one night in a $2500/night suite after starting out in a “non-smoking” room that smelled like an ashtray. They moved me out of the suite after one night, to a much nicer room than I started out in, but I’m quite sure that I’ll never actually pay that much for a night in a hotel room so was happy to have the experience.

FASTforward: Using Search to Achieve a Complete Customer View

Final session of the conference, and I’m in the financial services breakout track to hear Lee Atkinson, SVP at iDNA, and Marc Hebert, CMO of Virtusa, discuss a real-world example of using enterprise search to build a true 360-degree customer view. They started with some background on the problems with enterprise applications and how dynamic business applications (Forrester’s term for composite applications; he’s quoting from a Forrester report) are starting to take on some of the functionality. Enterprise search applications provide a layer on top of the enterprise applications, files and databases to create search solutions such as corporate search or intelligence solutions. They see enterprise search as a key enabler of dynamic business applications in order to provide both speed and agility of those applications.

The case study that they presented is the implementation of enterprise search in a top ten global bank, providing fast retrieval of structured and unstructured data, and integration with business process management (not a term that I expected to hear at this conference!) to manage events being generated by the system. Their technology platform included enterprise search, BPM, SOA and enterprise portals, and they addressed the 360-degree customer view and a GRC (governance, risk and compliance) event management system. They didn’t use search because of its inherent properties — in fact, the users don’t even see this as a search application, and the applications look like standard structured data extracted from operational systems — they use it to extract information quickly from heterogeneous data sources. Prior to this, the bank had point solutions, multiple data warehouses, and no commonality between systems and databases to provide any sort of consolidated customer view. They needed to integrate data from multiple sources, and also meet their compliance regulations.

They started by providing a compliance and risk view, plus product and customer profiling, in 2005, then expanded to include event-driven business processes and exception management in 2006. In 2007, they brought in the single customer view and more advanced business processes and analysis. Of course, there is no such thing as a single view of a customer in a large organization: different business areas (compliance, operations) have different needs, and each user type’s customized view of a customer is actually a subset of the entire customer model, with other supplementary information pulled in and additional analysis added appropriate to the task at hand. The result is a number of different end-to-end processes, such as know-your-client, anti-money laundering and sanction lists in account opening.

For them, search technology is a way to integrate legacy systems — not an application that I thought of when I considered search — although it requires a deep knowledge of the business domain and the nature of the underlying data. Once the core integration structure has been created, additional data sets and applications can be added quickly and at a fairly low cost. The focus on event-driven business applications based on search results is where search really contributes value to their applications.