HandySoft BizFlow BPM

I caught up with Garth Knudson from HandySoft a few weeks ago; I’ve looked at their BizFlow product previously, and they’re currently at version 11.3 so have a pretty long track record. Although HandySoft handles the same sort of structured processes as you see in most other BPMS vendors, they really focus on ad hoc and dynamic (unstructured) processes, where either a user needs to jump out of an existing process definition at a particular step to an unstructured flow and bring the results back to the structured process, or even create a new dynamically-defined process. Some processes just can’t be modeled in advance due to non-standard processes, changing roles and responsibilities, or process participants and actions being dependent on the participating user request: this is more like managing a project rather than a traditional process, but with BPM capabilities and structured applied to it rather than trying to manage this in email. These types of dynamic processes can form a huge portion of an organization’s processes: think of all the ad hoc processes that you have now in email, only with no control or monitoring. Some significant research efforts are underway on dealing with dynamic processes, as I saw at the academic conference in Ulm two weeks ago; Gartner and Forrester are all over this area as well, so I expect that we’ll see some advances from many vendors in this area in the next few years.

The structured parts of the process are managed by BizFlow BPM, whereas the unstructured workflow portions, whether spawned from a structured process or initiated directly, are managed by the OfficeEngine front-end application; in both cases, the process engine is BizFlow. Although you use an email-like interface to kick things off, and email is used as a transport for external recipients, this provides tracking of ad hoc processes that’s just not possible in email.

HandySoft: Specify ad hoc task detailTo start a completely ad hoc process, you create a task, specify properties such as instructions and deadlines, and attach any documents required or link to documents in a ECM repository using a URL in the rich comments field on the launch form. You use ActiveDirectory/LDAP or type in external email addresses to select participants, specify whether the participants can reassign the task further, and submit the task; then, the task is available for monitoring and you can see who has done what in a graphical view. Process participants receive tasks as calendar invitations, then click through to login to BizFlow and work on the task assigned to them, which may include adding other people to the collaboration. The web-based user interface includes a list of ad hoc tasks in which you are participating, a work list for your activities within structured processes, a launch pad for initiating new tasks or processes, and a graphical view of your SLA scorecard. From there, you can click through to the task monitor for ad hoc tasks that you have created, and see the state of each participant.

HandySoft: Task monitoringSince external participants can’t access BizFlow directly, they do their work outside the system and reply; replies from external participants are returned as a proxy, and an internal user must enter the response manually. This sort of one-step collaborative process – including multiple participants and reassignments – can replace the current practice of emailing around to multiple people for information or comments, then manually tracking to see who has responded. In an environment dominated by ad hoc processes in email, this provides a big benefit for tracking who is doing what, and when.

It’s fairly similar for launching an ad hoc task from a structured process: the structured process is modeled (using BPMN) in a similar fashion to other BPMS tools, and launched using a web form. From the participant’s UI at any step, however, you have an “Assign a task” tab that pops up the same form as was used for the purely ad hoc tasks; essentially, this allows delegating the structured process activity to the collaborative task, which can then include people who were not originally involved in the structured process. It doesn’t change the structured process; it just pops out to a collaborative task at this point, and when that completes, it returns to this step in the structured process and continues on. Just as with the standalone ad hoc tasks, this reduces the amount of unmonitored email activity that is prevalent in many structured processes where someone needs to request more information at a step.

In many BPM implementations, there is an attempt to capture all possible exceptions and collaborations as part of the structured process, but in reality, this just isn’t possible; they end up in email, phone calls and other untraceable activities. As Clay Richardson of Forrester pointed out in his vendor snapshot on HandySoft:

Traditional BPM platforms perpetuate the myth of neatly structured processes – with most vendors providing ample support for capturing reoccurring and well-defined workflows, but minimal support for managing unstructured and dynamic business processes. This chasm between the worlds of structured and unstructured processes forces teams to develop custom workarounds to handle ad hoc routing and collaborative interactions, ultimately increasing the time and cost to deliver BPM solutions.

HandySoft: Detailed stats of SLA violationAllowing an ad hoc, yet monitored, task to be launched from any point in a structured process has the effect of reducing the complexity of the structured process without sacrificing monitoring and auditability of the process. In BizFlow, launching an ad hoc task from a structured process causes an indicator to appear on the graphical view of the executing process to indicate that a task has been launched from that point, and the complete audit trail of structured and unstructured processes is maintained. If the ad hoc task isn’t completed within the specified deadline, that SLA violation shows on the structured process monitoring, and you can click to the OfficeEngine interface for detailed monitoring of the task.

On their product release agenda for later this year are a reporting service module – there’s already a fairly capable BAM functionality – and full rich internet application development capabilities to create more usable web forms for the user interface.

Vacation pause

For those of you who were kind enough to comment on blog posts from the BPM conference in Ulm last week, my apologies for being so late to approve the comments: I’ve been on vacation in Switzerland and Germany since then, and mostly off the internet due to being in small towns with crappy wifi coverage. To give them credit, the towns do have great wines, cycling along the river, boat rides, 15th century castles, all the schnitzel you can eat, and a number of other good points. 🙂

The short version of the trip report: after leaving Ulm (which was a really great place for recreation as well as an excellent conference), I headed to Zurich for the weekend to visit a friend, stayed in the old town on the east side of the river, visited the old cathedral, climbed a small mountain, went for a boat cruise on the lake, and went to a piano bar where I experienced a minor dancing injury (and I wasn’t even dancing). I then returned to Germany to spend a day in Baden-Baden and visit the incomparable Friedrichsbad Roman-style spa, then two days in Cochem on the Mosel River, complete with a 4-hour hike to Burg Eltz, a view of a castle from my hotel room and some lovely local wines, before heading back to Dusseldorf.

I’ll be getting on a flight home in a few minutes and back to work tomorrow, but will be jet-lagged so don’t expect too much witty repartee.

Wrap up notes on #BPM2009

Another great BPM conference: as with last year, I was impressed with the quality of the papers and the amount of interesting research going on.

The logistics here have also been excellent: everything well organized and executed, including someone in a “BPM2009 team” shirt meeting a group of us at the bus stop the first day to take us to the registration. The student volunteers ran the logistics on campus, including everything from staffing the coffee breaks to excellent signage everywhere we went. I haven’t discussed my usual conference complaint, wifi, since I’m on a university campus and the wifi rocks. It would be nice to have a few more power points around, but with my netbook’s 6-hour battery life, that’s much less of an issue than it used to be; since today is a short day, I didn’t even bring along a power cable.

This is primarily an academic conference, taking place on a university campus, which is much different than the usual commercial conference. Most attendees are staying in or near the historic city center, and the campus is a bit outside the center, but they provide us with a bus pass so that we can take advantage of the excellent transit system: easy to navigate even if you don’t speak German.

We had a very corporate-style reception dinner Monday night at BellaVista, a fancy rooftop restaurant with a great view of the cathedral; upscale food, and alcohol still flowing at 1am when I left, and last night we had the conference dinner at Stadthaus; also great food and wine. These dinner events were funded by the contributions of the conference sponsors: BizAgi, IBM, Metastorm, IDS Scheer and inubit. During the day at the university, however, we had lunch in the student cafeteria, which is really quite terrible. Short of catering in something else, this is the only real alternative considering our distance from town; maybe what they saved on lunch went into the bar tab at the two dinners.

Next year the conference moves to North America for the first time, where it will be hosted by Stevens Institute in Hoboken, New Jersey on September 13-16. It’s already on my calendar.

Discovering Reference Models by Mining Process Variants Using a Heuristic Approach #BPM2009

Chen Li of University of Twente gave the last presentation of the conference on process variant mining. We heard yesterday in the tutorial on flexibility about process variants; one issue with process variants is that there needs to be some way to identify which of the variants are important enough to update the original process model. The paper describes a heuristic search algorithm for determining which of the variants are important, by considering both the similarity to the original process model and the most common changes amongst the variants, e.g., the same new step is added to almost all process variants.

Since the process can be varied at runtime, the new process model doesn’t have to be perfect, it just has to be better than the original one. In general, the more activities contained in a candidate model and the more that its structure matches that of the variants, the better it is: they have created a fitness function that combines these two parameters and calculates how good a specific candidate model is. The search tree used to find the optimal candidate process model generates all potential candidates by changing one activity at a time, calculating the best fit, then replacing the original with the candidate if it is better than the original. This continues until no better candidate model can be calculated, or until you reach your maximum search distance (which would be set in order to bound computations).

The algorithm was evaluated with simulations, indicating that the most important changes tend to be performed at the beginning of the search.

Divide-and-Conquer Strategies for Process Mining #BPM2009

In the first of two papers in the final session of the conference, Josep Carmona of Universitat Politecnica de Catalunya presented on process mining calculation strategies. The theory of regions shows how to derive a Petri net representation of a process model from the process log, which shows the transition between states, but it’s very computationally expensive. This paper deals with ways of making that computation less expensive in order to deal effectively with large logs.

First is a decompositional strategy, which partitions the regions in a way that allows the identification of a set of state machines that cover all the events, then uses parallel composition to assemble the state machines into a Petri net.

The second approach is a higher-level divide-and-conquer strategy, where the event log is recursively partitioned by event class until the log sections are small enough to use other techniques. The clustering of the events is the key thing here: first, compute the causal dependency graph, then use spectral graph theory to find clusters of highly related events that will be partitioned off into their own section of the event log.

What they’ve seen in experiments using this technique is that there is a significant computational improvement (from minutes to seconds) from the decompositional approach, and that the divide-and-conquer approach allows for the processing of event logs that are just too large for other techniques.

You can get Genet, the tool that they developed to do this, here.

Discovering process models from unlabelled event logs #BPM2009

Diogo Ferriera of Universidade Tecnica de Lisboa presented a paper on process discovery based on unlabelled event logs: where the events in the log are only identified by the specific task, not by the process instance. Consider that a process instance may be executed via multiple paths through the process model, resulting in a different sequence of events logged: although you might know all possible paths through the model, you don’t know which one that any given instance followed. Also consider that processes will be executing simultaneously, so their events are intermingled.

Taking a probabilistic approach, you can take the event sequence and the source sequence (i.e., you know both the events and the instances that created them) and generate a matrix of probabilities that any given event will follow another within the same source instance: that’s some fairly standard math. He then took the event sequence and the matrix (which now represents a priori knowledge about how events interrelated), and did a fairly magic-looking calculation that calculated the source sequence based on that information.

The problem, of course, is that you don’t have the magic matrix, you only have the event sequence: initialize the matrix to something, then use the event sequence and the matrix to estimate the source sequence, then use the event sequence and the estimated source sequence to estimate the matrix. Wash, rinse, repeat until the matrix converges. You could initialize the matrix randomly, but that would take a while to converge (or would converge to a local maximum); instead, Ferreira pulled a rabbit out of his hat by stating that the matrix can be initialized with the transition probabilities present in the event sequence, that is, as if the event sequence were generated from a single source. As the paper states:

Even if x [event sequence] is the result of interleaving a number of sources, their underlying behaviour will be present in M+ [probability matrix] since consistent behaviour will stand out with stronger transition probabilities than the spurious effects of random interleaving. Therefore, M+ is a good initial guess for the estimation of M.

Amazingly, this works. Or maybe not so amazing, because I suppose there would be no research paper if this didn’t work. As the number of sources (instances) increases, the accuracy approaches that of when both the event and source sequence are known; as the number of overlapping sources increases, the accuracy drops to about 60% (by the time that you reach 10 overlapping instances), then flattens out.

There are a number of use cases for this: preprocessing for other process mining algorithms, or as a labeling mechanism to find the source instance when it is unknown. Or just if you want to show off some cool math at parties.

Tutorial: enabling flexibility in process-aware information systems #BPM2009

Manfred Reichert of Ulm University and Barbara Weber of University of Innsbruck presented a tutorial on the challenges, paradigms and technologies involved in enabling flexibility in process-aware information systems (PAIS). Process flexibility is important, but you have to consider both build time flexibility (how to quickly implement and configure new processes) and run time flexibility (how to deal with uncertainty and exceptional cases during execution), as well as their impact on process optimization.

We started by looking at the flexibility issues inherent in the imperative approach to BPM, where pre-defined process models are deployed and executed, and the execution logs monitored (in other words, the way that almost all BPMS work today). As John Hoogland discussed this morning, there are a number of flexibility issues at build time due to regional process variations or the lack of a sufficient information about decisions to build them into the process model. There’s also flexibility issues in the run time, mostly around exception handling and the need for ad hoc changes to the process. As all this rolls back in to the process analyst through the execution monitoring, it can be used to optimize the process model, which requires flexibility in evolving the process model and impacting work in progress. The key problem is that there are way too many variants in most real-life processes to realistically model all of them: there needs to be a way to model a standard process, then allow user-driven configuration (either explicitly or based on the instance parameters) at run time. The Provop approach presented in the tutorial allows for selective enabling and disabling of process steps in a master model based on the instance conditions, with a lot of the research based on the interaction between the parameters and the soundness of the resultant models.

Late binding and late modeling approaches use a pre-specified business process with one or more placeholder activities, then the placeholder activities are replaced with a process fragment at run time either from a pre-determined set of process fragments or a process fragment assembled by the user from existing activity templates (the latter is called the “pockets of flexibility” approach, a name that I find particularly descriptive).

Up to this point, the focus has been on changes to the process model to handle variability that are part of normal business, but possibly not considered exceptions. Next, we looked at runtime exception handling, such as the failure or unavailability of a web service that causes the normal process to halt. Exceptions that are expected (anticipated) can be handled with compensation, with the compensation events and handler built into the process model; unexpected exceptions may be managed with ad hoc process changes to that executing instance. Ad hoc process changes can be a bit tricky: they need to be done at a high level of abstraction in order to make it understandable to the user making the change, yet the correctness of the changes must be validated before continuing. This ability needs to be constrained to a subset of the users, and the users who can make the changes may require some assistance to do this correctly.

This was a good tutorial, but I wanted to catch the process mining session so skipped out at the break and missed the last half.

BPMN 2.0 tutorial #BPM2009

Hagen Völzer from IBM Research in Zurich – and a member of the BPMN standards group – gave a tutorial on BPMN 2.0, with a specific focus on the new execution semantics.

BPMN 1.1 has taken us a long way towards a standardized process modeling notation, but has several shortcomings: no standardized execution semantics, no standardized serialization, and lack of choreography support. BPMN 2.0 addresses these and some other issues, but does not include organizational models, data models, strategy and business rules. BPMN 2.0 is still being finalized and will be adopted as a standard sometime in 2010, but many vendors are starting to include the new features in their products. The challenge, as he points out, is representing the interrelationship between the new diagram types.

He walked through BPMN, highlighting the changes to the visual notation in 2.0:

  • Lanes can be used to structure visualization of activities by any suitable attribute, not just roles
  • Call activity type that can call another process or a global task.
  • Business rule activity type (although this could be handled with a service task type).
  • Non-interrupting boundary events, including all event types except error events.
  • Escalation events.
  • Event subprocesses, which is an event handler within a subprocess (hence has access to the subprocess context); this can be triggered multiple times for the containing subprocess, and all event subprocesses must terminate before the containing subprocess terminates.
  • Inline error handler, which appears similar to the event subprocesses but for handling errors; it can handle the error locally rather than propagating it up the scope tree.
  • Compensation event subprocess, similar to the previous two handlers but for handling compensation events.
  • Data are now first class elements, and can be used as an input or output to an activity using the new data association that describes data flow between data objects and activities or events.
  • Choreography diagrams containing (not surprisingly) choreography activities that represent the activities between the participants; the pools and message flows are hidden, and only the choreography activities and the linkages between them are modeled. The two diagram types can be combined to show the choreography in the context of the pools and message flows.
  • Conversations, which collapse multiple related message flows to visually simplify a diagram; conversations can include more than two participants.

He also walked through the execution semantics, which are based on token flows, with token being produced and consumed by activities, events and gateways. BPMN 2.0 also specifies the representation of an activity lifecycle to map all of the actions that can occur around an activity. We looked in detail at the token-based execution semantics for an inclusive join (where the number of tokens generated by the split is not known locally) in a variety of nested contexts. The complexity of this explains a lot about why the 7PMG guidelines recommend not using inclusive ORs: if it’s this hard to figure out all the possible execution semantics, it’s going to be complex for a human to understand the model visually.

He ended up with research opportunities that have been created by the new specification, such as formalizing semantics of choreography diagrams (now we know his motivation for sitting on the committee 🙂 ); it seems that there are many unresolved issues remaining.

It was useful to have a complete walkthrough of the specification as a refresher and to see the changes in context. I’m left with the impression that the event handlers and some of the other new features are exceptionally useful, but unlikely to be well-understood (and therefore used) by non-technical business analysts. As another one of the attendees pointed out, they’ve just invented a new visual programming language.

John Hoogland keynote: Change in Control #BPM2009

John Hoogland, CEO of Pallas Athena, gave the opening keynote this morning, discussing how BPM has changed since the early days of workflow and painfully customized EAI to today’s landscape of BPM and SOA. He also contrasted what he sees as the old style of assembly line-style control flow unique to US and UK companies, and the more “one and done” style prevalent in Europe.

He went through a montage of BPM vendor websites – pointing out that the smaller the company, the clearer the message – and noted that the key message on all of those sites in 2009 is agility. The promise of all of these vendors is that change is key, and that this change can be done by the business, without IT. BPM doesn’t bring flexibility: it brings control. It gives more flexibility than the processes within legacy applications, but is inherently less flexible than a process that is not under the control of a system. Agility is more than flexibility, it’s about change but in a controlled environment: change in control.

He did a closer look at many of the vendors’ messages, pointing out the nonsense of much of the marketing speak about how easy and beneficial that BPM is. Agility in that context means that the process can be re-modeled; in fact, that’s often the only point of agility that the BPM vendors allow (although I would argue that integration of rules management creates significantly more agility without process model changes).

From a modeling standpoint, having users involved in process modeling does bring agility through faster and better design decisions. Separating process flow from legacy applications also provides agility. However, these processes need to be released within an IT infrastructure, serve the needs of the entire enterprise, and tend to be fairly complex: all recipes for killing agility. Even looking at the vendors’ architecture diagrams of BPM, it is always shown as a very small component within a much larger IT ecosystem; this is typical of what I see in large implementations, where the BPM part of an enterprise project is just a tiny piece of the whole project, even though it is labeled as a BPM project. Other factors also impede agility: the issue of migrating work in progress when the process model changes, including changes to related services. The conclusion: although the vendors make process model changes look easy in the demo, it’s not that easy in reality.

Complexity of process models is also an issue. In practice, exception handling tends to cause process models to balloon far beyond the “happy path” that you model on day 1, as do regional differences, handling of unexpected external events, and interfaces between departments and with systems. For example, you really don’t want separate process models for each region, you want a single model that can handle the regional differences, but that creates both a more complex model plus a nightmare of change control. However, if business users have the ability to change their local processes through parameters without having to change the enterprise-wide process model, things become much simpler. What parameters should be available for change on a local level, then? Hoogland suggests rules/decisioning, prioritization, resource allocation including role assignment, deadlines/SLAs and other factors. This requires an explicit step during the process design to design what parameters have to be able to be changed by the business users on the fly without changing the process model (I am going through exactly this exercise now with a client, and completely agree with his assessment), and ensure that the BPMS is capable of providing those as parameters under direct control of the business.

He ended up with some final thoughts about how the market is confused by vendors and analysts leaping onto new concepts and techniques too quickly, and how we waste time and energy on non-issues such as BPM standards.

He mentioned my post yesterday about the process model comprehension tutorial, and agrees with my assessment that the vendors have a huge amount to learn from academic research. I’m beginning to think that my true calling is as an evangelist bringing the word of new and useful academic BPM research to the BPM vendor community, although I have no idea how to monetize that. 🙂

There was a good Q&A following, with discussions on the potential value of bringing academics and vendors closer together; he admitted that a year ago, if asked to attend this conference, he would have been concerned only with whether there would be customers here. Now – hopefully due to some of my prodding of the vendors encouraging them to attend – he understands that vendors have a lot to gain from seeing what the academic BPM community is doing. There’s definitely monetary issues: academics need funding in order to continue research, and much of the research is still sufficiently early that a significant amount of research and development would still be required by the vendors to bring it to market. Many small vendors just can’t afford their own pet researchers such as IBM and SAP have, and are very focused on getting a product out this year as opposed to investing in the long-term potential from BPM research.

There must be a way to have funding flow from BPM vendors to BPM researchers without having a vendor try to control the speed and direction of the research, and without even having a specific researcher tied to a specific vendor. Is there a place for a virtual institute of BPM research spanning the many great BPM research institutions, funded by a consortium of vendors, with all having access to the end results? I’m thinking that for less than the cost of having a booth at a Gartner conference, a BPM vendor could contribute significantly to BPM research and reap the rewards.