Month: October 2007

Policies, procedures, processes and rules

Most of my customers are large financial institutions, and I help them to select and implement BPM within their organizations. The technology is only one part of that, however: I’m almost always helping them with their business processes as well. Since policies and procedures drive processes, I often end up in the thick of their policies and procedures, and that’s where the confusion starts.

First of all, there’s the definition of terms. What’s the difference between a policy and a procedure, when many people lump them together as if PoliciesAndProcedures were one word? I like this definition:

Policy is a mandate and directive from the top of the organization. Its purpose is to influence behaviour. From it, management provide the overarching principles under which the business operates. It should not vary in its message or enforcement model.

Procedures are process specific and detail the steps taken to achieve an objective. Procedures include operations manual, user manual, and all manner of process documentation.

I see policies as the rules, or laws, of an organization, whereas the procedures are the processes used to enact the policies. The problem is, however, that many companies see policies and procedures as belonging to the Legal/Compliance department, and create another set of processes — usually referred to as an "operational guide" — that are created and maintained by the operational area that executes the actual processes. If you throw in a BPMS, then some (but rarely all) of these operational procedures may be further documented in the process descriptions within the BPMS or a BPA tool.

What’s the distinction between a policy and a procedure? Is there a difference between a procedure and the operational description of a business process? What about between the business process and the process model in a BPMS?

Secondly, there’s the responsibility issue that I referred to above: who’s responsible for each of these essential bits of corporate documentation? Legal/Compliance is almost always handed policies and procedures, but what about the case when the procedures are actually descriptions the operational business processes? Should policies be left with Legal, and procedures given to the operational areas, with Compliance there to make sure that everything matches up? Or are the operational process descriptions a separate, more fine-grained version of the procedures, leaving the procedures with Legal and the operational processes with Operations?

If process maps are created within a BPMS, do they become part of the business process documentation, replace part of it, or stay as a separate "implementation view" of the processes? I’ve definitely seen cases where the process maps in a BPMS bear little resemblance to what the business perceives as its processes, either due to limitations in the BPMS environment or to the business having an incomplete view of the process.

And if there’s four separate types of documentation — policies, procedures, business processes and BPMS process definitions — who’s responsible for keeping them all in synch?

Third is the whole technology issue: how is all of this information captured, published and synchronized? There are a tools such as RulesArts and RuleBurst (both of which I saw last week at the Business Rules Forum) that help to capture policies as high-level non-executable rules — an approach that makes more sense than just trying to document them free-form in a word processor while praying for consistency. Check out the Flash demo on the RuleBurst site to see what this can look like. Some of this systems are also business rules engines, that is, they execute the rules and can be called from other applications; some are just platforms for non-technical users to document policies, detect gaps and exceptions, and help to ensure compliance.

As we move into procedures, operational guides and process definitions, it’s all about processes. Processes based on rules (and what process isn’t?), but processes nonetheless. Those organizations documenting their policies in a word processor are likely also documenting their procedures in the same way — in fact, possibly in the same document — using descriptive text and a few diagrams. At some level of detail, someone starts drawing process maps, although these are usually as illustrations to the descriptive text rather than a replacement for it.

The two biggest issues in all of this technology are synchronization (usually manual, and therefore almost certainly out of date) and publishing (ditto). From the synchronization standpoint, there needs to be something that links the policies (rules) with the various granularities of process descriptions (both text and graphical) and either keeps them in synch or alerts someone when related pieces are modified. For publication, none of this information is of any use unless it’s in the hands of the people who need it; that means that there needs to be an easy (or automated) way for all of this information to be published within an enterprise and accessed with nothing more than a browser and network authentication.

What starts to become shockingly apparent as you dig into the technology is that policies are about rules, and procedures are about processes. Yeah, I know, I said that at the start of this post, but it’s not just some abstract concept, it’s about how you need to document and implement policies and procedures. The crux of the issue is in the crossover from rules to process, since a rule (policy) usually doesn’t dictate the operational procedure required to enact it, hence there’s not a clear technology path to map from policies to procedures. If policies are maintained in a high-level rule repository and procedures are maintained in some combination of descriptive text and process maps, what’s the missing link between them?

Policy and procedure documentation is just one place where business rules and business processes intersect (they touch again at the point of process execution), and I’m interested in exploring the ideas around this. I’ve put forward more questions than answers — feel free to join the conversation by commenting on this post, tracking back from your own post, or dropping me an email.

Webinar on Enterprise Decision Management tomorrow

After attending the Business Rules Forum last week, either I’m more aware of related events or I’m on more mailing lists for rules/decisioning vendors. In either case, Fair Isaac is putting on an Introduction to Enterprise Decision Management webinar tomorrow at 2pm Eastern. From their description of the event:

Learn how your organization can automate and improve decisions by:

  • Taking control of your decisions and make them a corporate asset
  • Separating operational decisions from processes and systems for maximum agility
  • Using business rules management systems to ensure decision consistency and speed
  • Applying different kinds of analytics – descriptive, predictive and decision – to make more precise and profitable decisions

It’s free, just sign up online.

Integration World next week

This week I’m not travelling — a chance to catch up on the work that I’m actually paid for 🙂 — but next week, I’m off to the Software AG/webMethods user conference, Integration World. Watch for my blogging coverage from there.

Disclosure: as with all vendor user conferences that I attend, Software AG is covering my expenses, although I am not paid to attend.

Meeting the bloggers at BRF

Last week at the Business Rules Forum gave me a chance to meet many people who I’ve never met face-to-face, but feel that I know from our exchanges of blog comments and emails: at one point, I was standing around talking to James Taylor, Rolando Hernandez and Scott Sehlhorst.

James was certainly the most prolific in blogging about the conference: he live-blogged the sessions that he attended (even mine), so you can compare with the posts on those sessions that I wrote. He has a wrap-up post with pointers to all of the blogs that he found with coverage of the event.

Smart Enough Systems

I’m sure that James Taylor has almost given up on me ever writing a book review of Smart Enough Systems: I wrote a brief advance review back in April that’s printed in the book, but nothing since it was released. This week, I’ve been immersed in business rules and decisioning, and had a chance to finally meet James face-to-face after a couple of years of emailing back and forth. Also, James’ situation has changed since the book was released: he’s left Fair Isaac and is now an independent, working (I think) with his co-author, Neil Raden. Neil, who I met very briefly this week, is an independent consultant and analyst who’s been focussed on business intelligence for quite a while; James refers to his work as “BI 2.0” (a term that I think that I invented here in early 2006). The two of them met through James’ blog and started the conversation about how someone needed to write a book about this crossover area between business rules and business intelligence.

Just to get started, here’s my pre-release review:

Taylor and Raden’s central manifesto highlights that it’s critical to embody more intelligence in today’s business decision-making and have consistent, automated decisioning built into business processes in order to remain agile and competitive in today’s fast-moving market. They take you through the core concepts of enterprise decision management (EDM), dive into the underlying technologies, then address how to integrate EDM into your business processes to create your own Smart (Enough) Systems.

By focusing on operational decisions that contribute to corporate strategy, Smart (Enough) Systems provide the ability not only to create agile business processes, but to have these processes be self-learning based on historical results. Instead of simply capturing operational process statistics in a data warehouse for later analysis, Smart (Enough) Systems use that knowledge to inform the business rules and allow them to adapt their guidance of the decision-making process. By extracting the decisions from legacy applications, static enterprise applications and manual procedures, and managing them within a shared enterprise decision management system, operational decisions can be applied consistently — and modified easily for processes in flight — across the enterprise.

What I like about the book is that it provides an overview that will get business people interested in the topic, as well as providing a practical guide to getting started. There’s a lot of books focussed on analytics and business rules already, but most of them assume that you already know what you’re going to do with these technologies; in many cases, it’s hard to discover the right decisions on which to focus, since some things are more important than you may have originally perceived when starting a project.

As I heard this week, there’s still a strong tendency to sell rules technology to IT as a way to implement systems fast and cheaper, instead of selling decisioning to the business as a way to solve business problems. For the most part, decisions in business processes are unconsciously relegated either to a developer coding them into an application, or to a front-line worker executing them on an ad hoc basis. Decisions, and the rules that underlie them, need to be made explicit: therein lies the path to both compliance and agility. I joked yesterday about Jan Venthienen’s presentation that he’d like to reduce all processes to a single activity and have all business logic in a rule set, but there’s definitely value in what he’s saying: we need to seek out the decisions in processes, capture the ever-changing business rules that are embedded within them, and move these out of the processes and into rules/decisioning systems. The reality is that process maps don’t change all that often if the decisions and business rules are stripped out of the processes: most business agility is based on rule changes, not process changes, especially for high-volume core transactional processes. However, since we often code rules directly into the business processes — making these processes some combination of processes and decision trees — we perceive the need to be that of changing processes rather than changing rules. And so (and I’ll be totally blackballed in the BPM community for this particular blasphemy), the entire BPM industry has focussed on building tools that allow for easy modification of process maps by the business, when maybe they really should have been focussed on pushing their customers towards the integration of business rules for decisioning in order to greatly reduce the need to modify process maps.

Climbing off my soapbox for a moment and returning to James and Neil’s book, they focus on a key benefit of this sort of smart decisioning in operational processes, which is the ability to replace the “irreplaceables”, such as insurance underwriters, before all the boomers take their knowledge and retire to Palm Beach. This greatly controls risk: not just the risk of the workers leaving, but the risk of bad or inconsistent decision-making. By allowing decisions to become more personalized based on the customer and scenario, this can also provide the ability to be more customer-centric, as well as agility in the face of changing regulations and market conditions.

They see a number of roadblocks to this sort of smart decisioning at the operational level:

  • Everyone is focussed on strategic issues and not taking their operations seriously; however, the aggregate effect of a small but poor decision on millions of operational transactions is, in fact, strategic. In other words, execution matters.
  • Business and IT have an antagonistic relationship. This is often blamed on IT being arrogant and not allowing the business to participate in technology initiatives, but there’s also some amount of the business not wanting to take responsibility since that means that they can’t blame IT if something goes wrong.

The ideas that James and Neil put forward in their book are a great place for business and IT to start collaborating on how to make smart enough systems.

You may also want to listen to the interview that Scott Sehlhorst did with James back in June (I know, I know, when the book was actually released).

BRF Day 3: Good Business Rules in Process — Eliminate 65% of the Activities

I couldn’t quite drag myself out of bed for the 8am sessions, but I did want to hear Kathy Long of the Process Renewal Group talking about process and rules. She talked about how to derive rules from processes, and use them as guides to the process. There’s a number of process-related problems that can occur when the rules are not explicit: assumed policies, activities with experience as the only guide, and inconsistent (and therefore likely non-compliant) processes. The key things to consider when analyzing the guides for a process can be focussed around what happens at a given activity (are what knowledge is required, what decisions are required, what reports have to be generated) as well as a number of other factors; she presents a number of different questions to ask in order to drive out the rules and make them explicit.

She also made a distinction between policies and rules, where the key differentiator is that rules are actionable, whereas policies must be interpreted into more concreted business rules in order to take action. Within rules, there’s both structural rules (can’t be broken) and operative rules (which have a bit more wiggle room); this sounds a bit like the distinction between a fact and a rule that I heard in a session yesterday, which makes me unsure that there’s a really common vocabulary for some of these things.

Looking at some of their process analysis techniques, she presented categories of activities as real value-added (impact the customer’s requirements), business value-added (required to run business, such as regulations), and non value-added (that 65-85% of work that doesn’t contribute to either RVA or BVA). There’s a whole list of verbs — adjust, approve, expedite, inspect, verify and many others — that tend to indicate that activities are NVA and should be considered for elimination. Many of these are because something wasn’t done right the first time; a lot of the NVA activities can be cut if there’s ways to reduce the error rates in the RVA and BVA activities. This isn’t, of course, really about rules: it’s about process improvement. Sure, the appropriate addition of business rules can certainly lead to process improvement, but it’s also about the myriad other ways that we improve processes, such as establishing accountability and eliminating unneeded steps that are only there for historical reasons. Some thoughts that Long gave us to take away:

  • The greatest opportunity to improve a process is by changing the rules
  • Challenge all policies
  • Validate all compliance interpretations
  • Eliminate the use of assumed policies
  • Ensure that all rules are documented including the use of experience/knowledge
  • Create consistent rules across the enterprise
  • Structure rules so that they can be easily changed
  • Allow the business to design its own processes

I was surprised that she didn’t talk at all about some of the technology issues such as how BPM and BR can be used together to improve processes, but her focus was not at all on technology: her only case study was about improving a process based on a manual procedural change.

I have to head off for a mid-day flight home, so that was the end of my Business Rules Forum experience. I’ve actually learned a lot here, which has made my time here definitely worthwhile. However, I’m still left with the feeling that I mentioned on my first post back on Tuesday: we need to start having much more crossover between different technology areas such as BPM and BR. I’ve been writing since mid-2005 about the importance of looking at BPM and BR together, but in spite of the technology advances that have occurred since then to facilitate this, I’m not seeing much happening in the real world.

BRF Day 2: How Business Rules Re(Define) Business Processes: A Service Oriented View

For the last session today, I attended Jan Venthienen’s session; he’s a professor at Katholieke Universiteit Leuven. He talked about different representations of rules, particularly decision tables (at length, although in an interesting way). He talked about the problems with maintaining decision trees, then as he moved on to business processes, he showed how a business process with the rules encoded in the process as routing logic was really just a form of decision tree, and therefore difficult to maintain from a rules integrity standpoint. As rules are distilled out of and separated from the processes, the processes become thinner and thinner, until you have a single branch straight-through flow. I have the feeling that he’d like to reduce the process to a single activity, where everything else is done in a complex rule called from that step. I’m not sure that I agree with that level of stripping of logic out of the process and into the rules; there’s value in having a business process that’s understandable by business users, and the more that the logic is encapsulated in rules, the harder it is to understand how the process flow works by looking at the process map. The critical thing is knowing which rules to strip out of the business process, and which to leave in.

He’s doing research now to determine if it’s possible to specify business rules, then automatically derive the business process from the rules; an interesting concept. In order to do this, there must be rules that constrain the permission and obligations of the actors in the process, e.g., an order must be accepted before the product is shipped. This presents two possible architectural styles: process first, or rules first. In either case, what is developed is an architecture of rules, events and services, with a top layer of business rules and processes, a middle layer of services and components, and a bottom layer of enterprise applications.

BRF Day 2: Using Business Rules to Enable a Closed Loop of Compliance

I’m eager to learn more about the relationship between policies, procedures and rules, and how they relate to compliance, so I sat in on a presentation by Peter Still of RuleBurst. There’s a pretty high percentage of vendors on the speaker roster, but so far the quality has been good so no complaints.

The theme of Still’s talk is that the business rules approach will only gain critical mass if it stops being a technical implementation tool and starts being a business problem-solving tool. The current pitch from the business rules vendors is that this is a way to implement systems faster and cheaper, while allowing the business to access some tuning parameters, but this is really focussed on the technological capabilities and not the business value of business rules. This is such a perfect mirror of the BPM field, where BPM has just barely moved from a purely technical sell to something that’s now being sold more and more to the business side of an organization, so I can completely understand where the business rules market is and the challenges that lie ahead in shifting the focus of their marketing message. Worldwide market for business rules product revenue is $250M — not a lot when you consider the size of related markets — and it could be a lot larger if there was greater recognition of the business benefits of business rules.

A perfect business case for re-targeting the business rules message is compliance: it’s an enterprise-wide initiative with executive support where business rules can be included in the decisioning at key points of the process. Although business rules aren’t the complete answer to compliance since compliance is a very process-focussed initiative, rules can be a significant contributor to compliance efforts. One of the difficulties with compliance is that many regulations, such as Sarbannes Oxley, are pretty vague since they have to deal with such a broad range of companies, and it’s difficult to determine precise business rules to implement them. Compliance at a transactional level is a mostly automated application of BPM and business rules, but as you move up to risk management and higher-level compliance factors, there’s less automation although still opportunities for business rules to be wrapped in a compliance framework, such as using business rules to classify a risk although the management of that risk may be done manually. Still maintains that there’s a link between transactional and operational compliance, and believes that business rules can help with that link although that’s not recognized by most business rules vendors.

As with most other complex applications of technology, you can solve this with an integrated compliance and rules solution from a single vendor, or go for a best-of-breed approach. Still recommends the former approach, and invited us to drop by his booth to check out what RuleBurst has to offer in this area.

BRF Day 2: True Adventures in Business Rules

Paul Armborst of the Westfield Group presented on his experiences in implementing business rules for their various insurance applications over the past 6 years. He sees one of the key problems is in documenting business rules, with two main choices for how to do it:

  • Define the rules in a repository first (a rules management/modelling tool). Although this is the most likely approach for the first definition of rules within an organization, they’ll become out of sync as soon as they are implemented since the rules will be modified in the executing code or BRE rather than in the repository.
  • Define the rules directly in a business rules engine, then generate the documentation in some automated fashion (likely provided by the BRE vendor)

He sees that the best way would be a combination of both approaches: a throw-away repository to document rules as they are discovered, and an extract from the BRE to provide ongoing documentation.

He pointed out one of the problems with introducing business rules: “real” developers want to write Java code, not this airy-fairy business rules stuff, which is an argument that I see against BPM as well. As IT budgets get squeezed, however, development teams will start to look for ways to reduce the amount of code that they have to write and pass some of the tuning capabilities directly over to the business; both BRE and BPM provide these types of capabilities.

He discussed various methods of implementing business rules:

  • Decision tables, noting that the BRE vendor needs to provide some sort of analysis tool to detect gaps and overlaps in the rules, with the caveat that it’s possible to construct a decision table that’s too big to reasonably maintain.
  • Decision trees, which can provide the same functionality in a decision table, but in a graphical form; if the decision points are not in the right order, the number of nodes can multiply unnecessarily, and overly-large trees can be difficult to follow.

He also discussed stateful and stateless implementations of business rules: although stateless is simpler to implement, stateful allows for running only the rules that use data that has changed since the last evaluation of each rule.

There were some last comments on end user rule maintenance: all of their rules are written by developers, but they’re thinking about how to offer some rule creation and modification capabilities to end users. It’s important to have a BRE that allows some sort of restricted view for less technical users, but it’s also necessary for the techies to do some helpful things like naming objects in a manner that users can understand rather than, say, reverse Hungarian notation. Users who have access to create rules need to have some notion of the logic required to do so, and there needs to be some provision for testing.