Building Scalable Business Automation with Microservices – a paper I created for @Camunda

scalable-business-automation-with-microservicesLast year, I did a few presentations for Camunda: a keynote at their main conference in Berlin, a webinar together with CEO Jakob Freund, and a presentation at their Camunda Day in Toronto, all on the similar theme of building a scalable digital automation platform using microservices and a BPMS.

I wrapped up my ideas for those presentations into a paper, which you can download from the Camunda website. Enjoy!

Blast from the past: what I was writing about on Column 2 in 2010

After I posted earlier this week about top 10 blog posts for 2019, I decided to take a look back at my archives and see what was happening 10 years ago. We were just starting to crawl out of a recession, and interesting things were happening in the industry.

Acquisitions! IBM had announced the acquisition of Lombardi just before Christmas 2009, and closed the deal in January 2010. Also that month, Progress Software acquired Savvion. Later that year, I ranted briefly about how when vendors acquire multiple overlapping products, it’s not good for the customers.

Standards! BPMN 2.0 neared release, and started to gain traction with many vendors. There was a fiery online debate about the use of BPMN by business people later in the year.

Conferences! The academic BPM conference came to North America for the first time, landing at the Stevens Institute in New Jersey. I spoke at the Software 2010 conference in Oslo on BPM and Enterprise 2.0 (what we would now call social BPM), a topic I’d been covering since 2006 and was “discovered” by the large analyst firms around 2010. I went to a lot of vendor conferences that year, and blogged about them while there.

Cloud! Faced with a growing number of vendors offering cloud BPM products, I climbed up on my usual soapbox about how geography does matter when it comes to cloud, at least US versus non-US hosting locations. It took some vendors a long time (and a few EU regulations) to realize this.

Case management! This was definitely the year that case management started to hit the BPM vendors’ radar, with many of them adding or acquiring capabilities to handle less-structured processes. I did a webinar with Keith Swenson on the top of agile and social BPM: while Keith and I don’t always agree, we always have interesting conversations.

New products! The creators of jBPM moved over to Alfresco and started the Activiti project. The reverberations of this are still felt today, with both of those creators having moved on, and at least two notable forks of Activiti currently available.

Column 2 turned five that year, which means that this year will be fifteen years that I’ve been blogging.

The top Column 2 posts of 2019

A couple of weeks ago, I wrote about my post that has had the most visits over all time, a 2007 post on policies, procedures, processes and rules.

Here’s what was the most popular in 2020:

  1. That same 2007 post, Policies, procedures, processes and rules. Obviously, this theme strikes deep with a broad range of people, and a recent comment on that post was from someone who had used it as a source in developing definitions for the PMI’s Project Management Body of Knowledge (PMBOK).
  2. An even older post from 2005, Adaptive approaches. This is about application development and deployment methodologies, and would now be called “Agile”. I also talk about my usual method of “get something simpler into production sooner”, which would now be called “minimum viable product”. The only thing that has really changed here is terminology.
  3. The first product-related post, my 2012 post Introduction to AWD 10. I was at the user conference for DST Systems, now part of SS&C Technologies, and wrote about what I saw at a session where they presented the new product version.
  4. Another terminology post, this one from 2017: What’s in a name? BPM and DPA. This was prompted by Forrester’s move to relabel business process management (BPM) systems as Digital Process Automation (DPA), and the ongoing confusion in terminology. This problem continues today, with Gartner sticking to the term iBPMS (Intelligent BPM Suite) but shifting it to mean low-code application development platform.
  5. The first that was originally published in 2019, Snowed in at the OpenText Analyst Summit 2019. In the midst of a massive snowstorm (I arrived on one of the last flights before the airport shut down), I attended OpenText’s analyst meetup in Boston, and this post was on the main keynote featuring CEO Mark Barrenechea’s vision for their future product direction.
  6. From the 2019 bpmNEXT conference, bpmNEXT 2019 demos: microservices, robots and intentional processes with Bonitasoft, Signavio and Flowable. My conference live-blogging is usually popular with those who can’t make it to the conference themselves, but this post was likely read more than most because it covered the Flowable chatbot + CMMN demo that went on to win the “Best in Show” award.
  7. Another throwback to 2005, Shallow vs. Deep Knowledge. I was writing in response to a post from EDS that said that they believe that someone working on a business application based on vendor components really had to see the vendor’s source code to do this right. I disagreed.
  8. A post on service-oriented architecture standards from 2009, The Open Group’s Service Integration Maturity Model and SOA Governance Framework. This was the result of a briefing that I had with them in advance of the standards’ release; to be honest, I’ve never used these frameworks and have no idea how broadly they were adopted.
  9. A post from this year’s academic BPM conference in Vienna, Day 1 @BPMConf opening keynote: Kalle Lyytinen on the role of BPM in design and business model innovation. The keynote discussed the concept of digital intensity, namely, the degree to which digitalization is required to perform a task, and how technology is changing the way that we do things on a micro level.
  10. Another post from the academic BPM conference, Workshop at @BPMConf on BPM in the era of Digital Innovation and Transformation. This workshop day preceded the keynote mentioned above, and covered a number of talks on digital transformation. This is the only one of the top 10 posts for 2019 that covers a presentation that I made, since I was invited to give a short talk at the end of the workshop.

I blogged quite a bit less in 2019 (in fact, my blogging has been a bit slow the past couple of years) although I had a lot of activity around conferences and a few product briefings. I’ve been fairly active on Twitter, and I’m looking at ways to bring together some of the links that I post there onto the blog for more discussion.

Looking forward to 2020!

Policies, procedures, processes and rules (redux)

For some reason, the most popular post of all time on this blog is from 2007, on policies, procedures, process and rules: definitions, differences, and how processes and decisions intersect both in modeling/documentation and implementation. If I look at the past year, quarter, month or week, it’s still the most viewed post in those time periods, even though it’s over 12 years old. Interesting to see what is still relevant after all this time, since blog posts typically have a half-life of only a couple of days.

A few days ago, someone named Jason Gorman (who did not note their affiliation) added a lengthy comment to the post, describing how he is reviewing the 6th edition of the PMBOK Guide, and has documented their definitions of policy, process, procedure and rule based on various sources including my original blog post. Glad I could be of service!

Conway’s Law and end-to-end processes – my new post on the @Trisotech blog

I’ve written another post for the Trisotech blog on end-to-end processes, and how Conway’s Law can prevent good horizontal integration across your organization’s processes. Conway’s Law is an adage that states that organizations design systems that mirror their own communication structure: basically, if your interdepartmental communications is not that great, the underlying systems are unlikely to do a good job of it either.

I go into some detail of the problems that occur when an organization is functionality siloed with poor communication between areas, and how you can start to find your way out of it.

You can find all of the posts that I’m creating for them here.

The Need for Goal Alignment — on the @trisotech blog

I’ll be writing a few guest posts over on the Trisotech blog, starting with this one on goal alignment through the hierarchy of your orgnization to make sure that you’re not only doing the right thing, but doing the thing right. As I mention over there, I have this conversation with almost every enterprise client that I talk to, and thought it would be good to put down some thoughts around a goal alignment structure like this:

This is (techinically) sponsored content, although I don’t discuss Trisotech products at all, so I’m not reprinting it here but encourage you to head over there and give it a read.

How to (not) become a digital enterprise – @jakobfreund CamundaCon keynote

I thought that I was done with my CamundaCon coverage, but noticed that Jakob Freund is blogging more details about what he covered in his keynote. I spent most of his keynote behind the curtain waiting for my turn to speak, but was able to see it again when they posted the video of his presentation.

He’s doing a five-part series on the themes that he covered, based in part on their experiences with their clients over the past years, with the first two available here:

  • Part 1: Intro to the four key elements of becoming a digital enterprise
  • Part 2: The first key element, customer-focused innovation

If he keeps to his posting schedule, the next one should be up tomorrow.

My upcoming webinar with @Camunda on scalable business automation and microservices

Last month at CamundaCon2019, I gave a keynote on using a BPMS and microservices to build a business automation platform. It’s not for everyone, but I specified some of the use cases and what needs to go into it. You can see the recording here:

On November 12, I’ll be expanding on some of those themes in a webinar sponsored by Camunda:

  • Why a monolithic architecture, whether a legacy application or an all-in-one business automation platform, lacks the agility and scalability required for businesses to survive and thrive
  • How a best-of-breed business automation platform can be assembled from a set of components in a microservices architecture
  • How to migrate from your legacy systems to a best-of-breed business automation platform

In particular, the new bit on migrating legacy systems is drawn from my best practices developed over years of consulting as a systems architect/designer.

You can sign up for the webinar here; if you can’t make it on that date, sign up anyway and you’ll get a link to the recording.

CamundaCon 2019: Monolith to microservices at Deutsche Telekom

Friedbert Samland from Deutsche Telekom IT and Willm Tüting from their technology partner conology presented on Telekom IT (the internal IT provider for Deutsche Telekom) migrating from monolithic systems to a microservices architecture while also moving from waterfall to Agile development methodologies. In 2017, they had a number of significant problems with their monolithic system for wholesale orders: time to market for new features was 12+ months, lots of missing functionality that required manual steps, vendor lock-in, large (therefore risky and time-consuming) releases, and more.

Willm Tüting and Friedbert Samland presenting on the problems with Telekom IT’s monolithic wholesale ordering system

They tried a variety of approaches to alleviate these problems, such as a partial Agile environment, but needed something more radical to make a difference. They identified four major drivers: microservices, cloud, SAFe (Scaled Agile Framework) and devops. I’m sure everyone in the audience was familiar with those concepts, but they went through how this actually works in a large organization like this, where it’s not always as easy as the providers say it will be. They learned a lot of lessons the hard way, such as the long learning curve of moving to cloud.

They were migrating a system built on the Oracle BPEL engine, starting by partitioning the monolith in terms of data and functionality (logic and processes) in order to identify three categories of microservices: business process microservices, data microservices, and domain-specific microservices. They balanced orchestration and choreography with a “choreographed orchestration” of the microservices, where the (Camunda) process orchestrations were embedded within the microservices for handling processes and inter-service communication. By having separate Camunda instances with separate databases for each microservice (which provides a high degree of scalability), they had to enhance the monitoring to get an aggregated view of all of the process flows.

This is a great example of a real-world large-scale implementation where a proprietary and monolithic iBPMS just would not work for the architecture that Telekom IT needed: Camunda BPM is embedded in the services, it doesn’t pre-suppose fixed orchestration at the top level of an application.

Although we’re just halfway through the last day, this was my last session at CamundaCon, I’m headed south for a short weekend break then DecisionCamp in Bolzano next week. Thanks to the entire Camunda team for putting on a great event, and inviting me to give a keynote yesterday.

CamundaCon 2019: @berndruecker on Zeebe and microservices

Camunda co-founder Bernd Rücker presented on some of the implementation issues with microservices, in particular following on from Susanne Kaiser’s keynote with the theme of having small delivery teams spend more of their time developing business capabilities and less on the “undifferentiated heavy lifting” infrastructure bits required to support those. This significantly reduces the cognitive load for the team, allowing them to build the best possible business capabilities without worrying about arcane configuration details. Interestingly, this is not that different from the argument to move from a business process embedded within a business system logic to an externalized process in a BPMS — something that Bernd has a long history with.

He went through an example of the services behind a train ticket booking, which requires payment, seat reservation and ticket generation services; there are issues of latency and uptime as well as the user experience of how the results of those services are presented to the customer. He referenced the Reactive Manifesto as a guideline for software design patterns that are “more robust, more resilient, more flexible and better positioned to meet modern demands”.

Event-driven choreography is a common pattern these days, but has the problem of not being able to visualize the overall process flow between services. This can be alleviated somewhat by using event monitoring overlaid on a process model — effectively process discovery if the flow is not standardized or when it changes — or even more so by orchestrating standard parts of the flow to combine event-driven and orchestration patterns. Orchestration has the advantage of relocating the coupling between services in an event-driven flow to the orchestration layer: although event choreography is seen as loosely-coupled, there’s a lot of event listening that has to be built into the services, which couples them more closely. It’s not that one is good and the other bad: there’s a place for both choreography and choreography patterns in software development.

He finished with a discussion of monolithic legacy software and how to deal with it: from the initial step of just adding APIs to access functionality, you gradually chip away at the monolith’s capabilities, ripping them out replacing with externalized services.