Philipp Klinger
Architect
Julian Leischner
Senior Consultant
In today’s fast-evolving digital landscape, software isn’t just a collection of functions performing specific tasks – in fact it’s a web of interwoven workflows that dictate how systems interact and respond to dynamic business needs.
At Senacor, we specialize in complex and large-scale IT transformations. Looking at those IT platforms, we witness firsthand the challenges and complexities of managing workflows. This is the case when projects are started from scratch, but it’s also the case for brownfield projects. This blog post will explore the significance of adopting a workflow engine for orchestrating software processes – rather than solely relying on hard-coded orchestration in the services to call other services and highlighting the benefits of making this pivotal decision early in a software project.
The Case for Workflow-Centric Applications
When embarking on a new development project, one of the first questions to address is whether the use case is workflow centric. We find them in various sectors across our client projects:
- E-Commerce (e.g., automotive sales processes)
- Banking (e.g., account openings, including KYC and product changes)
- Insurance (e.g., account onboarding, case management)
- Public Sector (e.g., administrative processes)
- Energy Sector (e.g. registration of metering devices)
Usually, projects in these domains are inherently complex and require meticulous orchestration of various tasks and services. Having a clear understanding of the workflow requirements early on can significantly improve how the software is shaped and written. This allows a clear focus on the core business logic during development.
The Complexity of Legacy Systems and Orchestration Silos
As software landscapes grow and evolve, transitioning to workflow orchestration can become increasingly complicated. Team members often find themselves working in „orchestrated silos“ – where individual teams assume they have solved their workflow challenges independently, without a broader understanding of the interconnections or implications across the entire IT landscape.
Have you ever overheard your (business) PO ask a question to the team, how IT/the development team actually implemented certain behavior or reaffirming how ‘the actual process’ is being handled in the software? The question could probably be avoided if Business and IT were synced over the same models as actual executable artifacts.
This disconnect leads to inefficient workflows, poor visibility, and a tangled architecture that is difficult to maintain over time.
Another motivation might be to focus on actual service implementation more instead of investing effort into writing glue code, meaning code used for integration purposes solely, that does not add any business value, but which is needed to make the software work. A workflow engine allows developers to focus on writing service endpoints while externalizing the integration parts.
Hence, it’s imperative to make the decision for workflow orchestration at the kickoff of a project, rather than as an afterthought.
Process Orchestration and Monitoring capability is not an afterthought
Deciding to make use of a workflow engine for business process orchestration and monitoring capabilities is not a mere “per product” decision; it’s a strategic initiative that should span domain, initiative, and cross-product areas within your entire IT transformation journey. Deciding early for orchestrating workloads across IT products enables you to reap the benefits of orchestration – such as monitoring, rollback capabilities, and process visibility – that would prove arduous to implement later.
By delaying this decision, you may find yourself building software without the essential orchestration capabilities. As a result, you would lose out on significant advantages that workflow engines provide – monitoring, transaction management, and simplified error handling – often leading to increased development overhead.
Avoiding this decision will put your teams to the only option they have left – doing all the integration work “by hand” in custom software-code per codebase, reinventing the wheel in every product, having to think about (business process) monitoring, rollback behavior, retry- and error handling with their integration partners respectively their own product. Adopting a workflow engine later on will require significant efforts when existing code and structures have to be migrated or made to work with the chosen engine, which leads to unnecessary overhead and more complicated maintenance.
The Case for Workflow Engines
Investing in an engine-orchestrated product-overarching workflow specifically addresses the pain points associated with entirely self-written and hard-coded services that resemble “implicit orchestration”. These benefits entail:
- Reduced Development Overhead
Using an orchestration engine eliminates the burden of hard-coding complex process integration. Developers can now concentrate on crafting core business logic, in return enhancing productivity having to focus less on integration code. - Visibility and Monitoring
Developers, business analysts and product owners gain enhanced insights into running workflows through built-in process monitoring features. Workflow engines offer tracing capabilities that hard-coded orchestration lacks, providing a clear overview of ongoing operations. The monitoring capabilities of the business process orchestration are “included” using a workflow solution, contrasting additional effort to gain visibility in the case of services calling other services directly, although the situation has improved in recent years with standards like OpenTelemetry on the technical level. - No Need to Reinvent the Wheel
With reusable workflow patterns and predefined tasks, teams can avoid duplicative efforts and streamline development processes, allowing for increased efficiency. - Rollback and Saga Pattern
Incorporating established patterns for managing transactions and errors is seamlessly integrated in a workflow engine, enhancing the reliability of your software. - Concentrate on Core Software
Workflow engines provide a solid backbone for orchestration, minimizing having to write glue code (custom integration logic). This allows product teams to focus on core business features.
This leaves us with the general notion to think about workflow orchestration potential in the first place and having to choose a Workflow Engine in the second place. Choosing an implementation is highly dependent on your projects’ requirements, but based on our experience* we have an opinion we would like to share.
Image 1: Hard Coded Process vs. Orchestrated Process
How Camunda improves our projects
Now that we’ve explained why a Workflow Engine can make sense in general, let’s talk about how this looks in practice with Camunda. From a business perspective Camunda provides us with a toolkit, that we can utilize throughout our development process and in production. You could say that while it ultimately helps your team implement well-orchestrated processes, it can also positively impact the way you reach this goal.
In our experience it offers a very good balance between transparency and visualization for everyone, who’s not deep in the technological details, while also not imposing restrictions on developers the way some low-code solutions tend to do. When we’re thinking along the whole value chain of our processes, everyone involved gains some sort of benefit.
Designing a business process
A very simple but common pitfall during any software development project is simply miscommunication. Especially during a stage where everything is still on an abstract level and no software or process exists yet, it is very helpful to have a common visual language in the form of BPMN diagrams.
When we’re using Camunda 8 we can easily collaborate on BPMN diagrams with product owners, business analysts and developers by using the Camunda Web Modeler, which allows multiple users to work together on one process. We recommend drawing out how a process is supposed to work as a BPMN anyway, but by using Camunda we are at this point already creating a functional structure that will be used in our production application. It’s also possible (and recommended) to make notes in the BPMN to explain how the process branches out, to group segments by overarching features, to represent calls from tasks to other systems or just to add relevant thoughts and ideas we want to keep in mind during development until we’ve solved the associated issue.
This preparation means our development team starts off with a framework for the overall orchestration and whenever we discuss what we’re building, all parties have a clear visual reference of how it’s supposed to function. Once the environment has been set up and Camunda is running, we assign rights so every team member can always view the currently deployed process version, including information about running instances (but more on that later). All of this contributes to making our next steps easier as well.
Software development
Once a business process has been designed, we break it down into user stories, that our developers can handle in a reasonable amount of time. For a business analyst it’s very easy to use the tasks within the Camunda orchestration as a guideline to cut stories. This also nicely compartmentalizes what a developer is working on and allows them to finish pieces of the whole process one by one.
Along with the organizational benefits, working with Camunda gives our developers some building blocks like timers, connectors and many different types of events, that help them deal with common issues while saving some effort by not having to write everything from scratch. At the same time a task in a Camunda diagram is just a shell that executes whatever code is put behind it, which gives developers the flexibility to implement custom solutions as required. We also often find ourselves either copying tasks from similar processes we’ve built previously or building a general feature such as a KYC check as entirely separate subprocesses, that can then be easily used by any number of workflows in Camunda. In a future blogpost we’ll go into more detail about what our development team appreciates in their work with Camunda.
Depending on the impact individual tasks have on our workflow we often mock their outputs at the beginning and continue to replace mocked parts with functioning code as we progress. With this approach we can execute and test the entire process end to end very early on, even if not every step has executable code behind it.
To keep the spirit of transparency throughout development we also like to highlight tasks in the BPMN diagrams with color until they have been implemented, which clearly indicates for every team member what is currently functional and what isn’t.
Testing our application
Once we are at a stage where we can test our features or have them reviewed by a test team, we can make use of another feature in Camunda. Camunda Operate (previously known as Cockpit) is a web interface, that allows users to view all processes deployed in the environment, which provides two big benefits.
Firstly, Operate visualizes past and present process instances within the BPMN diagram. Next to all tasks and events we can see the number of times the task or event has been executed and how often it failed. Individual tasks failing can point towards a logical issue within them or a problem with the incoming data from previous steps, while paths in the BPMN diagram not receiving as much traffic as expected can hint at a more general problem, that’s worth investigating.
Image 2: Camunda Operate; image source: Camunda
Secondly, looking at individual process instances helps us to track if a testcase behaves as expected or if it runs into a problem, that is caused by a bug or a logical oversight. We can then take a closer look at which task caused the failure and inspect the process instance’s variables in Camunda, where we store information relevant to individual instances. While Operate doesn’t replace logs to trace bugs and other issues, it offers a much more user-friendly way of finding and narrowing them down.
In addition to these aids for manual testing Camunda also supports running automated tests on the processes you create. This can add another safety net to your testing process to improve test coverage, avoid even more errors and potentially reduce the amount of manual testing needed, especially in the long run.
Camunda processes in production
Once we’ve deployed Camunda processes to a production environment we have yet another set of user groups, that can make use of its features.
In many cases being able to access Operate can help support agents find where a customer’s process instance ran into an issue. Depending on the assigned rights they can also manually retry failed tasks with a few clicks directly from this view, which is a very accessible way to resolve temporary issues without having to involve a DevOps colleague. If it does end up being a real bug, the process instance in question can be investigated in Operate by a business expert, resulting in better incident descriptions than “customers report the process doesn’t work right now”, once again saving time of the developers who work on the issue eventually.
Another way we can include other users in our usage of Camunda is by utilizing “User Tasks”, an integrated solution for manual steps in a BPMN diagram. Ideally a process is designed so that no manual intervention is required at any point. However, if at the time of development it is the best way to resolve a step in the workflow, then it’s a nice tool to have . A custom form can be built with the Camunda Modeler, including conditional fields, and once it’s tied to a User Task, users with appropriate rights can handle it through Camunda’s integrated frontend for User Tasks, called Tasklist . If you, like most of our customers, prefer a more tailored UI and UX for manual tasks, it’s common practice to build a frontend for such tasks as required and handle communication between tasks and the BPMN processes via an API. In our processes we try to stay as automated as possible, but for tasks that for example rely on external dependencies it’s helpful to easily be able to delegate a step to an employee, if the external dependency is out of order or misbehaving and causing the automated task to fail. This can save or speed up process instances, that would otherwise fail or wait until the external issue is resolved.
Lastly, there is one feature that offers transparency on a higher level. Camunda Optimize lets our customers create reports and track the usage of their processes, which gives helpful insight and is a big advantage compared to building a custom process orchestration from scratch without a BPMN Engine or associated process-tracking. Compared to tracking solutions like Google Analytics, where the focus is on the frontend usage, Optimize is an easy way to track what is happening in automated, asynchronous processes. This is especially valuable for decision makers in a company and gives the opportunity to present how your processes are used in a visually appealing and easily digestible form.
Image 3: Camunda Optimize; image source: Camunda
So, are we happy working with Camunda?
The short answer is yes, otherwise we would not have become a partner. Not every application needs a BPMN Engine and evaluating this is an important task in the early stages of a new project, as pointed out earlier in this article. If it does make sense though, Camunda brings advantages to multiple parties in the lifecycle of an application while imposing little restrictions on development. Ultimately what it’s supposed to do is help users create a better product and improve the experience of reaching this goal along the way, which in our opinion Camunda does very well.
*Disclosure: Senacor is a Camunda Partner.