Law #2: Complexity: you can move it around, or externalize it, but you can’t make it go away

Apr 9, 2011

Healthcare Interoperability projects are complex; they are made up of lots of people and a great number of data items, each with their own complexity, and usually with subtle interactions between the data, the business processes, and the people involved. Generally, the complexity arises from three different sources: * The innate complexity of the problem, the real underlying scientific, clinical and business processes involved

  • Diversity in approach and thinking between different instances of the same processes and data items due to different legacy practices, cultures and personalities
  • Human factors (poor communication, poor motivation, failure to understand, fraudulent behaviour)

It can be very difficult to tease these apart; very difficult to determine whether business process variability is justified or not. This is always aggravated by the fact that any gathering of ten scientists, managers, doctors or nurses etc will have at least ten ways of doing something, all of which are the only way. Even if you can properly resolve business variability and somehow correct for human factors, you’re still left with the innate complexity.

You can’t make the innate complexity go away: it has to exist somewhere. However you break up the project, divide the work amongst the project participants, and write the specifications, that complexity has to be somewhere. It’s not going to go away.

Of course, it’s always possible to make things worse, to divide the work up so badly or to describe it so incoherently that you make things much worse than they need to be. In fact, it often seems impossible to avoid this, impossible to do things cleanly: we keep our solutions as simple as we can, but they get complex all by themselves. Since this problem is so ubiquitous, it can be hard to know exactly how simple a solution gets before it’s gotten too simple, before it’s failing to deal with the complexity.

There’s two ways for a solution to be too simple: it just completely fails to account for the complexity of the problem, or it finds some way to externalize the complexity so it becomes Someone Else’s Problem. The first will inevitably turn into a problem during the implementation phase of a project; such projects often turn into death marches as the complexity slowly is dealt with and the successive cycles of reworking for discovered complexity turn the overall project into a permutational explosion of special cases.

The second kind of simple solution – one that externalizes the complexity – can be a very powerful solution indeed. There’s nothing like turning your own problem into someone else’s problem. One of the hallmarks of successful IT solutions is exactly that – that they solve a subset of the problems, the ones that can be solved neatly, and allow other people to extend the system one way or another to solve more complicated problems. On the other hand, this approach can also backfire badly – it all depends on just who the problems have been shifted to, and whether the cost of that is part of the project, or someone else’s project.

The bottom line is that the complexity that does exist must be handled somewhere, by someone. It’s not going to go away. So everyone involved in healthcare interoperability should account for where the complexity has gone throughout a project, and rigorously resist accepting overly simplistic solutions where it’s not clearly understood what happened to the complexity that does exist. Unfortunately this implies that requirements gathering should actually capture the extent of the complexity of a project, and this is most often a total failure - in particular, people seem hardwired to express their requirements as simple principles with some special cases.