Support for Testing in the #FHIR Specification

Sep 28, 2016

On the last day of the baltimore meeting, a small group of us met to talk about the future of testing in the FHIR specification. In particular, the participants included both Aegis and Mitre, who develop Touchstone and Crucible respectively. Out small group made 3 decisions: 1. We will work towards the adoption of a formal testing framework for FHIR interfaces

  1. We will simplify the existingTestScriptresource
  2. We will draft and propose a TestReport resource

Taking these backwards:

The TestReport will contain a summary of the outcomes a test run - which tests were run, whether they passed, and references to a further information (logs, etc). It’s for helping to maintain current registries of implementations and outcomes. I’ll blog about this further once there’s a draft to look at.

I’ll comment on #2 in time, when we have a draft for people to consider.

For #1, a testing framework, we have several motivations for this:

  • It’s well known that a problem with testing processes like IHE is that under the right care (e.g. configuration) and with some judicious off-road modifications, systems can pass their IHE testing - but that often doesn’t equate to conformant implementations in practice
  • Many systems undergo stringent testing in-house in continuous builds, but real world integrations are extremely hard to test
  • As a consequence, applications are often poorly tested prior to upgrade
  • Every real clinical system I’ve worked with has had test patients so that staff can check how the system works. In Australia, custom is that the patients are called Donald Duck, Mickey Mouse, etc. Staff just have to magically ‘know’ that these are test patients, and they pollute stats etc a little
  • One near exception is the Australian MyHR, for which there was no test system, and no possibility of test patients. In practice, people trying to use it had to use their own personal records as test patients (many told me that they did this, and bits of those records drifted into the media)

We’d like to normalise testing - to make clear expectations about how production systems could be tested, and to start encouraging production systems to actually provide such support. Two key corollaries of this - administrators can download the official tests (e.g. IHE, or others) and run them against their system with robots playing the part of the the other system. And also, the testing framework starts to acquire a clinical safety role - adminstrators can add new tests to their own automated tests for the application (e.g. prior to upgrade or configuration change) and assure themselves that known system requirements with regard to system safety are assured. Note that this is something you can only do in the context of a fully configured and linked system of systems - which is why support for testing in production systems is so important in this regard.

Anywhere, he’s a set of ideas to get people thinking about how this all might work:

  • Resources on the FHIR API are either ‘Production’ or ‘Test’. There will be a ‘test’ tag that labels all the test resources accordingly. No tag = not testing
  • Resources labelled ‘test’ are allowed to refer to production records (e.g. for configuration) , but never the other way around
  • Requests may be labelled ‘test only’ or ‘test and production’. Default is production.
  • Clients that have either ‘test’ or ‘test-only’ on requests SHALL indicate clearly to users that this is the case
  • Operations on test resources that use production resources internally on the server (e.g. $expand) are allowed in ‘test only mode’
  • Conformance resource : add a ‘testCompartment’ : boolean flag where true means that the system offers the same services as in production, but in test mode
  • If a system supports a test compartment, it also supports a $clear operation on the test compartment - it clears all the records associated with the test compartment (including audit records), so to reset for an automated test run
  • Testable Systems can be the target of a test engine that executes a set of tests, and which can produce a test report for archival / comparison purposes
  • System administrators can write tests against their system - e.g. clinical safety focused using TestScripts - and these test scripts can cross multiple systems
  • Systems are allowed to make their own arrangements regarding multiple test compartents (e.g. per authorised user)

Anyone with any engineering experience will immediately recognise that implementing this set of requirements is a major design undertaking. It’s not going to happen quickly in existing EHRs, unless s conceptually similar system already exists - but I’ve never seen one.

But if we regularise the interface, and the test servers implement it (not too hard in my server) - then we might encourage movement in this direction gradually, and that would make it cheaper to deliver truly reliable systems….