Hard #FHIR Safety Problem: Synchronization
May 21, 2019It’s a common thing for implementers to want to do with FHIR: connect to a FHIR server, and make a local copy of the information provided by the server, and then check back occasionally with the server for updates - that is, new resources, or changes to existing resources. (In fact, it might be argued that this is **the **thing that FHIR is for, based on actual usage).
This simple sounding intention turns out to be very difficult to get right, for all sorts of really important reasons. And that’s a real problem.
As an example, let’s assume that you want to maintain some kind of personal patient copy of their summary - meds, problems, allergies, care plans (e.g. using the Argonautinterface). We’ll focus on medications, but the problems are the same for any kind of resource that you want to synchronize like this .
The usual way to start these things is for the client that will keep it’s own copy to get authorized to use the server (say, using Smart-App-Launch) and then do a query like:
GET [base]/MedicationStatement?patient=X
where [X] is the id of the patient obtained from somewhere (exactly where is out of the scope of this log post). This call will return a list of MedicationStatement resources in a Bundle.
MedicationStatement 1234 v2: status=active, code=phenytoin
MedicationStatement 2346 v1: status=active, code=salbutamol
MedicationStatement 6234 v1: status=active, subject=warfarin
Note that the MedicationStatements may not include all the relevant medication information; implementers have to check whether MedicationRequest(/Order) and Dispense/Administration resources have other critical information, depending on applicable FHIR IGs, system behavior, clinical context etc. **We'd love to nail this down, but that's not the healthcare system we have right now**
So the client gets back this list of resources, stores them in it's own storage somewhere, and then displays them to the user. Note that the medication statements that come back have a version (how many updates there have been on the server), and the all important [status](http://hl7.org/fhir/R4/valueset-medication-statement-status.html):
|**Code**|**Definition**
|active|The medication is still being taken
|completed|The medication is no longer being taken
|entered-in-error|The medication was entered by mistake
|intended|The medication may be taken at some time in the future
|stopped|The medication was stopped being taken
|on-hold|The medication is not being taken right now
|unknown|The status of the medication is not known
|not-taken|This medication is not being taken (e.g. "I never took X")
Feedback from the EHR vendors reviewing real production apps presented to their app stores is very strong:
**Many Applications are ignoring the status code and not displaying it, and not checking it**.
That's a shocking unsafe practice. And the statement even applies to experienced healthcare developers. The status code is critical always forever, and applications can never ignore it (just how important it is will become clear further down). First point for implementers: always check the status. We're open to ideas about how to make it more likely that implementers will check for status - but since everyone ignores [the safety check list](http://hl7.org/fhir/R4/safety.html) already, it seems unlikely we can do more; others will have to take up the gavel on this one.
Note: under US regulations, EHR vendors can review and approve apps that institutions use, and force apps like this to get status information correct. But EHR vendors cannot review patient applications. It's an open question who is responsible for this, or who can do anything about it.
Btw, readers will have noticed the absolutely awful status value of '**unknown**' in the list, and are probably wondering just what an application is supposed to do with a medication of unknown status, and why we even added that to the FHIR standard. Well, welcome to the wonderful world of healthcare records, where critical legacy records that cannot be lost in history also are so unreliable that you don't know what they actually mean. No new record should ever be created with a status of 'unknown', but that won't stop them existing.
So applications should always display and check the status - but we've got a lot of problems to deal with yet in this post.
Our client now has those 3 resources in it's local store, stored against the id from the server. Some time later, it performs a follow up query:
```GET [base]/MedicationStatement?patient=[X]&_
_lastUpdated=gt[Y]
where [Y] is the the timestamp on the [http header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Last-Modified) from the last call. This call will returns a list of updated medications - anything that has changed since the last call.
```MedicationStatement 1234 v4: status=completed, code=phenytoin
MedicationStatement 7485 v1: status=active, subject=vancomycin
At first glance, this seems clear: there's been an update to 1234- the patient is no longer taking phenytoin, and now they have started taking vancomycin. The client adds the vancomycin to it's persistent store, and updates it's own copy of the MedicationStatement 1234.
Note, I'm assuming here that the resource ids of the medication statements are preserved - they **SHOULD** be for all sorts of reasons, but not all systems do. Those that don't: the consequence is that there's no way for any application to maintain a copy of the information downstream (a particularly subtle and pernicious way of information blocking).
So that's all good, but what about MedicationStatements 2346 and 6234?
Well, we presume that they haven't changed since the last query. So what if we didn't time limit that query, and just made the original query again? The client might assume that it would get all 4 records, but it might not. Say the client did the original query again, and got
```MedicationStatement 1234 v4: status=completed, code=phenytoin
MedicationStatement 6234 v1: status=active, subject=warfarin
MedicationStatement 7485 v1: status=active, subject=vancomycin
What happened to MedicationStatement 2346? It hasn't been returned - why? and what should the client do about it? Should it remove it from it's persistent store or not?
Here's a list of the more likely reasons why the record might not have been returned:
* The source resource(/record) was hard deleted from the origin server. And so the client should also delete it? Analysis: it's a**bad practice**to hard delete records that prior healthcare decisions might have been made from, or that might have been distributed. That's what we have**entered-in-error**for: to make that something was removed, and be able to distribute it. But of course, you guessed it - lots of systems just hard delete records when asked
* The record was marked as confidential on the server side, and policy is not to provide access to confidential information across the API.
Analysis: this is just a hard problem. Security isn't going to go away, and can always create problems like this. The resource will no longer be available, period. And it's computationally hard to recognise that a change somewhere in the security system means a resource is no longer available to a particular client that already accessed it
* The record was created against the wrong subject, and that's been fixed by simply changing the subject. The resource is no longer available for this patient. Analysis: this is a variant of the security problem, since security is by patient for patient access. Like deleting records, this is also bad practice (for the same reason). Applications should mark the old record as 'entered-in-error' and create a new correct record. But you guessed it... most don't
* The system may only be returning records with status active or completed unless the client specifically asks for other status codes (or even only active, unlike this example). Analysis: Some in-production Argonaut servers do this, in response to the status problem described above. We would rather they didn't do this, because it creates other problems - see[the long discussion on the subject](https://chat.fhir.org/#narrow/stream/179166-implementers/topic/Are.20search.20parameters.20like.20filters.3F)- but the problems this is addressing are real and serious. This situation can be rectified by the client by performing a GET [base]/MedicationStatement/2346 and seeing what is returned)
* The portal providing the API service may have
(temporarily?) lost access to the underlying record store from where MedicationStatement 2346 came from. Analysis: portals are often facades over much more complex underlying systems, so this is a real world possibility. There's no ideal solution other than to pretend that it will never be a problem
* The record may no longer available to the production system if enough years (5-7 or longer) have passed (and most EHR systems are at least that old, if not much older, no matter how recent the FHIR interface itself is)
Note: There's another bigger deal here - patient records may be [merged or unmerged](http://hl7.org/fhir/R4/patient.html#merge) which significantly complicates matters. Every system handles this slightly differently, and applications that maintain their own persistent store for more than one patient **cannot **ignore this - records may be moved, or come and go, and they just have to keep up. Of course, many/most don't do anything about patient record merge/link.
So: any client that it's keeping it's own persistent copy of data accessed by a FHIR Search like the above has to consult with each server to find out which of those possible reasons are applicable for the server, and decide what to do about it.
Applications that don't... are unsafe and shouldn't be written, marketed, or used. Of course, the users have no way of finding out how an application handles this. Maybe someone will take up the task of reviewing patient apps for safety, but it will be terribly difficult (=expensive) to do that.
I wish there was better advice I could provide, and we're working on getting better advice, but the problem is the trillions of dollars of legacy systems out there that healthcare providers don't seem to be in any hurry to replace (with systems that don't yet exist, mind).
The problems discussed here are already documented in unsafe outcomes for patients in production systems that access EHR clients using the FHIR interface (and in related forms they have been a documented problem for national patient record systems too)
Note that there are other ways for a client to synchronize data:
* Using the[history interaction](http://hl7.org/fhir/R4/http.html#history)
* Using[Subscriptions](http://hl7.org/fhir/R4/subscription.html)
These offer different APIs with different strengths and weaknesses - but since the underlying issues are record-keeping policy issues, not API surface issues, these generally don't make much difference, though we are working hard on the subscription route to make it possible to deal with some of these issues (more on that in a later post; and we probably will never resolve the security related issues).
```**MedicationStatement 1234 v4: status=completed, code=phenytoin
MedicationStatement 6234 v1: status=active, subject=warfarin
MedicationStatement 7485 v1: status=active, subject=vancomycin
What happened to MedicationStatement 2346? It hasn't been returned - why? and what should the client do about it? Should it remove it from it's persistent store or not?
Here's a list of the more likely reasons why the record might not have been returned:
* The source resource(/record) was hard deleted from the origin server. And so the client should also delete it? Analysis: it's a**bad practice**to hard delete records that prior healthcare decisions might have been made from, or that might have been distributed. That's what we have**entered-in-error**for: to make that something was removed, and be able to distribute it. But of course, you guessed it - lots of systems just hard delete records when asked
* The record was marked as confidential on the server side, and policy is not to provide access to confidential information across the API.
Analysis: this is just a hard problem. Security isn't going to go away, and can always create problems like this. The resource will no longer be available, period. And it's computationally hard to recognise that a change somewhere in the security system means a resource is no longer available to a particular client that already accessed it
* The record was created against the wrong subject, and that's been fixed by simply changing the subject. The resource is no longer available for this patient. Analysis: this is a variant of the security problem, since security is by patient for patient access. Like deleting records, this is also bad practice (for the same reason). Applications should mark the old record as 'entered-in-error' and create a new correct record. But you guessed it... most don't
* The system may only be returning records with status active or completed unless the client specifically asks for other status codes (or even only active, unlike this example). Analysis: Some in-production Argonaut servers do this, in response to the status problem described above. We would rather they didn't do this, because it creates other problems - see[the long discussion on the subject](https://chat.fhir.org/#narrow/stream/179166-implementers/topic/Are.20search.20parameters.20like.20filters.3F)- but the problems this is addressing are real and serious. This situation can be rectified by the client by performing a GET [base]/MedicationStatement/2346 and seeing what is returned)
* The portal providing the API service may have
(temporarily?) lost access to the underlying record store from where MedicationStatement 2346 came from. Analysis: portals are often facades over much more complex underlying systems, so this is a real world possibility. There's no ideal solution other than to pretend that it will never be a problem
* The record may no longer available to the production system if enough years (5-7 or longer) have passed (and most EHR systems are at least that old, if not much older, no matter how recent the FHIR interface itself is)
Note: There's another bigger deal here - patient records may be [merged or unmerged](http://hl7.org/fhir/R4/patient.html#merge) which significantly complicates matters. Every system handles this slightly differently, and applications that maintain their own persistent store for more than one patient **cannot **ignore this - records may be moved, or come and go, and they just have to keep up. Of course, many/most don't do anything about patient record merge/link.
So: any client that it's keeping it's own persistent copy of data accessed by a FHIR Search like the above has to consult with each server to find out which of those possible reasons are applicable for the server, and decide what to do about it.
Applications that don't... are unsafe and shouldn't be written, marketed, or used. Of course, the users have no way of finding out how an application handles this. Maybe someone will take up the task of reviewing patient apps for safety, but it will be terribly difficult (=expensive) to do that.
I wish there was better advice I could provide, and we're working on getting better advice, but the problem is the trillions of dollars of legacy systems out there that healthcare providers don't seem to be in any hurry to replace (with systems that don't yet exist, mind).
The problems discussed here are already documented in unsafe outcomes for patients in production systems that access EHR clients using the FHIR interface (and in related forms they have been a documented problem for national patient record systems too)
Note that there are other ways for a client to synchronize data:
* Using the[history interaction](http://hl7.org/fhir/R4/http.html#history)
* Using[Subscriptions](http://hl7.org/fhir/R4/subscription.html)
These offer different APIs with different strengths and weaknesses - but since the underlying issues are record-keeping policy issues, not API surface issues, these generally don't make much difference, though we are working hard on the subscription route to make it possible to deal with some of these issues (more on that in a later post; and we probably will never resolve the security related issues).
**```**MedicationStatement 1234 v4: status=completed, code=phenytoin
MedicationStatement 7485 v1: status=active, subject=vancomycin
At first glance, this seems clear: there's been an update to 1234- the patient is no longer taking phenytoin, and now they have started taking vancomycin. The client adds the vancomycin to it's persistent store, and updates it's own copy of the MedicationStatement 1234.
Note, I'm assuming here that the resource ids of the medication statements are preserved - they **SHOULD** be for all sorts of reasons, but not all systems do. Those that don't: the consequence is that there's no way for any application to maintain a copy of the information downstream (a particularly subtle and pernicious way of information blocking).
So that's all good, but what about MedicationStatements 2346 and 6234?
Well, we presume that they haven't changed since the last query. So what if we didn't time limit that query, and just made the original query again? The client might assume that it would get all 4 records, but it might not. Say the client did the original query again, and got
```MedicationStatement 1234 v4: status=completed, code=phenytoin
MedicationStatement 6234 v1: status=active, subject=warfarin
MedicationStatement 7485 v1: status=active, subject=vancomycin
What happened to MedicationStatement 2346? It hasn't been returned - why? and what should the client do about it? Should it remove it from it's persistent store or not?
Here's a list of the more likely reasons why the record might not have been returned:
* The source resource(/record) was hard deleted from the origin server. And so the client should also delete it? Analysis: it's a**bad practice**to hard delete records that prior healthcare decisions might have been made from, or that might have been distributed. That's what we have**entered-in-error**for: to make that something was removed, and be able to distribute it. But of course, you guessed it - lots of systems just hard delete records when asked
* The record was marked as confidential on the server side, and policy is not to provide access to confidential information across the API.
Analysis: this is just a hard problem. Security isn't going to go away, and can always create problems like this. The resource will no longer be available, period. And it's computationally hard to recognise that a change somewhere in the security system means a resource is no longer available to a particular client that already accessed it
* The record was created against the wrong subject, and that's been fixed by simply changing the subject. The resource is no longer available for this patient. Analysis: this is a variant of the security problem, since security is by patient for patient access. Like deleting records, this is also bad practice (for the same reason). Applications should mark the old record as 'entered-in-error' and create a new correct record. But you guessed it... most don't
* The system may only be returning records with status active or completed unless the client specifically asks for other status codes (or even only active, unlike this example). Analysis: Some in-production Argonaut servers do this, in response to the status problem described above. We would rather they didn't do this, because it creates other problems - see[the long discussion on the subject](https://chat.fhir.org/#narrow/stream/179166-implementers/topic/Are.20search.20parameters.20like.20filters.3F)- but the problems this is addressing are real and serious. This situation can be rectified by the client by performing a GET [base]/MedicationStatement/2346 and seeing what is returned)
* The portal providing the API service may have
(temporarily?) lost access to the underlying record store from where MedicationStatement 2346 came from. Analysis: portals are often facades over much more complex underlying systems, so this is a real world possibility. There's no ideal solution other than to pretend that it will never be a problem
* The record may no longer available to the production system if enough years (5-7 or longer) have passed (and most EHR systems are at least that old, if not much older, no matter how recent the FHIR interface itself is)
Note: There's another bigger deal here - patient records may be [merged or unmerged](http://hl7.org/fhir/R4/patient.html#merge) which significantly complicates matters. Every system handles this slightly differently, and applications that maintain their own persistent store for more than one patient **cannot **ignore this - records may be moved, or come and go, and they just have to keep up. Of course, many/most don't do anything about patient record merge/link.
So: any client that it's keeping it's own persistent copy of data accessed by a FHIR Search like the above has to consult with each server to find out which of those possible reasons are applicable for the server, and decide what to do about it.
Applications that don't... are unsafe and shouldn't be written, marketed, or used. Of course, the users have no way of finding out how an application handles this. Maybe someone will take up the task of reviewing patient apps for safety, but it will be terribly difficult (=expensive) to do that.
I wish there was better advice I could provide, and we're working on getting better advice, but the problem is the trillions of dollars of legacy systems out there that healthcare providers don't seem to be in any hurry to replace (with systems that don't yet exist, mind).
The problems discussed here are already documented in unsafe outcomes for patients in production systems that access EHR clients using the FHIR interface (and in related forms they have been a documented problem for national patient record systems too)
Note that there are other ways for a client to synchronize data:
* Using the[history interaction](http://hl7.org/fhir/R4/http.html#history)
* Using[Subscriptions](http://hl7.org/fhir/R4/subscription.html)
These offer different APIs with different strengths and weaknesses - but since the underlying issues are record-keeping policy issues, not API surface issues, these generally don't make much difference, though we are working hard on the subscription route to make it possible to deal with some of these issues (more on that in a later post; and we probably will never resolve the security related issues).
```**MedicationStatement 1234 v4: status=completed, code=phenytoin
MedicationStatement 6234 v1: status=active, subject=warfarin
MedicationStatement 7485 v1: status=active, subject=vancomycin
What happened to MedicationStatement 2346? It hasn't been returned - why? and what should the client do about it? Should it remove it from it's persistent store or not?
Here's a list of the more likely reasons why the record might not have been returned:
* The source resource(/record) was hard deleted from the origin server. And so the client should also delete it? Analysis: it's a**bad practice**to hard delete records that prior healthcare decisions might have been made from, or that might have been distributed. That's what we have**entered-in-error**for: to make that something was removed, and be able to distribute it. But of course, you guessed it - lots of systems just hard delete records when asked
* The record was marked as confidential on the server side, and policy is not to provide access to confidential information across the API.
Analysis: this is just a hard problem. Security isn't going to go away, and can always create problems like this. The resource will no longer be available, period. And it's computationally hard to recognise that a change somewhere in the security system means a resource is no longer available to a particular client that already accessed it
* The record was created against the wrong subject, and that's been fixed by simply changing the subject. The resource is no longer available for this patient. Analysis: this is a variant of the security problem, since security is by patient for patient access. Like deleting records, this is also bad practice (for the same reason). Applications should mark the old record as 'entered-in-error' and create a new correct record. But you guessed it... most don't
* The system may only be returning records with status active or completed unless the client specifically asks for other status codes (or even only active, unlike this example). Analysis: Some in-production Argonaut servers do this, in response to the status problem described above. We would rather they didn't do this, because it creates other problems - see[the long discussion on the subject](https://chat.fhir.org/#narrow/stream/179166-implementers/topic/Are.20search.20parameters.20like.20filters.3F)- but the problems this is addressing are real and serious. This situation can be rectified by the client by performing a GET [base]/MedicationStatement/2346 and seeing what is returned)
* The portal providing the API service may have
(temporarily?) lost access to the underlying record store from where MedicationStatement 2346 came from. Analysis: portals are often facades over much more complex underlying systems, so this is a real world possibility. There's no ideal solution other than to pretend that it will never be a problem
* The record may no longer available to the production system if enough years (5-7 or longer) have passed (and most EHR systems are at least that old, if not much older, no matter how recent the FHIR interface itself is)
Note: There's another bigger deal here - patient records may be [merged or unmerged](http://hl7.org/fhir/R4/patient.html#merge) which significantly complicates matters. Every system handles this slightly differently, and applications that maintain their own persistent store for more than one patient **cannot **ignore this - records may be moved, or come and go, and they just have to keep up. Of course, many/most don't do anything about patient record merge/link.
So: any client that it's keeping it's own persistent copy of data accessed by a FHIR Search like the above has to consult with each server to find out which of those possible reasons are applicable for the server, and decide what to do about it.
Applications that don't... are unsafe and shouldn't be written, marketed, or used. Of course, the users have no way of finding out how an application handles this. Maybe someone will take up the task of reviewing patient apps for safety, but it will be terribly difficult (=expensive) to do that.
I wish there was better advice I could provide, and we're working on getting better advice, but the problem is the trillions of dollars of legacy systems out there that healthcare providers don't seem to be in any hurry to replace (with systems that don't yet exist, mind).
The problems discussed here are already documented in unsafe outcomes for patients in production systems that access EHR clients using the FHIR interface (and in related forms they have been a documented problem for national patient record systems too)
Note that there are other ways for a client to synchronize data:
* Using the[history interaction](http://hl7.org/fhir/R4/http.html#history)
* Using[Subscriptions](http://hl7.org/fhir/R4/subscription.html)
These offer different APIs with different strengths and weaknesses - but since the underlying issues are record-keeping policy issues, not API surface issues, these generally don't make much difference, though we are working hard on the subscription route to make it possible to deal with some of these issues (more on that in a later post; and we probably will never resolve the security related issues).
****```**GET [base]/MedicationStatement?patient=[X]&_
_lastUpdated=gt[Y]
where [Y] is the the timestamp on the [http header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Last-Modified) from the last call. This call will returns a list of updated medications - anything that has changed since the last call.
```MedicationStatement 1234 v4: status=completed, code=phenytoin
MedicationStatement 7485 v1: status=active, subject=vancomycin
At first glance, this seems clear: there's been an update to 1234- the patient is no longer taking phenytoin, and now they have started taking vancomycin. The client adds the vancomycin to it's persistent store, and updates it's own copy of the MedicationStatement 1234.
Note, I'm assuming here that the resource ids of the medication statements are preserved - they **SHOULD** be for all sorts of reasons, but not all systems do. Those that don't: the consequence is that there's no way for any application to maintain a copy of the information downstream (a particularly subtle and pernicious way of information blocking).
So that's all good, but what about MedicationStatements 2346 and 6234?
Well, we presume that they haven't changed since the last query. So what if we didn't time limit that query, and just made the original query again? The client might assume that it would get all 4 records, but it might not. Say the client did the original query again, and got
```MedicationStatement 1234 v4: status=completed, code=phenytoin
MedicationStatement 6234 v1: status=active, subject=warfarin
MedicationStatement 7485 v1: status=active, subject=vancomycin
What happened to MedicationStatement 2346? It hasn't been returned - why? and what should the client do about it? Should it remove it from it's persistent store or not?
Here's a list of the more likely reasons why the record might not have been returned:
* The source resource(/record) was hard deleted from the origin server. And so the client should also delete it? Analysis: it's a**bad practice**to hard delete records that prior healthcare decisions might have been made from, or that might have been distributed. That's what we have**entered-in-error**for: to make that something was removed, and be able to distribute it. But of course, you guessed it - lots of systems just hard delete records when asked
* The record was marked as confidential on the server side, and policy is not to provide access to confidential information across the API.
Analysis: this is just a hard problem. Security isn't going to go away, and can always create problems like this. The resource will no longer be available, period. And it's computationally hard to recognise that a change somewhere in the security system means a resource is no longer available to a particular client that already accessed it
* The record was created against the wrong subject, and that's been fixed by simply changing the subject. The resource is no longer available for this patient. Analysis: this is a variant of the security problem, since security is by patient for patient access. Like deleting records, this is also bad practice (for the same reason). Applications should mark the old record as 'entered-in-error' and create a new correct record. But you guessed it... most don't
* The system may only be returning records with status active or completed unless the client specifically asks for other status codes (or even only active, unlike this example). Analysis: Some in-production Argonaut servers do this, in response to the status problem described above. We would rather they didn't do this, because it creates other problems - see[the long discussion on the subject](https://chat.fhir.org/#narrow/stream/179166-implementers/topic/Are.20search.20parameters.20like.20filters.3F)- but the problems this is addressing are real and serious. This situation can be rectified by the client by performing a GET [base]/MedicationStatement/2346 and seeing what is returned)
* The portal providing the API service may have
(temporarily?) lost access to the underlying record store from where MedicationStatement 2346 came from. Analysis: portals are often facades over much more complex underlying systems, so this is a real world possibility. There's no ideal solution other than to pretend that it will never be a problem
* The record may no longer available to the production system if enough years (5-7 or longer) have passed (and most EHR systems are at least that old, if not much older, no matter how recent the FHIR interface itself is)
Note: There's another bigger deal here - patient records may be [merged or unmerged](http://hl7.org/fhir/R4/patient.html#merge) which significantly complicates matters. Every system handles this slightly differently, and applications that maintain their own persistent store for more than one patient **cannot **ignore this - records may be moved, or come and go, and they just have to keep up. Of course, many/most don't do anything about patient record merge/link.
So: any client that it's keeping it's own persistent copy of data accessed by a FHIR Search like the above has to consult with each server to find out which of those possible reasons are applicable for the server, and decide what to do about it.
Applications that don't... are unsafe and shouldn't be written, marketed, or used. Of course, the users have no way of finding out how an application handles this. Maybe someone will take up the task of reviewing patient apps for safety, but it will be terribly difficult (=expensive) to do that.
I wish there was better advice I could provide, and we're working on getting better advice, but the problem is the trillions of dollars of legacy systems out there that healthcare providers don't seem to be in any hurry to replace (with systems that don't yet exist, mind).
The problems discussed here are already documented in unsafe outcomes for patients in production systems that access EHR clients using the FHIR interface (and in related forms they have been a documented problem for national patient record systems too)
Note that there are other ways for a client to synchronize data:
* Using the[history interaction](http://hl7.org/fhir/R4/http.html#history)
* Using[Subscriptions](http://hl7.org/fhir/R4/subscription.html)
These offer different APIs with different strengths and weaknesses - but since the underlying issues are record-keeping policy issues, not API surface issues, these generally don't make much difference, though we are working hard on the subscription route to make it possible to deal with some of these issues (more on that in a later post; and we probably will never resolve the security related issues).
```**MedicationStatement 1234 v4: status=completed, code=phenytoin
MedicationStatement 6234 v1: status=active, subject=warfarin
MedicationStatement 7485 v1: status=active, subject=vancomycin
What happened to MedicationStatement 2346? It hasn't been returned - why? and what should the client do about it? Should it remove it from it's persistent store or not?
Here's a list of the more likely reasons why the record might not have been returned:
* The source resource(/record) was hard deleted from the origin server. And so the client should also delete it? Analysis: it's a**bad practice**to hard delete records that prior healthcare decisions might have been made from, or that might have been distributed. That's what we have**entered-in-error**for: to make that something was removed, and be able to distribute it. But of course, you guessed it - lots of systems just hard delete records when asked
* The record was marked as confidential on the server side, and policy is not to provide access to confidential information across the API.
Analysis: this is just a hard problem. Security isn't going to go away, and can always create problems like this. The resource will no longer be available, period. And it's computationally hard to recognise that a change somewhere in the security system means a resource is no longer available to a particular client that already accessed it
* The record was created against the wrong subject, and that's been fixed by simply changing the subject. The resource is no longer available for this patient. Analysis: this is a variant of the security problem, since security is by patient for patient access. Like deleting records, this is also bad practice (for the same reason). Applications should mark the old record as 'entered-in-error' and create a new correct record. But you guessed it... most don't
* The system may only be returning records with status active or completed unless the client specifically asks for other status codes (or even only active, unlike this example). Analysis: Some in-production Argonaut servers do this, in response to the status problem described above. We would rather they didn't do this, because it creates other problems - see[the long discussion on the subject](https://chat.fhir.org/#narrow/stream/179166-implementers/topic/Are.20search.20parameters.20like.20filters.3F)- but the problems this is addressing are real and serious. This situation can be rectified by the client by performing a GET [base]/MedicationStatement/2346 and seeing what is returned)
* The portal providing the API service may have
(temporarily?) lost access to the underlying record store from where MedicationStatement 2346 came from. Analysis: portals are often facades over much more complex underlying systems, so this is a real world possibility. There's no ideal solution other than to pretend that it will never be a problem
* The record may no longer available to the production system if enough years (5-7 or longer) have passed (and most EHR systems are at least that old, if not much older, no matter how recent the FHIR interface itself is)
Note: There's another bigger deal here - patient records may be [merged or unmerged](http://hl7.org/fhir/R4/patient.html#merge) which significantly complicates matters. Every system handles this slightly differently, and applications that maintain their own persistent store for more than one patient **cannot **ignore this - records may be moved, or come and go, and they just have to keep up. Of course, many/most don't do anything about patient record merge/link.
So: any client that it's keeping it's own persistent copy of data accessed by a FHIR Search like the above has to consult with each server to find out which of those possible reasons are applicable for the server, and decide what to do about it.
Applications that don't... are unsafe and shouldn't be written, marketed, or used. Of course, the users have no way of finding out how an application handles this. Maybe someone will take up the task of reviewing patient apps for safety, but it will be terribly difficult (=expensive) to do that.
I wish there was better advice I could provide, and we're working on getting better advice, but the problem is the trillions of dollars of legacy systems out there that healthcare providers don't seem to be in any hurry to replace (with systems that don't yet exist, mind).
The problems discussed here are already documented in unsafe outcomes for patients in production systems that access EHR clients using the FHIR interface (and in related forms they have been a documented problem for national patient record systems too)
Note that there are other ways for a client to synchronize data:
* Using the[history interaction](http://hl7.org/fhir/R4/http.html#history)
* Using[Subscriptions](http://hl7.org/fhir/R4/subscription.html)
These offer different APIs with different strengths and weaknesses - but since the underlying issues are record-keeping policy issues, not API surface issues, these generally don't make much difference, though we are working hard on the subscription route to make it possible to deal with some of these issues (more on that in a later post; and we probably will never resolve the security related issues).
**```**MedicationStatement 1234 v4: status=completed, code=phenytoin
MedicationStatement 7485 v1: status=active, subject=vancomycin
At first glance, this seems clear: there's been an update to 1234- the patient is no longer taking phenytoin, and now they have started taking vancomycin. The client adds the vancomycin to it's persistent store, and updates it's own copy of the MedicationStatement 1234.
Note, I'm assuming here that the resource ids of the medication statements are preserved - they **SHOULD** be for all sorts of reasons, but not all systems do. Those that don't: the consequence is that there's no way for any application to maintain a copy of the information downstream (a particularly subtle and pernicious way of information blocking).
So that's all good, but what about MedicationStatements 2346 and 6234?
Well, we presume that they haven't changed since the last query. So what if we didn't time limit that query, and just made the original query again? The client might assume that it would get all 4 records, but it might not. Say the client did the original query again, and got
```MedicationStatement 1234 v4: status=completed, code=phenytoin
MedicationStatement 6234 v1: status=active, subject=warfarin
MedicationStatement 7485 v1: status=active, subject=vancomycin
What happened to MedicationStatement 2346? It hasn't been returned - why? and what should the client do about it? Should it remove it from it's persistent store or not?
Here's a list of the more likely reasons why the record might not have been returned:
* The source resource(/record) was hard deleted from the origin server. And so the client should also delete it? Analysis: it's a**bad practice**to hard delete records that prior healthcare decisions might have been made from, or that might have been distributed. That's what we have**entered-in-error**for: to make that something was removed, and be able to distribute it. But of course, you guessed it - lots of systems just hard delete records when asked
* The record was marked as confidential on the server side, and policy is not to provide access to confidential information across the API.
Analysis: this is just a hard problem. Security isn't going to go away, and can always create problems like this. The resource will no longer be available, period. And it's computationally hard to recognise that a change somewhere in the security system means a resource is no longer available to a particular client that already accessed it
* The record was created against the wrong subject, and that's been fixed by simply changing the subject. The resource is no longer available for this patient. Analysis: this is a variant of the security problem, since security is by patient for patient access. Like deleting records, this is also bad practice (for the same reason). Applications should mark the old record as 'entered-in-error' and create a new correct record. But you guessed it... most don't
* The system may only be returning records with status active or completed unless the client specifically asks for other status codes (or even only active, unlike this example). Analysis: Some in-production Argonaut servers do this, in response to the status problem described above. We would rather they didn't do this, because it creates other problems - see[the long discussion on the subject](https://chat.fhir.org/#narrow/stream/179166-implementers/topic/Are.20search.20parameters.20like.20filters.3F)- but the problems this is addressing are real and serious. This situation can be rectified by the client by performing a GET [base]/MedicationStatement/2346 and seeing what is returned)
* The portal providing the API service may have
(temporarily?) lost access to the underlying record store from where MedicationStatement 2346 came from. Analysis: portals are often facades over much more complex underlying systems, so this is a real world possibility. There's no ideal solution other than to pretend that it will never be a problem
* The record may no longer available to the production system if enough years (5-7 or longer) have passed (and most EHR systems are at least that old, if not much older, no matter how recent the FHIR interface itself is)
Note: There's another bigger deal here - patient records may be [merged or unmerged](http://hl7.org/fhir/R4/patient.html#merge) which significantly complicates matters. Every system handles this slightly differently, and applications that maintain their own persistent store for more than one patient **cannot **ignore this - records may be moved, or come and go, and they just have to keep up. Of course, many/most don't do anything about patient record merge/link.
So: any client that it's keeping it's own persistent copy of data accessed by a FHIR Search like the above has to consult with each server to find out which of those possible reasons are applicable for the server, and decide what to do about it.
Applications that don't... are unsafe and shouldn't be written, marketed, or used. Of course, the users have no way of finding out how an application handles this. Maybe someone will take up the task of reviewing patient apps for safety, but it will be terribly difficult (=expensive) to do that.
I wish there was better advice I could provide, and we're working on getting better advice, but the problem is the trillions of dollars of legacy systems out there that healthcare providers don't seem to be in any hurry to replace (with systems that don't yet exist, mind).
The problems discussed here are already documented in unsafe outcomes for patients in production systems that access EHR clients using the FHIR interface (and in related forms they have been a documented problem for national patient record systems too)
Note that there are other ways for a client to synchronize data:
* Using the[history interaction](http://hl7.org/fhir/R4/http.html#history)
* Using[Subscriptions](http://hl7.org/fhir/R4/subscription.html)
These offer different APIs with different strengths and weaknesses - but since the underlying issues are record-keeping policy issues, not API surface issues, these generally don't make much difference, though we are working hard on the subscription route to make it possible to deal with some of these issues (more on that in a later post; and we probably will never resolve the security related issues).
```**MedicationStatement 1234 v4: status=completed, code=phenytoin
MedicationStatement 6234 v1: status=active, subject=warfarin
MedicationStatement 7485 v1: status=active, subject=vancomycin
What happened to MedicationStatement 2346? It hasn't been returned - why? and what should the client do about it? Should it remove it from it's persistent store or not?
Here's a list of the more likely reasons why the record might not have been returned:
* The source resource(/record) was hard deleted from the origin server. And so the client should also delete it? Analysis: it's a**bad practice**to hard delete records that prior healthcare decisions might have been made from, or that might have been distributed. That's what we have**entered-in-error**for: to make that something was removed, and be able to distribute it. But of course, you guessed it - lots of systems just hard delete records when asked
* The record was marked as confidential on the server side, and policy is not to provide access to confidential information across the API.
Analysis: this is just a hard problem. Security isn't going to go away, and can always create problems like this. The resource will no longer be available, period. And it's computationally hard to recognise that a change somewhere in the security system means a resource is no longer available to a particular client that already accessed it
* The record was created against the wrong subject, and that's been fixed by simply changing the subject. The resource is no longer available for this patient. Analysis: this is a variant of the security problem, since security is by patient for patient access. Like deleting records, this is also bad practice (for the same reason). Applications should mark the old record as 'entered-in-error' and create a new correct record. But you guessed it... most don't
* The system may only be returning records with status active or completed unless the client specifically asks for other status codes (or even only active, unlike this example). Analysis: Some in-production Argonaut servers do this, in response to the status problem described above. We would rather they didn't do this, because it creates other problems - see[the long discussion on the subject](https://chat.fhir.org/#narrow/stream/179166-implementers/topic/Are.20search.20parameters.20like.20filters.3F)- but the problems this is addressing are real and serious. This situation can be rectified by the client by performing a GET [base]/MedicationStatement/2346 and seeing what is returned)
* The portal providing the API service may have
(temporarily?) lost access to the underlying record store from where MedicationStatement 2346 came from. Analysis: portals are often facades over much more complex underlying systems, so this is a real world possibility. There's no ideal solution other than to pretend that it will never be a problem
* The record may no longer available to the production system if enough years (5-7 or longer) have passed (and most EHR systems are at least that old, if not much older, no matter how recent the FHIR interface itself is)
Note: There's another bigger deal here - patient records may be [merged or unmerged](http://hl7.org/fhir/R4/patient.html#merge) which significantly complicates matters. Every system handles this slightly differently, and applications that maintain their own persistent store for more than one patient **cannot **ignore this - records may be moved, or come and go, and they just have to keep up. Of course, many/most don't do anything about patient record merge/link.
So: any client that it's keeping it's own persistent copy of data accessed by a FHIR Search like the above has to consult with each server to find out which of those possible reasons are applicable for the server, and decide what to do about it.
Applications that don't... are unsafe and shouldn't be written, marketed, or used. Of course, the users have no way of finding out how an application handles this. Maybe someone will take up the task of reviewing patient apps for safety, but it will be terribly difficult (=expensive) to do that.
I wish there was better advice I could provide, and we're working on getting better advice, but the problem is the trillions of dollars of legacy systems out there that healthcare providers don't seem to be in any hurry to replace (with systems that don't yet exist, mind).
The problems discussed here are already documented in unsafe outcomes for patients in production systems that access EHR clients using the FHIR interface (and in related forms they have been a documented problem for national patient record systems too)
Note that there are other ways for a client to synchronize data:
* Using the[history interaction](http://hl7.org/fhir/R4/http.html#history)
* Using[Subscriptions](http://hl7.org/fhir/R4/subscription.html)
These offer different APIs with different strengths and weaknesses - but since the underlying issues are record-keeping policy issues, not API surface issues, these generally don't make much difference, though we are working hard on the subscription route to make it possible to deal with some of these issues (more on that in a later post; and we probably will never resolve the security related issues).
******