CATCH-IT Draft Report: The unintended consequences of computerized order entry: findings from a mixed methods exploration.

Presentation

Introduction

The purpose of this report is to analyze the research presented in the paper by Ash et al; 2009(1). This paper provides an overview of the research methods and results of a 4 year research endeavor by a research group known as POET. Their work looked into the types, extent and importance of unintended adverse consequences (UAC) associated with the implementation of Computerized Provider Order Entry (CPOE) systems. However due to the large scope of the paper, a critical analysis of the research is limited by a paucity of details particularly in the reporting of results. As such research methods and results were also reviewed from 2 other papers (Ash et al; 2007(2), Campbell et al(3)).

The rationale presented in this report, for exploring this domain is related to the growing urgency(4) to implement CPOEs to address medical errors. As such it would be of significant value to understand if these solutions may be causing new forms of medical errors. Another rationale provided by Ash in a set of earlier writings pertains to how UACs may be a barrier to the adoption of CPOE and may explain the slow uptake of CPOE in US based medical facilities(5). From this second rationale we can then understand the use of Rogers diffusion of innovation theory as the theoretical framework in the qualitative work that is presented(3). Some of the value of this paper is in how it challenges presumptions about the value of IT in health care environments by exploring UACs. As such this body of research can be of value to a broad range of individuals (health care workers, IT and hospital administration, policy makers, researchers, CPOE vendors, advocacy groups and hospital associations). The value for these groups falls into 2 key areas; develop a better understanding of UACs in order to address them and as a political tool to change existing policies around CPOEs.

Objectives

As per the authors of Ash et al; 2009(1) the objectives are: To describe the activities, methods and results of a 4 year research project exploring the unintended consequences of CPOE.

In order to achieve this goal the POET group carried out their research in two separate but linked steps with the following objectives:

To identify the types of clinical unintended adverse consequences resulting from CPOE implementation(3) and develop a taxonomy of UACs. To discover the extent and importance of unintended adverse consequences related to CPOE in US hospitals(2). A possible unwritten objective may have been to provide evidence that the derived results from UAC identification through qualitative methodologies is generalizable.

Methods

The interventions being studied are CPOEs and the authors used the following definition for CPOE;

“Defined as direct entry of orders into the computer by physicians or others with the same ordering privileges.”(1)

The research methods used for the identification of types of UACs(3) was of a qualitative nature using semi-structured participant interviews, ethnography and focus groups. The setting for their work involved clinicians and administrators at 5 “excellent” US based acute care hospitals. Analysis was done by the POET group using a card sort method and a grounded theory approach.

The research method used for the identification of the extent and importance of UACs were 10-20 minute telephone surveys to collect quantitative and qualitative data. This survey was based on the taxonomy of UACs that was developed from the first phase of their work. It was pilot tested but was not tested for validity and reliability(2). The sample was all US based acute care hospitals that had implemented CPOE (n=561). Analysis utilized descriptive statistics, logistic regression and Spearman’s rho statistic to describe infusion of CPOE, compare responders vs. non responders and response rate to duration of CPOE use.

Results

From the work done to identify types of UACs(3), 324 types were found that were grouped into 9 categories. The authors concluded that this taxonomy could help identify and address UACs that arise in CPOE implementation. From the survey results were analyzed for 265/561 hospitals for a response rate of 47%. Logistic regression identified responder-non responder differences and there was no correlation (Spearman’s rho) between response rate and duration of CPOE use. The authors of this paper(2) concluded that these results verified the proposed taxonomy and that clinical decision support tools are related to many of the UACs. From the original paper (Ash et al; 2009) the authors go on to describe how an expert panel and the POET group went on to develop a series of tools that are available to help avoid or manage UACs with the implementation of a CPOE.

Limitations

From the paper identifying the types of UACs there are concerns as to whether the authors(3) truly achieved saturation from the work with the 5 excellent institutions. The reporting from both papers(1,3) was unclear as to whether data was collected till saturation was achieved. As such without further clarification this represents a significant methodological issue that would call into question the validity of the results. Further issues regarding whether the judgment sample should have included non-excellent institutions and fuller reporting around the analysis raise some concerns but are not critical methodological flaws, especially since the goal was not to provide an exhaustive list of all UACs associated with CPOE use.

The work looking at the extent and importance of UACs(2) did not seem to have any critical methodological flaws. However the single largest issue that significantly challenges the generalizability of the findings is the poor response rate. This coupled with the noted differences in characteristics between responders and non-responders further limits who these results can be applied to. The lack of survey validation is perhaps reflected in the confusing wording of questions and raises the question about whether this tool was actually measuring what it was intended to. Finally the authors raise the issue that the interviewees were mainly IT personnel and this may also have biased the result.

Discussion

Overall the reporting about the research methods in the Ash et al.; 2009 paper was difficult to follow and there was use of too many references to expand on key ideas. The taxonomy that was created is an interesting body of work that may already be a part of change management practices, as such perhaps more value could be derived from this work by highlighting UACs that are CPOE specific. Even if one were to put aside the issues raised earlier about the survey results, another issue that stands out is that the research does not provide the reader any sense of the magnitude of importance of these UACs especially in the context of other implementation issues, which was a stated objective of this research.


Questions for Authors

1. Can the authors of the Ash et al; 2006 paper elaborate on their data collecting protocols as it pertains to understanding whether saturation was achieved?
2. What if any measures were taken to improve the response rate?
3. Why did the authors not provide any further analysis of the responders who answered no and survey non responders?
4. Why did the authors not consider interviewing a broader range of individuals at the various institutions?

References

1. Ash JS, Sittig DF, Dykstra R, Campbell E, Guappone K. The unintended consequences of computerized provider order entry: findings from a mixed methods exploration. Int J Med Inform 2009;78 Suppl 1:S69-76.
2. Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007;14(4):415-23.
3. Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2006;13(5):547-56.
4. Institute of Medicine (U.S.). Committee on Quality of Health Care in America. Crossing the quality chasm : a new health system for the 21st century. Washington, D.C.: National Academy Press, 2001.
5. Ash JS, Gorman PN, Seshadri V, Hersh WR. Computerized physician order entry in U.S. hospitals: results of a 2002 survey. J Am Med Inform Assoc 2004;11(2):95-9.