This report will review and analyze the research presented in the paper by Ash et al; 2009(1). The research paper provides an overview of the methods and results of a 4 year NLM funded endeavor by a group known as Physician Order Entry Team (POET). This group primarily consists of researchers affiliated with the Oregon Health & Science University, who have published over 30 papers about Computerized Provider Order Entry (CPOE) implementation. The work presented in this paper looked at the types, extent and importance of unintended adverse consequences (UAC) associated with the implementation of CPOE systems. However due to the large scope of the paper, a critical analysis of the research is limited by a paucity of details particularly in the reporting of results. As such research methods and results were also reviewed from 2 other papers (Ash et al; 2007(2), Campbell et al(3)).
The rationale presented for exploring this domain is related to the growing urgency(4) in the United States to implement CPOEs to address medical errors. As such it would be valuable to understand if CPOEs may create new forms of medical errors. Another rationale provided by Ash pertains to how UACs may be a barrier to the adoption of CPOE possibly explaining its slow uptake in US based hospitals(5). Framing UACs as a barrier helps explain the use of Rogers diffusion of innovation theory as the framework for the qualitative work(3). Some of the value of Ash et al; 2009(1) is how it challenges presumptions about the value of IT in health care by exploring UACs. As such this research can be of value to a range of individuals; health care workers, IT and hospital administration, policy makers, researchers, CPOE developer/vendors, advocacy groups and hospital associations. The value for these groups can fall into 2 key areas;
1. To develop a better understanding of UACs in order to address them
2. As a political tool to change existing policies around CPOEs.
As per Ash et al; 2009(1) the objectives were:
1. To describe the activities, methods and results of a 4 year research project exploring the unintended consequences of CPOE.
To achieve this goal the POET group carried out their research in two separate but linked steps with the following objectives;
1. To identify the types of clinical unintended adverse consequences resulting from CPOE implementation(3) and develop a taxonomy of UACs.
2. To discover the extent and importance of unintended adverse consequences related to CPOE in US hospitals(2). An unwritten objective may be to provide evidence that the UACs identified through qualitative methodologies are generalizable beyond the study sample.
The intervention studied was CPOE and the following definition was used;
“Direct entry of orders into the computer by physicians or others with the same ordering privileges.”(1)
The research methods used for the identification of types of UACs(Campbell et al)(3) was qualitative in nature using semi-structured participant interviews, ethnography and focus groups. The setting involved clinicians and administrators at 5 “excellent” US based acute care hospitals. Excellent institutions were judged by POET and an expert panel as those with a reputation for excellence in using clinical information systems. Analysis was done by the POET group using a card sort method and a grounded theory approach.
The research method used to discover the extent and importance of UACs (Ash et al; 2007) (2) were 10-20 minute telephone surveys collecting quantitative and qualitative data. This survey was based on a taxonomy of UACs developed from the first phase of their work. It was pilot tested but was not further tested for validity and reliability(2). The sample was US based acute care hospitals that had implemented CPOE (n=561). Analysis utilized descriptive statistics, logistic regression and Spearman’s rho statistic to describe infusion of CPOE, compare responders vs. non responders and response rate to duration of CPOE use.
From Campbell et al(3), 324 types of UACs were found that were grouped into a taxonomy of 9 categories. The authors concluded that this taxonomy could help identify and address UACs that arise in CPOE implementation. The survey in Ash et al; 2007 had a response rate of 47% with a majority of respondents ranking 7/8 UACs categories as important. Logistic regression identified responder-non responder differences for location and management type. Finally there was no correlation (Spearman’s rho) between response rate and duration of CPOE use. The authors of Ash et al; 2007(2) concluded that these results verified the proposed taxonomy and that clinical decision support tools are related to many of the UACs. In the original paper (Ash et al; 2009)(1) the authors also describe the development of a series of tools to help avoid or manage UACs with the implementation of a CPOE.
From the Campbell et al (3) paper there are concerns as to whether the authors truly achieved saturation from the work with the judgment sample of 5 excellent institutions. The reporting from both papers(1, 3) was unclear as to whether data was collected till saturation was achieved. Without further clarification this represents a significant methodological issue that calls into question the validity of the results. Further issues regarding whether the judgment sample should have included non-excellent institutions and fuller reporting around the analysis raise some concerns but are not critical methodological flaws, especially since the goal was not to provide an exhaustive list of all UACs associated with CPOE use.
The Ash et al; 2007 paper(2) did not seem to have any critical methodological flaws. However the single largest issue that significantly challenges the generalizability of the findings is the poor response rate. This coupled with the noted differences in characteristics between responders and non-responders further limits who these results can be applied to. The lack of rigorous survey validation is perhaps reflected in the confusing wording of questions and raises the question about whether this tool was actually measuring what it was intended to. Finally the authors raise the issue that the interviewees were mainly IT personnel which may have biased the results. This sample will need to be expanded upon to further validate the taxonomy.
Overall the reporting about the research methods in the Ash et al.; 2009(1) paper was difficult to follow and there was use of too many references to expand on key ideas. The taxonomy that was created is an interesting body of work that has value for the design, development, implementation and ongoing management of CPOEs in the hospital environment. Putting aside the issues raised earlier about the survey results, another issue that stands out is that the research does not provide the reader a sense of the relative magnitude of importance of these UACs especially in the context of other implementation issues, which was a stated objective of this research.
Questions for Authors
1. Can the authors of the Ash et al; 2006 paper elaborate on their data collecting protocols as it pertains to understanding whether saturation was achieved?
2. What if any measures were taken to improve the response rate?
3. Why did the authors not provide any further analysis of the responders who answered no and survey non responders?
4. Why did the authors not consider interviewing a broader range of individuals at the various institutions?
1. Ash JS, Sittig DF, Dykstra R, Campbell E, Guappone K. The unintended consequences of computerized provider order entry: findings from a mixed methods exploration. Int J Med Inform 2009;78 Suppl 1:S69-76.
2. Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007;14(4):415-23.
3. Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2006;13(5):547-56.
4. Institute of Medicine (U.S.). Committee on Quality of Health Care in America. Crossing the quality chasm : a new health system for the 21st century. Washington, D.C.: National Academy Press, 2001.
5. Ash JS, Gorman PN, Seshadri V, Hersh WR. Computerized physician order entry in U.S. hospitals: results of a 2002 survey. J Am Med Inform Assoc 2004;11(2):95-9.
“CATCH-IT Reports” are Critically Appraised Topics in Communication, Health Informatics, and Technology, discussing recently published ehealth research. We hope these reports will draw attention to important work published in journals, provide a platform for discussion around results and methodological issues in eHealth research, and help to develop a framework for evidence-based eHealth. CATCH-IT Reports arise from “journal club” - like sessions founded in February 2003 by Gunther Eysenbach.
Sunday, November 22, 2009
CATCH-IT Final Report: The unintended consequences of computerized order entry: findings from a mixed methods exploration.