Final CATCH-IT Report: Clinical Decision Support capabilities of Commercially-available Clinical Information Systems

WRIGHT, A., SITTIG, D. F., ASH, J.S., SHARMA, S., PANG, J. E., and MIDDLETON, B. (2009). Clinical Decision Support capabilities of Commercially-available Clinical Information Systems. Journal of the American Medical Informatics Association, 16(5), 637 – 644.

Links: Abstract, Full Text, Presentation, Draft Report

Introduction

Clinical information systems (CISs) are helping with the production and maintenance of an increased amount of health information that can be used for clinical decision support (CDS) using CDS systems (1). Recent studies have reported that the CDS applications built in-house produce the best results (2). However, there is not much research done for the CDS capabilities of commercially available CISs (2). The paper by Wright et al. (2), “Clinical Decision Support Capabilities of Commercially-available Clinical Information Systems”, wishes to fill this gap in research by evaluating the CDS capabilities of 9 commercially available CCHIT certified EHR systems using a 42-element functional taxonomy. The study findings suggest that while some CDS capabilities are commonly available in the evaluated systems, some other capabilities are not very well covered among most systems (2).

This report is based on an evaluation of the study in the CATCH-IT Journal Club (3). It reports the key methodological issues of the study as a result of the CATCH-IT analysis. These issues are discussed in the following, and it is expected that consideration of this evaluation will enhance the quality of research performed by the research community.

A majority of the researchers involved with this study are affiliated with the Partners Healthcare System (4,5), with one being a candidate for a Masters degree at the Oregon Health & Science University (6). The key authors have had numerous publications in the area of CDS (7). The authors seem to have a sizable research network, with the primary author having the first publication in 2007 (7).

Study Background

The authors claimed that since no other functional taxonomy existed for CDS evaluation, the study utilized a self-developed functional taxonomy. The taxonomy, developed at the Partners Healthcare System, describes CDS capabilities along four axes (triggers, input data elements, interventions, and offered choices) for the evaluation of the CDS-capabilities in this research. In order to establish a baseline, the study used CCHIT-certified EHR systems to ensure that the selected systems meet a particular quality and have comparable features.

A background study has been conducted to determine the availability of any other functional taxonomies for CDS evaluation. It has failed to identify any other functional taxonomy that could be applicable for this research. In addition, functional features such as the workflow capabilities is considered as one of the most important features for the success of a CDS (1), the selected taxonomy does not use this in any of its axes.

The background study has identified that even though the taxonomy research (8) (titled “A Description and Functional Taxonomy of Rule-Based Decision Support Content at a Large Integrated Delivery Network”) was published in 2007, to date there are only 5 journal articles that reference the study (9). Four of these articles are self-authored (9). The only one reference from a neutral authorship does not make any comment or use of the taxonomy itself (10). Thus, the survey has failed to identify any neutral opinion about the developed taxonomy, raising concerns of self-boasting by the authors’ use of a self-developed taxonomy that lacks an apparent acceptance in the research community.

It must be noted that CCHIT uses a matrix of requirements based on its domain (such as for ambulatory care and outpatient care) and aspect (such as for EMR storage and CDS) of use (3). The report is unclear about the steps taken to ensure the alignment of the selected taxonomy with the CDS –specific CCHIT requirements.

Methods

A preliminary set of CCHIT-certified EHR systems was identified based on figures from Klas and HIMSS Analytics. The vendors involved with the development of these systems and the customers of these systems were then contacted, based on which a sample of 9 systems were selected for this study. Three of the authors interviewed ‘knowledgeable individuals’ (2) within the vendor organizations and the results from these interviews were used to evaluate the CDS capabilities of each system against the 42-element taxonomy. If there were any doubts about the availability of a particular feature in the systems, the authors contacted other members of the vendor organization, read product manuals, and conducted hands-on evaluation to determine availability of a feature.

The methods do not provide details about the sample size (number of systems originally reviewed), nature of the communications with the vendors and customers, and the short-listing criteria. While the reader is unable to find such basic information about the study, it also remains unclear on whether the study had followed an effective selection procedure with no external influence or bias. It is also difficult for the readers to the procedure used that resulted in short-listing to 9 systems.

The report has omitted key details on the data collection mechanisms. Inefficient data collection procedures can be a cause of unreliable study data. For example, it is not known whether the authors used one-on-one interviews or panel interviews for collecting the data, the number of interviews conducted with the same interviewee, number of interviews conducted with representatives from the same vendor, the style of the interviews (such as open-ended or close-ended interviews), number and types of questions asked, the follow-up procedure, the interview preparation procedure, and whether the interviewers had to reach a consensus or whether different interviewers made their own decisions.

The report has missing details on the interviewees, and who the authors referred to as the ‘knowledgeable individuals’. It is unlikely that all members of the vendor organization can answer the technical questions that the authors may have had. These raise concerns of bias and lack of knowledge among the interviewees. And this leads to the question as to what method the authors used to ensure that information provided by the interviewees are valid. In addition, the report is not clear about the strategy used by the authors to validate the interviewees’ responses when in doubt (2). Such omissions can lead to the audience’s inability to assess the legibility of the method used and determination of whether the study results can be trusted.

Results

The study results are pseudonymously presented (to respect the software vendors’ privacy) in a tabular form for each of the axis of the taxonomy. A binary-style evaluation with a yes (available) or no (unavailable) for applicable systems, or with a N/A (not applicable) was used. The final result was represented with a count of unavailable features for each system. In the authors’ view, the system with the least number of unavailable features is the best system.

Based on the description of the study methods, there seems to be a mismatch between the collection and usage of data in this research. Perhaps this is due to the missing details about the steps that the researchers took to reach a conclusion about a particular feature based on the collected data. This raises concerns about potential bias in the evaluation of feature availability without the inclusion of completeness, usefulness, and applicability of CDS features in evaluating each system. These features can have a great impact on the proper adoption of a CDS application (1). As the selected taxonomy does not incorporate the criteria above, it raises concerns about the validity of the results of the study.

The representation of the final results in this research by tallying the number of unavailable features provides limited context to the reader. With pseudonymous representation of the systems, the audience is left unclear about which features are lacking in each systems. And if this is not the conclusion that the authors had intended to reach, it the report remains unclear about the actual intent of this research. In addition, the authors’ selection of the best system as the one with the least number of unavailable features fails to assess the impact that each unavailable feature could have on the success of the system. It is possible that the absence of five infrequently used features may be better than the absence of one crucial feature in a system. As a result, the authors’ approach in not weighing the importance of a feature in the system evaluation is not useful for such comparative evaluations.

Conclusion

The discussion presented in this report highlight a few important issues with the study. It seems that the report has left out important details such as the identification of the systems in the results, detailed description of the study methods, and appropriate usage of the data collected. Perhaps if the report was more appropriately written with these details, these issues may have been minimized or eliminated. Based on its current status, the study fails to add much value to the research community, as neither can its pseudonymous results be used for ongoing CDS research, nor can its usage of the evaluation methodology with an untested taxonomy be utilized for evaluating commercially available CDS-enabled EHR systems in a reliable manner.

The use of a comprehensively developed CDS-based taxonomy with the use of weighed features based on their importance in different healthcare settings may have helped immensely in making the study results more reliable.

Questions for the authors

1. What was the reason behind the use of a taxonomy which is not yet well-received in the research community? Do you feel that the other taxonomies may have been used with minor modifications?
2. Why were both inpatient and outpatient systems with potentially different capabilities selected for the study? What was done to ensure that the selected taxonomy and the two types of systems were in alignment in terms of their features?
3. What was the reason for which vendors of the systems were contacted for evaluating the systems? What was done to ensure unbiased information from the vendors?
4. What methodological procedure, including the details of each step, was undertaken to validate the information collected from the vendors and customers?
5. What was evaluation procedure undertaken to use the qualitative information gathered from the interviewees in order to conclude about the availability of a feature in a system? Why was this detail not included in the report?
6. What was the reason behind counting the number of unavailable features rather than available ones? Did you not want to deal with the complexity of working with N/A? Or was it done as a workaround to deal with the two types of inherently different types of systems (inpatient and outpatient)?
7. What led to the linear treatment of the capabilities without any discussion about the importance or usefulness of the capabilities? What impact do you feel that the inclusion of such details would have had on the study results?
8. What was the actual objective of the study? Was it to demonstrate a CDS evaluation methodology or was it to identify the best CDS-enabled EHR system currently in the market?

Acknowledgements

The author thanks all the members of the CATCH-IT Journal Club, including Professor Gunther Eysenbach and Professor Nancy Martin-Ronson, for their insightful comments that helped with the evaluation of the study.

Bibliography

1. Sittig D F WAOJAMBTJMAJSCEBDW. Grand challenges in clinical decision support. Journal of Biomedical Informatics. 2008; 41(2): p. 387-392.
   
   
2. Wright A SDFAJSSSPJEaMB. Clinical Decision Support capabilities of Commercially-available Clinical Information Systems. Journal of the American Medical Informatics Association. 2009; 16(5): p. 637-644.
   
   
3. CCHIT. Concise Guide to CCHIT Certification Criteria. [Online].; 2009 [cited 2009 October 10. Available from: http://www.cchit.org/sites/all/files/ConciseGuideToCCHIT_CertificationCriteria_May_29_2009.pdf.
   
   
4. Clinical and Quality Analysis ISPHS. Clinical and Quality Analysis Staff. [Online]. [cited 2009 October 18. Available from: http://www.partners.org/cqa/Staff.htm.
   
   
5. Healthcare P. What is Partners? [Online]. [cited 2009 October 20. Available from: http://www.partners.org/about/about_whatis.html.
   
   
6. OHSU. DMICE: People – Students, Department of Medical Informatics & Clinical Epidemiology, Oregon Health & Science University. [Online]. [cited 2009 October 20. Available from: http://www.ohsu.edu/ohsuedu/academic/som/dmice/people/students/index.cfm.
   
   
7. Experts B. BioMed Experts. [Online].; 2009 [cited 2009 October 15. Available from: http://www.biomedexperts.com.
   
   
8. Wrigh A GHHTaMB. A Description and Functional Taxonomy of Rule-Based Decision Support Content at a Large Integrated Delivery Network. Journal of the American Medical Informatics Association. 2007; 14(4): p. 489-496.
   
   
9. Scopus. Scopus Journal Search. [Online].; 2009 [cited 2009 October 22. Available from: http://simplelink.library.utoronto.ca/url.cfm/54186.
   
   
10. Chused A E KGJaSPD. Alert Override Reasons: A Failure to Communicate. American Medical Informatics Association - Annual Symposium Proceedings 2008. 2008;: p. 111-115.