Final CATCH-IT Report: Clinical Decision Support capabilities of Commercially-available Clinical Information Systems

WRIGHT, A., SITTIG, D. F., ASH, J.S., SHARMA, S., PANG, J. E., and MIDDLETON, B. (2009). Clinical Decision Support capabilities of Commercially-available Clinical Information Systems. Journal of the American Medical Informatics Association, 16(5), 637 – 644.

Links: Abstract, Full Text, Presentation, Draft Report

Introduction

Clinical information systems (CISs) are helping with the production and maintenance of an increased amount of health information that can be used for clinical decision support (CDS) using CDS systems (1). Recent studies have reported that the CDS applications built in-house produce the best results (2). However, there is not much research done for the CDS capabilities of commercially available CISs (2). The paper by Wright et al. (2), “Clinical Decision Support Capabilities of Commercially-available Clinical Information Systems”, wishes to fill this gap in research by evaluating the CDS capabilities of 9 commercially available CCHIT certified EHR systems using a 42-element functional taxonomy. The study findings suggest that while some CDS capabilities are commonly available in the evaluated systems, some other capabilities are not very well covered among most systems (2).

This report is based on an evaluation of the study in the CATCH-IT Journal Club (3). It reports the key methodological issues of the study as a result of the CATCH-IT analysis. These issues are discussed in the following, and it is expected that consideration of this evaluation will enhance the quality of research performed by the research community.

A majority of the researchers involved with this study are affiliated with the Partners Healthcare System (4,5), with one being a candidate for a Masters degree at the Oregon Health & Science University (6). The key authors have had numerous publications in the area of CDS (7). The authors seem to have a sizable research network, with the primary author having the first publication in 2007 (7).

Study Background

The authors claimed that since no other functional taxonomy existed for CDS evaluation, the study utilized a self-developed functional taxonomy. The taxonomy, developed at the Partners Healthcare System, describes CDS capabilities along four axes (triggers, input data elements, interventions, and offered choices) for the evaluation of the CDS-capabilities in this research. In order to establish a baseline, the study used CCHIT-certified EHR systems to ensure that the selected systems meet a particular quality and have comparable features.

A background study has been conducted to determine the availability of any other functional taxonomies for CDS evaluation. It has failed to identify any other functional taxonomy that could be applicable for this research. In addition, functional features such as the workflow capabilities is considered as one of the most important features for the success of a CDS (1), the selected taxonomy does not use this in any of its axes.

The background study has identified that even though the taxonomy research (8) (titled “A Description and Functional Taxonomy of Rule-Based Decision Support Content at a Large Integrated Delivery Network”) was published in 2007, to date there are only 5 journal articles that reference the study (9). Four of these articles are self-authored (9). The only one reference from a neutral authorship does not make any comment or use of the taxonomy itself (10). Thus, the survey has failed to identify any neutral opinion about the developed taxonomy, raising concerns of self-boasting by the authors’ use of a self-developed taxonomy that lacks an apparent acceptance in the research community.

It must be noted that CCHIT uses a matrix of requirements based on its domain (such as for ambulatory care and outpatient care) and aspect (such as for EMR storage and CDS) of use (3). The report is unclear about the steps taken to ensure the alignment of the selected taxonomy with the CDS –specific CCHIT requirements.

Methods

A preliminary set of CCHIT-certified EHR systems was identified based on figures from Klas and HIMSS Analytics. The vendors involved with the development of these systems and the customers of these systems were then contacted, based on which a sample of 9 systems were selected for this study. Three of the authors interviewed ‘knowledgeable individuals’ (2) within the vendor organizations and the results from these interviews were used to evaluate the CDS capabilities of each system against the 42-element taxonomy. If there were any doubts about the availability of a particular feature in the systems, the authors contacted other members of the vendor organization, read product manuals, and conducted hands-on evaluation to determine availability of a feature.

The methods do not provide details about the sample size (number of systems originally reviewed), nature of the communications with the vendors and customers, and the short-listing criteria. While the reader is unable to find such basic information about the study, it also remains unclear on whether the study had followed an effective selection procedure with no external influence or bias. It is also difficult for the readers to the procedure used that resulted in short-listing to 9 systems.

The report has omitted key details on the data collection mechanisms. Inefficient data collection procedures can be a cause of unreliable study data. For example, it is not known whether the authors used one-on-one interviews or panel interviews for collecting the data, the number of interviews conducted with the same interviewee, number of interviews conducted with representatives from the same vendor, the style of the interviews (such as open-ended or close-ended interviews), number and types of questions asked, the follow-up procedure, the interview preparation procedure, and whether the interviewers had to reach a consensus or whether different interviewers made their own decisions.

The report has missing details on the interviewees, and who the authors referred to as the ‘knowledgeable individuals’. It is unlikely that all members of the vendor organization can answer the technical questions that the authors may have had. These raise concerns of bias and lack of knowledge among the interviewees. And this leads to the question as to what method the authors used to ensure that information provided by the interviewees are valid. In addition, the report is not clear about the strategy used by the authors to validate the interviewees’ responses when in doubt (2). Such omissions can lead to the audience’s inability to assess the legibility of the method used and determination of whether the study results can be trusted.

Results

The study results are pseudonymously presented (to respect the software vendors’ privacy) in a tabular form for each of the axis of the taxonomy. A binary-style evaluation with a yes (available) or no (unavailable) for applicable systems, or with a N/A (not applicable) was used. The final result was represented with a count of unavailable features for each system. In the authors’ view, the system with the least number of unavailable features is the best system.

Based on the description of the study methods, there seems to be a mismatch between the collection and usage of data in this research. Perhaps this is due to the missing details about the steps that the researchers took to reach a conclusion about a particular feature based on the collected data. This raises concerns about potential bias in the evaluation of feature availability without the inclusion of completeness, usefulness, and applicability of CDS features in evaluating each system. These features can have a great impact on the proper adoption of a CDS application (1). As the selected taxonomy does not incorporate the criteria above, it raises concerns about the validity of the results of the study.

The representation of the final results in this research by tallying the number of unavailable features provides limited context to the reader. With pseudonymous representation of the systems, the audience is left unclear about which features are lacking in each systems. And if this is not the conclusion that the authors had intended to reach, it the report remains unclear about the actual intent of this research. In addition, the authors’ selection of the best system as the one with the least number of unavailable features fails to assess the impact that each unavailable feature could have on the success of the system. It is possible that the absence of five infrequently used features may be better than the absence of one crucial feature in a system. As a result, the authors’ approach in not weighing the importance of a feature in the system evaluation is not useful for such comparative evaluations.

Conclusion

The discussion presented in this report highlight a few important issues with the study. It seems that the report has left out important details such as the identification of the systems in the results, detailed description of the study methods, and appropriate usage of the data collected. Perhaps if the report was more appropriately written with these details, these issues may have been minimized or eliminated. Based on its current status, the study fails to add much value to the research community, as neither can its pseudonymous results be used for ongoing CDS research, nor can its usage of the evaluation methodology with an untested taxonomy be utilized for evaluating commercially available CDS-enabled EHR systems in a reliable manner.

The use of a comprehensively developed CDS-based taxonomy with the use of weighed features based on their importance in different healthcare settings may have helped immensely in making the study results more reliable.

Questions for the authors

1. What was the reason behind the use of a taxonomy which is not yet well-received in the research community? Do you feel that the other taxonomies may have been used with minor modifications?
2. Why were both inpatient and outpatient systems with potentially different capabilities selected for the study? What was done to ensure that the selected taxonomy and the two types of systems were in alignment in terms of their features?
3. What was the reason for which vendors of the systems were contacted for evaluating the systems? What was done to ensure unbiased information from the vendors?
4. What methodological procedure, including the details of each step, was undertaken to validate the information collected from the vendors and customers?
5. What was evaluation procedure undertaken to use the qualitative information gathered from the interviewees in order to conclude about the availability of a feature in a system? Why was this detail not included in the report?
6. What was the reason behind counting the number of unavailable features rather than available ones? Did you not want to deal with the complexity of working with N/A? Or was it done as a workaround to deal with the two types of inherently different types of systems (inpatient and outpatient)?
7. What led to the linear treatment of the capabilities without any discussion about the importance or usefulness of the capabilities? What impact do you feel that the inclusion of such details would have had on the study results?
8. What was the actual objective of the study? Was it to demonstrate a CDS evaluation methodology or was it to identify the best CDS-enabled EHR system currently in the market?

Acknowledgements

The author thanks all the members of the CATCH-IT Journal Club, including Professor Gunther Eysenbach and Professor Nancy Martin-Ronson, for their insightful comments that helped with the evaluation of the study.

Bibliography

1. Sittig D F WAOJAMBTJMAJSCEBDW. Grand challenges in clinical decision support. Journal of Biomedical Informatics. 2008; 41(2): p. 387-392.
   
   
2. Wright A SDFAJSSSPJEaMB. Clinical Decision Support capabilities of Commercially-available Clinical Information Systems. Journal of the American Medical Informatics Association. 2009; 16(5): p. 637-644.
   
   
3. CCHIT. Concise Guide to CCHIT Certification Criteria. [Online].; 2009 [cited 2009 October 10. Available from: http://www.cchit.org/sites/all/files/ConciseGuideToCCHIT_CertificationCriteria_May_29_2009.pdf.
   
   
4. Clinical and Quality Analysis ISPHS. Clinical and Quality Analysis Staff. [Online]. [cited 2009 October 18. Available from: http://www.partners.org/cqa/Staff.htm.
   
   
5. Healthcare P. What is Partners? [Online]. [cited 2009 October 20. Available from: http://www.partners.org/about/about_whatis.html.
   
   
6. OHSU. DMICE: People – Students, Department of Medical Informatics & Clinical Epidemiology, Oregon Health & Science University. [Online]. [cited 2009 October 20. Available from: http://www.ohsu.edu/ohsuedu/academic/som/dmice/people/students/index.cfm.
   
   
7. Experts B. BioMed Experts. [Online].; 2009 [cited 2009 October 15. Available from: http://www.biomedexperts.com.
   
   
8. Wrigh A GHHTaMB. A Description and Functional Taxonomy of Rule-Based Decision Support Content at a Large Integrated Delivery Network. Journal of the American Medical Informatics Association. 2007; 14(4): p. 489-496.
   
   
9. Scopus. Scopus Journal Search. [Online].; 2009 [cited 2009 October 22. Available from: http://simplelink.library.utoronto.ca/url.cfm/54186.
   
   
10. Chused A E KGJaSPD. Alert Override Reasons: A Failure to Communicate. American Medical Informatics Association - Annual Symposium Proceedings 2008. 2008;: p. 111-115.

CATCH-IT Draft: Clinical Decision Support capabilities of Commercially-available Clinical Information Systems

WRIGHT, A., SITTIG, D. F., ASH, J.S., SHARMA, S., PANG, J. E., and MIDDLETON, B. (2009). Clinical Decision Support capabilities of Commercially-available Clinical Information Systems. Journal of the American Medical Informatics Association, 16(5), 637 – 644.

Background
Recent studies have reported that the CDS applications built in-house produce the best results. However, there is not much research done for the CDS capabilities of commercially available clinical information systems (CIS). The cited paper wishes to fill this gap in research by evaluating the CDS capabilities of 9 commercially available CCHIT certified EHR systems using a 42-element functional taxonomy. The evaluations are based on information collected from the vendors and customers of the EHR systems. The study finds that while capabilities for ‘triggers’ in CDS are well covered among the systems, many capabilities for ‘offered choices’ are not present. The results of the study are presented pseudonymously to respect privacy of the customer.

This report is based on an evaluation of the study in the CATCH-IT Journal Club. It reports the key points raised about the methodological issues of the study as a result of the CATCH-IT analysis. These issues are discussed in the following, and it is expected that consideration of this evaluation will enhance the quality of the research performed by the research community.

Methodological Issues
There are several methodological issues with the original study that can be highlighted. These methodological issues can be causes of potential concerns that may hinder the validity of the research findings. The following sections discuss the methodological issues in more detail.

Use of CCHIT Certified EHR Systems
The authors indicated about the use of CCHIT certification as a baseline for the selected systems in the study. As a result of establishing such a baseline, the authors have ensured that the selected systems meet a particular quality and are have comparable features.

An investigation of the CCHIT certification requirements has indicated that the CCHIT certification criteria are continuously evolving, with additional requirements being added each year. CCHIT uses a matrix of requirements with specific requirements relating to a system’s domain of use (such as ambulatory care and outpatient care) and the system’s aspect of use (such as for EMR storage and CDS). While CCHIT certification has been used as a baseline for the selection procedure of the systems, the authors have not discussed details about what year the selected systems were certified, and whether their certification has been renewed with the evolving CCHIT requirements. In addition, it is unclear as to what the authors have done to ensure that the features in the selected taxonomy are in alignment with the CDS –specific requirements CCHIT.

System Selection Procedure
The authors indicated in the methods that a preliminary set of CCHIT certified EHR systems was identified based on figures from Klas and HIMSS Analytics. The vendors involved with the development of these systems and the customers of these systems were then contacted, based on which a sample of 9 systems were selected for this study.

The immediate concerns that arise regarding the selection procedure is that it is unclear as to how many systems were originally selected, what was the nature of such communications (such as questions asked, and type of information requested), and what was the criteria for short listing the selected EHRs for the study. Without such details, the study cannot illustrate to the audience that the study followed an effective selection procedure in which there was no external influence, and that a specific system was included or excluded from the study due to potential bias.

Taxonomy Selection
For the purpose of determining the availability of a certain CDS capability in the selected systems, the authors selected a self-developed functional taxonomy that combined common CDS capabilities along four axes – triggers, input data elements, interventions, and offered choices. The authors mention that the taxonomy was developed based on a research at the Partners HealthCare System, by emphasizing on the fact that there was no other functional taxonomy available for use in this research.

It has been determined that the taxonomy has been developed based on the numerous clinical rules used at the Partners HealthCare System in Boston. While Partners is evidently a large healthcare system involving a blend of healthcare provider types, it must be noted that the developed taxonomy has not been validated by employing it in other healthcare organizations outside of Partners.

Due to the concerns raised by the use of an un-validated self-developed taxonomy in this research, an investigative approach has been used to determine how well the taxonomy has been received by the research community. Findings suggest that even though the taxonomy research was published in 2007, to date there are only 5 journal articles that reference that research study. There is only one article that has not been authored by any of the researchers involved with the taxonomy development. However, that article does not make any specific reference to the taxonomy or its development. As a result, research has failed to identify any neutral opinion about the developed taxonomy, raising concerns of self-boasting by the authors’ use of a self-developed taxonomy that lacks an apparent acceptance in the research community. However, this raises serious concerns about the findings of the study, since the study evaluation is wholly based on the CDS capabilities indentified in the taxonomy.

Data Collection Procedure
In this study, the vendors and customers of the 9 systems were contacted and interviewed by three of the authors. The outcomes of these interviews were used to evaluate the CDS capabilities of each system against the 42-element taxonomy. The authors reported that if there was any doubt about the availability of a particular feature in any of the systems, they contacted other customers, read product manuals, and conducted hands-on evaluation to determine availability of a feature.

The paper suggests that three of the authors were involves with the data collection procedure. The authors have not specified what kind of data collection mechanisms were used for collecting the data. Data collection procedures can be a cause of bad data for which the study is based. As a result, the researchers must demonstrate the validity of their data collection procedure. For example, it is not known whether the authors used one-on-one interviews or panel interviews for collecting the data, how many interviews were conducted with the same interviewee, were the interviews open-ended or close-ended, how many questions were involved, what was the follow-up procedure, and what did the authors do to prepare for the interviews. The audience of the research can easily raise questions about the procedures and argue about the limitations of the procedures used. A well-written report will typically avoid letting such concerns settle in the mind of its audience.

Apart from the concerns about the data collection procedure, there are concerns about the validity of the collected data. It is unclear who the researchers spoke with in each of these interviews with the vendors or customers. Not all members of the vendor organization are able to answer the same question about the availability of a particular feature. In the case of a vendor, there may be bias in the answers about the availability of a certain feature. At the same time, asking the customers about the availability of a feature may raise concerns about the knowledge of customer about the product itself. And this leads to the question as to what method did the authors use to ensure that 1) what the vendors and customers are saying are actually valid, 2) how was the collected data validated, 3) what is it that raised doubts in the researchers’ minds because of which they conducted further investigation, and 4) what determined whether a feature is actually available.

Results Interpretation
The results of the study have been presented in a tabular form for each of the axis of the taxonomy by evaluating the systems against the features in the axes. To respect the software vendors’ right to privacy, the results were pseudonymously represented by identifying each system with a number. In their evaluation, the authors used a binary-style evaluation, where the result is either yes (available) or no (unavailable). Since both in-patient and outpatient were used, the inapplicable criterion for a system was marked as N/A (not applicable). The final result was represented with a count of unavailable features by each system by each axis. In the authors’ view, the system with the least number of unavailable features is the best system.
Although the authors have mentioned this as a limitation of their study, but the binary-style evaluation does not match the way that the data for this study has been collected and used. The data collected in the study was qualitative, and it has been used to evaluate a question to yes or no. Similar to the concern about the validity of collected data, this raises serious questions about the validity of the evaluation that the authors have performed. For example, even if a feature is available, what did the authors do to evaluate how well that feature has been implemented by the software developers? How complete is the feature? How usable is it? How applicable is it for a particular setting?

The representation of the final result in this research by tallying the number of unavailable features is all but useful. The authors have failed to use key concepts of importance, usefulness, and frequency of use of an available feature. For example, a feature may be useful, but may not be frequently used. Or perhaps a feature is not frequently used but is very important for the success of a CDS application. This directly impacts the final results of the study where the authors chose the system with the least number of unavailable features as the best system. Since the scoring system used in this research is weak, it can be argued that the results are invalid.

Questions for the authors
1. What led to the linear treatment of the capabilities?
2. What was the reason behind the use of a taxonomy which is not yet well-received in the research community?
3. What were the steps taken to validate the information gathered from the vendors and customers?
4. What was the reasoning behind counting the number of unavailable features rather than available ones? Did you not want to deal with the complexity of working with N/A?
5. Why were both inpatient and outpatient systems with potentially different capabilities selected for the study?

References
1. Wright A, Sittig D F, Ash J S, Sharma S, Pang J E, and Middleton B. Clinical Decision Support capabilities of Commercially-available Clinical Information Systems. Journal of the American Medical Informatics Association 2009; 16(5): 637-644.

2. Parners Healthcare. What is Partners?. Accessed via http://www.partners.org/about/about_whatis.html. Accessed on October 20, 2009

3. Scopus. Scopus Journal Search. Accessed via http://simplelink.library.utoronto.ca/url.cfm/54186. Accessed on October 22, 2009

4. BioMed Experts. Accessed via http://www.biomedexperts.com. Accessed on October 15, 2009.

5. DMICE: People – Students. Department of Medical Informatics & Clinical Epidemiology, Oregon Health & Science University. Accessed via http://www.ohsu.edu/ohsuedu/academic/som/dmice/people/students/index.cfm. Accessed on October 20, 2009

6. Clinical and Quality Analysis, Information Systems. Clinical and Quality Analysis Staff. Accessed via http://www.partners.org/cqa/Staff.htm. Accessed on October 18, 2009.

7. Wrigh A, Goldberg H, Hongsermeier T, and Middleton B. A Description and Functional Taxonomy of Rule-Based Decision Support Content at a Large Integrated Delivery Network. Journal of the American Medical Informatics Association 2007; 14(4): 489-496.

8. CCHIT. Concise Guide to CCHIT Certification Criteria. Accessed via http://www.cchit.org/sites/all/files/ConciseGuideToCCHIT_CertificationCriteria_May_29_2009.pdf. Accessed on October 10, 2009.

9. Sittig DF, Wright A, Osheroff JA, Middleton B, Teich JM, Ash JS, Campbell E, Bates DW. Grand challenges in clinical decision support. Journal of Biomedical Informatics 2008; 41(2):387-392.