CATCH-IT Final Report: Effect of guideline based computerised decision support on decision making of multidisciplinary teams

CATCH-IT Draft: Effect of guideline based computerised decision support on decision making of multidisciplinary teams: cluster randomised trial in cardiac rehabilitation

Goud R, de Keizer NF, ter Riet G, Wyatt JC, Hasman A, Hellemans IM, Peek N. Effect of guideline based computerised decision support on decision making of multidisciplinary teams: cluster randomised trial in cardiac rehabilitation. BMJ. 2009;338:b1440.

Abstract & Initial Comments / Full Text / Slideshow / CATCH-IT Draft & Comments

Introduction

In this article in BMJ, the authors investigated the effect of computerized decision support (CDS) on guideline adherence to recommended therapeutic decisions in multidisciplinary teams. They investigated the use of the Cardiac Rehabilitation Decision Support System (CARDSS) in promoting adherence to the Dutch Cardiac Rehabilitation Guidelines in cardiac rehabilitation centres in the Netherlands. It was a registered trail and the first to my knowledge to evaluate the effect of CDS on decision making in teams. The article would be of interest to health care settings with multidisciplinary teams who are thinking about adding CDS to their center.

Discussion

The study design was a randomized clustered trial. In order to investigate the effect of CARDSS on the care providers, the trial had to randomize by team/center rather than randomize on the patient level to avoid contamination at the physician level. Although a randomized clustered design has less precision and requires a greater sample size, due to the nature of the trial it was a necessity.

There was some concern for the high attrition rate in the study. Initially, there were 31 randomized centers, but 5 centres discontinued participation and another 5 were excluded. However, the authors also took steps to reduce this bias such as blinding the investigators during randomization allocation, use of objective outcome measures, and involvement of an external evaluator and statistician.

With this high attrition rate, there was a possible issue with the study having insufficient statistical power as the authors stated that they required 36 participating centers during their six month follow up to detect a 10% absolute difference in adherence rate with 80% power at a type I error risk (a) of 5%. However, in the end they were only able to include 21 centres in their analysis. Although, the 21 centers may have provided a large enough sample size by having a larger number of patients per a center, their analysis still resulted in wide confidence intervals such as the borderline significance found for exercise therapy (95% CI 0% to 5.4%). In addition, the authors did not explain how they calculated the adjusted difference or confidence intervals. It was not clear that the Improvement seen in the intervention group in comparison with control group was not influenced by the confounding factors. There was also no explanation on how the covariates (age, sex, diagnosis, weekly volume of new patients, whether the center is a specialized rehabilitation center or part of academic hospital) influenced the rate of adherence. The investigators were able to detect a significant change for three of the four therapies, but they also noticed a considerable undertreatment of patients where patients did not receive the treatment they were suppose to according to the guidelines and therefore were incorrectly given other treatments. This undertreatment was due to two issues: the therapy was not available at the center or patient non-adherence. The investigators could have controlled for the first issue by excluding centers which did not offer all four therapies, but that would have reduced the number of participating centers even further. The second issue was more of a concern since the trial was investigating care team adherence to guidelines; however, this relied on the patient adhering to them as well.

To account for the initial learning curve to the CARDSS, the authors provided a standardized training course for the care providers and excluded the data of patients enrolled in the first two weeks of CARDSS being used in each participating centre from the analyses. The authors stated that CARDSS was judged favourably in a usability study. However, the fact that three centres were excluded from the study for not recording their decisions on CARDSS and two centers for too much missing data may indicate a possible usability issue. It may be possible to re-engineer the software in a way to reduce missing recorded decisions and data. The data audit which the authors used to exclude the centres by comparing the record keeping on CARDSS and the paper based patient record was one of the strengths of the study. If there were discrepancies in the data between the two records, the authors considered the centre in question to be unreliable and excluded them from the analyses. If a centre passed the data audit but data analysis indicated that 20% or more of a centre’s patients’ records had missing data, the authors also excluded that centre from the analyses. The authors did perform a follow up qualitative study performed afterwards which looked into how CARDSS affected the main barriers to the implementation of guidelines for cardiac rehabilitation. In this follow up study, there was more qualitative user feedback from the interviews on the usability of CARDSS.

Although the authors stated that the study design ensured that there would be no bias from the Hawthorne effect, they did not go into on detail how this was accomplished. There could have been a possible Hawthorne effect since the care providers were aware they were in a study with their performance being monitored and reported which may have prompted them to increase their adherence to guidelines. There was also the potential bias resulting from having the care providers record their reason for cases where they do not adhere to the guidelines. They may simply go along with the recommended therapy from CARDSS since it would prevent additional work.

It would have been also beneficial if the investigators collected baseline adherence data prior to implementing the CARDSS at the centres. This would allow comparisons to be made between pre-implementation and post-implementation of CARDSS. The control group in the current study design is unable to do this since they were still given a CARDSS. Even if their version had limited functionality, it would have an effect on the results. It would also have been interesting to investigate guideline adherence rates in the control group after they have been given full functionality of the CARDSS in a follow up trial. The authors do have an ongoing follow up trial which may be looking into this aspect.

Finally, there was the issue regarding the unnecessity of ethics approval for the study. Since patients are involved in the study, one would assume ethics approval would be required; however, it was stated in the article that ethics approval was not needed in their country according to the medical ethics committee of the Academic Medical Centre in Amsterdam.

Conclusion

The authors concluded that CARDSS improved adherence to guideline recommendations with respect to three out of the four therapies: exercise, education, and relaxation therapy. There was no effect on lifestyle change therapy which may be partly due to the fact that the majority of clinics did not have the therapy program available. Although there are some weaknesses and limitations in this study, the authors hope to address most of them in their follow up randomized cluster study.

Questions to the authors

• Why was baseline adherence data not collected?
• How do the covariates affect guideline adherence?
• How were the adjusted differences and CI values calculated?
• What is the reason for large variation in adherence between centers?
• Why was CARDSS not designed to be interoperable with other information systems?
• How did the authors account for the effect of the control group knowing they were in the control group?
• How significant of a factor was patient non-adherence and how do you plan on controlling it for the follow up trial?

CATCH-IT FINAL: Individualized electronic decision support and reminders to improve diabetes care in the community: COMPETE II randomized trial.

Holbrook, A., Thebane, L., Keshavjee, K., Dolovich, L., Bernstein, B., Chan, D., Troyan, S., Foster, G., Gerstein, H. Individualized electronic decision support and reminders to improve diabetes care in the community: COMPETE II randomized trial. CMAJ. 2009 Jul 7;181(1-2):37-44

Links: Abstract - Full Paper - Draft Report - Presentation

Introduction

Globally, it is estimated that 18 million people are living with Diabetes Mellitus, accounting for 5% of all deaths worldwide (1). The condition affects approximately 7% of the populations of Canada and the United States (2). Any intervention that is capable of reducing the burden that such patients pose on the health care system, and in a cost-effective manner, would have huge potential in relieving and reducing future diabetes related health care costs. To this end, the paper by Holbrooke et al (2), Individualized electronic decision support and reminders can improve diabetes care in the community: COMPETE II randomized trial, represents a significant attempt at addressing diabetes care in a community-based care setting using an information technology based intervention. The paper is a pragmatic randomized trial that evaluates a Web-based decision support enabled diabetes tracker, the Computerization of Medical Practices for the Enhancement of Therapeutic Effectiveness (COMPETE) Project (http://compete-study.com/), and the general application of eHealth technologies in community-based care settings.

The authorship team of this paper have extensive experience in the area of clinical decision support and conducting research trials in the relatively emergent area of eHealth and clinical informatics, and have published several papers on information technology-based interventions for chronic diseases. This paper is one of just many publications put forth over the past decade with regard to the COMPETE project. Phase II of project, conducted under the auspice of Health Canada, was funded $2.378 million (3) by the Canada Health Infostructure Partnerships Program (CHIPP). The COMPETE project site is publicly available (http://www.compete-study.com/).

Objectives of the study

The aim of the study was to evaluate whether the COMPETE II intervention would result in improved quality of diabetes care. Holbrook and colleagues (2) describe their multi-modal intervention as having many components including

shared access by the primary care provider and the patient to a Web-based, colour-coded diabetes tracker, which provided sequential monitoring values for 13 diabetes risk factors, their respective targets and brief, prioritized messages of advice.

The intervention also included monthly-automated telephone reminders, 3 incidences of lab work, and frequent physician visits. The authors cite that there is a lack of high-quality research on the effect of computerized decision support systems. However, the limited studies available suggest that CDS systems can positively change provider behavior.

Methods

The study design was a randomized trial over the time period of approximately one year. A total of 46 primary care providers (43 physicians and 3 nurse practitioners) were recruited – all having an electronic medical record in their practice. Participants were selected randomly using allocation concealment and computer generation of group assignment from the care providers’ roster and recruited by invitations sent via mail. Patients in the intervention group were instructed to have relevant blood tests followed by a visit with their family physician a week later. In addition, intervention arm patients had access to the web-based tracker, were mailed a paper version of the tracker page twice, and were sent monthly reminders for medication, lab, and physician visits via phone. The study duration was 6 months. No description of primary care provider recruitment was provided.

Results were computed using two scoring methods: a process composite score and a clinical outcome score. The process composite score resulted from the sum of 8 variables measuring the frequency of care (e.g., quarterly or semiannual). The variables were modeled from recommendations by professional diabetes associations, expert opinion, and literature reviews. Each variable was given a maximum score (either 1 or 2), though no description of how the authors determined this score was given. Clinical outcomes were scored based on targets on an 8-item composite for clinical marker outcomes. A special subset, labeled ‘ABC’ (glycated hemoglobin, blood pressure, and cholesterol [LDL]) was also evaluated.

Results

A total of 511 participants were recruited, 253 and 258 assigned to the intervention and control arms respectively. The intervention group improved the total process composite score when compared to the control group (1.33 v 0.06; difference 1.27, 95% CI 0.79 to 1.75, p <>p = 0.036) and the ABC composite score (0.34, 95% CI 0.04 to 0.65, p = 0.028). The results, although modest, are compatible with other studies (4,5) of eHealth interventions targeting chronic diseases.

Limitations

Although the authors suggest that diabetes quality of care can be improved with electronic tracking and decision support shared by providers and patients, it is unclear whether this effect is attributable to the intervention or merely due to increased frequency of doctor visits. The study design specifies 3 care provider visits for the intervention arm, compared to none in the control arm. This alone could have impacted both process and clinical outcome scores. The authors do not report the actual number of primary care provider (PCP) visits of patients within either groups. A difference of 0.66 is reported, though absolute numbers have been omitted.

There are also concerns regarding the frequency of lab visits. Appendix 3 states that patients in the intervention arm had a total of 3 lab visits during the 6-month course of the study. There is no mention of the number of lab visits for patients in the control arm. Moreover, the paper does not state what type of lab work is done which would alter the process score - for example, if albuminuria, glycated hemoglobin, and/or LDL cholesterol were measured - thereby completely undermining the assertion that the observations were a result of the decision support system.

Discussion

Holbrook and colleagues have taken a valuable stride in the implementation of complex eHealth innovations, specifically in a community-based care setting. The complexities of such implementations present many obstacles in eHealth evaluation. It would be valuable to understand the role that patients themselves play with regard to eHealth tools in non-disease specific contexts – topics that warrant further investigation.

Although the study concept is unique, the manner in which the study was reported was problematic in numerous areas. The paper does not reveal the utilization rates of the actual decision support tool. As a result, it is difficult to deduce which portion of the tool was useful to both physicians and patients. The paper also states that there were “technical difficulties” (2) with the tool itself. No description is provided. Therefore, the reader is left wondering the extent to which the tool was actually used. Similarly, there is no description of how the system interfaced with the providers’ EMRs. The paper briefly discusses the “inability to completely integrate the tracker’s decision-support system with each of the 5 different types of electronic medical records” (2). To this end, new questions surrounding data quality emerge. Readers are unaware of the accuracy of the data, who entered in data, how often, or the motivations to do so. Such information would have been beneficial in guiding further development of similar clinical informatics interventions.

There is some concern with regard to some of the data presented in the paper. Table 3 reports a score of 1.00 and 1.00 in the ‘Before’ column of the intervention arm for exercise and smoking respectively. The ‘After’ column reports scores of 0.69 and 0.69 respectively. This implies that in this cohort of patients, 100% of the processes were measured before the intervention with much fewer being measured after. In addition, the standard deviation in the ‘Before’ column is 0.06 for smoking, despite a score of 1.00. How is this possible? The authors do not address this anomaly within the paper.

Questions also arise regarding the timeframe of the study. Participants were tracked for 6 months, although the total period for the study was over a year. Why did the authors choose a 6-month time period when many of the process variables were measured on a biannual basis? In addition, the authors do not address the issue of bias towards null. In other words, physicians in the control arm could have possibly learned from their rostered patients in the intervention arm, thereby altering the effect of the decision support tool. A remedy for this issue, a cluster randomized trial, was not mentioned.

Although the study has many limitations, many of which could be reduced through more thorough reporting, the implications arevast. Conducting randomized trials of eHealth applications are especially onerous and difficult to carry out. This study exemplifies a successful randomized trial using eHealth innovations in a community-based setting. The findings have significance in the area of chronic disease management and will aid in the future development of eHealth tools which specifically target chronic diseases. Nonetheless, this study highlights the necessity of more rigorous standards for research in health information technology in which tools can be meticulously evaluated.

Questions for the Authors

1. What were the technical difficulties in integrating the tracker with the care providers’ EMR? How did this affect the study?

2. How was the process composite score validated? How did the authors decide to assign differential scores to BP (2) vs. glycate HgB (1)?

3. Why was the study conducted within such a short time frame (6 months)?

4. Why were only the intervention arm participants sent for initial lab tests and follow up physician visits? How were the baseline values measured for the control arm?

5. What were the utilization rates of the electronic decision support system? Which components worked/were used?

6. How were patients educated on using the intervention?

7. What were the reasons for patients withdrawing from the study?

8. Why did the authors not perform a cluster randomized trial in order to address bias towards null?

9. How often did patients in the control arm visit primary care providers and have relevant lab tests conducted?

10. How were primary care providers recruited?

11. Did the authors consider the ‘checklist effect’ bias whereby filling out a questionnaire can alter behavior?

Acknowledgements

The author would like to thank the members of the CATCH-IT Journal Club at the University of Toronto and the Centre for Global eHealth Innovation, Toronto, for their feedback and critical analysis that have greatly contributed to this report.

References

(1) World Health Organization Diabetes Programme. 2008; Available at: http://www.who.int/mediacentre/factsheets/fs312/en/index.html. Accessed Nov/7, 2009.

(2) Holbrook A, Thabane L, Keshavjee K, Dolovich L, Bernstein B, Chan D, et al. Individualized electronic decision support and reminders to improve diabetes care in the community: COMPETE II randomized trial. CMAJ 2009 Jul 7;181(1-2):37-44.

(3) COMPETE II. Available at: http://www.hc-sc.gc.ca/hcs-sss/pubs/chipp-ppics/2003-compete/final-eng.php. Accessed Oct 15, 2009.

(4) Trudel M, Cafazzo JA, Hamill M, Igharas W, Tallevi K, Picton P, et al. A mobile phone based remote patient monitoring system for chronic disease management. Stud.Health Technol.Inform. 2007;129(Pt 1):167-171.

(5) Wyne K. Information technology for the treatment of diabetes: improving outcomes and controlling costs. J.Manag.Care.Pharm. 2008 Mar;14(2 Suppl):S12-7.

CATCH-IT Draft Report: The unintended consequences of computerized order entry: findings from a mixed methods exploration.

Presentation

Introduction

The purpose of this report is to analyze the research presented in the paper by Ash et al; 2009(1). This paper provides an overview of the research methods and results of a 4 year research endeavor by a research group known as POET. Their work looked into the types, extent and importance of unintended adverse consequences (UAC) associated with the implementation of Computerized Provider Order Entry (CPOE) systems. However due to the large scope of the paper, a critical analysis of the research is limited by a paucity of details particularly in the reporting of results. As such research methods and results were also reviewed from 2 other papers (Ash et al; 2007(2), Campbell et al(3)).

The rationale presented in this report, for exploring this domain is related to the growing urgency(4) to implement CPOEs to address medical errors. As such it would be of significant value to understand if these solutions may be causing new forms of medical errors. Another rationale provided by Ash in a set of earlier writings pertains to how UACs may be a barrier to the adoption of CPOE and may explain the slow uptake of CPOE in US based medical facilities(5). From this second rationale we can then understand the use of Rogers diffusion of innovation theory as the theoretical framework in the qualitative work that is presented(3). Some of the value of this paper is in how it challenges presumptions about the value of IT in health care environments by exploring UACs. As such this body of research can be of value to a broad range of individuals (health care workers, IT and hospital administration, policy makers, researchers, CPOE vendors, advocacy groups and hospital associations). The value for these groups falls into 2 key areas; develop a better understanding of UACs in order to address them and as a political tool to change existing policies around CPOEs.

Objectives

As per the authors of Ash et al; 2009(1) the objectives are: To describe the activities, methods and results of a 4 year research project exploring the unintended consequences of CPOE.

In order to achieve this goal the POET group carried out their research in two separate but linked steps with the following objectives:

To identify the types of clinical unintended adverse consequences resulting from CPOE implementation(3) and develop a taxonomy of UACs. To discover the extent and importance of unintended adverse consequences related to CPOE in US hospitals(2). A possible unwritten objective may have been to provide evidence that the derived results from UAC identification through qualitative methodologies is generalizable.

Methods

The interventions being studied are CPOEs and the authors used the following definition for CPOE;

“Defined as direct entry of orders into the computer by physicians or others with the same ordering privileges.”(1)

The research methods used for the identification of types of UACs(3) was of a qualitative nature using semi-structured participant interviews, ethnography and focus groups. The setting for their work involved clinicians and administrators at 5 “excellent” US based acute care hospitals. Analysis was done by the POET group using a card sort method and a grounded theory approach.

The research method used for the identification of the extent and importance of UACs were 10-20 minute telephone surveys to collect quantitative and qualitative data. This survey was based on the taxonomy of UACs that was developed from the first phase of their work. It was pilot tested but was not tested for validity and reliability(2). The sample was all US based acute care hospitals that had implemented CPOE (n=561). Analysis utilized descriptive statistics, logistic regression and Spearman’s rho statistic to describe infusion of CPOE, compare responders vs. non responders and response rate to duration of CPOE use.

Results

From the work done to identify types of UACs(3), 324 types were found that were grouped into 9 categories. The authors concluded that this taxonomy could help identify and address UACs that arise in CPOE implementation. From the survey results were analyzed for 265/561 hospitals for a response rate of 47%. Logistic regression identified responder-non responder differences and there was no correlation (Spearman’s rho) between response rate and duration of CPOE use. The authors of this paper(2) concluded that these results verified the proposed taxonomy and that clinical decision support tools are related to many of the UACs. From the original paper (Ash et al; 2009) the authors go on to describe how an expert panel and the POET group went on to develop a series of tools that are available to help avoid or manage UACs with the implementation of a CPOE.

Limitations

From the paper identifying the types of UACs there are concerns as to whether the authors(3) truly achieved saturation from the work with the 5 excellent institutions. The reporting from both papers(1,3) was unclear as to whether data was collected till saturation was achieved. As such without further clarification this represents a significant methodological issue that would call into question the validity of the results. Further issues regarding whether the judgment sample should have included non-excellent institutions and fuller reporting around the analysis raise some concerns but are not critical methodological flaws, especially since the goal was not to provide an exhaustive list of all UACs associated with CPOE use.

The work looking at the extent and importance of UACs(2) did not seem to have any critical methodological flaws. However the single largest issue that significantly challenges the generalizability of the findings is the poor response rate. This coupled with the noted differences in characteristics between responders and non-responders further limits who these results can be applied to. The lack of survey validation is perhaps reflected in the confusing wording of questions and raises the question about whether this tool was actually measuring what it was intended to. Finally the authors raise the issue that the interviewees were mainly IT personnel and this may also have biased the result.

Discussion

Overall the reporting about the research methods in the Ash et al.; 2009 paper was difficult to follow and there was use of too many references to expand on key ideas. The taxonomy that was created is an interesting body of work that may already be a part of change management practices, as such perhaps more value could be derived from this work by highlighting UACs that are CPOE specific. Even if one were to put aside the issues raised earlier about the survey results, another issue that stands out is that the research does not provide the reader any sense of the magnitude of importance of these UACs especially in the context of other implementation issues, which was a stated objective of this research.


Questions for Authors

1. Can the authors of the Ash et al; 2006 paper elaborate on their data collecting protocols as it pertains to understanding whether saturation was achieved?
2. What if any measures were taken to improve the response rate?
3. Why did the authors not provide any further analysis of the responders who answered no and survey non responders?
4. Why did the authors not consider interviewing a broader range of individuals at the various institutions?

References

1. Ash JS, Sittig DF, Dykstra R, Campbell E, Guappone K. The unintended consequences of computerized provider order entry: findings from a mixed methods exploration. Int J Med Inform 2009;78 Suppl 1:S69-76.
2. Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007;14(4):415-23.
3. Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2006;13(5):547-56.
4. Institute of Medicine (U.S.). Committee on Quality of Health Care in America. Crossing the quality chasm : a new health system for the 21st century. Washington, D.C.: National Academy Press, 2001.
5. Ash JS, Gorman PN, Seshadri V, Hersh WR. Computerized physician order entry in U.S. hospitals: results of a 2002 survey. J Am Med Inform Assoc 2004;11(2):95-9.

Web-based Weight Loss in Primary Care: A Randomized Controlled Trial

Bennett GG, Herring SJ, Puleo E, Stein EK, Emmons KM and Gillman MW. Web-based Weight Loss in Primary Care: A Randomized Controlled Trial. Obesity (2009) doi:10.1038/oby.2009.242

Full text: click here

Abstract
Evidence is lacking regarding effective and sustainable weight loss approaches for use in the primary care setting. We conducted a 12-week randomized controlled trial to evaluate the short-term efficacy of a web-based weight loss intervention among 101 primary care patients with obesity and hypertension. Patients had access to a comprehensive website that used a moderate-intensity weight loss approach designed specifically for web-based implementation. Patients also participated in four (two in-person and two telephonic) counselling sessions with a health coach. Intent-to-treat analysis showed greater weight loss at 3 months (-2.56 kg; 95% CI -3.60, -1.53) among intervention participants (-2.28 +/- 3.21 kg), relative to usual care (0.28 +/- 1.87 kg). Similar findings were observed among intervention completers (-3.05 kg; 95% CI -4.24, -1.85). High rates of participant retention (84%) and website utilization were observed, with the greatest weight loss found among those with a high frequency of website logins (quartile 4 vs. 1: -4.16 kg; 95% CI -1.47, -6.84). The intervention's approach promoted moderate weight loss at 12 weeks, though greater weight loss was observed among those with higher levels of website utilization. Efficacious web-based weight loss interventions can be successfully offered in the primary care setting.

Additional resources to assess study:
1. CONSORT for Non Pharmacological Treatments: click here
2. STARE-HI (Talmon J et al. International Journal of Medical Informatics
Volume 78, Issue 1, January 2009, Pages 1-9)

CATCH-IT Draft: Clinical Decision Support capabilities of Commercially-available Clinical Information Systems

WRIGHT, A., SITTIG, D. F., ASH, J.S., SHARMA, S., PANG, J. E., and MIDDLETON, B. (2009). Clinical Decision Support capabilities of Commercially-available Clinical Information Systems. Journal of the American Medical Informatics Association, 16(5), 637 – 644.

Background
Recent studies have reported that the CDS applications built in-house produce the best results. However, there is not much research done for the CDS capabilities of commercially available clinical information systems (CIS). The cited paper wishes to fill this gap in research by evaluating the CDS capabilities of 9 commercially available CCHIT certified EHR systems using a 42-element functional taxonomy. The evaluations are based on information collected from the vendors and customers of the EHR systems. The study finds that while capabilities for ‘triggers’ in CDS are well covered among the systems, many capabilities for ‘offered choices’ are not present. The results of the study are presented pseudonymously to respect privacy of the customer.

This report is based on an evaluation of the study in the CATCH-IT Journal Club. It reports the key points raised about the methodological issues of the study as a result of the CATCH-IT analysis. These issues are discussed in the following, and it is expected that consideration of this evaluation will enhance the quality of the research performed by the research community.

Methodological Issues
There are several methodological issues with the original study that can be highlighted. These methodological issues can be causes of potential concerns that may hinder the validity of the research findings. The following sections discuss the methodological issues in more detail.

Use of CCHIT Certified EHR Systems
The authors indicated about the use of CCHIT certification as a baseline for the selected systems in the study. As a result of establishing such a baseline, the authors have ensured that the selected systems meet a particular quality and are have comparable features.

An investigation of the CCHIT certification requirements has indicated that the CCHIT certification criteria are continuously evolving, with additional requirements being added each year. CCHIT uses a matrix of requirements with specific requirements relating to a system’s domain of use (such as ambulatory care and outpatient care) and the system’s aspect of use (such as for EMR storage and CDS). While CCHIT certification has been used as a baseline for the selection procedure of the systems, the authors have not discussed details about what year the selected systems were certified, and whether their certification has been renewed with the evolving CCHIT requirements. In addition, it is unclear as to what the authors have done to ensure that the features in the selected taxonomy are in alignment with the CDS –specific requirements CCHIT.

System Selection Procedure
The authors indicated in the methods that a preliminary set of CCHIT certified EHR systems was identified based on figures from Klas and HIMSS Analytics. The vendors involved with the development of these systems and the customers of these systems were then contacted, based on which a sample of 9 systems were selected for this study.

The immediate concerns that arise regarding the selection procedure is that it is unclear as to how many systems were originally selected, what was the nature of such communications (such as questions asked, and type of information requested), and what was the criteria for short listing the selected EHRs for the study. Without such details, the study cannot illustrate to the audience that the study followed an effective selection procedure in which there was no external influence, and that a specific system was included or excluded from the study due to potential bias.

Taxonomy Selection
For the purpose of determining the availability of a certain CDS capability in the selected systems, the authors selected a self-developed functional taxonomy that combined common CDS capabilities along four axes – triggers, input data elements, interventions, and offered choices. The authors mention that the taxonomy was developed based on a research at the Partners HealthCare System, by emphasizing on the fact that there was no other functional taxonomy available for use in this research.

It has been determined that the taxonomy has been developed based on the numerous clinical rules used at the Partners HealthCare System in Boston. While Partners is evidently a large healthcare system involving a blend of healthcare provider types, it must be noted that the developed taxonomy has not been validated by employing it in other healthcare organizations outside of Partners.

Due to the concerns raised by the use of an un-validated self-developed taxonomy in this research, an investigative approach has been used to determine how well the taxonomy has been received by the research community. Findings suggest that even though the taxonomy research was published in 2007, to date there are only 5 journal articles that reference that research study. There is only one article that has not been authored by any of the researchers involved with the taxonomy development. However, that article does not make any specific reference to the taxonomy or its development. As a result, research has failed to identify any neutral opinion about the developed taxonomy, raising concerns of self-boasting by the authors’ use of a self-developed taxonomy that lacks an apparent acceptance in the research community. However, this raises serious concerns about the findings of the study, since the study evaluation is wholly based on the CDS capabilities indentified in the taxonomy.

Data Collection Procedure
In this study, the vendors and customers of the 9 systems were contacted and interviewed by three of the authors. The outcomes of these interviews were used to evaluate the CDS capabilities of each system against the 42-element taxonomy. The authors reported that if there was any doubt about the availability of a particular feature in any of the systems, they contacted other customers, read product manuals, and conducted hands-on evaluation to determine availability of a feature.

The paper suggests that three of the authors were involves with the data collection procedure. The authors have not specified what kind of data collection mechanisms were used for collecting the data. Data collection procedures can be a cause of bad data for which the study is based. As a result, the researchers must demonstrate the validity of their data collection procedure. For example, it is not known whether the authors used one-on-one interviews or panel interviews for collecting the data, how many interviews were conducted with the same interviewee, were the interviews open-ended or close-ended, how many questions were involved, what was the follow-up procedure, and what did the authors do to prepare for the interviews. The audience of the research can easily raise questions about the procedures and argue about the limitations of the procedures used. A well-written report will typically avoid letting such concerns settle in the mind of its audience.

Apart from the concerns about the data collection procedure, there are concerns about the validity of the collected data. It is unclear who the researchers spoke with in each of these interviews with the vendors or customers. Not all members of the vendor organization are able to answer the same question about the availability of a particular feature. In the case of a vendor, there may be bias in the answers about the availability of a certain feature. At the same time, asking the customers about the availability of a feature may raise concerns about the knowledge of customer about the product itself. And this leads to the question as to what method did the authors use to ensure that 1) what the vendors and customers are saying are actually valid, 2) how was the collected data validated, 3) what is it that raised doubts in the researchers’ minds because of which they conducted further investigation, and 4) what determined whether a feature is actually available.

Results Interpretation
The results of the study have been presented in a tabular form for each of the axis of the taxonomy by evaluating the systems against the features in the axes. To respect the software vendors’ right to privacy, the results were pseudonymously represented by identifying each system with a number. In their evaluation, the authors used a binary-style evaluation, where the result is either yes (available) or no (unavailable). Since both in-patient and outpatient were used, the inapplicable criterion for a system was marked as N/A (not applicable). The final result was represented with a count of unavailable features by each system by each axis. In the authors’ view, the system with the least number of unavailable features is the best system.
Although the authors have mentioned this as a limitation of their study, but the binary-style evaluation does not match the way that the data for this study has been collected and used. The data collected in the study was qualitative, and it has been used to evaluate a question to yes or no. Similar to the concern about the validity of collected data, this raises serious questions about the validity of the evaluation that the authors have performed. For example, even if a feature is available, what did the authors do to evaluate how well that feature has been implemented by the software developers? How complete is the feature? How usable is it? How applicable is it for a particular setting?

The representation of the final result in this research by tallying the number of unavailable features is all but useful. The authors have failed to use key concepts of importance, usefulness, and frequency of use of an available feature. For example, a feature may be useful, but may not be frequently used. Or perhaps a feature is not frequently used but is very important for the success of a CDS application. This directly impacts the final results of the study where the authors chose the system with the least number of unavailable features as the best system. Since the scoring system used in this research is weak, it can be argued that the results are invalid.

Questions for the authors
1. What led to the linear treatment of the capabilities?
2. What was the reason behind the use of a taxonomy which is not yet well-received in the research community?
3. What were the steps taken to validate the information gathered from the vendors and customers?
4. What was the reasoning behind counting the number of unavailable features rather than available ones? Did you not want to deal with the complexity of working with N/A?
5. Why were both inpatient and outpatient systems with potentially different capabilities selected for the study?

References
1. Wright A, Sittig D F, Ash J S, Sharma S, Pang J E, and Middleton B. Clinical Decision Support capabilities of Commercially-available Clinical Information Systems. Journal of the American Medical Informatics Association 2009; 16(5): 637-644.

2. Parners Healthcare. What is Partners?. Accessed via http://www.partners.org/about/about_whatis.html. Accessed on October 20, 2009

3. Scopus. Scopus Journal Search. Accessed via http://simplelink.library.utoronto.ca/url.cfm/54186. Accessed on October 22, 2009

4. BioMed Experts. Accessed via http://www.biomedexperts.com. Accessed on October 15, 2009.

5. DMICE: People – Students. Department of Medical Informatics & Clinical Epidemiology, Oregon Health & Science University. Accessed via http://www.ohsu.edu/ohsuedu/academic/som/dmice/people/students/index.cfm. Accessed on October 20, 2009

6. Clinical and Quality Analysis, Information Systems. Clinical and Quality Analysis Staff. Accessed via http://www.partners.org/cqa/Staff.htm. Accessed on October 18, 2009.

7. Wrigh A, Goldberg H, Hongsermeier T, and Middleton B. A Description and Functional Taxonomy of Rule-Based Decision Support Content at a Large Integrated Delivery Network. Journal of the American Medical Informatics Association 2007; 14(4): 489-496.

8. CCHIT. Concise Guide to CCHIT Certification Criteria. Accessed via http://www.cchit.org/sites/all/files/ConciseGuideToCCHIT_CertificationCriteria_May_29_2009.pdf. Accessed on October 10, 2009.

9. Sittig DF, Wright A, Osheroff JA, Middleton B, Teich JM, Ash JS, Campbell E, Bates DW. Grand challenges in clinical decision support. Journal of Biomedical Informatics 2008; 41(2):387-392.

Nov 16 - Effectiveness of Active-Online, an Individually Tailored Physical Activity Intervention, in a Real-Life Setting: Randomized Controlled Trial

Wanner M., Martin-Diener E., Braun-Fahrländer C., Bauer G., Martin B.W. (2009). Effectiveness of Active-Online, an Individually Tailored Physical Activity Intervention, in a Real-Life Setting: Randomized Controlled Trial. J Med Internet Res, 11 (3): e23.

Full Text - Abstract Only - CATCH-IT Draft

Background: Effective interventions are needed to reduce the chronic disease epidemic. The Internet has the potential to provide large populations with individual advice at relatively low cost.

Objective: The focus of the study was the Web-based tailored physical activity intervention Active-online. The main research questions were (1) How effective is Active-online, compared to a nontailored website, in increasing self-reported and objectively measured physical activity levels in the general population when delivered in a real-life setting? (2) Do respondents recruited for the randomized study differ from spontaneous users of Active-online, and how does effectiveness differ between these groups? (3) What is the impact of frequency and duration of use of Active-online on changes in physical activity behavior?

Methods: Volunteers recruited via different media channels completed a Web-based baseline survey and were randomized to Active-online (intervention group) or a nontailored website (control group). In addition, spontaneous users were recruited directly from the Active-online website. In a subgroup of participants, physical activity was measured objectively using accelerometers. Follow-up assessments took place 6 weeks (FU1), 6 months (FU2), and 13 months (FU3) after baseline.

Results: A total of 1531 respondents completed the baseline questionnaire (intervention group n = 681, control group n = 688, spontaneous users n = 162); 133 individuals had valid accelerometer data at baseline. Mean age of the total sample was 43.7 years, and 1146 (74.9%) were women. Mixed linear models (adjusted for sex, age, BMI category, and stage of change) showed a significant increase in self-reported mean minutes spent in moderate- and vigorous-intensity activity from baseline to FU1 (coefficient = 0.14, P = .001) and to FU3 (coefficient = 0.19, P < .001) in all participants with no significant differences between groups. A significant increase in the proportion of individuals meeting the HEPA recommendations (self-reported) was observed in all participants between baseline and FU3 (OR = 1.47, P = .03), with a higher increase in spontaneous users compared to the randomized groups (interaction between FU3 and spontaneous users, OR = 2.95, P = .02). There were no increases in physical activity over time in any group for objectively measured physical activity. A significant relation was found between time spent on the tailored intervention and changes in self-reported physical activity between baseline and FU3 (coefficient = 1.13, P = .03, intervention group and spontaneous users combined). However, this association was no longer significant when adjusting for stage of change. Conclusions: In a real-life setting, Active-online was not more effective than a nontailored website in increasing physical activity levels in volunteers from the general population. Further research may investigate ways of integrating Web-based physical activity interventions in a wider context, for example, primary care or workplace health promotion.

Acceptability of a Personally Controlled Health Record in a Community-Based Setting: Implications for Policy and Design

Weitzman ER, Kaci L, Mandl KD. Acceptability of a Personally Controlled Health Record in a Community-Based Setting: Implications for Policy and Design. J.Med.Internet Res. 2009 Apr 29;11(2):e14.


Full-text article: http://www.jmir.org/2009/2/e14/HTML
J Med Internet Research Vol 11, No 2 (2009)


ABSTRACT

Background: Consumer-centered health information systems that address problems related to fragmented health records and disengaged and disempowered patients are needed, as are information systems that support public health monitoring and research. Personally controlled health records (PCHRs) represent one response to these needs. PCHRs are a special class of personal health records (PHRs) distinguished by the extent to which users control record access and contents. Recently launched PCHR platforms include Google Health, Microsoft’s HealthVault, and the Dossia platform, based on Indivo.

Objective: To understand the acceptability, early impacts, policy, and design requirements of PCHRs in a community-based setting.

Methods: Observational and narrative data relating to acceptability, adoption, and use of a personally controlled health record were collected and analyzed within a formative evaluation of a PCHR demonstration. Subjects were affiliates of a managed care organization run by an urban university in the northeastern United States. Data were collected using focus groups, semi-structured individual interviews, and content review of email communications. Subjects included: n = 20 administrators, clinicians, and institutional stakeholders who participated in pre-deployment group or individual interviews; n = 52 community members who participated in usability testing and/or pre-deployment piloting; and n = 250 subjects who participated in the full demonstration of which n = 81 initiated email communications to troubleshoot problems or provide feedback. All data were formatted as narrative text and coded thematically by two independent analysts using a shared rubric of a priori defined major codes. Sub-themes were identified by analysts using an iterative inductive process. Themes were reviewed within and across research activities (ie, focus group, usability testing, email content review) and triangulated to identify patterns.

Results: Low levels of familiarity with PCHRs were found as were high expectations for capabilities of nascent systems. Perceived value for PCHRs was highest around abilities to co-locate, view, update, and share health information with providers. Expectations were lowest for opportunities to participate in research. Early adopters perceived that PCHR benefits outweighed perceived risks, including those related to inadvertent or intentional information disclosure. Barriers and facilitators at institutional, interpersonal, and individual levels were identified. Endorsement of a dynamic platform model PCHR was evidenced by preferences for embedded searching, linking, and messaging capabilities in PCHRs; by high expectations for within-system tailored communications; and by expectation of linkages between self-report and clinical data.

Conclusions: Low levels of awareness/preparedness and high expectations for PCHRs exist as a potentially problematic pairing. Educational and technical assistance for lay users and providers are critical to meet challenges related to: access to PCHRs, especially among older cohorts; workflow demands and resistance to change among providers; inadequate health and technology literacy; clarification of boundaries and responsibility for ensuring accuracy and integrity of health information across distributed data systems; and understanding confidentiality and privacy risks. Continued demonstration and evaluation of PCHRs is essential to advancing their use.