“CATCH-IT Reports” are Critically Appraised Topics in Communication, Health Informatics, and Technology, discussing recently published ehealth research. We hope these reports will draw attention to important work published in journals, provide a platform for discussion around results and methodological issues in eHealth research, and help to develop a framework for evidence-based eHealth. CATCH-IT Reports arise from “journal club” - like sessions founded in February 2003 by Gunther Eysenbach.

Thursday, October 15, 2009

CATCH-IT Draft: The Relationship between Electronic Health Record Use and Quality of Care over Time

Zhou L, Soran CS, Jenter CA, Volk LA, Orav EJ, Bates DW, Simon, SR. Relationship between EHR use and Quality of Care over Time. J Am Med Inform Assoc. Jul-Aug 2009;16:457– 464.

Abstract and initial discussion

CATCH-IT Final Report

The paper chosen for this particular CATCH-IT report is entitled, The Relationship between Electronic Health Record Use and Quality of Care over Time. It was written by the research team of Zhou, Soran, Jenter, Volk, Orav, Bates and Simon. This team consisted of the leading researchers in electronic health records (EHR) from Columbia and Harvard Universities. This paper can be considered a high quality paper because this group has conducted what little research there has been in this field. The results of this paper may have a potential impact on the decision of hospital administrators and clinicians to use EHRs in their respective practice settings. It was also picked because electronic health records are currently a very timely issue in political landscapes of Ontario and America.

This paper would be interesting for a number of stakeholders. It would interest researchers because it suggests that further research is required. There are also the quality measurement criteria that could be improved. It may also appeal to researchers because there is a potential for other quality of care data sets to be used to compare against individual clinician EHR usage. It would be of interest to politicians, especially the opposition because the conclusions drawn state that the use of EHR is not associated with improved healthcare outcomes. This can be used as an argument for government excess or spending. The media may also use the results of this study, especially in Ontario, to show that there may not be any inherent value in having an EHR at all. However, since this paper only discusses the quality aspect of care, this may not be as much of a disincentive to stop the investment in EHRs.

Some of the background information drawn for this paper is the methodology for the clinician EHR usage survey. To obtain this (and the survey) one would have to check the reference list and contact the author of that particular study which only measured physician EHR usage. It was noted upon reading the reference paper by Simon et al. that there were many physicians who had moved, were deceased or were no longer practicing medicine who received the survey.

This article studied the use of EHR and the quality of care given by clinicians in various practices settings. The study employed the use of two data sets. It is of interest to note that both data sets used are secondary data and were not created for the purpose intended by the study authors. One survey measured physicians’ adoption and use of EHR in the state of Massachusetts. It used an eight page questionnaire with a stratified random sample, i.e., hospital-based vs. large vs. rural practices, by specialty, etc. Sample weights were used to adjust for overall representation. For example, if there was a total of 70 physicians, and only 50 responded, the authors would perform an adjustment calculation, i.e., 50/70 multiplied by each answer to represent the entire population. The second data set was statewide data on physicians’ quality of care as indicated by their performance on widely used quality measures. The secondary data set was drawn from a HEDIS database. A HEDIS database is populated using insurance claims data. There were certain exclusion criteria set for the HEDIS data which made respondents younger, more recently graduated, in smaller practices with more patient visits and more likely to be female. These two data sets were integrated and linked for each physician. The outcome measures used were core EHR functions and associated features (i.e., health information and data, result management, order entry and management, decision support, and electronic communication and connectivity) as well as the HEDIS quality measuses which were aggregated into six clinical categories (i.e., asthma care, behavioural and mental health, cancer screening, diabetes care, well child and adolescent visit, and women’s health). A suggestion for future health outcome measures would be to use clinical guidelines for measuring quality. Also, the use of a feature such as decision support in this type of measure is misleading. The extent of decision support varies as well as the functionality as this feature has progressed from 2001 through 2005.

As mentioned, this study examined the cross-sectional relationship between having an EHR and concurrent indicators of quality of care. The authors carried out a longitudinal analysis to study the trend of EHR adoption and usage as well as to examine the association between the duration of EHR usage and quality of care. They compared the HEDIS quality measures of respondents to the length of time using an HER. The results show that EHR adoption increased over time. For the purposes of this study, adoption was equated to the availability of an EHR in a particular practice. This may not be the case. There can be physician who have an EHR available yet choose not to use it. The results also showed that the availability and use of EHR core functions increased. The usage of specific features was measured and presented in the paper as used most of the time, some of the time or none of the time. There was no statistical difference between EHR users and non-users. Finally, and most importantly, usage of an EHR was not related to increased quality of care.The results are valid. This paper addressed the research question it proposed from the outset. The data used shows that there is no clear connection between EHR usage and quality of care. However, it should be noted that the results are shown with adjusted values. It should be recognized that it is difficult to conduct these types of studies. There are measurement issues, one cannot prove correlation can be equated to causation and there are adjustments for variables. Initially, health care quality cannot be associated with systems. It is much easier to document negative outcomes.

There were only two references cited for relationship of EHR use and quality of care. This would seem to indicate a need for more research to be conducted in this field. However, the difficulties with conducting this type of research have already been explained. The implications for policy makers would be that there may be a need to pay more for higher quality care. This may ensure that physicians are using EHRs to their full capacity. This can be achieved through education and workflow transformation. There may need to be stronger incentives or more extensive programs to support physician office transformation. One strong argument the authors suggest is that The 2005 EHR adoption rate will need to triple in less than a decade (for MA) for 100% physician EHR usage. This represents a challenge for health care policy.


  1. Hi Daniel,
    Well written summary. Just some points to clarify, your suggestion for "future health outcome measures to use clinical guidelines", were the clinical categories (e.g., asthma care, diabetes..) not based on clinical guidelines?
    I am not entirely sure I understand the last couple of paragraphs. Your point that the results are valid and "usage of an EHR was not related to increased quality of care." Was it "quality of care" that was measured? What do you mean by "health care quality cannot be associated with systems?" I am not sure that I am following your argument that "implications for policy makers would be that there may be a need to pay more for higher quality care."
    Hope this is useful.

  2. Hey Daniel,

    Interesting paper. Some of my thoughts are:
    -I don’t quite understand where you say “It would interest researchers because it suggests that further research is required”. Could you clarify
    -I like how you mentioned the media seeing as this topic is highly politicized (at least I find this extremely interesting).
    -I think that it would be valuable to mention how difficult it is to measure quality of care. This study makes an excellent attempt, but it is still extremely difficult to conceptualize and objectively study.
    -What are your thoughts on the future of this topic. What other questions do you think should be asked as a result of the study?

  3. Good observations Daniel.
    I found the section where you discuss various stakeholders a very interesting one. The author is likely to be interested in your comments. However I found it somewhat confusing to comprehend the statement- "However, since this paper only discusses the quality aspect of care, this may not be as much of a disincentive to stop the investment in EHRs." Was wondering if you would consider clarifying it in your final report.

  4. Hi Daniel, nice work on the paper and the presentation.

    I wanted to bring up that in the section about interest for various stakeholders if you might want to include policy makers (more inclusive than ust politicians).

    Also n terms of the conclusions of this paper I take issue with their assertion that the lack of effect on HEDIS data represents a lack of impact on quality.

  5. Wanted to add one comment - Although the class may be interested in the Ontario perspective, if the paper is meant to go out to a broad range of readers, would consider leaving this focus for the classroom only and writing for the interest of the broadest possible audience.

  6. This comment has been removed by the author.

  7. Hi Daniel,
    I liked how you thought about the potential use of this report by all the stakeholders. However, I agree with others that it was a bit difficult to follow some of your points for example as Shamsa mentioned the following statement was not clear to me either: "However, since this paper only discusses the quality aspect of care, this may not be as much of a disincentive to stop the investment in EHRs." Could you provide more clarification for this statement, please.
    I also, liked your idea about providing more support for the healthcare professionals who use EMR; educating our doctors on the usage of EMR and smoothening the workflow transformation for them and their staff. I believe the doctors at least the first generation of EMR users can use all support the can form their government.
    Moreover, I think it is useful to question the authors about what they meant by "the quality of care"
    How did they define the quality of care; and may also be useful to clarify from which perspective; healthcare professionals, patients, or the healthcare system perspective? Thank you.

  8. Note that the final CATCH-IT report has been published at http://ehealth-catchit.blogspot.com/2009/10/catch-it-final-report-relationship.html