“CATCH-IT Reports” are Critically Appraised Topics in Communication, Health Informatics, and Technology, discussing recently published ehealth research. We hope these reports will draw attention to important work published in journals, provide a platform for discussion around results and methodological issues in eHealth research, and help to develop a framework for evidence-based eHealth. CATCH-IT Reports arise from “journal club” - like sessions founded in February 2003 by Gunther Eysenbach.

Tuesday, November 17, 2009

Syndromic Surveillance Using Ambulatory Electronic Health Records

Hripcsak G, Soulakis ND, Li L, Morrison FP, Lai AM, Friedman C, Calman NS, Mostashari F. (2009). Journal of the American Medical Informatics Association, 16(3):354-61. Epub 2009 Mar 4.

Full Text

Objective: To assess the performance of electronic health record data for syndromic surveillance and to assess the feasibility of broadly distributed surveillance.

Design: Two systems were developed to identify influenza-like illness and gastrointestinal infectious disease in ambulatory electronic health record data from a network of community health centers. The first system used queries on structured data and was designed for this specific electronic health record. The second used natural language processing of narrative data, but its queries were developed independently from this health record. Both were compared to influenza isolates and to a verified emergency department chief complaint surveillance system.

Measurements: Lagged cross-correlation and graphs of the three time series.

Results: For influenza-like illness, both the structured and narrative data correlated well with the influenza isolates and with the emergency department data, achieving cross-correlations of 0.89 (structured) and 0.84 (narrative) for isolates and 0.93 and 0.89 for emergency department data, and having similar peaks during influenza season. For gastrointestinal infectious disease, the structured data correlated fairly well with the emergency department data (0.81) with a similar peak, but the narrative data correlated less well (0.47).

Conclusions: It is feasible to use electronic health records for syndromic surveillance. The structured data performed best but required knowledge engineering to match the health record data to the queries. The narrative data illustrated the potential performance of a broadly disseminated system and achieved mixed results.


  1. A process of manual review was described in order to create a gold standard for the Narrative Data System. My question is why this wasn't considered for the Structured Data System. There is other literature that describes validating a query by taking a subset of data (from a larger database), manually reviewing it, then validating the query against the manual review. This would have allowed them to calculate sensitivity, specificity, precision, etc.

  2. It would appear to me from this study that the authors are attempting to establish the concurrent validity of 4 tools as it compares to established validated tools in the areas of public health monitoring of ILI and GIID.

    However I am unclear about what their hypothesis is. Is it that tools that monitor EMR notes can be superior to what is currently available to monitor outbreaks? If so how are they superior, this is what I find to be very unclear to me.

    If the limitation is that ambulatory clinics and ERs and not always comparable for timings for illness presentaion then why analyze the data the way that they did? In fact what were the authors hoping to accomplish by comparing isolates, ER CC complaints and IFH data?

    Further there is not nearly enough back ground information about there ambulatory centre in terms of how it compares to other ambulatory centres to be able to generalize these findings to all EMRs in ambulatory settings.

  3. This was quite a technical paper in nature. I really don’t have too many questions about the paper – mainly because I had a difficult time comprehending some of the technical jargon. My question is, how did the authors optimized the syndrome definitions? They just say that one physician and one expert defined them. Is this valid? (I am not too sure)

  4. 1. For the comparison made in Table 1 with the WHO isolates for ILI and GIID, I am not too sure if the time frame studied were the same for this study compared to that of WHO. Assuming it was, I was not able to locate it in the article.

    2. What was the justification behind using MedLEE as opposed to other NLP systems? Did the text provide any rationale?

  5. Narrative free text are unstructured, ungrammatical, non-standardized, and fragmented making it difficult to access reliablity given that the variety of expressions is vast and specific to institutions. From the readings, the type of clinicians at the IFH are different from those at the CUMC. The patient population would therefore not be the same!!! The validity of developing the queries for the Natural Language Processing (NLP) at the Columbia University and not modifying or tailoring these for the IFH data is of concern. This may greatly influence the results of the Narrative data system.

  6. I found this study difficult to follow because I was unclear as to what the study was trying to prove. Are the authors trying to prove that EHRs as a surveillance tool are better than existing ones? Perhaps the paper would have been easier to follow had the hypothesis been more explicit.

  7. I was hoping that the researchers did some work in managing the typographical errors with the manual entry of the information when using the narrative text. Although it's not clear, to me it appears that part of the activity that the authors had done at the time when they were studying the data. So my question is, what potential impact could it have on the study results if the authors did not incorporating critical information missed with those typos?

  8. I like that this article uses a novel approach to using EHRs. (At least, novel to me). Instead of using them for primary care purposes, these tools can be harnessed for public health. This has major implications for vendors, policy makers and reserachers. What other uses could we find for these systems? This could revolutionize the way public health information and communications can be delivered to the general public. The issues of standards and interoperability can be cited as barriers, but once implemented, these systems can be used to help the population as a whole. This is yet another reason why eHealth must be pushed forward in Ontario. I liked this article.

  9. My general comments are:

    currently due to the lack of standardization of codes for reason-for- visits and the fact that each institution creates their own terminologies it is very challenging to connect institutions and share data. Also, another challenge that makes such surveillance systems not viable is the lack of a unified and standardized detection method.

    In addition, What happens when many jurisdictions across the country run syndromic surveillance simultaneously? Surveillance systems may be vulnerable to declaring false positives meaning that they may falsely detect an event that isn’t there. What should be an acceptable rate of false positives?

  10. This comment has been removed by the author.

  11. I find syndromic surveillance very interesting but the process of surveillance described by the paper seems very complex. For the structured data, if ICD9 Code 784.0 is relatively specific for influenza and that 86% of such patients were selected by the developed query, how much value does the query add?

  12. This comment has been removed by the author.

  13. Sorry Cyn, what i meant is,
    I wonder if the analysis could be done with with a query using only ICD 9 Code 784 instead of the their more comprehensive ILI query. But then including fever,cough may be needed to increase predictive value.
    Narrative searches can be very difficult, as there are so many ways to describe things and typos.

  14. Given a lack of health care standards, do the benefits of a broad surveillance system based on EHR data outweigh the costs of tailoring various local data elements?

    Are there plans to compare syndromic surveillance based on EHR data to other data sources besides ED chief complaints, or to investigate whether EHR data included with other data sources significantly improved surveillance over other data sources alone?