Introduction
The purpose of this report is to analyze the research presented in the paper by Ash et al; 2009(1). This paper provides an overview of the research methods and results of a 4 year research endeavor by a research group known as POET. Their work looked into the types, extent and importance of unintended adverse consequences (UAC) associated with the implementation of Computerized Provider Order Entry (CPOE) systems. However due to the large scope of the paper, a critical analysis of the research is limited by a paucity of details particularly in the reporting of results. As such research methods and results were also reviewed from 2 other papers (Ash et al; 2007(2), Campbell et al(3)).
The rationale presented in this report, for exploring this domain is related to the growing urgency(4) to implement CPOEs to address medical errors. As such it would be of significant value to understand if these solutions may be causing new forms of medical errors. Another rationale provided by Ash in a set of earlier writings pertains to how UACs may be a barrier to the adoption of CPOE and may explain the slow uptake of CPOE in US based medical facilities(5). From this second rationale we can then understand the use of Rogers diffusion of innovation theory as the theoretical framework in the qualitative work that is presented(3). Some of the value of this paper is in how it challenges presumptions about the value of IT in health care environments by exploring UACs. As such this body of research can be of value to a broad range of individuals (health care workers, IT and hospital administration, policy makers, researchers, CPOE vendors, advocacy groups and hospital associations). The value for these groups falls into 2 key areas; develop a better understanding of UACs in order to address them and as a political tool to change existing policies around CPOEs.
Objectives
As per the authors of Ash et al; 2009(1) the objectives are: To describe the activities, methods and results of a 4 year research project exploring the unintended consequences of CPOE.
In order to achieve this goal the POET group carried out their research in two separate but linked steps with the following objectives:
To identify the types of clinical unintended adverse consequences resulting from CPOE implementation(3) and develop a taxonomy of UACs. To discover the extent and importance of unintended adverse consequences related to CPOE in US hospitals(2). A possible unwritten objective may have been to provide evidence that the derived results from UAC identification through qualitative methodologies is generalizable.
Methods
The interventions being studied are CPOEs and the authors used the following definition for CPOE;
“Defined as direct entry of orders into the computer by physicians or others with the same ordering privileges.”(1)
The research methods used for the identification of types of UACs(3) was of a qualitative nature using semi-structured participant interviews, ethnography and focus groups. The setting for their work involved clinicians and administrators at 5 “excellent” US based acute care hospitals. Analysis was done by the POET group using a card sort method and a grounded theory approach.
The research method used for the identification of the extent and importance of UACs were 10-20 minute telephone surveys to collect quantitative and qualitative data. This survey was based on the taxonomy of UACs that was developed from the first phase of their work. It was pilot tested but was not tested for validity and reliability(2). The sample was all US based acute care hospitals that had implemented CPOE (n=561). Analysis utilized descriptive statistics, logistic regression and Spearman’s rho statistic to describe infusion of CPOE, compare responders vs. non responders and response rate to duration of CPOE use.
Results
From the work done to identify types of UACs(3), 324 types were found that were grouped into 9 categories. The authors concluded that this taxonomy could help identify and address UACs that arise in CPOE implementation. From the survey results were analyzed for 265/561 hospitals for a response rate of 47%. Logistic regression identified responder-non responder differences and there was no correlation (Spearman’s rho) between response rate and duration of CPOE use. The authors of this paper(2) concluded that these results verified the proposed taxonomy and that clinical decision support tools are related to many of the UACs. From the original paper (Ash et al; 2009) the authors go on to describe how an expert panel and the POET group went on to develop a series of tools that are available to help avoid or manage UACs with the implementation of a CPOE.
Limitations
From the paper identifying the types of UACs there are concerns as to whether the authors(3) truly achieved saturation from the work with the 5 excellent institutions. The reporting from both papers(1,3) was unclear as to whether data was collected till saturation was achieved. As such without further clarification this represents a significant methodological issue that would call into question the validity of the results. Further issues regarding whether the judgment sample should have included non-excellent institutions and fuller reporting around the analysis raise some concerns but are not critical methodological flaws, especially since the goal was not to provide an exhaustive list of all UACs associated with CPOE use.
The work looking at the extent and importance of UACs(2) did not seem to have any critical methodological flaws. However the single largest issue that significantly challenges the generalizability of the findings is the poor response rate. This coupled with the noted differences in characteristics between responders and non-responders further limits who these results can be applied to. The lack of survey validation is perhaps reflected in the confusing wording of questions and raises the question about whether this tool was actually measuring what it was intended to. Finally the authors raise the issue that the interviewees were mainly IT personnel and this may also have biased the result.
Discussion
Overall the reporting about the research methods in the Ash et al.; 2009 paper was difficult to follow and there was use of too many references to expand on key ideas. The taxonomy that was created is an interesting body of work that may already be a part of change management practices, as such perhaps more value could be derived from this work by highlighting UACs that are CPOE specific. Even if one were to put aside the issues raised earlier about the survey results, another issue that stands out is that the research does not provide the reader any sense of the magnitude of importance of these UACs especially in the context of other implementation issues, which was a stated objective of this research.
Questions for Authors
1. Can the authors of the Ash et al; 2006 paper elaborate on their data collecting protocols as it pertains to understanding whether saturation was achieved?
2. What if any measures were taken to improve the response rate?
3. Why did the authors not provide any further analysis of the responders who answered no and survey non responders?
4. Why did the authors not consider interviewing a broader range of individuals at the various institutions?
References
1. Ash JS, Sittig DF, Dykstra R, Campbell E, Guappone K. The unintended consequences of computerized provider order entry: findings from a mixed methods exploration. Int J Med Inform 2009;78 Suppl 1:S69-76.
2. Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007;14(4):415-23.
3. Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2006;13(5):547-56.
4. Institute of Medicine (U.S.). Committee on Quality of Health Care in America. Crossing the quality chasm : a new health system for the 21st century. Washington, D.C.: National Academy Press, 2001.
5. Ash JS, Gorman PN, Seshadri V, Hersh WR. Computerized physician order entry in U.S. hospitals: results of a 2002 survey. J Am Med Inform Assoc 2004;11(2):95-9.
Sunday, November 8, 2009
CATCH-IT Draft Report: The unintended consequences of computerized order entry: findings from a mixed methods exploration.
Web-based Weight Loss in Primary Care: A Randomized Controlled Trial
Full text: click here
Abstract
Evidence is lacking regarding effective and sustainable weight loss approaches for use in the primary care setting. We conducted a 12-week randomized controlled trial to evaluate the short-term efficacy of a web-based weight loss intervention among 101 primary care patients with obesity and hypertension. Patients had access to a comprehensive website that used a moderate-intensity weight loss approach designed specifically for web-based implementation. Patients also participated in four (two in-person and two telephonic) counselling sessions with a health coach. Intent-to-treat analysis showed greater weight loss at 3 months (-2.56 kg; 95% CI -3.60, -1.53) among intervention participants (-2.28 +/- 3.21 kg), relative to usual care (0.28 +/- 1.87 kg). Similar findings were observed among intervention completers (-3.05 kg; 95% CI -4.24, -1.85). High rates of participant retention (84%) and website utilization were observed, with the greatest weight loss found among those with a high frequency of website logins (quartile 4 vs. 1: -4.16 kg; 95% CI -1.47, -6.84). The intervention's approach promoted moderate weight loss at 12 weeks, though greater weight loss was observed among those with higher levels of website utilization. Efficacious web-based weight loss interventions can be successfully offered in the primary care setting.
Additional resources to assess study:
1. CONSORT for Non Pharmacological Treatments: click here
2. STARE-HI (Talmon J et al. International Journal of Medical Informatics
Volume 78, Issue 1, January 2009, Pages 1-9)
Saturday, November 7, 2009
CATCH-IT Draft: Clinical Decision Support capabilities of Commercially-available Clinical Information Systems
Background
Recent studies have reported that the CDS applications built in-house produce the best results. However, there is not much research done for the CDS capabilities of commercially available clinical information systems (CIS). The cited paper wishes to fill this gap in research by evaluating the CDS capabilities of 9 commercially available CCHIT certified EHR systems using a 42-element functional taxonomy. The evaluations are based on information collected from the vendors and customers of the EHR systems. The study finds that while capabilities for ‘triggers’ in CDS are well covered among the systems, many capabilities for ‘offered choices’ are not present. The results of the study are presented pseudonymously to respect privacy of the customer.
This report is based on an evaluation of the study in the CATCH-IT Journal Club. It reports the key points raised about the methodological issues of the study as a result of the CATCH-IT analysis. These issues are discussed in the following, and it is expected that consideration of this evaluation will enhance the quality of the research performed by the research community.
Methodological Issues
There are several methodological issues with the original study that can be highlighted. These methodological issues can be causes of potential concerns that may hinder the validity of the research findings. The following sections discuss the methodological issues in more detail.
Use of CCHIT Certified EHR Systems
The authors indicated about the use of CCHIT certification as a baseline for the selected systems in the study. As a result of establishing such a baseline, the authors have ensured that the selected systems meet a particular quality and are have comparable features.
An investigation of the CCHIT certification requirements has indicated that the CCHIT certification criteria are continuously evolving, with additional requirements being added each year. CCHIT uses a matrix of requirements with specific requirements relating to a system’s domain of use (such as ambulatory care and outpatient care) and the system’s aspect of use (such as for EMR storage and CDS). While CCHIT certification has been used as a baseline for the selection procedure of the systems, the authors have not discussed details about what year the selected systems were certified, and whether their certification has been renewed with the evolving CCHIT requirements. In addition, it is unclear as to what the authors have done to ensure that the features in the selected taxonomy are in alignment with the CDS –specific requirements CCHIT.
System Selection Procedure
The authors indicated in the methods that a preliminary set of CCHIT certified EHR systems was identified based on figures from Klas and HIMSS Analytics. The vendors involved with the development of these systems and the customers of these systems were then contacted, based on which a sample of 9 systems were selected for this study.
The immediate concerns that arise regarding the selection procedure is that it is unclear as to how many systems were originally selected, what was the nature of such communications (such as questions asked, and type of information requested), and what was the criteria for short listing the selected EHRs for the study. Without such details, the study cannot illustrate to the audience that the study followed an effective selection procedure in which there was no external influence, and that a specific system was included or excluded from the study due to potential bias.
Taxonomy Selection
For the purpose of determining the availability of a certain CDS capability in the selected systems, the authors selected a self-developed functional taxonomy that combined common CDS capabilities along four axes – triggers, input data elements, interventions, and offered choices. The authors mention that the taxonomy was developed based on a research at the Partners HealthCare System, by emphasizing on the fact that there was no other functional taxonomy available for use in this research.
It has been determined that the taxonomy has been developed based on the numerous clinical rules used at the Partners HealthCare System in Boston. While Partners is evidently a large healthcare system involving a blend of healthcare provider types, it must be noted that the developed taxonomy has not been validated by employing it in other healthcare organizations outside of Partners.
Due to the concerns raised by the use of an un-validated self-developed taxonomy in this research, an investigative approach has been used to determine how well the taxonomy has been received by the research community. Findings suggest that even though the taxonomy research was published in 2007, to date there are only 5 journal articles that reference that research study. There is only one article that has not been authored by any of the researchers involved with the taxonomy development. However, that article does not make any specific reference to the taxonomy or its development. As a result, research has failed to identify any neutral opinion about the developed taxonomy, raising concerns of self-boasting by the authors’ use of a self-developed taxonomy that lacks an apparent acceptance in the research community. However, this raises serious concerns about the findings of the study, since the study evaluation is wholly based on the CDS capabilities indentified in the taxonomy.
Data Collection Procedure
In this study, the vendors and customers of the 9 systems were contacted and interviewed by three of the authors. The outcomes of these interviews were used to evaluate the CDS capabilities of each system against the 42-element taxonomy. The authors reported that if there was any doubt about the availability of a particular feature in any of the systems, they contacted other customers, read product manuals, and conducted hands-on evaluation to determine availability of a feature.
The paper suggests that three of the authors were involves with the data collection procedure. The authors have not specified what kind of data collection mechanisms were used for collecting the data. Data collection procedures can be a cause of bad data for which the study is based. As a result, the researchers must demonstrate the validity of their data collection procedure. For example, it is not known whether the authors used one-on-one interviews or panel interviews for collecting the data, how many interviews were conducted with the same interviewee, were the interviews open-ended or close-ended, how many questions were involved, what was the follow-up procedure, and what did the authors do to prepare for the interviews. The audience of the research can easily raise questions about the procedures and argue about the limitations of the procedures used. A well-written report will typically avoid letting such concerns settle in the mind of its audience.
Apart from the concerns about the data collection procedure, there are concerns about the validity of the collected data. It is unclear who the researchers spoke with in each of these interviews with the vendors or customers. Not all members of the vendor organization are able to answer the same question about the availability of a particular feature. In the case of a vendor, there may be bias in the answers about the availability of a certain feature. At the same time, asking the customers about the availability of a feature may raise concerns about the knowledge of customer about the product itself. And this leads to the question as to what method did the authors use to ensure that 1) what the vendors and customers are saying are actually valid, 2) how was the collected data validated, 3) what is it that raised doubts in the researchers’ minds because of which they conducted further investigation, and 4) what determined whether a feature is actually available.
Results Interpretation
The results of the study have been presented in a tabular form for each of the axis of the taxonomy by evaluating the systems against the features in the axes. To respect the software vendors’ right to privacy, the results were pseudonymously represented by identifying each system with a number. In their evaluation, the authors used a binary-style evaluation, where the result is either yes (available) or no (unavailable). Since both in-patient and outpatient were used, the inapplicable criterion for a system was marked as N/A (not applicable). The final result was represented with a count of unavailable features by each system by each axis. In the authors’ view, the system with the least number of unavailable features is the best system.
Although the authors have mentioned this as a limitation of their study, but the binary-style evaluation does not match the way that the data for this study has been collected and used. The data collected in the study was qualitative, and it has been used to evaluate a question to yes or no. Similar to the concern about the validity of collected data, this raises serious questions about the validity of the evaluation that the authors have performed. For example, even if a feature is available, what did the authors do to evaluate how well that feature has been implemented by the software developers? How complete is the feature? How usable is it? How applicable is it for a particular setting?
The representation of the final result in this research by tallying the number of unavailable features is all but useful. The authors have failed to use key concepts of importance, usefulness, and frequency of use of an available feature. For example, a feature may be useful, but may not be frequently used. Or perhaps a feature is not frequently used but is very important for the success of a CDS application. This directly impacts the final results of the study where the authors chose the system with the least number of unavailable features as the best system. Since the scoring system used in this research is weak, it can be argued that the results are invalid.
Questions for the authors
1. What led to the linear treatment of the capabilities?
2. What was the reason behind the use of a taxonomy which is not yet well-received in the research community?
3. What were the steps taken to validate the information gathered from the vendors and customers?
4. What was the reasoning behind counting the number of unavailable features rather than available ones? Did you not want to deal with the complexity of working with N/A?
5. Why were both inpatient and outpatient systems with potentially different capabilities selected for the study?
References
1. Wright A, Sittig D F, Ash J S, Sharma S, Pang J E, and Middleton B. Clinical Decision Support capabilities of Commercially-available Clinical Information Systems. Journal of the American Medical Informatics Association 2009; 16(5): 637-644.
2. Parners Healthcare. What is Partners?. Accessed via http://www.partners.org/about/about_whatis.html. Accessed on October 20, 2009
3. Scopus. Scopus Journal Search. Accessed via http://simplelink.library.utoronto.ca/url.cfm/54186. Accessed on October 22, 2009
4. BioMed Experts. Accessed via http://www.biomedexperts.com. Accessed on October 15, 2009.
5. DMICE: People – Students. Department of Medical Informatics & Clinical Epidemiology, Oregon Health & Science University. Accessed via http://www.ohsu.edu/ohsuedu/academic/som/dmice/people/students/index.cfm. Accessed on October 20, 2009
6. Clinical and Quality Analysis, Information Systems. Clinical and Quality Analysis Staff. Accessed via http://www.partners.org/cqa/Staff.htm. Accessed on October 18, 2009.
7. Wrigh A, Goldberg H, Hongsermeier T, and Middleton B. A Description and Functional Taxonomy of Rule-Based Decision Support Content at a Large Integrated Delivery Network. Journal of the American Medical Informatics Association 2007; 14(4): 489-496.
8. CCHIT. Concise Guide to CCHIT Certification Criteria. Accessed via http://www.cchit.org/sites/all/files/ConciseGuideToCCHIT_CertificationCriteria_May_29_2009.pdf. Accessed on October 10, 2009.
9. Sittig DF, Wright A, Osheroff JA, Middleton B, Teich JM, Ash JS, Campbell E, Bates DW. Grand challenges in clinical decision support. Journal of Biomedical Informatics 2008; 41(2):387-392.
Nov 16 - Effectiveness of Active-Online, an Individually Tailored Physical Activity Intervention, in a Real-Life Setting: Randomized Controlled Trial
Full Text - Abstract Only - CATCH-IT Draft
Background: Effective interventions are needed to reduce the chronic disease epidemic. The Internet has the potential to provide large populations with individual advice at relatively low cost.
Objective: The focus of the study was the Web-based tailored physical activity intervention Active-online. The main research questions were (1) How effective is Active-online, compared to a nontailored website, in increasing self-reported and objectively measured physical activity levels in the general population when delivered in a real-life setting? (2) Do respondents recruited for the randomized study differ from spontaneous users of Active-online, and how does effectiveness differ between these groups? (3) What is the impact of frequency and duration of use of Active-online on changes in physical activity behavior?
Methods: Volunteers recruited via different media channels completed a Web-based baseline survey and were randomized to Active-online (intervention group) or a nontailored website (control group). In addition, spontaneous users were recruited directly from the Active-online website. In a subgroup of participants, physical activity was measured objectively using accelerometers. Follow-up assessments took place 6 weeks (FU1), 6 months (FU2), and 13 months (FU3) after baseline.
Results: A total of 1531 respondents completed the baseline questionnaire (intervention group n = 681, control group n = 688, spontaneous users n = 162); 133 individuals had valid accelerometer data at baseline. Mean age of the total sample was 43.7 years, and 1146 (74.9%) were women. Mixed linear models (adjusted for sex, age, BMI category, and stage of change) showed a significant increase in self-reported mean minutes spent in moderate- and vigorous-intensity activity from baseline to FU1 (coefficient = 0.14, P = .001) and to FU3 (coefficient = 0.19, P < .001) in all participants with no significant differences between groups. A significant increase in the proportion of individuals meeting the HEPA recommendations (self-reported) was observed in all participants between baseline and FU3 (OR = 1.47, P = .03), with a higher increase in spontaneous users compared to the randomized groups (interaction between FU3 and spontaneous users, OR = 2.95, P = .02). There were no increases in physical activity over time in any group for objectively measured physical activity. A significant relation was found between time spent on the tailored intervention and changes in self-reported physical activity between baseline and FU3 (coefficient = 1.13, P = .03, intervention group and spontaneous users combined). However, this association was no longer significant when adjusting for stage of change. Conclusions: In a real-life setting, Active-online was not more effective than a nontailored website in increasing physical activity levels in volunteers from the general population. Further research may investigate ways of integrating Web-based physical activity interventions in a wider context, for example, primary care or workplace health promotion.
Monday, November 2, 2009
Acceptability of a Personally Controlled Health Record in a Community-Based Setting: Implications for Policy and Design
Full-text article: http://www.jmir.org/2009/2/e14/HTML
J Med Internet Research Vol 11, No 2 (2009)
ABSTRACT
Background: Consumer-centered health information systems that address problems related to fragmented health records and disengaged and disempowered patients are needed, as are information systems that support public health monitoring and research. Personally controlled health records (PCHRs) represent one response to these needs. PCHRs are a special class of personal health records (PHRs) distinguished by the extent to which users control record access and contents. Recently launched PCHR platforms include Google Health, Microsoft’s HealthVault, and the Dossia platform, based on Indivo.
Objective: To understand the acceptability, early impacts, policy, and design requirements of PCHRs in a community-based setting.
Methods: Observational and narrative data relating to acceptability, adoption, and use of a personally controlled health record were collected and analyzed within a formative evaluation of a PCHR demonstration. Subjects were affiliates of a managed care organization run by an urban university in the northeastern United States. Data were collected using focus groups, semi-structured individual interviews, and content review of email communications. Subjects included: n = 20 administrators, clinicians, and institutional stakeholders who participated in pre-deployment group or individual interviews; n = 52 community members who participated in usability testing and/or pre-deployment piloting; and n = 250 subjects who participated in the full demonstration of which n = 81 initiated email communications to troubleshoot problems or provide feedback. All data were formatted as narrative text and coded thematically by two independent analysts using a shared rubric of a priori defined major codes. Sub-themes were identified by analysts using an iterative inductive process. Themes were reviewed within and across research activities (ie, focus group, usability testing, email content review) and triangulated to identify patterns.
Results: Low levels of familiarity with PCHRs were found as were high expectations for capabilities of nascent systems. Perceived value for PCHRs was highest around abilities to co-locate, view, update, and share health information with providers. Expectations were lowest for opportunities to participate in research. Early adopters perceived that PCHR benefits outweighed perceived risks, including those related to inadvertent or intentional information disclosure. Barriers and facilitators at institutional, interpersonal, and individual levels were identified. Endorsement of a dynamic platform model PCHR was evidenced by preferences for embedded searching, linking, and messaging capabilities in PCHRs; by high expectations for within-system tailored communications; and by expectation of linkages between self-report and clinical data.
Conclusions: Low levels of awareness/preparedness and high expectations for PCHRs exist as a potentially problematic pairing. Educational and technical assistance for lay users and providers are critical to meet challenges related to: access to PCHRs, especially among older cohorts; workflow demands and resistance to change among providers; inadequate health and technology literacy; clarification of boundaries and responsibility for ensuring accuracy and integrity of health information across distributed data systems; and understanding confidentiality and privacy risks. Continued demonstration and evaluation of PCHRs is essential to advancing their use.
Nov 9: User-designed information tools to support communication and care coordination in a trauma hospital
Full-Text · Abstract Only · Slideshow · Draft CATCH-IT
BACKGROUND: In response to inherent inadequacies in health information technologies, clinicians create their own tools for managing their information needs. Little is known about these clinician-designed information tools. With greater appreciation for why clinicians resort to these tools, health information technology designers can develop systems that better meet clinicians' needs and that can also support clinicians in design and use of their own information tools.
OBJECTIVE: To describe the design characteristics and use of a clinician-designed information tool in supporting information transfer and care coordination
DESIGN: Observations, semi-structured interviews, and photographing were used to collect data. Participants were six nurse coordinators in a high-volume trauma hospital. Content analysis was carried out and interactions with information tools were analyzed.
RESULTS: Nurse coordinators used a paper-based information tool (a nurse coordinator's clipboard) that consisted of the compilation of essential data from disparate information sources. The tool was assembled twice daily through (1) selecting and formatting key data from multiple information systems (such as the unit census and the EHR), (2) data reduction (e.g., by cutting and whitening out non-essential items from the print-outs of computerized information systems), (3) bundling (e.g., organizing pieces of information and taping them to each other), and (4) annotating (e.g., through the use of colored highlighters and shorthand symbols). It took nurse coordinators an average of 41min to assemble the clipboard. The design goals articulated by nurse coordinators to fit the tool into their tasks included (1) making information compatible with the mobile nature of their work, (2) enabling rapid information access and note-taking under time pressure, and (3) supporting rapid information processing and attention management through the effective use of layout design, shorthand symbols, and color-coding.
CONCLUSIONS: Clinicians design their own information tools based on the existing health information technologies to meet their information needs. The characteristics of these clinician-designed tools provide insights into the "realities" of how clinicians work with health information technologies. The findings suggest an often overlooked role for health information technologies: facilitating user creation of information tools that will best meet their needs.
Sunday, November 1, 2009
Abstract , Full text , Presentation, Final report
Introduction
The authors started the report by an introduction on e-consultation and the increase in the use of the Internet as a source of health information. Authors stated the rationale for doing the study as low usage of e-consultation despite the fact that e-consultation offered many benefits. The authors identified the need for this research by reviewing an up-to-date literature regarding the use of Internet in communication between physicians and patients. The benefits of e-consultation was stated as increasing access to care enabling patients to ask questions regardless of time and place; allowing for anonymous consultation regarding sensitive questions; increase in self-management support for individuals with significant medical problems; and reducing health system cost by having the capability of responding to increasing demands for care in an aging society.
The purpose of the study
The objective of the study was clearly stated as to identify factors that could increase the use of e-consultation among nonusers, who were patients with access to the Internet, but with no previous e-consultation experience.
Methodology & Data collection
To collect data an online survey was carried out among non-users, patients with access to the Internet but no prior e-consultation experience, in order to assess their barriers towards e-consultation, their demands regarding e-consultation and their motivations to use e-consultation. The authors also investigated the motivation for using two types of e-consultation, which were provided in the Netherlands:
• Direct e-consultation: consulting a GP through secured e-mail.
• Indirect e-consultation: consulting a GP through secured email with intervention of a Web-based triage system. The survey was available for the period of 11 weeks.
Participants were required to response to a 5-point likert scale from 1 representing
“strongly disagree” to 5 representing “strongly agree”.
Although the authors addressed in their paper that the online survey was pre-tested but it was not clear how the survey was developed, including whether the usability and technical functionality of the questioner had been tested. Moreover, authors did not mentioned how they dealt with possible methodological issues regarding the online survey.
The authors could have improved the use of the online survey by addressing some methodological issues that are associated with the use of online surveys for data collection (Eysenbach, 2004).
Despite the fact that online surveys have several advantages such as reaching a certain population more accessibly and achieving sample sizes that exceed mail and telephone surveys (Kraut et al., 2004), there are several limitations and biases associated with the use of online survey including the non-representativeness nature of the Internet population, and the self-selection of participants. When interpreting the results of online surveys, it is necessary to consider both who responded to the survey and who did not .Among the responses self-selection; multiple submissions, non-serious responses, and dropouts are especially problematic in web-based designs (Eysenbach, 2004). For instance, since there is a lack of control over who responses to the survey, there may be multiple responses and/or non-serious responses.
It is also possible that the responders answer and submit their questionnaire multiple times. The authors did not address how they dealt with these issues in their paper. For, example some ways that they could have handled these issues were to require participants to provide a unique identifier at the beginning of the survey or to have a couple of questions with definite answers (e.g., birth date) repeated in the survey as well as using the IP address of the client computer to identify potential duplicate entries from the same user or use cookies to indicate whether cookies were used to assign a unique user identifier to each client computer (Eysenbach, 2004).
In addition, there are also several biases regarding likert scale (central tendency bias) This is where survey respondents may avoid using extreme response categories such as strongly disagree or strongly agree; and respondents may agree with statements as they are presented for example if the statement is positive, respondents are more likely to agree with statement as presented and answer in the affirmative. And, finally, the respondents are more likely to represent themselves or their opinion in a more favorable and sociably desired way (Eysenbach, 2009).
Ethical consideration
It was not clear from the article whether the study had been approved by ERB (Ethics Review Board). Authors did not address the ethical considerations regarding the online survey. The process of informed consent and data protection were not mentioned within the paper. It was also not clear whether the patients were told about the length of time of the survey,
Sampling
Participants were 18 years and older; they were recruited through banners on frequently visited websites of 26 well-trusted patient organizations that were all member organizations of the Dutch Federation of Patients and Consumer Organizations.
One of the issues with the online surveys is that they often do not have a defined sampling frame; therefore, it is impossible to calculate the response rate for such studies (Couper, 2000).
Data analysis
Authors performed descriptive and inferential statistics to identify factors that could enhance the use of e-consultation in primary settings.
Authors appropriately collapsed “agree” and “strongly agree” into one category ; “disagree” and “strongly disagree” to another category and the “neutral” to a third category; then used barcharts to present the responses of each question in a percentage format. The authors used means of likert scales to compare patient groups on perceived berries towards e-consultation, demands regarding e-consultation, motivation to use e-consultation and motivation to use direct and indirect e-consultation. However, using mean for comparison by age, education level, medication use and frequency of GP visit may not have been suitable since mean and standard deviation are not proper summary of likert scale. Authors could have alternatively use median or mode to perform their comparison more appropriately. Also, to perform the comparisons, they could have used percentages of “agree” or “disagree”. The authors could have simplified the survey data further by combining strongly agree, agree, disagree, strongly disagree into two nominal categories, such as agree/disagree. This would have offered other analysis possibilities such as chi square test and logistic regression.
Conclusion
Authors concluded that In order to promote the use of e-consultation in primary care both GPs and non-users must be informed about the possibilities and consequences of e-consultation through tailored education and instruction. Furthermore, authors mentioned that patient profiles and their specific demands for e-consultation should be also taken into account and special attention should be paid to patients who can benefit the most from e-consultation while also facing the greatest chance of being excluded from the service (Nijland et al, 2009).
The barriers, demands and motivation towards e-consultation that were identified throughout this paper, were compatible with other studies. However, the author’s conclusion was based on the population who had access to the Internet; therefore, it may not have been representative of those who did not have access to the Internet.
Moreover, some of the statistically significant results identified throughout this study could be as result of improper use of mean technique. The authors could have used other statistical techniques such as mode, median or percentages instead of means of likert scales to present the results and perform inferential statistics. In general, the statistical technique that uses mean for comparison or assessing association have more power compare to techniques that use mode or median, Therefore, the results might have been different, if the authors had performed modes and medians.
Furthermore, the authors could have used a different data collection technique to collect data. For instance, they could have surveyed patients within the General Practitioner offices to access patients with and without access to the Internet. This sampling technique would have also eliminated several online survey limitations. Moreover, the study would have been stronger if the authors had taken a mix method approach using both quantitative and qualitative methodologies to better identify barriers, demands and motivations regarding use of e-consultation.
References:
Couper.M.P (2000) Review: Web Surveys: A Review of Issues and Approaches. Public Opinion Quarterly, 64, 464-494.
Eysenbach Gunther. Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). J Med Internet Res. 2004 Sep 29;6(3):e34. doi: 10.2196/jmir.6.3.e34. http://www.jmir.org/2004/3/e34/v6e34 [PubMed].
Eysenbach Gunther. Class Discussion. Health Policy, Management & Evaluation Department & HAD5726: Design and Evaluation in eHealth Innovation and Information Management. 26 October 2009.
Kraut, R., Olson, J., Banaji, M., Bruckman, A., Cohen, J., & Couper, M. (2004). Psychological research online: Report of board of scientific affairs' advisory group on the conduct of research on the internet. American Psychologist, 59, 105-117.
Nijland N., van Gemert-Pijnen J. E.W.C., Boer H., Steehouder M. F., and Seydel E. R.(2009). Increasing the use of e-consultation in primary care: Results of an online survey among non-users of e-consultation. International Journal of Medical Informatics 78(10), 688-703.
Posts
