Skip Navigation

This Article
Right arrow Extract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrow Search for citing articles in:
ISI Web of Science (10)
Right arrowRequest Permissions
Google Scholar
Right arrow Articles by Doll, L.
Right arrow Articles by Binder, S.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Doll, L.
Right arrow Articles by Binder, S.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?

Epidemiol Rev 2003;25:51-59
Copyright © 2003 by the Johns Hopkins Bloomberg School of Public Health

METHODOLOGICAL ISSUES

Evaluation of Interventions Designed to Prevent and Control Injuries

Lynda Doll, Tom Bartenfeld and Sue Binder

From the National Center for Injury Prevention and Control, Centers for Disease Control and Prevention, Atlanta, GA.

Received for publication August 20, 2002; accepted for publication March 21, 2003.

Abbreviation: IRB, institutional review board.

Violent and unintentional injuries place a severe physical, emotional, and financial burden on US communities. Injuries affect people of all ages, from infants through older adults, and are the leading cause of death in the first four decades of life (1). In 1995, the economic cost of injuries was estimated to be over $260 billion, including the costs of health care and lost productivity (2).

Despite the large burden on US communities, injury prevention is still a relatively new area of public health. In 1992, Congress mandated the establishment of the Centers for Disease Control and Prevention’s National Center for Injury Prevention and Control to coordinate research and programmatic responses to the problem of injuries in this country (3). In partnership with other federal, state, and local organizations and universities, the Center has encouraged the use of a population-based or public health approach to injury research and programs. This approach complements engineering, mental health, and criminal justice approaches to injury prevention and control.

Evaluation of interventions is an important aspect of injury prevention. Results of evaluations have the power to change injury prevention practice. For example, bicycle helmet use increased following demonstration of the impact of helmets on brain injury (4–6). The number of programs promoting installation of smoke detectors and provision of fire injury prevention education grew once these programs were shown to be highly effective and to save money (7, 8). The number of states with laws prohibiting blood alcohol concentrations of 0.08 percent or higher increased after such laws were shown to decrease fatal motor vehicle injuries (9–11).

In this paper, our goal is to discuss the evaluation of injury prevention interventions in two contexts—intervention research (efficacy or effectiveness trials) and program evaluation. Our goal is not to present an exhaustive methodological review of evaluation strategies. The reader is referred to several excellent articles and texts that provide overviews of evaluation methods used in research and program evaluation (12–17). Rather, our intent is to review selected broad issues and challenges faced by injury researchers and practitioners as they undertake an evaluation activity. Attention to these issues is important to ensure that evaluation findings are both credible and useful.


   EVALUATION DEFINITIONS AND TYPES
TOP
 EVALUATION DEFINITIONS AND TYPES
TYPES OF EVALUATION
 ISSUES IN EVALUATION
 CHALLENGES IN EVALUATION OF...
 REFERENCES
 
Evaluation, to borrow the definition from The American Heritage Dictionary of the English Language (18), is action "to ascertain or fix the value or worth of" something, in this case injury prevention interventions. Scriven provides a well-known definition in the field of program evaluation, namely, "the process of determining the merit, worth, or value of something or the product of that process" (19, p. 139). Evaluation is ultimately about making judgments—judgments by scientists about whether an intervention is effective in reducing injury risk and judgments by practitioners about whether an intervention should be continued, modified, or eliminated from an injury prevention program. Evaluation is the overarching guide that helps to assure effective and efficient use of limited resources.

In this paper, we artificially divide the overall evaluation process into two areas: intervention research and program evaluation. "Intervention" refers to an activity or device employed with the goal of reducing or preventing injuries. Interventions can include medical activities, for example, acute care of the injured in emergency departments or rehabilitation care for patients with traumatic brain injury. In this paper, we emphasize nonmedical interventions aimed at primary prevention of injuries. These interventions can be of many kinds: behavioral (brief motivational interviews in emergency departments to reduce problem drinking and driving (20)), social (family therapy interventions for youth at risk for violent behavior and institutionalization (21)), community (multilevel intervention to reduce alcohol-related motor vehicle crashes and assaults (22)), policy (passage of 0.08 blood alcohol concentration laws (9)), environmental (traffic calming measures (23)), or technologic or biomechanical (hip pads for preventing fractures in the frail elderly (24)). We define intervention research as systematic investigations conducted to create generalizable knowledge about effective injury prevention interventions (25). Intervention research is typically aimed at evaluating new and innovative strategies (efficacy trials (20, 24, 26, 27)) or testing the application of proven interventions in a setting or population significantly different from that in which they were originally tested (effectiveness trials (21, 26, 27)). As we note below, intervention research is typically more researcher-driven and, as conditions allow, uses rigorous research methods.

Program evaluations are usually practitioner-driven, providing credible information on whether to implement, improve, continue, or expand a specific intervention program rather than generating knowledge that is generalizable to other situations. For example, Luria et al. (28) evaluated Safety City, a program designed to increase children’s safety knowledge and behaviors that was implemented in Columbus, Ohio, city schools in 1989. Evaluation findings showed that the program as it was currently being practiced was not successful in changing children’s knowledge about safety procedures for crossing streets, dialing 911, and avoiding strangers. The sponsoring agency made changes in the program based on the evaluation, including ensuring that the curriculum used age-appropriate language, providing more specific information on teacher activities, and providing a letter to parents highlighting parental strategies for encouraging child safety behavior. Because of a variety of social and economic circumstances, program evaluation activities may not attain the level of research rigor seen in intervention research.

Although the lines between intervention research and program evaluation are not always clear, in general intervention research will involve a more complex design and require more data collection. Whereas control and comparison groups are critical to research, time and financial limitations often limit the use of these groups in program evaluations. Considering the viewpoints of various stakeholders, such as program managers, staff, and clients, in developing evaluation questions is a cornerstone of program evaluation. Intervention research questions may be driven by identified gaps in knowledge and by the needs of public agencies that fund intervention research.


   TYPES OF EVALUATION
TOP
 EVALUATION DEFINITIONS AND TYPES
 TYPES OF EVALUATION
ISSUES IN EVALUATION
 CHALLENGES IN EVALUATION OF...
 REFERENCES
 
Evaluations may be designed to answer several kinds of questions related to the process, outcome, cost, or impact of interventions. In process evaluations, details are gathered on intervention implementation and quality. Outcome evaluations assess changes in proximal indicators, for example, attitudes, beliefs, and behaviors, or environmental and policy changes that are believed to be related to reductions in injuries and deaths. Evaluations of the cost or cost-effectiveness of injury-related interventions have been infrequently conducted. Such evaluations describe the effectiveness of a particular intervention in the context of the costs of implementation, the costs to study participants, and the costs associated with any side effects of the intervention (8, 29–32). Legislators are heavily influenced by cost-benefit data in setting budget priorities (33).

Impact evaluations assess changes in the frequency or severity of injuries and deaths resulting from a particular intervention. In deciding which interventions should be further studied or implemented more widely, public health practitioners and policy makers often ask for data on impact or cost-effectiveness. However, there are many circumstances in which impact data may be difficult to obtain. When the impact one would want to measure is rare (e.g., the killing of a child by a teenage parent) or the impact of the intervention might be difficult to disentangle from the impacts of other concurrent changes (e.g., changes in the economy), it may be very important to measure proximal changes (e.g., improved parent-child interaction). Potential problems with the use of proximal measures as evidence of intervention effectiveness are discussed below. The acceptance of more proximal changes may be greater in circumstances where it is considered unethical not to intervene or where failure to conduct an intervention would have an apparently negative outcome.


   ISSUES IN EVALUATION
TOP
 EVALUATION DEFINITIONS AND TYPES
 TYPES OF EVALUATION
 ISSUES IN EVALUATION
CHALLENGES IN EVALUATION OF...
 REFERENCES
 
A series of issues should be considered in all evaluations, regardless of the context in which the evaluations are conducted. The Centers for Disease Control and Prevention’s "Framework for Program Evaluation in Public Health," published in 1997, serves as one foundation for our discussion (34–36). This document describes six activities that should be considered in conducting any program evaluation in public health. Although the Framework focuses on program evaluation, many of these activities are important in conducting any type of evaluation.

Before the intervention begins

It is best to determine the evaluation design before the intervention begins. Once an intervention has started, the opportunity to collect critical baseline data or early process data may be lost if these factors were not built into the evaluation design. Furthermore, if control or comparison groups are not selected before initiation of the intervention, it may be impossible to collect data that will allow appropriate comparison. With research studies, the unit of analysis (e.g., the individual, school, or community) must be identified early to ensure that there is sufficient power (e.g., sample size) to draw conclusions from the study. In addition, developing the design before the intervention begins allows stakeholders to be engaged early in the process. This often leads to a better evaluation and enhances the likelihood that stakeholders will use the findings.

Not infrequently, program evaluations cannot be initiated until after intervention implementation has begun. Despite the loss of some opportunities, important evaluation questions can still be answered. Intervention participants can provide information on program benefits and satisfaction, and surveillance data can be used to assess possible impacts.

Engage stakeholders

The Centers for Disease Control and Prevention’s program evaluation framework identifies three groups of potential stakeholders for program evaluations: persons involved in program operations, persons who are served or affected by the program, and primary users of the evaluation (34). Within these groups may be clients, funders, policy makers, program managers, researchers, and other persons who are positively or negatively, directly or indirectly affected by the intervention.

Evaluation activities are key to building high-quality injury prevention programs. Key to conducting high-quality, useful evaluations is collaboration between the researcher and community stakeholders, particularly injury program personnel. While the level of collaboration will differ according to the purpose and scope of the evaluation, neither group should work in a vacuum, since the resulting product is often compromised. Relationships between researchers and stakeholders are often initiated when the evaluation begins. These short-term relationships can be helpful; however, it is primarily through ongoing relationships that researchers and injury program staff develop a level of trust and understanding of each other’s skills and perspectives that can greatly enhance the evaluation outcomes.

Many different models exist for involving stakeholders in program evaluations (37–42). These range from no involvement to having stakeholders involved in several aspects of the evaluation (e.g., hypothesis generation or questionnaire development) to empowerment evaluations, in which scientists serve more as coaches, with the bulk of the evaluation responsibilities being carried out by stakeholders (43). Involvement of community stakeholders may not be as relevant in some instances—for example, in the laboratory evaluation of devices such as hip pads designed to reduce injuries related to falls. However, even in this context, stakeholder involvement is necessary at some point to ensure that the device or other intervention is acceptable to manufacturers or end users.

Research or program evaluations of interventions that are conducted in communities almost always include stakeholder input at various levels. For example, in community-based research studies, community members may participate with scientists in generating study questions, devising the intervention, interviewing participants, and disseminating results. In a recent example, Evans et al. (37) from several public and private community agencies used a three-tiered collaborative model to conduct a randomized community trial of two interventions aimed at reducing early childhood violence and enhancing healthy psychosocial development (). This approach resulted in a clear structure of collaboration such that the team could balance the need for community input and support while also completing study activities in a timely fashion. Maintenance of intervention activities is more likely to occur after completion of the study if community members are engaged in the research (44, 45).

Specify the intervention

The components of the intervention to be evaluated should be clearly specified, as should the relation of the intervention to risk and protective factors. While this seems like an obvious step, it is surprising how frequently an intervention is pulled "off the shelf" without consideration for its relevance to the desired outcomes. If there is a gap between the known risk and protective factors and the intervention, the intervention is unlikely to be successful. Logic models can be particularly useful for specifying the intervention and selecting outcome measures (34, 46). Logic models describe the relations among risk and protective factors, intervention components, and desired outcomes. They describe both direct and indirect relations, including moderators and mediators, based on data from a variety of sources such as epidemiologic or other research and theory (47–49).

Once the intervention has been specified, the consistency of implementation becomes important. Training and written materials should be provided to staff. Staff should not be encouraged to "make up" their own interventions. Although interventions should be tailored for the gender, age, or culture of the population of interest (27), standardization helps ensure that staff deliver intervention elements or messages that are critical to success. Inconsistencies in delivering interventions make it difficult to find an intervention effect or attribute an effect to a particular intervention.

Design the evaluation

Once the evaluation questions have been determined and the intervention described, the next steps are to select the evaluation design and the measurements to be made. During efficacy trials, researchers are likely to conduct controlled trials that maximize internal validity through random assignment of individuals, schools, or communities. The level of randomization must match the proposed intervention and the proposed unit of analysis. For example, if the intervention targets an entire community, participating communities should be randomized to the experimental or control condition, with analyses evaluating differences between communities as a whole. Quasi-experimental designs are common in effectiveness trials and program evaluations and are used when randomization is not feasible and baseline differences between groups can be controlled in other ways (e.g., statistical controls). An evaluation of a community-wide bicycle helmet campaign (50) and an evaluation of changes in bicycle helmet laws and an educational campaign (51) are two examples of evaluations using quasi-experimental designs. In both studies, interventions were ongoing, and thus randomization of communities was not possible.

Given some of the logistical and ethical concerns about experimental studies and some quasi-experimental techniques, alternative approaches to evaluation have been developed. For example, Cook and Campbell (14) describe the nonequivalent, dependent-variables design, in which the results of an intervention are predicted on the basis of intervention theory and the underlying logic model. If the observed evaluation data resemble the data expected on the basis of the proposed logic model and theory, and the data are inconsistent with expected results, assuming that the most likely alternative logic models and hypotheses were operative, then causal inferences can be made. This technique is likely to have more credibility for gross comparisons than for subtle ones (52).

While most evaluation activities rely on quantitative assessment to measure effects, qualitative methods, such as individual and focus group interviews (53, 54) and case studies (55), may be useful for planning interventions, conducting process evaluations, or interpreting evaluation results (16, 56). For example, in a study comparing peer-led unintentional injury prevention programs for high-risk adolescents with programs led by health care professionals, quantitative measures showed that neither intervention resulted in changes in knowledge, attitudes, or behavioral intentions in comparison with a control group. However, open-ended questions also asked on the questionnaire provided insights into possible effects of the interventions, such as increased reflection before acting and awareness of the possible consequences of one’s behavior (57). Another qualitative study provided important information on battered women’s perspectives on the effects of their partners’ participation in batterer interventions on the partners’ behavior as well as their own (58). There are excellent software packages available for coding and analyzing qualitative data (59).

There are instances in which interventions are very difficult to evaluate. For example, in the case of multilevel interventions, assessing the independent effects of the various components is difficult without a very expensive and complex multifactorial evaluation design (60). In effectiveness research, there is a tendency to evaluate multilevel interventions as a package, especially in those cases where many of the individual components have already been tested in efficacy trials or where it is believed that intervention components may be synergistic (22).

In some cases, data collected for other purposes, such as medical record data or surveillance data, can be helpful in program evaluation. This is particularly true for evaluations of policy changes, where data from several time periods before policy implementation may show that the prevalence of the outcome of interest remained steady for many months or years and then dropped or rose shortly after implementation. An evaluation of the impact of child-resistant drug packaging (61) illustrates the usefulness of both multiple assessments and active or passive surveillance systems in monitoring such trends. In this case, trend data from poison control centers documented steep declines in unintentional poisoning deaths and ingestions by children after new packaging policies were put into place (61). A review of data on annual mortality rates associated with the unintentional ingestion of oral prescription drugs for children under age 5 years also showed that mortality rates declined 45 percent from the period before policy implementation through 1992 (62). Two evaluations of suicide prevention programs (63, 64) have also used existing data to assess program effectiveness. Air Force surveillance systems, including annual health surveys, showed a decline in suicide rates among Air Force personnel following initiation of community-wide, multistrategy suicide prevention programs (63). Similarly, Zenere and Lazarus (64) used Miami school data on student suicides for the years 1980–1994 to evaluate suicide prevention programs that were in effect during the years 1989–1990 and 1993–1994. While it is not possible to make a definitive judgment about cause and effect in these instances, such data certainly support the argument that a relation exists.

Control and comparison groups

Evaluation designs may incorporate one of several different types of control or comparison groups. Independent control or comparison groups are almost always used in efficacy and effectiveness trials. The use of an independent control group permits comparison of one group that has received an intervention with another group that has, in theory, received no intervention. When comparison rather than control groups are used, it is possible to contrast the effects of two or more interventions.

As an alternative to an independent control or comparison group, study participants may serve as their own controls through the use of a pre-/post-design. For example, research has shown that a retrospective pretest design can provide accurate information. This design involves interviewing study participants after the intervention about their current (postintervention) knowledge, attitudes, or behaviors and also their knowledge, attitudes, or behaviors referenced to a specific point in time before the intervention (65).

In program evaluations, independent control or comparison groups are used less frequently, often because of time, resource, and other constraints. Wright et al. (66) described several methodological challenges they faced in evaluating Think First, a widely used, school-based head and spinal cord injury prevention program. For example, many school principals felt that allowing their school to serve as a control school was a waste of students’ and teachers’ time. Other school staff were concerned about the length of the questionnaire. In response to these concerns, these researchers shortened the survey instrument considerably and had to be content with only one control school.

Choice of measurements

Measures used to assess intervention effects depend on the project’s evaluation questions and the resources available for collecting data. The typical evaluation project collects outcome and impact data before and after the intervention is implemented. Intermediate assessments may provide important additional information on when, how much, and in what domains change occurs. Caution should always be taken in interpreting the relation between proximal indicators and impact. For example, Stevens et al. (67, 68) used a randomized controlled trial design to evaluate the effect of a multilevel intervention (home visits, home hazard assessment, and installation of safety devices) on reductions in the number of environmental hazards in the home and on the frequency of falls among older adults. While the intervention resulted in small but significant reductions in four of the five most prevalent hazards, the incidence of falls was not reduced (67, 68). In other cases, intermediate measures may underestimate the effectiveness of interventions that will ultimately be shown to be effective (69–71). This may occur when the effects of an intervention are cumulative or when community-level interventions aim to change social norms.

The quality of data collected during an evaluation is critical to its outcome. In the case of cognitive, behavioral, or laboratory data, it is often best to identify measures that have been used previously and for which there are psychometric data (e.g., reliability data) or laboratory standards. For other data, such as rates of injury or death, care must be taken to compare data with similar denominators. As interest grows in social and environmental interventions, measurement of constructs related to these interventions, such as social norms, social capital, community cohesiveness, walkability, green space, and mixed-use and high-density housing, are receiving greater attention. We refer the reader to several texts and articles (72–76) for examples of constructs and measures.

Number of assessments

Another critical design issue is the question of the number of assessments to conduct before, during, and after the intervention is implemented. Perhaps the most common design has been the single pre-/post-design. However, multiple outcome assessments made before, during, and after intervention implementation can provide data on three important issues: 1) secular trends in policy implementation or environmental changes; 2) intermediate effects of the intervention; and 3) maintenance of effects after the intervention is over. Multiple pre- and postintervention assessments can provide useful information for interpreting results in the absence of an independent control group. Unfortunately, the limited funding available for evaluations usually results in a tradeoff between the frequency of data collection and the depth and amount of data collected.

Maintenance of intervention effects beyond the immediate implementation period is a key concern for public health practitioners. Clearly, those interventions that promise long-term effects are preferred over others, if only for the cost implications. Assessments made over time provide information on whether the intervention has a long-term effect or whether the effect is diluted over time. In an evaluation of a fire prevention program entailing smoke-alarm giveaways and education, on-site fire alarm inspections were conducted 3, 12, and 48 months after the program began. Proper installation and functioning of the alarms declined over time, though no fire-related injuries occurred in the targeted areas (7). In a series of studies evaluating the impact of prenatal and infancy home visitations conducted by nurses over a 2-year period, intervention assessments were conducted periodically during pregnancy, during the first 4 years of the child’s life, and at age 15 years. The intervention significantly reduced harm to the children of low-income, unmarried teenagers, including fewer cases of child abuse and neglect and fewer injuries. At age 15 years, intervention children had fewer arrests, fewer convictions and violations of probation, and fewer days of alcohol consumption (26).

Ethical considerations

There are a number of important ethical considerations in conducting intervention evaluations in research or programs. Researchers conducting studies funded by the US Department of Health and Human Services are familiar with the regulations regarding the protection of human subjects in Title 45, Part 46 of the Code of Federal Regulations (25). They mandate that researchers consider the potential costs (i.e., risks or discomforts) and benefits of individuals’ participation in their research and that these be explicitly described in the informed consent document. All research, that is, systematic investigations conducted to create generalizable knowledge, must receive approval from an institutional review board (IRB). All universities and most larger health and social service organizations have IRBs that approve protocols for research studies.

In contrast to research, program evaluations that are conducted to improve an existing intervention program are considered public health practice and therefore do not usually require IRB approval (25). Local IRBs differ on whether they require approval for program evaluations, and they should be consulted before the evaluation activity begins. However, regardless of IRB considerations, practitioners conducting program evaluations should use consent forms and other strategies for the protection of human subjects to ensure that clients understand the risks and benefits of the activities and are not harmed by participation.

Another issue that must be considered in evaluating interventions is the type of intervention to be provided to control group members. Ethical guidelines suggest that control group members should not be denied something of potential or known benefit, that is, they should not be denied the best available intervention in their community (77). In a recent evaluation of a community-level intervention targeting sexual risk behaviors, consulting ethicists suggested that all persons participating in assessments in control communities should be offered the opportunity to attend small-group, cognitive behavioral interventions, which were the best proven intervention available in the local communities at that time (78). In research studies, the IRB overseeing the conduct of research at a particular university or organization can provide consultation on the ethics of interventions to be provided to control group members. In the case of program evaluations, biomedical ethicists at universities are available as consultants, if such dilemmas arise.

A third ethical issue is the extent to which injury researchers or practitioners are obligated to provide the experimental intervention to the control group, if that intervention is found to be effective. In clinical trials, once a medical procedure or drug has been found effective, control group members are offered the treatment (77). In a recent evaluation of an intervention to increase bicycle helmet use among low-income children, schools were randomized to the experimental condition or the control condition. Evaluation results showed that the intervention had been successful, and control group schools were offered the option of presenting the helmet program to their students (79). Unfortunately, many research grants provide insufficient funds to support the dissemination of intervention activities in schools or communities. When community stakeholders have been engaged in the intervention research or program evaluation, local agencies or health departments may be motivated to add the effective intervention to their repertoire once the research is complete (44, 45).

Unintended outcomes

During intervention implementation and evaluation, it is not uncommon to have multiple outcomes, including some that are unintended or unexpected. For example, in their review of youth violence studies, Dishion et al. (80) found that, contrary to the positive contributions of peers in childhood, delivering interventions in adolescent peer groups can increase adolescent problem behaviors (e.g., smoking and delinquent behaviors). However, implementation can also yield added benefits. Williams et al. (81) found that enhanced enforcement of seat-belt laws resulted in increased arrests for other crimes. Evaluators must take care not to so focus their energies on a few stated outcomes that they fail to identify other beneficial and undesired outcomes or to determine why programs do or do not work (82, 83).

The US Task Force on Community Preventive Services investigates positive and negative side effects in its assessment of the effectiveness of community-based interventions. Overall, the Task Force has found that the identification and assessment of side effects is inconsistent in the literature. For example, extensive literature reviews of interventions encouraging the use of safety belts (84) and interventions for reducing alcohol-impaired driving (9) yielded inconsistent evidence of side effects resulting from the interventions. It is unclear whether the problem is an artifact of lack of reporting or inadequate identification and assessment of such outcomes. Working with various stakeholders can be particularly helpful in identifying and assessing intervention outcomes, especially those that are unintended. Focus groups, surveys, or informal interviews of study participants can also provide input on the benefits and problems associated with their participation in the intervention.

Disseminate evaluation findings

Findings from an intervention research project or program evaluation must be shared with stakeholders in a timely manner. Discussions with stakeholders often improve interpretation of results and help initiate discussions about maintaining interventions found to be effective in the school, community, or other venue.

Researchers and practitioners also have an obligation to do what they can to encourage the use of their data in public health programs (85). Researchers typically (but not always) publish their evaluation results in peer-reviewed journals. This is particularly the case with studies in which there are positive outcomes. However, researchers and practitioners also have an obligation to make null or negative findings available in order to build the science of injury prevention and control (28, 66, 68, 86). Similarly, they have a responsibility to publish their results in venues where practitioners will have an opportunity to learn how to implement the intervention. Editors of peer-reviewed journals often discourage authors from including details on the intervention, target populations, and leadership and other information needed to conduct the intervention. Venues for publication of these types of data include training manuals, newsletters, and websites. Only with these details can a practitioner try to reproduce or use an intervention shown to be effective.

Recently, several groups conducting systematic review processes, such as the Cochrane Collaboration (87) and the creators of the Guide to Community Preventive Services (88), have synthesized information about the efficacy or effectiveness of injury interventions. These processes allow more robust recommendations than can often be made on the basis of individual studies. For example, recent evaluations of the Community Guide have concluded that strong evidence exists favoring a variety of means of protecting motor vehicle occupants (89), the use of home visitation programs to prevent child abuse (90), and the use of tenant-based rental assistance programs to prevent youth crime (76).


   CHALLENGES IN EVALUATION OF INTERVENTIONS FOR PREVENTING INJURIES
TOP
 EVALUATION DEFINITIONS AND TYPES
 TYPES OF EVALUATION
 ISSUES IN EVALUATION
 CHALLENGES IN EVALUATION OF...
REFERENCES
 
As is true in all public health endeavors, evaluations of injury-related prevention interventions face many challenges. These include issues related to scarce resources, the relation of injury control efforts to interventions targeting other public health problems, and changing social norms and other societal issues that affect rates of injury.

Resources

Limited availability of financial resources is often the reason why evaluations are not done or are done minimally. Public health responses to the injury problem can be viewed as underfunded when they are examined in the context of the size of the problem (1). Increasing the funds available for public health actions designed to prevent and control injuries would result in an increase in resources for evaluation, which, in turn, could result in increased funding when policy makers learned of cost-beneficial programs.

Interventions with multiple outcomes

As we noted above, injury interventions may have an impact on other health and societal outcomes besides rates of injury or violence. When appropriate, injury prevention studies should include noninjury outcomes. For example, exercise programs such as tai chi that are part of multifaceted fall prevention programs for the elderly may prove to be at least as beneficial to cardiovascular and mental health endpoints as to injury outcomes.

Secular trends

The impact of societal changes may increase as the length of evaluations increases. For example, large decreases in firearm-related deaths occurred throughout the United States between 1993 and 1998 and affected both suicide and homicide rates (91). Such societal trends may make it difficult to measure the impact of an intervention, especially if that impact is relatively small. Growing attention to social determinants of health may, over time, permit better measurement and statistical control of these influences.

Implementing interventions without evidence of effectiveness

In the field of public health, the importance of scientific evidence as a basis for choosing interventions to implement has increasingly been emphasized. However, there continue to be a large number of injury interventions that are used but have never been evaluated. The program "Scared Straight" was devised to deter future criminal activity by organizing visits to prisons by children engaging in or at risk for criminal behavior. The intervention was highly regarded for years, until an evaluation showed that it was having an effect opposite of that desired (92). Examples of widely used interventions that have not yet been adequately evaluated include suicide hotlines, batterer intervention programs, parental training programs designed to reduce unintentional and violent injuries to young children, and emergency medical treatment systems. Public health professionals vary greatly in whether they recommend widespread use of interventions that have not been rigorously evaluated but have face validity or fit with logic models.

Randomized controlled trials as the "gold standard"

Above, we discussed alternatives to the randomized controlled trial. However, there are a number of examples of recommendations that had to be reversed when results from randomized controlled trials contradicted results obtained from observational and other study designs. The recent reversal of the recommendation that hormone replacement therapy be used to prevent cardiovascular disease in postmenopausal women illustrates this phenomenon (93).

Much recent debate has centered around the level and type of evidence needed to determine the effectiveness of an intervention (94). Some experts, particularly in the developing world, have noted the western cultural bias found in the traditional randomized controlled trial and program evaluation approaches utilized in the United States (95, 96) and have called for the expansion of acceptable designs and types of data used in evaluations.

As we noted above, a growing number of injury intervention evaluations do not easily lend themselves to randomized controlled trial approaches. New approaches to evaluation are needed, as are methods that allow us to use existing tools better (e.g., better ways to control for confounding factors in nonrandomized studies). "Rules of evidence" should be developed for interpreting results from community and multilevel intervention studies, paralleling what has been recently used in the Guide to Community Preventive Services (97).

Conclusion

Despite the difficulty of conducting quality evaluations of health interventions, we as public health professionals have a responsibility to do our best to ensure that those interventions we espouse make a difference. We have a responsibility to conduct the best evaluations we can, taking into account the kinds of issues raised in this article.


   FOOTNOTES
 
Reprint requests to Dr. Lynda Doll, National Center for Injury Prevention and Control, Centers for Disease Control and Prevention, 4770 Buford Highway, K-02, Atlanta, GA 30341 (e-mail: lsd1{at}cdc.gov).


   REFERENCES
TOP
 EVALUATION DEFINITIONS AND TYPES
 TYPES OF EVALUATION
 ISSUES IN EVALUATION
 CHALLENGES IN EVALUATION OF...
 REFERENCES
 

  1. Bonnie RJ, Fulco CE, Liverman CT, eds. Reducing the burden of injury: advancing prevention and treatment. Washington, DC: National Academy Press, 1999.
  2. National Center for Injury Prevention and Control, Centers for Disease Control and Prevention. Web-based Injury Statistics Query and Reporting System. Atlanta, GA: National Center for Injury Prevention and Control, 2001. (World Wide Web URL: www.cdc.gov/ncipc/wisqars/). (Accessed August 15, 2002).
  3. Web-based Statistics Query and Reporting System (WISQARS). Fed Reg 1992;57(129):29732.
  4. Thompson RS, Rivara FP, Thompson DC. A case-control study of the effectiveness of bicycle safety helmets. N Engl J Med 1989;320:1361–7.[Abstract]
  5. Thompson DC, Rivara FP, Thompson RS. Effectiveness of bicycle safety helmets in preventing head injuries: a case-control study. JAMA 1996;276:1968–73.[Abstract/Free Full Text]
  6. Rivara FP, Thompson DC, Thompson RS, et al. The Seattle children’s bicycle helmet campaign: changes in helmet use and head injury admissions. Pediatrics 1994;93:567–9.[Abstract/Free Full Text]
  7. Mallonee S, Istre GR, Rosenberg M, et al. Surveillance and prevention of residential-fire injuries. N Engl J Med 1996;335:27–31.[Abstract/Free Full Text]
  8. Haddix AC, Mallonee S, Waxweiler R, et al. Cost effectiveness analysis of a smoke alarm giveaway program in Oklahoma City, Oklahoma. Inj Prev 2001;7:276–81.[Abstract/Free Full Text]
  9. Shults RA, Elder RW, Sleet DA, et al. Reviews of evidence regarding interventions to reduce alcohol-impaired driving. Am J Prev Med 2001;21(suppl):66–88.[Web of Science][Medline]
  10. Shults RA, Elder RW, Sleet DA, et al. The Guide to Community Preventive Services: systematic reviews and evidence-based recommendations for interventions to reduce motor vehicle occupant injuries. Presented at the SafeUSA Conference, Atlanta, Georgia, December 2001.
  11. Elder RW, Shults RA, Sleet DA. Do .08 percent blood alcohol concentration laws save lives? Washington Post 2001;Nov 26(suppl).
  12. Chen H. Theory-driven evaluations. Newbury Park, CA: Sage Publications, 1990.
  13. Rootman I, Goodstadt M, Hyndman B, et al. Evaluation in health promotion: principles and perspectives. Copenhagen, Denmark: World Health Organization, 2001.
  14. Cook TD, Campbell DT. Quasi-experimentation: design and analysis issues for field settings. Chicago, IL: Rand McNally and Company, 1979.
  15. Rivara FP, Cummings P, Koepsell TD. Injury control: a guide to research and program evaluation. New York, NY: Cambridge University Press, 2001.
  16. Thompson RS, Sacks JJ. Evaluating an injury intervention or program. In: Rivara FP, Cummings P, Koepsell TD, eds. Injury control: a guide to research and program evaluation. New York, NY: Cambridge University Press, 2001:196–216.
  17. Bryant KJ, Windle M, West SG. The science of prevention. Washington, DC: American Psychological Association, 1997.
  18. Soukhanov AH, ed. The American heritage dictionary of the English language. 3rd ed. Boston, MA: Houghton-Mifflin Company, 1996.
  19. Scriven M. Evaluation thesaurus. 4th ed. Thousand Oaks, CA: Sage Publications, 1991.
  20. Monti PM, Colby SM, Barnett NP, et al. Brief intervention for harm reduction with alcohol-positive older adolescents in a hospital emergency department. J Consult Clin Psychol 1999;67:989–94.[CrossRef][Web of Science][Medline]
  21. Institute of Behavioral Science, University of Colorado. Blueprints for violence prevention: functional family therapy. Boulder, CO: University of Colorado, 1998.
  22. Holder HD, Gruenewald PJ, Ponicki WR, et al. Effect of community-based interventions on high-risk drinking and alcohol-related injuries. JAMA 2000;284:2341–7.[Abstract/Free Full Text]
  23. Van Houten R, Malefant L. The influence of signs prompting motorists to yield before marked crosswalks on motor vehicle-pedestrian conflicts at crosswalks with flashing amber. Accid Anal Prev 1992;24:217–25.[CrossRef][Web of Science][Medline]
  24. Kannus P, Parkkari J, Niemi S, et al. Prevention of hip fracture in elderly people with use of a hip protector. N Engl J Med 2000;343:1506–13.[Abstract/Free Full Text]
  25. US Department of Health and Human Services. Protection of human research subjects. Title 45, Code of Federal Regulations, Part 46. Washington, DC: US Department of Health and Human Services, 1996.
  26. Institute of Behavioral Science, University of Colorado. Blueprints for violence prevention: prenatal and infancy home visitation by nurses. Boulder, CO: University of Colorado, 1998.
  27. Istre GR, McCoy MA, Womack KN, et al. Increasing the use of child restraints in motor vehicles in a Hispanic neighborhood. Am J Public Health 2002;92:1096–9.[Abstract/Free Full Text]
  28. Luria JW, Smith GA, Chapman JI. An evaluation of a safety education program for kindergarten and elementary school children. Arch Pediatr Adolesc Med 2000;154:227–31.[Abstract/Free Full Text]
  29. Haddix AC, Teutsch SM, Shaffer PA, et al. Prevention effectiveness. New York, NY: Oxford University Press, 1996.
  30. Rice DP, MacKenzie EJ. Cost of injury in the United States: a report to Congress. San Francisco, CA: Institute for Health and Aging, University of California, and Injury Prevention Center, The Johns Hopkins University, 1989.
  31. Greenwood PW, Model KE, Rydell CP, et al. Diverting children from a life of crime: measuring costs and benefits. Santa Monica, CA: RAND Corporation, 1996.
  32. Jones AS. The cost of batterer programs. J Interpers Violence 2000;15:566–86.[Abstract/Free Full Text]
  33. Research!America. Prevention Research Initiative National Poll. Alexandria, VA: Research!America, 2000.
  34. Framework for program evaluation in public health. MMWR Morb Mortal Wkly Rep 1999;48:1–40.[Medline]
  35. Thompson NJ, McClintock HO. Demonstrating your program’s worth. Atlanta, GA: National Center for Injury Prevention and Control, Centers for Disease Control and Prevention, 1998.
  36. Baker QE, Davis DA, Gallerani R, et al. An evaluation framework for community health programs. Durham, NC: The Center for the Advancement of Community Based Public Health, 2000.
  37. Evans GD, Rey J, Hemphill MM. Academic-community collaboration: an ecology for early childhood violence prevention. Am J Prev Med 2001;20(suppl):22–30.[CrossRef][Web of Science][Medline]
  38. Wandersman A. Community mobilization for prevention and health promotion can work. In: Schneiderman N, Speers M, Silva JM, et al, eds. Integrating behavioral and social sciences with public health. Washington, DC: American Psychological Association, 2001:231–47.
  39. Seifer SD, Connors KM. Community-campus partnerships for health: a guide for developing community-responsive models in health professions education. San Francisco, CA: UCSF Center for the Health Professions, University of California, San Francisco, 1997.
  40. Green L, Daniel M, Novick L. Partnerships and coalitions for community-based research. Public Health Rep 2001;116(suppl):20–31.[CrossRef][Web of Science][Medline]
  41. Doll LS, Dino G, Deutsch C, et al. Linking research and practice: two academic/public health collaborations that are working. Health Promot Pract 2001;2:295–300.[CrossRef]
  42. Sanstad KH, Stall R, Doll LS. Collaborative community research: partnerships between research and practice. Health Educ Behav 1999;26:165–283.[Abstract/Free Full Text]
  43. Fetterman DM. Steps of empowerment evaluation: from California to Cape Town. Eval Program Plann 1994;17:305–13.[CrossRef][Web of Science]
  44. Littleton MA, Cornell CE, Dignan M, et al. Lessons learned from the Uniontown Community Health Project. Am J Health Behav 2002;26:34–42.[Web of Science][Medline]
  45. Manfredi C, Crittenden K, Cho YI, et al. Maintenance of a smoking cessation program in public health clinics beyond the experimental evaluation period. Public Health Rep 2001;116(suppl):120–35.[CrossRef][Web of Science][Medline]
  46. Bartholomew LK, Parcel GS, Kok G, et al. Intervention mapping: designing theory- and evidence-based health promotion programs. Mountain View, CA: Mayfield Publications, 2001.
  47. Glanz K, Lewis FM, Rimer B, eds. Health behavior and health education: theory, research and practice. San Francisco, CA: Jossey-Bass, Inc, 1991.
  48. Gielen AC, Girasek DC. Integrating perspectives on the prevention of unintentional injuries. In: Schneiderman N, Speers M, Silva JM, et al, eds. Integrating behavioral and social sciences with public health. Washington, DC: American Psychological Association, 2001:203–27.
  49. Weinstein ND. Testing four competing theories of health protective behavior. Health Psychol 1993;12:325–33.
  50. DiGuiseppi CG, Rivara FP, Koepsell TD. Bicycle helmet use by children: evaluation of a community-wide helmet campaign. JAMA 1989;262:2256–61.[Abstract/Free Full Text]
  51. Dannenberg AL, Gielen AC, Beilenson PL, et al. Bicycle helmet laws and educational campaigns: an evaluation of strategies to increase children’s helmet use. Am J Public Health 1993;83:667–74.[Abstract/Free Full Text]
  52. Yin RK. The abridged version of case study research: design and method. In: Bickman L, Rog DJ, eds. Handbook of applied social research methods. Thousand Oaks, CA: Sage Publications, 1998:229–59.
  53. Bernard HR. Research methods in anthropology: qualitative and quantitative approaches. 3rd ed. Walnut Creek, CA: Altamira Press, 2001.
  54. Miles MB, Huberman AM. Qualitative data analysis: an expanded sourcebook. 2nd ed. Thousand Oaks, CA: Sage Publications, 1994.
  55. Yin RK, Campbell DT. Case study research: design and methods. 2nd ed. Thousand Oaks, CA: Sage Publications, 1994.
  56. Grossman DC, Rhodes L. Qualitative methods in injury research. In: Rivera FP, Cummings P, Koepsell TD, eds. Injury control: a guide to research and program evaluation. New York, NY: Cambridge University Press, 2001:104–15.
  57. Tenn L, Dewis ME. An evaluation of a Canadian peer-driven injury prevention programme for high-risk adolescents. J Adv Nurs 1996;23:329–37.[CrossRef][Web of Science][Medline]
  58. Gregory C, Erez E. The effects of batterer intervention programs. Violence Against Women 2002;8:206–32.[Abstract/Free Full Text]
  59. Weitzman EA, Miles MB. Computer programs for qualitative data analysis: a software sourcebook. Thousand Oaks, CA: Sage Publications, 1995.
  60. Ytterstad B, Wasmuth HH. The Harstad Injury Prevention Study: evaluation of hospital-based injury recording and community-based intervention for traffic injury prevention. Accid Anal Prev 1995;27:111–23.[CrossRef][Web of Science][Medline]
  61. Walton WW. An evaluation of the Poison Prevention Packaging Act. Pediatrics 1982;69:363–70.[Abstract/Free Full Text]
  62. Rodgers GB. The safety effects of child-resistant packaging for oral prescription drugs: two decades of experience. JAMA 1996;275:1661–5.[Abstract/Free Full Text]
  63. Suicide prevention among active duty Air Force personnel—United States, 1990–1999. MMWR Morb Mortal Wkly Rep 1999;48:1053–7.[Medline]
  64. Zenere FJ, Lazarus PJ. The decline of youth suicidal behavior in an urban, multicultural public school system following the introduction of a suicide prevention and intervention program. Suicide Life Threat Behav 1997;27:387–403.[Web of Science][Medline]
  65. Pratt CC, McGuigan WM, Katzev AR. Measuring program outcomes: using retrospective pretest methodology. Am J Eval 2000;21:341–9.
  66. Wright M, Rivara FP, Ferse D. Evaluation of the Think First head and spinal cord injury prevention program. Inj Prev 1995;1:81–5.[Abstract/Free Full Text]
  67. Stevens M, Holman CD, Bennett N. Preventing falls in older people: impact of an intervention to reduce environmental hazards in the home. J Am Geriatr Soc 2001;49:1442–7.[CrossRef][Web of Science][Medline]
  68. Stevens M, Holman CD, Bennett N, et al. Preventing falls in older people: outcome evaluation of a randomized controlled trial. J Am Geriatr Soc 2001;49:1448–55.[CrossRef][Web of Science][Medline]
  69. Wolf SL, Sattin RW, Kuttner M, et al. Intense tai chi exercise training and fall occurrences in older, transitionally frail adults: a randomized clinical trial. Geriatrics (in press).
  70. The CDC AIDS Community Demonstration Projects Research Group. Community-level HIV intervention in 5 cities: final outcome data from the CDC AIDS Community Demonstration Projects. Am J Public Health 1999;89:336–45.[Abstract/Free Full Text]
  71. McAlister A, Johnson W, Guenther-Grey C, et al. Behavioral journalism for HIV prevention: community newsletters influence risk-related attitudes and behavior. Journ Mass Commun Q 2000;77:143–59.
  72. Dahlberg LL, Toal SB, Behrens CB. Measuring violence-related attitudes, beliefs, and behaviors among youths: a compendium of assessment tools. Atlanta, GA: National Center for Injury Prevention and Control, Centers for Disease Control and Prevention, 1998.
  73. Berkman LF, Kawachi I. Social epidemiology. New York, NY: Oxford University Press, 2000.
  74. Laflamme L. Explaining socio-economic differences in injury risks. Inj Control Saf Promot 2001;8:149–53.
  75. Diez Roux AV. Invited commentary: places, people, and health. Am J Epidemiol 2002;155:516–19.[Free Full Text]
  76. Anderson LM, Shinn C, St. Charles J, et al. Community interventions to promote healthy social environments: early childhood development and family housing. A report on recommendations of the Task Force on Community Preventive Services. MMWR Recomm Rep 2002;51(no. RR-1):1–8.[Medline]
  77. World Medical Association. World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA 2000;284:3043–5.[Free Full Text]
  78. National Center for HIV, STD, and TB Prevention. Consultation on ethical considerations in choosing a comparison group for a community-level behavioral intervention efficacy trial. Atlanta, GA: National Center for HIV, STD, and TB Prevention, Centers for Disease Control and Prevention, 1998.
  79. Hendrickson SG, Becker H. Impact of a theory-based intervention to increase bicycle helmet use in low income children. Inj Prev 1998;4:126–31.[Abstract/Free Full Text]
  80. Dishion TJ, McCord J, Poulin F. When interventions harm: peer groups and problem behavior. Am Psychol 1999;54:755–64.[CrossRef][Medline]
  81. Williams AF, Reinfurt D, Wells JW. Increasing seat belt use in North Carolina. J Saf Res 1996;27:33–41.[CrossRef]
  82. Sherrill S. Identifying and measuring unintended outcomes. Eval Prog Plann 1984;7:27–34.
  83. Posavac EJ, Carey RO. Program evaluation: methods and case studies. Upper Saddle River, NJ: Prentice Hall, 1997.
  84. Dinh-Zarr TB, Sleet DA, Shults RA, et al. Reviews of evidence regarding interventions to increase the use of safety belts. Am J Prev Med 2001;21(suppl):48–65.[Web of Science][Medline]
  85. Caburnay CA, Kreuter MW, Donlin MJ. Disseminating effective health promotion programs from prevention research to community organizations. J Public Health Manag Pract 2001;7:81–9.[Medline]
  86. Stevens MM, Olson AL, Gaffney CA, et al. A pediatric, practice-based, randomized trial of drinking and smoking prevention and bicycle helmet, gun, and seatbelt safety promotion. Pediatrics 2002;109:490–7.[Abstract/Free Full Text]
  87. Cochrane Collaboration. Cochrane Library. (Online database). San Diego, CA: Cochrane Collaboration, 2002. (World Wide Web URL: http://www.cochranelibrary.com/enter/). (Accessed August 15, 2002).
  88. Truman BI, Smith-Akin CK, Hinman AR, et al. Developing the Guide to Community Preventive Services—overview and rationale. Am J Prev Med 2000;18(suppl):18–26.[Web of Science][Medline]
  89. The guide to community prevention services. Reducing injuries to motor vehicle occupants: systematic reviews of evidence, recommendations from the Task Force on Community Prevention Services, and expert commentary. Am J Prev Med 2001;21(suppl):1–90.[Web of Science][Medline]
  90. Hahn RA, Bilukha OO, Snyder S, et al. Early childhood home visitation for the prevention of violence. J Prev Med (in press).
  91. Gotsch KE, Annest JL, Mercy JA, et al. Surveillance for fatal and nonfatal firearm-related injuries—United States, 1993–1998. MMWR CDC Surveill Summ 2001;50(SS-2):1–34.[Medline]
  92. Petrosino A, Turpin-Petrosino C, Buehler J. "Scared Straight" and other juvenile awareness programs for preventing juvenile delinquency (Cochrane Review). In: The Cochrane Library, issue 2, 2003. Oxford, United Kingdom: Update Software Ltd, 2003.
  93. Grady D, Herrington D, Bittner V, et al. Cardiovascular disease outcomes during 6.8 years of hormone therapy: Heart and Estrogen/progestin Replacement Study follow-up (HERSII). JAMA 2002;228:49–57.
  94. McQueen DV. The evidence debate. J Epidemiol Community Health 2002;56:83–4.[Free Full Text]
  95. McQueen DV. Strengthening the evidence base for health promotion. Health Prom Int 2001;16:261–8.
  96. Goodman RM. Evaluation of community-based health programs: an alternative perspective. In: Schneiderman N, Speers M, Silva JM, et al, eds. Integrating behavioral and social sciences with public health. Washington, DC: American Psychological Association, 2001:293–304.
  97. Briss PA, Zaza S, Pappaioanou M, et al. Developing an evidence-based Guide to Community Preventive Services—methods. Am J Prev Med 2000;18(suppl):35–43.[CrossRef][Web of Science][Medline]

Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us    What's this?


This article has been cited by other articles:


Home page
 Inj. Prev.Home page
M. Beth Phelan, M. E Falimirski, D. E Simpson, M. L Czinner, and S. W Hargarten
Competency-based strategies for injury control and prevention curriculums in undergraduate medical education
Inj. Prev., February 1, 2007; 13(1): 6 - 9.
[Abstract] [Full Text] [PDF]

Home page
 Inj. Prev.Home page
P Nilsen
The theory of community based health and safety programs: a critical examination.
Inj. Prev., June 1, 2006; 12(3): 140 - 145.
[Abstract] [Full Text] [PDF]

Home page
 AJPHHome page
B. C. Lee, J. D. Westaby, and R. L. Berg
Impact of a National Rural Youth Health and Safety Initiative: Results from a Randomized Controlled Trial
Am J Public Health, October 1, 2004; 94(10): 1743 - 1749.
[Abstract] [Full Text] [PDF]

Home page
 Int J EpidemiolHome page
M. L Browne
Commentary: Priorities in epidemiological studies of drowning prevention
Int. J. Epidemiol., October 1, 2004; 33(5): 1063 - 1064.
[Full Text] [PDF]

Home page
 Epidemiol RevHome page
S. B. Thacker and E. J. MacKenzie
Preface: The Role of the Epidemiologist in Injury Prevention and Control--An Unmet Challenge
Epidemiol. Rev., August 1, 2003; 25(1): 1 - 2.
[Full Text] [PDF]

Home page
 Epidemiol RevHome page
F. P. Rivara
Introduction: The Scientific Basis for Injury Control
Epidemiol. Rev., August 1, 2003; 25(1): 20 - 23.
[Full Text] [PDF]

This Article
Right arrow Extract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrow Search for citing articles in:
ISI Web of Science (10)
Right arrowRequest Permissions
Google Scholar
Right arrow Articles by Doll, L.
Right arrow Articles by Binder, S.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Doll, L.
Right arrow Articles by Binder, S.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?