Editorial: A Call for Replications of Research in Pediatric Psychology and Guidance for Authors

Abstract

Various scientists have underscored the critical need for replication of research findings in the development of science, including behavioral science (Cohen, 1994; Hedges, 1987; Robinson & Levin, 1997; Shaver & Norton, 1980). However, replication of research in behavioral science in general and pediatric psychology in particular has been limited (Kline, 2004, 2009). My review of articles in the Journal of Pediatric Psychology (JPP) for the past several years indicate that replications of research findings in our field are rare. There are several reasons for this: The editorial process for scholarly journals places a high premium on new and innovative research findings rather than replications. Grant review criteria used by the National Institutes of Health (NIH) that emphasize innovative research may also inhibit research replications. Although the reasons are understandable, the relative absence of replication limits the cumulative impact, generalizability, and ultimately the validity of science (Cohen, 1994; Robinson & Levin, 1997). To address the need for more frequent research replications in the field of pediatric psychology, the purpose of this editorial is to describe the importance of research replication, types and exemplars of study replication, issue a call for replication research, and provide guidance for authors in submitting replications to JPP. Although replication is necessary if not critical for the development of all areas of science, there are several reasons why it is especially important in the field of pediatric psychology: Pediatric psychology is still a relatively new area of science in which many measures and intervention methods are not well understood and are in various phases of validation. For this reason, replications assume special importance in developing generalizable scientific knowledge. The methodological characteristics of published research in the field of pediatric psychology in journals such as JPP also heighten the need for replication. For example, sample sizes in published research in the field of pediatric psychology tend to be relatively small (100 or less), and estimates of sampling error are much greater in small samples than larger ones (Kline, 2004). Replication increases confidence in the validity of findings that are gathered in small samples. In addition, many studies published in JPP involve data that are gathered only at one site. This increases the likelihood of sampling bias attributable to specific characteristics of local samples, health care settings, etc. that reduces generalizablility of findings (Drotar & Riekert, 2000). Again, replication of findings at different sites with different samples increases the validity and generalizability of data that are gathered at a single site. Replication will also enhance the validity and generalizability of clinically relevant research in pediatric psychology. For example, the development of measures and evidence-based assessments is an important priority for the field of pediatric psychology (Holmbeck & Devine, 2009; Quittner, Modi, Lemanek, Ievers-Landis, & Rapoff, 2008). Measurement properties of new assessment instruments (or more established instruments for that matter) can vary considerably across different samples and require replication. Such replication is particularly important for measures of general constructs (e.g., family functioning) that have been initially validated in samples of children with a specific chronic condition. In addition, the validity and clinical utility of a measure depend upon the generalizability of psychometric properties to a broad range of samples including clinical populations. Thus, clinical significance will be enhanced by replications of measurement properties in multiple samples, especially populations who are seen in clinical practice. The development and clinical application of evidence-based interventions is another priority area for the field of pediatric psychology (Spirito & Kazak, 2006). Replication of intervention effects in different settings (ideally by different investigatory teams) is necessary for an intervention model to be considered as well established (Chambless & Hollon, 1998). Moreover, replication of research-based interventions in clinical practice settings is critical to demonstrate clinical utility and facilitate the dissemination of evidence-based interventions (Drotar, 2006). Unfortunately, replications of studies involving evidence-based assessments and interventions are the exception rather than the norm in the field of pediatric psychology. External replication, which involves data gathered from new samples collected in different settings and/or at different times than an original sample, is the kind of replication that is most familiar to pediatric psychologists. External replication can be conducted by the same investigator who conducted the original study or ideally by different investigatory teams. External replication can take several different forms: for example, in exact or literal replication all of the primary features of an original study (e.g., design, methods, and outcomes) are repeated as closely as possible (Kline, 2004). Owing to the expense and logistical considerations, exact replications of clinical research are rarely conducted. On the other hand, operational replication in which some (but not all) methods or measures from an original study are repeated (Kline, 2004) is more feasible and frequent. Operational replication of some of the initial study methods together with an extension that includes new methods or design features is particularly useful because it combines the advantages of replication with the potential for innovation. Comparison of a behavioral intervention previously demonstrated to be effective with a new intervention model hypothesized to be even more powerful is an example of operational replication. Internal replication can take any number of forms: One example is cross-validation, which is accomplished by dividing the total sample at random into a derivation sample and a cross validation sample and conducting identical analyses on each one. Cross-validation can be a very useful way to document the validity of a new measure. Internal replication can also involve statistical methods that recombine cases in a data set to estimate statistical precision of relevant parameters. Such methods, (e.g., bootstrapping, jackknife, and randomization procedures) involve computer generation of large numbers of random samples (e.g., greater than 1000) from an original data set, computing an estimator in each generated sample and constructing an empirical frequency distribution of that estimator across all generated samples (Kline, 2004). Published exemplars of replications related to theoretical models, interventions, and measurement from the field of pediatric psychology are now presented in order to inspire and guide other replications. One of the most cited (Aylward, Roberts, Columbo, & Steele, 2008) theoretical models that has stimulated research replications in pediatric psychology is Wallander, Varni, and colleagues’ Disability, Stress, and Coping Model of psychological adjustment in pediatric chronic illness (Walander, Varni, Babani, Banis, & Wilox, 1988). A primary contribution of this model was the distinction between risk factors (e.g., severity of illness, life stressors, etc.) hypothesized to increase risk for psychological adjustment problems versus resistance or coping resources (e.g., coping behaviors and self esteem) that lessen the impact of risk factors. Examples of replications that support this model with various chronic pediatric conditions includes work done by Wallander and colleagues (Wallander, Thompson, Alrikson-Schmidt, 2003; Wallander et al., 1988) and other research teams (Casey, Brown, & Bakeman, 2000). Randomized controlled trials (RCTs) of intervention are arguably the gold standard for testing the efficacy of interventions in our field (Spirito & Kazak, 2006). Although such trials are time consuming and expensive, time and expense do not negate the need for and importance of replication for establishing empirical support, generalizability, and, ultimately, the clinical utility of interventions. Wysocki and his colleagues’ research on Behavioral Family Systems Therapy (BFST) in the management of type 1 diabetes (Wysocki et al., 2000, 2006) is an example of a replication and extension of research on intervention. In an initial study, Wysocki et al. (2000) conducted an RCT that compared BFST with an Education and Support (ES) intervention that had an equivalent number of sessions as BFST and with current medical therapy (CT) that was directed by physicians. Compared with ES and CT, BFST demonstrated greater improvement in parent-adolescent relationships and reduced diabetes-specific conflict. However, no effects on treatment adherence were found. In order to enhance the impact of BFST on treatment adherence and blood sugar control, Wysocki et al. (2006) developed a new intervention model that retained the core elements of BFST such as family problem solving and communication training but also included a greater focus on diabetes management (e.g., targeting of diabetes-specific behavior problems, behavioral contracting, etc.). When compared with the same conditions: ES and SC as in the initial study (Wysocki et al., 2000), the revised intervention: BFST for Diabetes (BFST-D) was associated with better family conflict and treatment adherence. Moreover, both BFST-D and ES were associated with reductions in blood sugar control, particularly among adolescents with high blood sugar control (e.g., glycosolated hemoglobin levels above 9.0%) (Wysocki et al., 2006). A premier example of both programmatic research and replications involving measurement is Varni and his colleagues’ work on health-related quality of life (HRQOL) assessed with the PedsQL. Replication of research based on the PedsQL has been facilitated by the inclusion of items that measure generic HRQOL that are potentially applicable to any sample of children, including children with chronic conditions, along with condition-specific modules that have been developed for many chronic conditions. There are any number of published replications in the extensive track record of publications using the PedsQL. For example, Varni, Limbers, and Burwinkle (2007) compared scores from the PedsQL 4.0 Generic Core Scales in a sample of more than 2500 pediatric patients from 10 different physician-diagnosed chronic illnesses and 33 subgroups of various disease categories and severities with more than 9500 healthy children. In another example, Limbers, Newman, and Varni (2008) identified an equivalent five-factor structure on the PedsQL 4.0 Generic Core Scales across more than 11 different chronic conditions (N = 3607) and healthy children (N = 2826). In order to enhance the cumulative impact and generalizability of science in the field of pediatric psychology, replication of research will be encouraged in submissions to JPP. However, recommendations are one thing but implementing them is quite another. For this reason, guidance for authors in developing manuscripts that involve research on replications is presented in the next section. Not every published research study has sufficient scientific significance to warrant replication. On the other hand, research that has influenced other investigators and helped to shape theory and stimulate research in the field of pediatric psychology warrants replication. For example, Wallander et al.’s (1988) conceptual model has resulted in published replications because it was theoretically interesting and helped to explain the extraordinary individual differences in the outcomes of children with chronic illness (Drotar, 2006). As noted earlier, studies that extend the validity, generalizability, and clinical significance of evidence-based measures and treatments by conducting replications are very important. For example, replications and extensions that test the efficacy of specific components of multifaceted evidence-based interventions to identify those components that have the most powerful impact on clinically relevant outcomes are particularly significant. Priority will be given to manuscripts (e.g., descriptive, measurement, or intervention) that feature replication of results in multiple samples and settings. Such replication, including cross-validation, is particularly advantageous in studies that report on the development of new measures (Holmbeck & Devine, 2009). Research that replicates theoretically relevant and scientifically interesting measurement models of important constructs or tests explanatory models of processes that mediate intervention effects in different samples is another priority. It is not uncommon for researchers to produce interesting findings that fail to replicate previous findings in strong tests of hypotheses and are contrary to prevailing scientific paradigms. Such findings from well-designed and well-powered studies can certainly be published in JPP. As is the case for any published manuscript, authors need to articulate the scientific and/or clinical significance of the design and findings of their studies for the field of pediatric psychology (Drotar, 2009). However, the significance of a replication will generally not be based as much on the innovation of the study design (though creative replications and extensions are clearly possible) as it will be on the degree to which the study enhances cumulative scientific knowledge and/or clinical significance. For this reason, authors should be as explicit as possible in describing the scientific and clinical significance of their study designs and the findings derived from their replications. Authors face the challenges of convincing reviewers that: (1) the original research on which the replication is based is sufficiently important and significant to be replicated (Drotar, 2009); and (2) that the replication has sufficient significance and potential scientific impact to warrant publication. Both conditions are necessary for research to be published in JPP. In order to establish significance, investigators need to be as creative and convincing as possible in describing the need, rationale, and relevance for the replication and/or extension of findings. Careful and complete articulation of the original study methods and the replication are very important to facilitate review of the manuscript and additional replications, as relevant. For this reason, important but extensive information such as treatment manuals, a complete description of study procedures, etc. that will facilitate replication of treatments by other investigators should be submitted as supplementary material to be posted on the JPP website. By now, readers of this editorial may be questioning the wisdom of this author (it would neither be the first time nor the last). Prospective authors may wonder: What are my chances of publishing a replication in JPP? Even if I could, what is my incentive for doing so? Should I invest precious time and energy in conducting replication when this is not an NIH priority? The short answer to the last question is: yes, because it will develop the science of pediatric psychology and child health. Moreover, one potential incentive is that replications will be encouraged for submission and publication in JPP. The limited number of replications that are submitted to JPP may reflect authors’ disbelief that such research could be competitive for publication has value to the field, or to their career development. I recognize that authors’ beliefs and submission practices concerning replications may not easily change as a function of this editorial. Nevertheless, I’m hopeful that greater numbers of replications will be submitted to JPP. Authors who are interested in submitting a manuscript based on a replication of research should feel free to consult the editor if they have questions about whether a specific replication would be of interest to JPP and potentially competitive for publication. It is important to emphasize that replications are not the only way to develop and extend the impact of the cumulative body of knowledge in the field of pediatric psychology. For example, meta-analysis in which studies are synthesized using the metric of effect size provides an excellent way to evaluate the body of scientific research in specific content areas (Hunt, 1997; Kline, 2004). Meta-analyses will continue to have publication priority in the ongoing JPP series on review articles. I am hopeful that this editorial and the priority that will be placed on publishing replications will stimulate submissions of replications and extensions of research to JPP, especially those that are focused on measurement, intervention, and tests of theoretical models. Such work will ultimately enhance the validity, generalizability, and cumulative scientific impact of research in our evolving field. The suggestions made by the Associate Editors of JPP to improve the clarity and impact of this editorial are gratefully acknowledged as is the assistance of Meggie Bonner in typing this manuscript. Conflicts of interest: None declared.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 34 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.