In a previous blog post, we discussed how meta-analytic methods can help to address the replication crisis in psychology by providing a more accurate and comprehensive view of the research literature. In this follow-up post, I discuss some of the challenges and considerations that researchers may encounter in meta-analysis (see our previous blog post for background information on meta-analysis).
One common challenge in meta-analysis is publication bias, which refers to the tendency for positive or statistically significant results to be more likely to be published than negative or non-significant results (Sterling, 1959). This can lead to an overestimation of the true effect size, as the studies included in the meta-analysis may not be representative of the entire research literature (Rosenthal, 1979). To address this issue, researchers can use techniques such as funnel plots or trim-and-fill analysis to identify and adjust for publication bias (Egger, Davey Smith, Schneider, & Minder, 1997; Duval & Tweedie, 2000).
Another consideration in meta-analysis is the use of statistical methods to combine the effect sizes from different studies. There are several approaches that can be used, such as fixed-effects models, which assume that the studies are estimating the same underlying effect (Hedges & Olkin, 1985), or random-effects models, which allow for the possibility of between-study variability (DerSimonian & Laird, 1986). It is important to choose the appropriate model based on the characteristics of the studies being analyzed (Borenstein et al., 2009).
In conclusion, conducting a meta-analysis involves a number of steps and considerations, from identifying and selecting the studies to analyzing and interpreting the results (Cooper, 2009). Meta-analyses can provide valuable insights into the research literature and help to identify reliable and valid findings in psychology (Lipsey & Wilson, 2001). However, it is important to carefully consider the limitations and potential sources of bias in the meta-analysis process (Hedges & Olkin, 1985).
References:
Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to meta-analysis. West Sussex, UK: John Wiley & Sons, Ltd.
Cooper, H. (2009). Research synthesis and meta-analysis: A step-by-step approach (4th ed.). Los Angeles, CA: Sage.
DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7(3), 177-188.
Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455-463.
Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315(7109), 629-634.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. San Diego, CA: Academic Press.
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638-641.
Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American Statistical Association, 54(285), 30-34.
#metaanalysis #researchsynthesis #publicationbias
One common challenge in meta-analysis is publication bias, which refers to the tendency for positive or statistically significant results to be more likely to be published than negative or non-significant results (Sterling, 1959). This can lead to an overestimation of the true effect size, as the studies included in the meta-analysis may not be representative of the entire research literature (Rosenthal, 1979). To address this issue, researchers can use techniques such as funnel plots or trim-and-fill analysis to identify and adjust for publication bias (Egger, Davey Smith, Schneider, & Minder, 1997; Duval & Tweedie, 2000).
Another consideration in meta-analysis is the use of statistical methods to combine the effect sizes from different studies. There are several approaches that can be used, such as fixed-effects models, which assume that the studies are estimating the same underlying effect (Hedges & Olkin, 1985), or random-effects models, which allow for the possibility of between-study variability (DerSimonian & Laird, 1986). It is important to choose the appropriate model based on the characteristics of the studies being analyzed (Borenstein et al., 2009).
In conclusion, conducting a meta-analysis involves a number of steps and considerations, from identifying and selecting the studies to analyzing and interpreting the results (Cooper, 2009). Meta-analyses can provide valuable insights into the research literature and help to identify reliable and valid findings in psychology (Lipsey & Wilson, 2001). However, it is important to carefully consider the limitations and potential sources of bias in the meta-analysis process (Hedges & Olkin, 1985).
References:
Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to meta-analysis. West Sussex, UK: John Wiley & Sons, Ltd.
Cooper, H. (2009). Research synthesis and meta-analysis: A step-by-step approach (4th ed.). Los Angeles, CA: Sage.
DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7(3), 177-188.
Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455-463.
Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315(7109), 629-634.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. San Diego, CA: Academic Press.
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638-641.
Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American Statistical Association, 54(285), 30-34.
#metaanalysis #researchsynthesis #publicationbias
RSS Feed
