International Society for Data Science and Analytics, 2022 Meeting of ISDSA

Font Size: 
Meta-Analysis of Correlation Coefficients: A Cautionary Tale on Treating Measurement Error
Qian Zhang

Building: Corbett Family Hall
Room: E108
Date: 2022-05-31 01:00 PM – 01:30 PM
Last modified: 2022-05-07

Abstract


A scale to measure a psychological construct is subject to measurement error. When meta-analyzing correlations obtained from scale scores, many researchers recommend correcting for measurement error. I considered three caveats when correcting for measurement error in meta-analysis of correlations: (1) the distribution of true scores can be non-normal, resulting in violation of the normality assumption for raw correlations and Fisher's z transformed correlations; (2) coefficient alpha is often used as the reliability, but correlations corrected for measurement error using alpha can be inaccurate when some assumptions of alpha (e.g., tau-equivalence) are violated; and (3) item scores are often ordinal, making the disattenuation formula potentially problematic. Via three simulation studies, I examined the performance of two meta-analysis approaches – with raw correlations and z scores. In terms of estimation accuracy and coverage probability of the mean correlation, results showed that (1) considering the true-score distribution alone, estimation of the mean correlation was slightly worse when true scores of the constructs were skewed rather than normal; (2) when the tau-equivalence assumption was violated and coefficient alpha was used for correcting measurement error, the mean correlation estimates can be biased and coverage probabilities can be low; and (3) discretization of continuous items can result in biased estimates and under-coverage of the mean correlations even when tau-equivalence was satisfied. With more categories and/or items on a scale, results can improve whether tau-equivalence was met or not. Based on these findings, I gave recommendations for conducting meta-analyses of correlations.


Conference registration is required in order to view papers.