Bethany Hamilton Bhat – Meta-Analytic Pooling of Intraclass Correlation Coefficient Estimates

Meta-Analytic Pooling of Intraclass Correlation Coefficient Estimates

Bhat, B. H., & Beretvas, S. N. (2026). Meta-analytic pooling of intraclass correlation coefficient estimates. Research Synthesis Methods, 1–34. doi:10.1017/rsm.2026.10077

Overview

Motivation
Review of Concepts
Research Questions
Study Design
Results
Conclusions

Background

ICCs required for secondary analyses:
- Prospective power analyses for clustered designs
- Correcting effect sizes in meta-analysis for clustering (Hedges and Hedberg 2007) or psychometric artifacts (Wiernik and Dahlke 2020)
ICCs have been used as an outcome in meta-analysis to summarize reliability or clustering effects across studies

ICC estimate

One-way random effects model:

\[ \begin{split} Level \ 1: Y_{ij} = \beta_{0j} + e_{ij} \\ Level \ 2: \beta_{0j} = \gamma_{00} + u_{0j} \end{split} \] where \(e_{ij} \sim N(0, \sigma_e^2)\) and \(u_{0j} \sim N(0, \sigma^2_u)\)

ICC (McGraw and Wong 1996; Fisher 1925):

\[ \rho=\frac{\sigma^2_u}{\sigma^2_u+\sigma^2_e}\]

Sensitivity of Power Analysis to ICC Choice

Example: \(\hat{\rho} \in \{0.05, 0.12, 0.18, 0.25\}\)

Note. Assumptions: \(n = 20\); \(1 - \beta = 0.80\); \(\alpha = 0.05\); \(d = 0.3\); power calculation based on design effect approximation (i.e., DEFF=\(1 +(n-1) \times \rho\), where \(n\) is the average cluster size)

To motivate why we need a principled way to combine ICC estimates, consider a simple hypothetic example.

Suppose I want to conduct a cluster-randomized trial evaluating the effectiveness of the Marte Meo method on children’s communications skills in German daycares. I plan to use a cluster randomized trial and assume approximately 20 children per daycare and a cohen’s d of \(0.3\).

To determine the required number of day cares I need to recruit for my study, I need an assumption about the ICC. So, I identify four prior studies that have similar study characteristics and outcome that report these plausible ICC values: \(\hat{\rho}_1 = 0.05\), \(\hat{\rho}_2 = 0.12\), \(\hat{\rho}_3 = 0.18\), and \(\hat{\rho}_4 = 0.25\).

Although some of these values are relatively close in magnitude, they result in very different required number of clusters and sample sizes.

If I choose a value that is too large, I risk overestimating the ICC and wasting resources by planning an unnecessarily large study. And if I choose a value that is too small, I risk underpowering the study.

One option would be to take a simple average of these ICCs. However, this ignores the fact that these estimates come from studies with different sample sizes and therefore different levels of precision.

This motivates treating these ICCs as estimates of a common underlying parameter and combining them using a meta-analytic pooling, where estimates are weighted by their precision.

This provides a principled way to obtain a single representative ICC value for use in design planning.

Why Meta-Analysis?

Meta-analysis:

Combines ICC estimates using weighted pooling
Gives more weight to more precise studies

Random-Effects Pooling:

\[\hat{\mu} = \frac{\sum_{i=1}^k w_i T_i}{\sum_{i=1}^k w_i},\] where \(k\) denotes the number of studies, \(w_i = \frac{1}{\hat{v}_i+\hat{\tau}^2}\), \(T_i\) is the study-specific effect size estimate.

Note. Corresponds to random-effects meta-analytic model with variability from: between-study heterogeneity and within-study sampling error

Sampling Variances of ICC Estimates

Sampling Variance Formulas
Formula	Equation	Notes
Fisher (1925)	\(v_{\hat{\rho}}=\frac{2(1-\rho)^2[1+(n-1)\rho]^2}{n(n-1)(j-1)}\)	Requires average cluster size and number of clusters
Swiger et al. (1964)	\(v_{\hat{\rho}}=\frac{2(N-1)(1-\rho^2)[1+(n_0-1)\rho]^2}{n_0^2(N-j)(j-1)}\)	Scaled by total \(N\)
Hedges et al. (2012)	\(v_{\hat{\rho}}=\frac{(1-\rho)^2v_2}{(\sigma_e^2+\sigma^2_u)^2}\)	Requires reported variance components and the variance of the level-2 variance
Donner and Koval (1980)	\(v_{\hat{\rho}}=\frac{2N(1-\rho)^2}{N\sum_{i=1}^{j}n_i(n_i-1)\hat{V_i}\hat{W_i}^{-2}-\rho^2\Big[\sum_{i=1}^{j}n_i(n_i-1)\hat{W_i}^{-1}\Big]^2}\)	Requires how many level-1 units there are in each cluster, but reduces to Fisher when balanced
Smith (1957)	\(v_{\hat{\rho}}=\frac{2(1-\rho)^2}{n_0^2}\Big(\frac{[1+\rho(n_0-1)]^2}{N-j}+\frac{(j-1)(1-\rho)[1+\rho(2n_0-1)]+\rho^2[\sum n_i^2-2N^{-1}\sum n_i^3+N^{-2}(\sum n_i^2)^2]}{(j-1)^2}\Big)\)	Requires how many level-1 units there are in each cluster
Fisher TF	\(v_{\hat{\rho}}=0.5\{(j-1)^{-1}+(N-j)^{-1}\}\)	Normalizing transformation; fewest inputs

Note. \(n_0 = \frac{1}{j-1}\left[N-\sum_{i=1}^j\frac{n_i^2}{N}\right]\) is the weighted mean cluster size; \(\hat{W_i} = 1 + (n_i-1)\rho\); \(\hat{V_i} = 1 + (n_i-1)\rho^2\); \(v_2\) is the variance of the level-2 variance \(\sigma^2_u\); ICC estimates are normally transformed using: \(Z_F = \frac{1}{2}\text{ln}\frac{1+(n_0-1)\hat{\rho}}{1- \hat{\rho}}\).

Meta-Analytic Estimation Methods

Assuming a random effects working model with independent ICC estimates, we evaluated:

Robust Variance Estimation (RVE) (Hedges et al. 2010) (Sandwich / Huber–White-type estimator)
- Uses a method-of-moments estimate for \(\tau^2\)
- Robust to weight specification
- Agnostic to distributional assumptions
Restricted Maximum Likelihood (REML) (Viechtbauer et al. 2015)
- Standard likelihood-based \(\tau^2\) estimator
- Widely used in meta-analysis

Note. Differences are expected primarily in estimated standard error, not in the pooled ICC.

Research Question

How well does pooling of ICC estimates recover the population ICC value:

under a random-effects meta-analytic model,
using RVE versus REML, and
across different sampling variance estimators in the inverse variance weights?

Study Design

Primary study data were generated and an ICC was estimated for each study (\(k\)); meta-analytic datasets contained \(k = 20, 50,\) or \(100\) studies.

Primary study data generation and ICC estimation
Component	Values
Number of clusters (\(j\))	\(\mathcal{U}\)[30, 50], \(\mathcal{U}\)[50, 100]
Average units per cluster (\(\overline{n}_j\))	\(\mathcal{U}\)[10, 30], \(\mathcal{U}\)[30, 50]
Degree of imbalance in cluster sizes (\(\zeta\))	0.1, 0.5 Note. \(n_j \sim \mathcal{U}[\bar{n}_j(1-\zeta),\, \bar{n}_j(1+\zeta)]\)
True ICC (\(\rho\))	0.05; 0.10; 0.15; 0.25; 0.50; 0.90
Between-study heterogeneity (\(\tau_{\rho}\))	0.01, 0.02 Note. \(\tilde{\rho} = \rho + \mathcal{U}(-2\tau_\rho, 2\tau_\rho)\)
ICC estimation at primary study level	Unconditional two-level model with REML

Bias (PB, RPB)
RMSE
SE bias (RSEB)

In the simulation design, we consider two stages: a primary-study data generation and estimation stage, followed by a meta-analytic synthesis stage.

In the simulation design, we consider a two-stage framework consisting of a primary-study data generation and estimation stage, followed by a meta-analytic synthesis stage.

In the primary-study stage, we generate clustered data under a range of conditions that vary the number of clusters, the average cluster size, the degree of cluster-size imbalance, and the true ICC value. Cluster sizes are generated with controlled heterogeneity around the mean cluster size, where the degree of imbalance parameter governs the width of the uniform distribution from which cluster sizes are drawn. This produces bounded variation in cluster sizes across clusters within each study.

For each primary study, we first define a true ICC based on specified level-1 and level-2 variance components. We then introduce study-level variation in the ICC using a symmetric uniform distribution controlled by a heterogeneity parameter, τ. This yields a study-specific ICC value, which is then mapped back into valid variance components while holding the level-1 variance fixed. This ensures that each dataset remains a valid two-level random-intercept model.

This produces study-level ICC estimates, which are then used as observed effect sizes in the meta-analysis.

For the ICC values in particular, we selected a range that reflects different applied contexts.

Lower values, such as 0.05 to 0.25, are commonly observed in clustered data settings like education or health studies.

Higher values, such as 0.50 or 0.90, are more typical in reliability contexts, where measurements within a unit are expected to be highly similar.

This range allows us to evaluate performance across both clustering and reliability applications.

In the meta-analytic stage, we synthesize ICC estimates across studies, varying the number of studies in each meta-analytic dataset. We compare inverse-variance weighting approaches using different sampling variance formulas, as well as two estimation methods: REML and robust variance estimation.

Performance is evaluated using bias, relative bias, RMSE, and relative standard error bias for the pooled ICC, as well as bias for the variance components at the primary study level.

Results - PB for pooled ICC

Results - RPB for Pooled ICC

Results - Relative standard error bias (\(\tau_{\rho}\))

Conclusions

Meta-analysis provides a reliable way to obtain a single representative ICC estimate.

REML and RVE performed equivalently for pooling ICCs
Number of studies (\(k\)) had little impact on pooled ICC accuracy
Fisher’s transformation variance formula is recommended:
- Lowest bias for small ICCs
- Comparable performance for moderate and large ICCs
- Most stable standard error estimates
- Requires minimal information from primary studies (unlike other sampling variance formulas)

To conclude, we find that REML and RVE perform very similarly when pooling ICC estimates.

The number of studies included in the meta-analysis had little impact on the accuracy of the pooled ICC.

Across the variance estimators we evaluated, the Fisher transformation approach performed best overall.

It showed the lowest bias for smaller ICC values and performed comparably for moderate and larger ICCs.

It also provided the most stable standard error estimates across conditions.

Importantly, it requires relatively minimal information from primary studies, which makes it especially practical in applied settings where reporting is often incomplete.

More broadly, our results suggest that meta-analysis is a reliable and principled way to obtain a single representative ICC estimate when multiple estimates are available.

So in practice, when multiple ICC estimates are available, meta-analysis provides a principled and reliable way to choose a single value.

For moderate ICCs (\(\rho = .50\)), slightly underestimated ICCs when cluster sizes were small, but bias was very small (\(\leq -0.018\)).

Future Directions

Extend to non-normal outcomes (binary, count data)
Evaluate reliability ICCs with fewer level-1 units
Extend to three-level and more complex hierarchical structures
Consider settings with multiple dependent ICCs per study

Because this study focused on continuous, normally distributed outcomes in a two-level random-intercept setting, future work should extend these results to more complex and applied data structures.

In particular, many ICCs in practice arise from binary or count outcomes, where the distributional assumptions underlying both ICC estimation and variance approximations may behave differently.

Additionally, we only considered single ICC estimates per study, whereas in practice studies often report multiple ICCs derived from related outcomes or models, introducing dependence that is not captured here.

Another important direction is reliability-focused ICCs based on smaller numbers of level-1 units, which may behave differently in terms of estimation error and variance stability.

Finally, extending to three-level or more complex hierarchical structures would help generalize these findings to broader applied settings.

Thank you! Questions?

References

Donner, Allan, and John J. Koval. 1980. “The Estimation of Intraclass Correlation in the Analysis of Family Data.” Biometrics. Journal of the International Biometric Society 36 (1): 19–25. https://doi.org/10.2307/2530491.

Fisher, R. A. 1925. “Theory of Statistical Estimation.” Mathematical Proceedings of the Cambridge Philosophical Society 22 (5): 700–725. https://doi.org/10.1017/S0305004100009580.

Hedges, Larry V., and E. C. Hedberg. 2007. “Intraclass Correlation Values for Planning Group-Randomized Trials in Education.” Educational Evaluation and Policy Analysis 29 (1): 60–87. https://doi.org/10.3102/0162373707299706.

Hedges, Larry V., E. C. Hedberg, and Arend M. Kuyper. 2012. “The Variance of Intraclass Correlations in Three- and Four-Level Models.” Educational and Psychological Measurement 72 (6): 893–909. https://doi.org/10.1177/0013164412445193.

Hedges, Larry V., Elizabeth Tipton, and Matthew C. Johnson. 2010. “Robust Variance Estimation in Meta-Regression with Dependent Effect Size Estimates.” Research Synthesis Methods 1 (1): 39–65. https://doi.org/10.1002/jrsm.5.

McGraw, Kenneth O., and S. P. Wong. 1996. “Forming Inferences about Some Intraclass Correlation Coefficients.” Psychological Methods 1 (1): 30–46. https://doi.org/10.1037/1082-989X.1.1.30.

Smith, C. a. B. 1957. “On the Estimation of Intraclass Correlation.” Annals of Human Genetics 21 (4): 363–73. https://doi.org/10.1111/j.1469-1809.1972.tb00291.x.

Swiger, L. A., W. R. Harvey, D. O. Everson, and K. E. Gregory. 1964. “The Variance of Intraclass Correlation Involving Groups with One Observation.” Biometrics. Journal of the International Biometric Society 20 (4): 818. https://doi.org/10.2307/2528131.

Viechtbauer, Wolfgang, José Antonio López-López, Julio Sánchez-Meca, and Fulgencio Marín-Martínez. 2015. “A Comparison of Procedures to Test for Moderators in Mixed-Effects Meta-Regression Models.” Psychological Methods, Meta-Analysis Topics, vol. 20 (3): 360–74. https://doi.org/10.1037/met0000023.

Wiernik, Brenton M., and Jeffrey A. Dahlke. 2020. “Obtaining Unbiased Results in Meta-Analysis: The Importance of Correcting for Statistical Artifacts.” Advances in Methods and Practices in Psychological Science 3 (1): 94–123. https://doi.org/10.1177/2515245919885611.

Extra Slides

Results - RMSE for Pooled ICC

Results - Relative Standard Error Bias (\(\overline{n}_j\))

RPB \(k\)

PB \(k\)

L2 Variance by \(\tau_{\rho}\)

L2 Variance by Number of Clusters

L1 Variance by Clustersize