Meta-Analytic Pooling of CLES Estimates: A Comparison of Transformation Approaches

Motivation

Common measures for two-group comparisons on quantitative outcomes:

• Mean differences
• Standardized mean differences (\(d\))
• Log ratio of means (lnRR)
• Measures of stochastic superiority (\(\theta\))

Stochastic Superiority (\(\theta\))

The probability that a randomly selected person from one group has a higher outcome than a randomly selected person from another group

\[\theta = P(Y>X)+\frac{1}{2}P(Y=X)\]

• \(\frac{1}{2}P(Y=X)\) term handles ties
• Several parametric and non-parametric estimators:
  • CLES, AUC, Wilcoxon U, Concordance probability, Cliff’s \(\delta\)

McGraw and Wong (1992);Hanley and McNeil (1982); DeLong et al. (1988);Newson (2002);Cliff (1993)

Estimating \(\theta\): CLES

• Proposed by McGraw and Wong (1992) under normality and homoscedasticity
• Under these assumptions, \(d\) completely determines any measure of overlap between two distributions

\[\hat{\theta} = \Phi\left(\frac{d}{\sqrt{2}}\right), \quad \mathrm{var}(\hat{\theta}) = \frac{e^{-d^2 / 2}}{4\pi} \left(\frac{1}{n_1} + \frac{1}{n_2} + \frac{d^2}{2(n_1+n_2)} \right)\]

• There is no comprehensive evaluation on how to best pool CLES estimates

Meta-analyzing CLES Estimates

• CLES:
  • \(\hat{\theta} \in [0, 1]\)
  • Sampling distribution is skewed near 0 and 1
• Three transformations of \(\hat{\theta}_i\):
  • logit, arcsine, and probit/SMD
• Two back-transformations to obtain \(\hat{\mu}_{\theta}\):
  • Standard
  • Integral

Back-transformation Methods for \(\hat{\mu}_{\theta}\)

Two approaches to estimate \(\mu_{\theta}\) after pooling on the transformed scale:

• Standard:
  • Returns the median of the distribution of true effects, not the mean (Jensen’s inequality)
• Integral:
  • Integrates \(g^{-1}\) over the normal random-effects distribution
  • Recovers the mean of the distribution of true effects
  • See Igelmann et al. (2026)

Data Generation

1. Sample \(k\) true CLES values (\(\theta_i\)) from \(\text{Beta}(\alpha, \beta)\) with moments \((\mu_\theta, \tau^2_\theta)\)

2. Sample study sample sizes from right-skewed distribution (\(\gamma_1 \approx 1.45\)):

where \(\overline{n}\) is the average sample size for a primary study and assuming equal group sizes Marín-Martínez and Sánchez-Meca (1998); Sanchez-Meca and Marin-Martinez (1998)

3. Generated observed CLES estimates (\(\hat{\theta}_i\)) implied by the model that

• Assumed normally distributed outcomes with equal population variances
• \(\sigma_X = \sigma_Y = 1\) and \(\mu_X = 0\)
• \(\mu_Y\) chosen so that \(P(Y>X)=\theta_i\)

Simulation Design

• 600 conditions \(\times\) 10,000 replications each
• \(\mu_{\theta}\) and \(\tau_{\theta}\) choosen to correspond to commonly observed SMD values
• 7 pooling methods; REML estimation (DL when convergence failed); Knapp-Hartung test
• Performance criteria: bias and RMSE of \(\hat{\mu}_{\theta}\); 95% CI coverage; rejection rates and power for test of \(H_0: \mu_{\theta} = 0.5\) and Q-test of \(H_0: \tau^2 = 0\); bias in \(\hat{\tau}_{\theta}\)

Results: Parameter Bias

Results: RMSE

Results: Coverage

Results: Coverage – Integral Methods Only

Conclusions and Discussion

• Standard back-transformations are acceptable at low heterogeneity, but integral back-transformation is recommended when \(\tau_\theta\) is large
• Under normality and homoscedasticity, SMD with integral back-transformation is the recommended
  • arcsine and logit transformations with the integral back-transformation were comparable across most conditions (except for coverage for moderate \(\tau_{\theta}\) and large \(k\))
  • Direct pooling should be avoided
    
• Next Study:
  • Extend to non-normal outcomes and unequal population variances and evaluate other estimates (and corresponding sampling variances) of stochastic superiority

Transformations of CLES Estimates

Back-transformation Methods

Further Details on Between-study Heterogeneity

• Specified \(\mu_{\theta}\) and \(\tau_{\theta}^2\) values as the moments of the beta distribution

• Found the \(\alpha\) and \(\beta\) parameters of the beta distribution by: \(\alpha = \mu_{\theta} \times \nu\) and \(\beta = (1 - \mu_{\theta})\nu\) where \(\nu = (\alpha + \beta) =\frac{\mu_{\theta}(1 - \mu_{\theta})}{\tau^2_{\theta}} - 1\).

  • Note. values must meet the following condition: \(\nu = (\alpha + \beta) > 0\), so therefore \(\tau^2_{\theta} < \mu_{\theta}(1 - \mu_{\theta})\)

• For the specified \(\mu_{\theta}\) and \(\tau_{\theta}^2\), we derived them from SMD parameters through numerical integration

  • specified all 20 pairwise combinations of the following: \(\mu_d = \{0, 0.2, 0.5, 0.8, 1.1, 1.4\}\) and \(\tau_d = \{0, 0.05, 0.20, 0.40, 0.60\}\)

Beta Distributions

Obtaining CLES Estimates

1. After obtaining true CLES values, then transform to true SMD values:

3. Generate observed SMD estimates from a scaled non-central \(t\)-distribution assuming homoscedastic variances: