Font Size: 

LOO and WAIC become widely used for model selection in Bayesian statistics. Most studies select the model with the smallest value of them based on the point estimates, neither considering the difference of the fit indices between candidate models, nor the uncertainty of the estimates. Accordingly, we propose a sequential method comparing models based on confidence intervals for delta LOO or delta WAIC. A simulation study comparing this method and the point method in selecting mixed-effects location–scale models (MELSMs) is conducted.

Our study revealed that the sequential methods had higher accuracy rate of model selection than the point methods when the true model was simple, or had large magnitude of random intercept in the scale-model, or large sample size. For the most complex model, although the sequential methods preferred simper models, they performed as good as the point methods with larger sample size and more serious heterogeneity of residual variances. Moreover, the sequential methods, especially using 90%CI, tended to have higher power, lower Width95, and smaller SE than the point methods. Meanwhile, the differences between LOO and WAIC were obvious only when level-1 sample size was small, where LOO performed better when True model had either homogeneous residual variances, or serious heterogenous residual variance. Finally, as SE of delta LOO and delta WAIC used to construct their CI could be calculated by R package brms (or LOO) conveniently, we suggest apply the proposed methods in evaluation and selection for MELSMs in practice.

Leave-one-out cross-validation (LOO); Widely applicable information criterion (WAIC); mixed-effects location–scale models (MELSMs); confidence interval