Diagnostic Measures for Model Criticism

We discuss the problem of model criticism, with emphasis on developing summary diagnostic measures. We approach model criticism by identifying possible troublesome features of the currently entertained model, embedding the model in an elaborated model, and measuring the value of elaborating. This requires three elements: a model elaboration, a prior distribution and a utility function. Each triplet generates a different diagnostic measure. We focus primarily on the measure given by a Kullback-Leibler divergence between the marginal prior and posterior distributions on the elaboration parameter. We also develop a linearized version of this diagnostic and use it to show our procedure is related to other tools commonly used for model diagnostic, such as Bayes factors and the score function. One attraction of this approach is that it allows model criticism to be performed jointly with parameter inference and prediction. Also, this diagnostic approach aims at maintaining an exploratory nature to the criticism process, while affording feasibility of implementation. In this article we present the general outlook and discuss general families of elaborations for use in practice; the exponential connection elaboration plays a key role. We then describe model elaborations for use in diagnosing: departures from normality, goodness of fit in generalized linear models, and variable selection in regression and outlier detection. we illustrate our approach with two applications.