pmseCV {MMIX}R Documentation

Model performance indicators PMSE and AUC

Description

pmseCV and aucCV calculate the Predictive Mean Square Error (PMSE) by "leave-np-out" cross-validation and Area Under Roc Curve (AUC) by "leave-np-pair-out" cross-validation. They can be applied to models created using fullModel, stepSel, bmaBic, mixAic and arms.

Usage

pmseCV(data, method = 1, np, random = TRUE, npermu = 100, file = NULL, ...)

aucCV(data, method = 1, np, random = TRUE, npermu = 100, file = NULL, ...)

Arguments

data a data frame including the response variable (first column) and the explanatory variables. All the variables must be numeric and the response variable value must be 0 or 1 for the logistic model.
method the statistical method used to estimate the model parameters. method = 1 for fullModel, method = 2 for stepSel, method = 3 for bmaBic, method = 4 for mixAic, method = 5 for arms.
np number of observations (pmseCV) or pairs of observations (aucCV) left out for computing the PMSE or AUC.
random observations are selected at random if TRUE. random can be FALSE only if np = 1. In this case all the possible sets are selected.
npermu number of random samples of np observations if random = TRUE.
file the path of the file where the results are stored during the run. If file = NULL no file is created.
... the specific arguments of the called method

Details

These two cross validation procedures are implemented to assess the accuracy of model predictions. Linear models should be evaluated using pmseCV and logistic models should be evaluated using aucCV. In the aucCV procedure, data are held out by pair (one data from each class, 0 and 1) at each iteration in order to calculate the corresponding AUC. The PMSE (AUC) estimated by cross validation is the mean of the PMSE (AUC) calculated for possible sets of np observations (pairs). If file is not NULL, the np predictions (first column) and the np corresponding observations (second column) are saved at each iteration.

Value

pmseCV (aucCV) returns a one-row data frame including the PMSE (AUC) calculated from the whole sample in the first column and by cross validation in the other one.

Warning

These functions do not accept or treat the missing values.

Author(s)

Marie Morfin and David Makowski

References

Sing, T., Sander, O., Beerenwinkel, N. and Lengauer, T. (2005) ROCR: visualizing classifier performance, Bioinformatics applications note 21, 3940-3941.

Yuan, Z. and Ghosh, D. (2008) Combining Multiple Biomarker Models in Logistic Regression, Biometrics 64, 431-439.

Yuan, Z. and Yang, Y. (2005) Combining Linear Regression Models: When and How?, Journal of the American Statistical Association 100, 1202-1214.

See Also

fullModel, stepSel, bmaBic, mixAic, arms

Examples

##Data 
#Explanatory variables 
X1<-c(-0.2,-2.4,-0.7,1.2,0.0,-1.1,-2.1,-0.3,2.0,-1.7,1.4,-1.3,-3.4,0.4,-1.3,
-4.8)
X2<- c(-3,  2,  1, -2, -2, -4,  0,  1,  1, -1, -1, -4,  0,  2,  0, -4)
X3<-c(2,1,0,-2,1,-2, 0, -1, -4, 1, -3, -3, -3, -1, 0, 2)

#Linear model
Y1<- c(8.7, 6, 9.1, 10.4, 7.6 ,10.4,  7.9, 11.9, 18, 10.5, 16.5, 8.8, 7.7, 13.5,
 8.2, 0.8)
data1<-data.frame(Y1,X1,X2,X3)
#Logistic model
Y2<-c(1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1)
data2<-data.frame(Y2,X1,X2,X3)

##Linear models evaluated by pmseCV
#for a stepwise selection
pmseStepBic<-pmseCV(data=data1,method=2,np=1,random=FALSE,direction="both",
criterion="bic",trace=0)
pmseStepBic
#for the BMA method
pmseBMA<-pmseCV(data=data1,method=3,np=1,random=FALSE)
pmseBMA

##Logistic models evaluated by aucCV
#for a stepwise selection
aucStepBic<-aucCV(data=data2,method=2,np=1,random=FALSE,direction="both",
criterion="bic",trace=0)
aucStepBic
#for the BMA method
aucBMA<-aucCV(data=data2,method=3,np=1,random=FALSE)
aucBMA



[Package MMIX version 1.1 Index]