Questions tagged [model-evaluation]

On evaluating models, either in-sample or out-of-sample.

In-sample model evaluation techniques can be based on measures of fit or , but note that in-sample fit will typically increase spuriously as the model becomes more complex, which is called . For this reason, typically in-sample fit is penalized based on model complexity, like adjusted , or . AIC and BIC are also examples of information criteria, which can also be used in-sample.

Out-of-sample model evaluation usually relies on predictive accuracy and again on . Distributional predictions can be evaluated using .

1143 questions
258
votes
12 answers

Why is accuracy not the best measure for assessing classification models?

This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference. Accuracy , the proportion of correct classifications…
Tim
  • 146,039
67
votes
3 answers

How to select a clustering method? How to validate a cluster solution (to warrant the method choice)?

One of the biggest issue with cluster analysis is that we may happen to have to derive different conclusion when base on different clustering methods used (including different linkage methods in hierarchical clustering). I would like to know your…
58
votes
7 answers

Best PCA algorithm for huge number of features (>10K)?

I previously asked this on StackOverflow, but it seems like it might be more appropriate here, given that it didn't get any answers on SO. It's kind of at the intersection between statistics and programming. I need to write some code to do PCA…
dsimcha
  • 9,067
47
votes
5 answers

Optimized implementations of the Random Forest algorithm

I have noticed that there are a few implementations of random forest such as ALGLIB, Waffles and some R packages like randomForest . Can anybody tell me whether these libraries are highly optimized? Are they basically equivalent to the random…
Henry B.
  • 1,639
44
votes
3 answers

Classification/evaluation metrics for highly imbalanced data

I deal with a fraud detection (credit-scoring-like) problem. As such there is a highly imbalanced relation between fraudulent and non-fraudulent observations. http://blog.revolutionanalytics.com/2016/03/com_class_eval_metrics_r.html provides a great…
41
votes
3 answers

Can AUC-ROC be between 0-0.5?

Can AUC-ROC values be between 0-0.5? Does the model ever output values between 0 and 0.5?
Aman
  • 653
  • 1
  • 8
  • 10
36
votes
1 answer

Why is LogLoss preferred over other proper scoring rules?

It seems anytime people care about estimating probabilities accurately they choose LogLoss as the evaluation metric. But there are many other evaluation metrics which will prefer accurate estimation of probabilities (and not only ranking or…
Meir Maor
  • 610
36
votes
1 answer

Cross-validation misuse (reporting performance for the best hyperparameter value)

Recently I have come across a paper that proposes using a k-NN classifier on an specific dataset. The authors used all the data samples available to perform k-fold cross validation for different k values and report cross validation results of the…
32
votes
2 answers

Proper scoring rule when there is a decision to make (e.g. spam vs ham email)

Among others on here, Frank Harrell is adamant about using proper scoring rules to assess classifiers. This makes sense. If we have 500 $0$ s with $P(1)\in[0.45, 0.49]$ and 500 $1$ s with $P(1)\in[0.51, 0.55]$ , we can get a perfect classifier by…
29
votes
4 answers

AUC and class imbalance in training/test dataset

I just start to learn the Area under the ROC curve (AUC). I am told that AUC is not reflected by data imbalance. I think it means that AUC is insensitive to imbalance in test data, rather than imbalance in training data. In other words, only…
Munichong
  • 2,135
27
votes
3 answers

Evaluating logistic regression and interpretation of Hosmer-Lemeshow Goodness of Fit

As we all know, there are 2 methods to evaluate the logistic regression model and they are testing very different things Predictive power: Get a statistic that measures how well you can predict the dependent variable based on the independent…
23
votes
2 answers

What is the difference between $R^2$ and variance score in Scikit-learn?

I was reading about regression metrics in the python scikit-learn manual and even though each one of them has its own formula, I cannot tell intuitively what is the difference between $R^2$ and variance score and therefore when to use one or another…
21
votes
1 answer

How to Compute the Brier Score for more than Two Classes

tl;dr How do I correctly compute the Brier score for more than two classes? I got confusing results with different approaches. Details below. As suggested to me in a comment to this question , I would like to evaluate the quality of a set of…
20
votes
2 answers

Why not use evaluation metrics as the loss function?

Most algorithms use their own loss function for optimization. But these loss functions are always different from metrics used for actual evaluation. For example, for building binary classification models, log loss is normally used as the loss…
etang
  • 1,047
20
votes
3 answers

Should I use the cross validation score or the test score to evaluate a machine learning model?

Let's say I want to compare two machine learning models (A and B) on a classification problem. I split my data into train (80%) and test set (20%). Then I perform 4-fold cross-validation on the training set (so every time my validation set has 20%…
Marios
  • 301
1
2 3
…
76 77