Questions tagged [hyperparameter]

A parameter that is not strictly for the statistical model (or data generating process), but a parameter for the statistical method. It could be a parameter for: a family of prior distributions, smoothing, a penalty in regularization methods, or an optimization algorithm.

A is a parameter that is not strictly for the statistical model (or data-generating process), but a parameter for the statistical method. It could be a parameter for: a family of prior distributions, smoothing, a penalty in regularization methods, or an optimization algorithm.

652 questions
66
votes
5 answers

Practical hyperparameter optimization: Random vs. grid search

I'm currently going through Bengio's and Bergstra's Random Search for Hyper-Parameter Optimization [1] where the authors claim random search is more efficient than grid search in achieving approximately equal performance. My question is: Do people…
Bar
  • 3,002
61
votes
2 answers

Do we have to tune the number of trees in a random forest?

Software implementations of random forest classifiers have a number of parameters to allow users to fine-tune the algorithm's behavior, including the number of trees $T$ in the forest. Is this a parameter that needs to be tuned, in the same way as…
Sycorax
  • 97,234
43
votes
3 answers

Guideline to select the hyperparameters in Deep Learning

I'm looking for a paper that could help in giving a guideline on how to choose the hyperparameters of a deep architecture, like stacked auto-encoders or deep believe networks. There are a lot of hyperparameters and I'm very confused on how to choose…
Jack Twain
  • 8,481
43
votes
4 answers

How should Feature Selection and Hyperparameter optimization be ordered in the machine learning pipeline?

My objective is to classify sensor signals. The concept of my solution so far is : i) Engineering features from raw signal ii) Selecting relevant features with ReliefF and a clustering approach iii) Apply N.N, Random Forest and SVM However I am…
Grunwalski
  • 685
36
votes
3 answers

How to build the final model and tune probability threshold after nested cross-validation?

Firstly, apologies for posting a question that has already been discussed at length here , here , here , here , here , and for reheating an old topic. I know @DikranMarsupial has written about this topic at length in posts and journal papers, but…
Dr. Andi Lowe
  • 461
33
votes
6 answers

Is hyperparameter tuning on sample of dataset a bad idea?

I have a dataset of 140000 examples and 30 features for which I am training several classifiers for a binary classification (SVM, Logistic Regression, Random Forest etc) In many cases hyperparameter tuning on the whole dataset using either Grid or…
LetsPlayYahtzee
  • 658
  • 2
  • 8
  • 18
33
votes
2 answers

What is the reason that the Adam Optimizer is considered robust to the value of its hyper parameters?

I was reading about the Adam optimizer for Deep Learning and came across the following sentence in the new book Deep Learning by Bengio, Goodfellow and Courville: Adam is generally regarded as being fairly robust to the choice of hyper parameters,…
Charlie Parker
  • 7,538
30
votes
3 answers

How to get hyper parameters in nested cross validation?

I have read the following posts for nested cross validation and still am not 100% sure what I am to do with model selection with nested cross validation: Nested cross validation for model selection Model selection and cross-validation: The right way…
Heavy Breathing
  • 551
26
votes
2 answers

Natural interpretation for LDA hyperparameters

Can somebody explain what is the natural interpretation for LDA hyperparameters? ALPHA and BETA are parameters of Dirichlet distributions for (per document) topic and (per topic) word distributions respectively. However can someone explain what it…
abhinavkulkarni
  • 886
25
votes
1 answer

Is decision threshold a hyperparameter in logistic regression?

Predicted classes from (binary) logistic regression are determined by using a threshold on the class membership probabilities generated by the model. As I understand it, typically 0.5 is used by default. But varying the threshold will change the…
Nick
  • 453
25
votes
4 answers

How bad is hyperparameter tuning outside cross-validation?

I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one you used to tune the features. What I'm…
Ben Kuhn
  • 5,766
22
votes
2 answers

Advantages of Particle Swarm Optimization over Bayesian Optimization for hyperparameter tuning?

There's substantial contemporary research on Bayesian Optimization (1) for tuning ML hyperparameters. The driving motivation here is that a minimal number of data points are required to make informed choices about what points are worthwhile to try…
Sycorax
  • 97,234
20
votes
3 answers

Hyper parameters tuning: Random search vs Bayesian optimization

So, we know that random search works better than grid search, but a more recent approach is Bayesian optimization (using gaussian processes). I've looked up a comparison between the two, and found nothing. I know that at Stanford's cs231n they…
Yoni Keren
  • 576
20
votes
2 answers

Efficiently tuning Hyperparameters in Gaussian Process Regression

I am trying to tune the hyperparameters of the gaussian process regression algorithm I've implemented. I simply want to maximize the log marginal likelihood given by the formula $$\log(\mathbf{y}|X,\mathbf{\theta})=-\frac{1}{2}…
bfaskiplar
  • 632
19
votes
5 answers

What's in a name: hyperparameters

So in a normal distribution, we have two parameters: mean $\mu$ and variance $\sigma^2$. In the book Pattern Recognition and Machine Learning , there suddenly appears a hyperparameter $\lambda$ in the regularization terms of the error function. What…
cgo
  • 9,587
1
2 3
…
43 44