Questions tagged [neural-networks]

Artificial neural networks (ANNs) are a broad class of computational models loosely based on biological neural networks. They encompass feedforward NNs (including "deep" NNs), convolutional NNs, recurrent NNs, etc.

Artificial neural networks (ANN), are composed of neurons --- programming constructs that mimic the properties of biological neurons. A set of weighted connections between the neurons allows information to propagate through the network to adapt to specific tasks automatically, without the designer hand-crafting inputs and features.

Neural networks have a wide variety of uses and flavors. Some prominent artificial neural network types are , , (e.g. ). Modern feedforward neural networks are often comprised of many "layers", and often termed "deep networks" (a.k.a. "deep learning"), in contrast to so-called "shallow" networks comprised of a single hidden layer. Sometimes neural networks are used to train computers to learn strategies, e.g. in chess, Go, or video games; these models use approaches, e.g. .

Among freely available, high-quality resources on neural networks, the Deep Learning book by Ian Goodfellow, Yoshua Bengio, and Aaron Courville can be mentioned.

8123 questions
831
votes
10 answers

How to choose the number of hidden layers and nodes in a feedforward neural network?

Is there a standard and accepted method for selecting the number of layers, and the number of nodes in each layer, in a feed-forward neural network? I'm interested in automated ways of building neural networks.
Rob Hyndman
  • 60,133
395
votes
5 answers

What is the trade-off between batch size and number of iterations to train a neural network?

When training a neural network, what difference does it make to set: batch size to $a$ and number of iterations to $b$ vs. batch size to $c$ and number of iterations to $d$ where $ ab = cd $? To put it otherwise, assuming that we train the neural…
Franck Dernoncourt
  • 49,212
  • 35
  • 186
  • 308
377
votes
9 answers

What should I do when my neural network doesn't learn?

I'm training a neural network but the training loss doesn't decrease. How can I fix this? I'm not asking about overfitting or regularization. I'm asking about how to solve the problem where my network's performance doesn't improve on the training…
Sycorax
  • 97,234
331
votes
11 answers

What exactly are keys, queries, and values in attention mechanisms?

How should one understand the keys, queries, and values that are often mentioned in attention mechanisms? I've tried searching online, but all the resources I find only speak of them as if the reader already knows what they are. Judging by the paper…
307
votes
6 answers

What is batch size in neural network?

I'm using Python Keras package for neural network. This is the link . Is batch_size equals to number of test samples? From Wikipedia we have this information: However, in other cases, evaluating the sum-gradient may require expensive evaluations of…
user2991243
  • 4,301
240
votes
9 answers

What are the advantages of ReLU over sigmoid function in deep neural networks?

The state of the art of non-linearity is to use rectified linear units (ReLU) instead of sigmoid function in deep neural network. What are the advantages? I know that training a network when ReLU is used would be faster, and it is more biological…
217
votes
8 answers

What does 1x1 convolution mean in a neural network?

I am currently doing the Udacity Deep Learning Tutorial. In Lesson 3, they talk about a 1x1 convolution. This 1x1 convolution is used in Google Inception Module. I'm having trouble understanding what is a 1x1 convolution. I have also seen this post…
213
votes
4 answers

What does the hidden layer in a neural network compute?

I'm sure many people will respond with links to 'let me google that for you', so I want to say that I've tried to figure this out so please forgive my lack of understanding here, but I cannot figure out how the practical implementation of a neural…
187
votes
2 answers

A list of cost functions used in neural networks, alongside applications

What are common cost functions used in evaluating the performance of neural networks? Details (feel free to skip the rest of this question, my intent here is simply to provide clarification on notation that answers may use to help them be more…
Phylliida
  • 2,955
179
votes
11 answers

What is the difference between a neural network and a deep neural network, and why do the deep ones work better?

I haven't seen the question stated precisely in these terms, and this is why I make a new question. What I am interested in knowing is not the definition of a neural network, but understanding the actual difference with a deep neural network. For…
Nicolas
  • 2,001
172
votes
7 answers

Where should I place dropout layers in a neural network?

Is there any general guidelines on where to place dropout layers in a neural network?
Franck Dernoncourt
  • 49,212
  • 35
  • 186
  • 308
156
votes
6 answers

Why are neural networks becoming deeper, but not wider?

In recent years, convolutional neural networks (or perhaps deep neural networks in general) have become deeper and deeper, with state-of-the-art networks going from 7 layers ( AlexNet ) to 1000 layers ( Residual Nets) in the space of 4 years. The…
149
votes
4 answers

Difference between neural net weight decay and learning rate

In the context of neural networks, what is the difference between the learning rate and weight decay?
Ryan Zotti
  • 6,837
144
votes
6 answers

How is it possible that validation loss is increasing while validation accuracy is increasing as well

I am training a simple neural network on the CIFAR10 dataset. After some time, validation loss started to increase, whereas validation accuracy is also increasing. The test loss and test accuracy continue to improve. How is this possible? It seems…
142
votes
4 answers

What is the difference between convolutional neural networks, restricted Boltzmann machines, and auto-encoders?

Recently I have been reading about deep learning and I am confused about the terms (or say technologies). What is the difference between Convolutional neural networks (CNN), Restricted Boltzmann machines (RBM) and Auto-encoders?
1
2 3
…
99 100