Transformers are part of a deep learning model family using the attention mechanism to learn how to make prediction over sequential data without the need for recursion.
Questions tagged [transformers]
178 questions
24
votes
4 answers
What is the role of feed forward layer in Transformer Neural Network architecture?
What is the role and purpose of the fully connected layer after the attention layer in Transformer architecture ?
al pal
- 341
22
votes
1 answer
Why are residual connections needed in transformer architectures?
Residual connections are often motivated by the fact that very deep neural networks tend to "forget" some features of their input data-set samples during training. This problem is circumvented by summing the input x to the result of a typical…
Ramiro Hum-Sah
- 367
14
votes
4 answers
Why are the embeddings of tokens multiplied by $\sqrt D$ (note not divided by square root of D) in a transformer?
Why does the transformer tutorial in PyTorch have a multiplication by sqrt number of inputs? I know there is a division by sqrt(D) in the multiheaded self attention, but why is there something similar to with the output of the encoder? Especially…
Charlie Parker
- 7,538
14
votes
1 answer
When calculating self-attention for Transformer ML architectures, why do we need both a key and a query weight matrix?
I'm trying to understand the math behind Transformers, specifically self-attention. This link , and many others, gives the formula to compute the output vectors from the input embeddings as: $$Q=XW_Q,\;\;\;K=XW_K,\;\;\;V=XW_V$$…
itrase
- 143
13
votes
1 answer
Formula to compute approximate memory requirements of Transformer models
I would like to roughly estimate the memory requirement of training an arbitrary Transformer model $M$ , with $l$ layers, $h$ attention heads, an embedding dimension of $d$ , and an input dimension of $t$ tokens. What is the formula to compute this…
ClonedOne
- 243
13
votes
1 answer
Deciding between Decoder-only or Encoder-only Transformers (BERT, GPT)
I just started learning about transformers and looked into the following 3 variants The original one from Attention Is All You Need (Encoder & Decoder) BERT (Encoder only) GPT-2 (Decoder only) How does one generally decide whether their transformer…
Athena Wisdom
- 239
11
votes
2 answers
Why transformer in deep learning is called transformer?
Where does the name "transformer" come from in deep learning? I want to know more about the correlation between its name and its working principle.
Johnny Tam
- 383
9
votes
1 answer
What is the purpose of Add & Norm layers in Transformers?
According to the article Attention Is All You Need , each sublayer in each one of the encoder and decoder layers is wrapped by a residual connection followed by layer normalization ("Add & Norm" in the following figure): I more or less understand…
kodkod
- 223
9
votes
2 answers
Meaning of the value matrix in self-attention
I'm trying to understand how the self-attention mechanism of the transformer architecture (as proposed by Vaswani et al. ) works in detail. I get that self-attention is attention from a token of a sequence to the tokens of the same sequence. The…
Vale Boca
- 93
8
votes
1 answer
Why are Transformers "suboptimal" for language modeling but not for translation?
Language Models with Transformers states: Transformer architectures are suboptimal for language model itself. Neither self-attention nor the positional encoding in the Transformer is able to efficiently incorporate the word-level sequential context…
MWB
- 1,367
7
votes
1 answer
What advantage do sinusoidal positional encodings have over binary positional encodings in transformer LLMs?
I've recently come across an article that discusses the reasons why large language models use sinusoidal functions to generate positional encodings — as per the famous paper Attention Is All You Need (page 6) — as opposed to simpler, more intuitive…
Philip Voinea
- 165
- 1
- 8
7
votes
1 answer
Where is dropout placed in the original transformer?
I wanted to know where dropout was placed in the original transformer. According to the original paper Attention Is All You Need they say: Residual Dropout We apply dropout [27] to the output of each sub-layer, before it is added to the sub-layer…
Charlie Parker
- 7,538
6
votes
1 answer
How is padding masking considered in the Attention Head of a Transformer?
For purely educational purposes, my goal is to implement basic Transformer architecture from scratch. So far I focused on the encoder for classification tasks and assumed that all samples in a batch have the same length. This means, I didn't care…
Christian
- 219
6
votes
3 answers
Intuitive explanation for summing the embedding and positional encoding in the Transformer's embedding
In the Transformer model, the embedding and positional encoding are summed together to represent a word in each location ('positional embedding' from now on). This way, each cell contains semantic and positional information. However, since two types…
Emil
- 371
6
votes
2 answers
Why do large LMs use the transpose of the word embeddings matrix in the classification head?
All literature, guides and tutorials describing the construction of language models have used two separate matrices for the input and output projections: To project one-hot token IDs into hidden states, a matrix $U$ of size $\mathrm{VocabSize}…
Avelina
- 1,218