Probably would have been better to term it as linear transformations (dense layers) rather than…

…y the simplest part. It means that we’re applying this to… itself. Our vectors q and k are actually neural networks (typically linear) and if they have the same input (q(x), k(x)), then they are self attending (our …
Training Compact Transformers from Scratch in 30 Minutes with PyTorch
844
13
Steven Walton
Sayak Paul
·Follow
Jun 29, 2021
--
Probably would have been better to term it as linear transformations (dense layers) rather than neural networks.
--
--
Written by Sayak Paul839 Followers
·31 Following
ML at 🤗 | Netflix Nerd | Personal site: https://sayak.dev/
Responses (1)
Help
Status
About
Careers
Press
Blog
Privacy
Terms
Text to speech
Teams