Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?
Existing analyses of the expressive capacity of Transformer models have required excessively deep layers for data memorization, leading to a discrepancy with the Transformers actually used in practice. This is primarily due to the interpretation of the softmax function as an approximation of the har...
Guardado en:
| Autores principales: | , |
|---|---|
| Format: | Artículo |
| Udgivet: |
2023
|
| Fag: | |
| Online adgang: | https://arxiv.org/abs/2307.14023 http://arxiv.org/abs/2307.14023 |
| Tags: |
Ingen Tags, Vær først til at tagge denne postø!
|