ALiBi: Attention with Linear Biases
Understand ALiBi, the position encoding method that adds linear biases to attention scores, enabling exceptional length extrapolation without position embeddings.
19 min readConcept
Tagged with
position-encoding
Explore machine learning concepts related to position-encoding. Clear explanations and practical insights.
2
Concepts Found
Concepts Related to position-encoding
Rotary Position Embeddings (RoPE)
Understand Rotary Position Embeddings, the elegant position encoding method that encodes relative positions through rotation matrices, used in LLaMA, GPT-NeoX, and most modern LLMs.
8 min readConcept