ALiBi: Attention with Linear Biases
Understand ALiBi, the position encoding method that adds linear biases to attention scores, enabling exceptional length extrapolation without position embeddings.
19 min readConcept
Tagged with
transformers
Explore machine learning concepts related to transformers. Clear explanations and practical insights.
18
Concepts Found
Concepts Related to transformers
MHA vs GQA vs MQA: Choosing the Right Attention
Compare Multi-Head, Grouped-Query, and Multi-Query Attention mechanisms to understand their trade-offs and choose the optimal approach for your use case.
9 min readConcept
Attention Sinks: Stable Streaming LLMs
Understand attention sinks, the phenomenon where LLMs concentrate attention on initial tokens, and how preserving them enables infinite-length streaming inference.
17 min readConcept
Cross-Attention: Bridging Different Modalities
Understand cross-attention, the mechanism that enables transformers to align and fuse information from different sources, sequences, or modalities.
15 min readConcept
Grouped-Query Attention (GQA)
Learn how Grouped-Query Attention balances the quality of Multi-Head Attention with the efficiency of Multi-Query Attention, enabling faster inference in large language models.
7 min readConcept
Linear Attention Approximations
Explore linear complexity attention mechanisms including Performer, Linformer, and other efficient transformers that scale to very long sequences.
6 min readConcept
Masked and Causal Attention
Learn how masked attention enables autoregressive generation and prevents information leakage in transformers, essential for language models and sequential generation.
7 min readConcept
Multi-Query Attention (MQA)
Understand Multi-Query Attention, the radical efficiency optimization that shares keys and values across all attention heads, enabling massive memory savings for inference.
7 min readConcept
Rotary Position Embeddings (RoPE)
Understand Rotary Position Embeddings, the elegant position encoding method that encodes relative positions through rotation matrices, used in LLaMA, GPT-NeoX, and most modern LLMs.
8 min readConcept
Scaled Dot-Product Attention
Master the fundamental building block of transformers - scaled dot-product attention. Learn why scaling is crucial and how the mechanism enables parallel computation.
6 min readConcept
Sliding Window Attention
Learn how Sliding Window Attention enables efficient processing of long sequences by limiting attention to local context windows, used in Mistral and Longformer.
14 min readConcept
Sparse Attention Patterns
Explore sparse attention mechanisms that reduce quadratic complexity to linear or sub-quadratic, enabling efficient processing of long sequences.
7 min readConcept
Prompt Influence Flow: How Instructions Propagate Through Model Layers
Deep dive into how different prompt components influence model behavior across transformer layers, from surface patterns to abstract reasoning.
6 min readConcept
Cross-Encoder vs Bi-Encoder
Understand the fundamental differences between independent and joint encoding architectures for neural retrieval systems.
7 min readConcept
Context Windows: The Memory Limits of LLMs
Interactive visualization of context window mechanisms in LLMs - sliding windows, expanding contexts, and attention patterns that define what models can "remember".
6 min readConcept
KV Cache: The Secret to Fast LLM Inference
Interactive visualization of key-value caching in LLMs - how caching transformer attention states enables efficient text generation without quadratic recomputation.
7 min readConcept
Tokenization: Converting Text to Numbers
Interactive exploration of tokenization methods in LLMs - BPE, SentencePiece, and WordPiece. Understand how text becomes tokens that models can process.
5 min readConcept
Mixture of Experts (MoE)
Understanding sparse mixture of experts models - architecture, routing mechanisms, load balancing, and efficient scaling strategies for large language models
6 min readConcept