CLS Token in Vision Transformers
Learn how the CLS token acts as a global information aggregator in Vision Transformers, enabling whole-image classification through attention mechanisms.
8 min readConcept
Tagged with
attention
Explore machine learning concepts related to attention. Clear explanations and practical insights.
21
Concepts Found
Concepts Related to attention
Hierarchical Attention in Vision Transformers
Explore how hierarchical attention enables Vision Transformers (ViT) to process sequential data by encoding relative positions.
6 min readConcept
Multi-Head Attention in Vision Transformers
Explore how multi-head attention enables Vision Transformers (ViT) to process sequential data by encoding relative positions.
6 min readConcept
Positional Embeddings in Vision Transformers
Explore how positional embeddings enable Vision Transformers (ViT) to process sequential data by encoding relative positions.
5 min readConcept
Interactive Look: Self-Attention in Vision Transformers
Interactively explore how self-attention allows Vision Transformers (ViT) to understand images by capturing global context. Click, explore, and see how it differs from CNNs.
6 min readConcept
ALiBi: Attention with Linear Biases
Understand ALiBi, the position encoding method that adds linear biases to attention scores, enabling exceptional length extrapolation without position embeddings.
19 min readConcept
MHA vs GQA vs MQA: Choosing the Right Attention
Compare Multi-Head, Grouped-Query, and Multi-Query Attention mechanisms to understand their trade-offs and choose the optimal approach for your use case.
9 min readConcept
Attention Sinks: Stable Streaming LLMs
Understand attention sinks, the phenomenon where LLMs concentrate attention on initial tokens, and how preserving them enables infinite-length streaming inference.
17 min readConcept
Cross-Attention: Bridging Different Modalities
Understand cross-attention, the mechanism that enables transformers to align and fuse information from different sources, sequences, or modalities.
15 min readConcept
Grouped-Query Attention (GQA)
Learn how Grouped-Query Attention balances the quality of Multi-Head Attention with the efficiency of Multi-Query Attention, enabling faster inference in large language models.
7 min readConcept
Linear Attention Approximations
Explore linear complexity attention mechanisms including Performer, Linformer, and other efficient transformers that scale to very long sequences.
6 min readConcept
Masked and Causal Attention
Learn how masked attention enables autoregressive generation and prevents information leakage in transformers, essential for language models and sequential generation.
7 min readConcept
Multi-Query Attention (MQA)
Understand Multi-Query Attention, the radical efficiency optimization that shares keys and values across all attention heads, enabling massive memory savings for inference.
7 min readConcept
Rotary Position Embeddings (RoPE)
Understand Rotary Position Embeddings, the elegant position encoding method that encodes relative positions through rotation matrices, used in LLaMA, GPT-NeoX, and most modern LLMs.
8 min readConcept
Scaled Dot-Product Attention
Master the fundamental building block of transformers - scaled dot-product attention. Learn why scaling is crucial and how the mechanism enables parallel computation.
6 min readConcept
Sliding Window Attention
Learn how Sliding Window Attention enables efficient processing of long sequences by limiting attention to local context windows, used in Mistral and Longformer.
14 min readConcept
Sparse Attention Patterns
Explore sparse attention mechanisms that reduce quadratic complexity to linear or sub-quadratic, enabling efficient processing of long sequences.
7 min readConcept
Adaptive Tiling: Efficient Visual Token Generation
Understanding adaptive tiling in vision transformers - a technique that dynamically adjusts image partitioning based on complexity to optimize token usage while preserving detail.
7 min readConcept
Prompt Influence Flow: How Instructions Propagate Through Model Layers
Deep dive into how different prompt components influence model behavior across transformer layers, from surface patterns to abstract reasoning.
6 min readConcept
Context Windows: The Memory Limits of LLMs
Interactive visualization of context window mechanisms in LLMs - sliding windows, expanding contexts, and attention patterns that define what models can "remember".
6 min readConcept
Flash Attention: IO-Aware Exact Attention
Interactive visualization of Flash Attention - the breakthrough algorithm that makes attention memory-efficient through tiling, recomputation, and kernel fusion.
7 min readConcept