AI
11 min read
A practical explanation of context windows, model parameters, attention, KV cache memory, and why a million-token prompt is useful but not the same as perfect memory.
Highlights
- • Tokenization and embeddings
- • Self-attention, KV cache, positional encoding, and context caching
- • Dense versus mixture-of-experts parameter usage
- • Long-context limits and retrieval trade-offs
#LLM#Transformers#Long Context#AI Systems