IA-010 / 2023
AttentionViz is a scatter plotA graph where each point represents a pair of values, showing the relationship between two sets of numbers on a coordinate plane.A graph of paired numerical values, with one variable on the horizontal axis and the corresponding value of a second variable on the vertical axis, used to reveal relationships or association between the variables.glossary showing attentionHow the model decides which earlier words matter when it is working on the current one.Which positions a model attends to, and how strongly.glossary, about attention-headsThe parts that decide which earlier words matter for the next one.What an attention head does, on a chosen task or across the full data distribution.glossary.
- ID
- IA-010
- Name
- AttentionViz
- Artefact type
- Interactive tool
- Visual form
- Scatter plotA graph where each point represents a pair of values, showing the relationship between two sets of numbers on a coordinate plane.A graph of paired numerical values, with one variable on the horizontal axis and the corresponding value of a second variable on the vertical axis, used to reveal relationships or association between the variables.glossary
- Makes visible
- AttentionHow the model decides which earlier words matter when it is working on the current one.Which positions a model attends to, and how strongly.glossary
- Role of the visual
- Instrument
- Runnable by a visitor
- Unknown
- Date published
- 2023-05-04
- Year
- 2023
- Link status
- Live
- Built by (people)
- Catherine Yeh; Yida Chen; Aoyu Wu; Cynthia Chen; Fernanda Viégas; Martin Wattenberg
- Organisation
- Harvard University
- Venue / published in
- IEEE VIS / TVCG 2023; arXiv
- Sector
- Academia
- Country / region
- US
- Open source
- Unknown
- Method / technique
- Joint embedding of the query and key vectors used to compute attention, projected to two dimensions across many input sequences
- Model(s) studied
- Language and vision transformers (specific models not named in the abstract)
- Access required
- Open weights
- What it visualises
- Global attention patterns across many sequences at once — query and key vectors in a shared space, so recurring head behaviours appear as spatial structure rather than per-prompt heatmaps
- Interaction affordances
- Browse heads; switch model and layer; zoom into a single head's query-key space
- Pragmatic vs Basic science
- Basic science
- Reverse-eng vs Concept-based
- Reverse-engineering
- Observational vs Interventional
- Observational
- Intended audience
- Researchers
- Description (card)
- Visualises a joint embedding of the query and key vectors a transformer uses to compute attention, which makes it possible to see global patterns across many input sequences rather than one prompt at a time.
- Why it matters
- Shifted attention visualisation from single-example to distribution-level — the exact move Anthropic's HeadVis names as its closest prior work three years later.
- Visual / design notes
- The joint query-key scatter is the whole idea: spatial proximity encodes attention affinity, so a head's behaviour becomes visible geometry.
- Tags
- modality:languageText. Models that read and write words.Models that generate or process text.glossarymodality:visionImages. Models that look at pictures.Models that process static images.glossarymethod:attention-visualisationShowing which words the model looked at, and how hard.Displaying which positions a model attends to, and how strongly.glossarymethod:dimensionality-reductionFlattening very high-dimensional data down to two or three dimensions so it can be drawn.Projecting high-dimensional activations into two or three dimensions so their structure can be seen.glossaryphenomenon:attention-headsThe parts that decide which earlier words matter for the next one.What an attention head does, on a chosen task or across the full data distribution.glossary
- Citation
- Yeh, C., Chen, Y., Wu, A., Chen, C., Viégas, F. and Wattenberg, M., 2023. AttentionViz: A Global View of Transformer Attention. arXiv:2305.03210
- Related entries
- IA-005; IA-011
- Confidence
- High
- Source of info
- Read the arXiv abstract page in full, 2026-08-06. The live demo at attentionviz.com was not inspected — client-rendered, fetch returned an empty page shell.
- Date added
- 2026-08-06
- Added by
- Claude
- Notes
- HeadVis (IA-005) explicitly names this as its closest prior work — a documented lineage between two rows in this ledger.