Split the text into tokens
Text is cut into subword pieces from a fixed vocabulary. BERT-style encoders also add a [CLS] token at the start and a [SEP] token at the end.
- WordPiece
- [CLS]
- [SEP]
← All tools01 / Tokenize 02 / Embed 03 / Attend both ways 04 / Stack layers 05 / Pool 06 / Compare
Lab / Encoders
Inside a text encoder
An encoder reads a whole input at once and turns it into vectors: one per token, and one for the whole input once those are pooled. This toy uses six readable dimensions instead of hundreds of learned ones, so you can watch the numbers move.
Interactive field guide
Inside a text encoder
Follow a sentence from token lookup, through one layer of context, to a single vector you can compare.
01 / Embeddings
Every token starts as a row in a table.
02 / Bidirectional context
An encoder reads the whole sentence at once.
Before the layer
After one layer of attention
03 / Sentence embedding
Pool the tokens into one vector, then compare.
- the bank approved my loan
- she deposited cash at the bank
- we sat by the river bank
- fish swim near the river bank
- the dog chased a duck
- a cat ate the fish
sentencesentence
Field guide
Six steps from text to vector
The same pipeline sits behind BERT, sentence-embedding models and the text side of retrieval systems.
Look each token up in a table
Each token ID selects a learned row of numbers, and a position embedding is added so order is not lost. At this point “bank” has the same vector in every sentence.
- Token embedding
- Position embedding
Let every token read the whole sentence
Encoder self-attention has no causal mask, so a token can use words on both sides. That is how “bank” can lean towards rivers or money depending on what follows it. A decoder only looks left.
- Bidirectional
- No causal mask
Repeat attention and a feed-forward block
Each layer is attention followed by a small MLP, with residual connections around both. A real encoder stacks 6 to 24 of them; the toy in this tool runs one attention step.
- Residual
- Feed-forward
- Layer norm
Turn many vectors into one
To represent the whole sentence, take the [CLS] output or average the token outputs. Sentence-embedding models are fine-tuned so that this pooled vector places similar meanings close together.
- Mean pooling
- [CLS] pooling
Measure the angle between vectors
Cosine similarity compares the direction of two vectors and ignores their length. It is the usual score behind semantic search, clustering and retrieval.
- Cosine similarity
- Retrieval
- Clustering