Semantic Similarity Checker

What the number means

Cosine similarity runs from -1 to 1 and says how small the angle between two vectors is. With sentence embeddings the useful range is narrower, because two unrelated sentences do not point in opposite directions, they point in unrelated ones, and unrelated in 384 dimensions is close to perpendicular. That is why "The cat sat on the mat." against "Quarterly revenue rose by 12 percent." scores 0.003 rather than something negative.

The bands the tool prints are ours, read off the pairs below. They fit all-MiniLM-L6-v2 and no other model.

Two things stand out. The password row is the one the model is for. Two sentences that share two words out of thirteen and mean the same request land at 0.669. The router row is its mirror, five words out of seven in common and a different request, at 0.508. A word-overlap check ranks those two rows the wrong way round, and every keyword search does the same.

The second thing is the feline. A paraphrase that swaps every content word lands at 0.536, in the same band as the router. We expected higher. The model has seen "cat" next to "feline" often enough, but "rested on the rug" is far from "sat on the mat" in its training data, and a 22-million-parameter model does not have room for every synonym. all-mpnet-base-v2 puts the same pair at 0.687, and drops the negation pair to 0.879, so the bigger model is better on both counts without being right on either. The lesson is that a threshold of 0.8 will miss real paraphrases, and a threshold of 0.5 will accept the router. There is no setting that gets both, only a choice of which error you prefer.

From a sentence to 384 numbers

The model is a BERT-style transformer with six layers, fine-tuned by the sentence-transformers project on 1.17 billion sentence pairs with a contrastive objective. Given one sentence of a pair, it had to pick the partner out of a batch of a thousand candidates. Whatever geometry makes that task solvable is what the vectors encode. The full training recipe is on the model card of sentence-transformers/all-MiniLM-L6-v2.

Three steps happen between the text field and the score, and the tool draws the result of each.

1. The tokenizer lowercases the text and cuts it into WordPiece tokens from a 30,522-entry English vocabulary. Common words stay whole, rare ones split: "login" becomes "log" and "##in", "unbelievably" becomes "un", "##bel", "##ie", "##va", "##bly". A [CLS] token is added at the front and [SEP] at the end, which is why a six-word sentence shows eight or nine pieces.
2. The transformer turns every piece into a 384-dimensional vector, with the context of the whole sentence mixed in. "reset" in "reset my password" and "reset" in "reset my router" come out different.
3. The piece vectors are averaged into one sentence vector, mean pooling, and that vector is scaled to length 1. On unit vectors the cosine is the plain dot product, so the score is 384 multiplications and a sum. The third strip in the tool shows those 384 products. Where both vectors agree in sign the product is positive and pushes the score up, where they disagree it pulls down.

Because of the mean in step three, each value of the final vector is small, typically between -0.15 and 0.15, and no single dimension means anything on its own. Dimension 17 is not "about passwords". Meaning lives in the direction of the whole vector, which is why the strips of two similar sentences look alike as patterns without matching cell for cell.

Where the model is confidently wrong

Three rows in the table above carry the label "and wrong", and each one is a class of failure rather than a single bad pair.

Negation is the famous one. "Delete the user account" and "Do not delete the user account" score 0.935, because the embedding is a mean over eight token vectors and the two extra tokens move that mean very little. The model encodes what a sentence is about, and both sentences are about deleting an account. The same holds for "I love this product" against "I hate this product" at 0.692, and for any pair where the meaning hinges on one word. Nothing built on a bag of token vectors will catch this, and a cross-encoder or a natural-language-inference model is the tool for contradiction.

Word order is the second. "The cat sat on the mat." and "The mat sat on the cat." score 0.966. Position embeddings exist in the model, but for sentence similarity the training signal rarely rewarded telling these apart, so it did not learn to.

Language is the third. The German sentence and its English translation score 0.140, below "Paris" against "Berlin". The vocabulary is English, so "Katze" is cut into pieces the model has only seen in English words, and the pieces mean nothing. A multilingual model puts that pair above 0.8.

There is a quieter fourth. "Python is a programming language." against "Pythons are large snakes." scores 0.524, comfortably in the related band, because the surface form dominates when the sentences are short. Short inputs have few tokens to average, and the shared rare word weighs more than the difference in meaning. The model works best on full sentences with some context, worse on two-word titles and single keywords.

Finding duplicates in a list

Switch the tool to one per line and every pair of lines is scored, which is n·(n-1)/2 cosines. Fourteen support tickets are 91 pairs, five hundred lines are 124,750, and the arithmetic is the cheap part. The embedding is what costs, one model call per line, and the page caches every vector, so pasting a longer version of the same list only embeds the new lines.

On the fourteen sample tickets the model pairs "Cannot log in after resetting my password" with "I reset my password and now I cannot sign in" at 0.89, and the two VAT complaints at the same score. The two settings crashes come in at 0.78 and the two CSV export requests at 0.77, both under a threshold of 0.80 and both real duplicates. Lower the threshold to 0.75 and they appear, and so does nothing false, on this list. The one it misses outright is the pair about the cancelled subscription and the repeated charge, at 0.44, the same complaint in two vocabularies. That is the class of miss to expect from a 23 MB model, and the reason to read the matrix rather than only the list.

The full matrix is drawn up to 40 lines and the hover shows every value, which is faster for finding a threshold than any formula. Past a few thousand items the full matrix stops being practical, and that is where approximate nearest-neighbour indexes such as HNSW take over, in FAISS, pgvector or a hosted vector database. They return the top matches for each item without scoring every pair, at the cost of occasionally missing one.

Every pair with its score can be downloaded as CSV, sorted from most to least similar.

The same score in Python and JavaScript

sentence-transformers is the reference. The number it prints for the negation pair is the fp32 counterpart of the 0.935 the tool shows.

$ python -c "from sentence_transformers import SentenceTransformer, util; m = SentenceTransformer('all-MiniLM-L6-v2'); a, b = m.encode(['Delete the user account', 'Do not delete the user account']); print(util.cos_sim(a, b).item())"
0.9458737373352051

The JavaScript version uses the same ONNX export the page runs, through transformers.js, in Node or in the browser:

import { pipeline } from '@huggingface/transformers';

const embed = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2', { dtype: 'q8' });
const [a, b] = await Promise.all(['Delete the user account', 'Do not delete the user account']
	.map((t) => embed(t, { pooling: 'mean', normalize: true })));
const cosine = a.data.reduce((sum, x, i) => sum + x * b.data[i], 0);

Two details matter. The pooling and normalize options are what turn token vectors into a sentence vector, and leaving them off returns a matrix of token vectors instead. And dtype q8 picks the 23 MB file, while the default in Node is the 90 MB fp32 model.

For numpy without a framework, cosine is np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)), and for a whole list at once util.paraphrase_mining(model, sentences) returns every pair above zero, sorted.

Which embedding model

all-MiniLM-L6-v2 is on this page because it is the smallest model that produces usable sentence vectors, and small is the only way a model runs in a tab without a wait. It is not the best model, and the sentence-transformers authors say so themselves. Their default recommendation for English quality is all-mpnet-base-v2, twelve layers and 768 dimensions at 438 MB, which scores a few points higher on their benchmarks and takes about five times as long per sentence.

The token limit is the other axis. MiniLM-L6 reads 256 word pieces and was trained on 128, and paraphrase-multilingual-MiniLM-L12-v2 stops at 128 outright. bge-small-en-v1.5 and multilingual-e5-small read 512 at the same 384 dimensions, which makes them the better choice for paragraphs. The e5 models want a "query: " or "passage: " prefix in front of every text and lose quality without it.

Hosted embeddings from OpenAI, Cohere or Voyage produce longer vectors, 1024 to 3072 dimensions, from far larger models, and they are better on every benchmark that matters. They also mean every text leaves your machine, which is the trade this page exists to avoid. We take the small local model for duplicate detection on our own data and for anything under NDA, and it costs us the last few benchmark points. The cases it gets wrong are the cases listed above, and no model of any size gets those right by embedding alone.

One rule holds across all of them. Scores from two models are not on the same scale, so a threshold tuned for one is a random number for another.

Vectors, scores and thresholds