word2vec-style vector arithmetic on docs embeddings

2025 October 29

word2vec popularized the idea of representing words as vectors where semantically similar words are positioned close to each other in the vector space. Nowadays these vectors are usually called embeddings.

A neat consequence of the word2vec approach is that adding and subtracting vectors produces semantically logical results. From Efficient Estimations of Word Representations in Vector Space (the word2vec paper):

Using a word offset technique where simple algebraic operations are performed on the word vectors, it was shown for example that vector("King") - vector("Man") + vector("Woman") results in a vector that is closest to the vector representation of the word Queen.

Does word2vec-style vector arithmetic work in technical writing contexts?

Experiments§

word2vec was published in 2013. Embedding models have come a long way since then. word2vec models could only operate on single words. A vector always represented a single word. Modern embedding models can operate on arbitrary text. A vector can now represent a word, paragraph, section, document, set of documents, etc.

My experiments follow the same basic pattern of vector("King") - vector("Man") + vector("Woman"), with one difference. The experiments start out with a vector representing the full text of a document, not a single-word vector.

Same topic, different domain§

This is the first experiment. Starting with the vector for the full text of Testing Your Database from the Supabase docs, subtract the vector for the word supabase, and then add the vector for the word angular. The resultant vector should be semantically close to the concept of “testing in Angular”.

Different topic, same domain§

This is the second experiment. Starting with the vector for the full text of Testing Your Database from the Supabase docs, subtract the vector for the word testing, and then add the vector for the word vectors. The resultant vector should be semantically close to the concept of “vectors in Supabase”.

Task types§

From previous research I’ve learned that task types noticeably affect Gemini Embedding’s outputs. EmbeddingGemma (the model I’ll be using in the experiments) also supports tasks types. I’ll run both experiments twice: once with default task types, and again with customized task types.

Verification§

There’s no way to directly verify that the resultant vectors are semantically close to the expected concepts. What I can do instead is generate vectors from the full texts of various docs, and then compare the resultant vectors from the experiments against the vectors of these various docs using cosine similarity.

Here’s the full list of docs that I use in the experiments:

• Background Processing Using Web Workers (Angular)

• Refer To Locales By ID (Angular)

• Testing (Angular)

• Testing Services (Angular)

• LINESTRING (CockroachDB)

• Test Your Application Locally (CockroachDB)

• analysis_test (Skylib)

• bzl_library (Skylib)

• diff_test (Skylib)

• Actionability (Playwright)

• JUnit (Playwright)

• Writing Tests (Playwright)

• Branching (Supabase)

• Testing Your Database (Supabase)

• Testing Your Edge Functions (Supabase)

• Vector Columns (Supabase)

For the Same topic, different domain experiment (Testing Your Database - supabase + angular) I expect the resultant vector to be most similar to Testing or Testing Services from the Angular docs. And for the Different topic, same domain experiment (Testing Your Database - testing + vectors) I expect the resultant vector to be most similar to Vector Columns from the Supabase docs.

Note that I picked short docs because EmbeddingGemma only supports 2048 tokens of input and I didn’t feel like dealing with chunking. Most of the docs revolve around testing.

Results§

In the Same topic, different domain experiment (Testing Your Database - supabase + angular) the resultant vector is most similar to Testing and Testing Services from the Angular docs, as expected, when custom task types are enabled:

When using the default task types, the resultant vector is most similar to Testing Your Database i.e. the doc that the experiment started with:

In the Different topic, same domain experiment (Testing Your Database - testing + vectors) the resultant vector is most similar to Vector Columns, regardless of whether default or custom task types were used.

Custom task types:

Default task types:

So, yes, it seems like word2vec-style vector arithmetic can work in technical writing contexts. Make sure to set your task types correctly.

Discussion§

I still don’t really understand how it’s possible to semantically represent an entire document as a single vector, let alone how adding and subtracting single-word vectors from full-document vectors works.

How do we actually use this in technical writing workflows or documentation experiences? I’m not sure. I was just curious to learn whether or not it would work.

Appendix§

Source code§

experiments.py:

data.json:

Note that I forgot to pin the URLs to specific commits. I.e. I used the HEAD version of each URL. If you run the experiments a year or two from now (October 2025), your cosine similarity scores will probably be different, because the underlying text of the documents will probably have changed.