What are embeddings? Search policies by meaning, not keywords

What are embeddings? They turn a piece of text into a list of numbers (a vector) so that two texts with similar meaning get similar numbers. That lets your code find "refund" when a customer writes "compensate me", even though the two phrases share no words. Want a chatbot that understands what customers say instead of counting words? Let's build it.

Why isn't keyword search enough?

In the previous post, the chatbot for Mộc (a small Vietnamese coffee shop that sells online; the shop and its data in this series are made up) learned to read policies the RAG way. It split the policy into 14 sections, found the few that matched a question by counting shared words, and sent only those to the model. That works well when customers use the document's own words. It breaks when customers talk the way they talk: "compensate me" instead of "refund", "money back" instead of "refunded to the same payment method".

I don't want to conclude anything from a few questions I already knew would fail. So before changing anything, I wrote 12 questions in the style customers tend to use, and for each one I wrote down which policy sections must be present for the answer to be possible. For example, "I cancelled my bank transfer order, how long until the money is back?" needs both the Cancelling an order section and the Refunds section, because the number "three working days" lives in the latter. I'll rerun this set every time I change how the search works.

I should say something up front. Twelve questions is very few, and I wrote them myself after reading the policy, so the numbers below are a hint, not a rigorous measurement. Even so, having a set of questions beats guessing by feel.

What is an embedding, and how do you measure similarity?

An embedding model takes a piece of text and returns a vector, here 384 numbers. The model is trained so that two texts with the same idea get nearby vectors, and two texts with different ideas get distant ones. You don't need to understand what those 384 numbers mean, and in fact nobody understands any single one. You only need to compare two vectors and see how close they are.

The most common way to compare is cosine similarity, which measures the angle between two vectors. The closer the value is to 1, the more the two vectors point the same way, which means the two texts are closer in meaning. If the vectors are normalized to length 1, as I'll do below, cosine similarity is just the dot product of the two vectors, so one loop of multiply-and-add is enough.

First I want you to see what a vector looks like. The file below embeds three sentences and prints the similarity between each pair. The first two sentences have the same idea (a customer wanting money back for a badly made drink, and the refund policy), and the third is about something else entirely (discount codes):

// Three example sentences to see how an embedding measures "similar meaning". No policy needed, no language model called.
import { similarity, embedQuestion } from "./embed.ts";

const SENTENCES = [
  "My coffee was made wrong, I want the shop to compensate me",
  "The shop issues a refund within three working days",
  "A discount code can't be combined with loyalty points",
];

const vectors = await Promise.all(SENTENCES.map((sentence) => embedQuestion(sentence)));
console.log(`Each sentence becomes a vector of ${vectors[0].length} numbers. First three numbers of sentence 1: ${vectors[0].slice(0, 3).map((x) => x.toFixed(4)).join(", ")}\n`);
for (const [a, b] of [[0, 1], [0, 2], [1, 2]]) {
  console.log(`${similarity(vectors[a], vectors[b]).toFixed(3)}  sentence ${a + 1} vs sentence ${b + 1}`);
}

The two functions embedQuestion and similarity live in the file embed.ts, which I'll write next. Here's what running it prints:

Each sentence becomes a vector of 384 numbers. First three numbers of sentence 1: 0.0374, -0.0205, -0.0485

0.863  sentence 1 vs sentence 2
0.846  sentence 1 vs sentence 3
0.820  sentence 2 vs sentence 3

Look at the numbers. Sentence 1 versus sentence 2 scores 0.863, higher than the 0.846 and 0.820 of the other two pairs, so the pair with the same idea comes first, as expected. But the gap is surprisingly small: the "unrelated" pair still scores 0.85. That's a trait of the e5 model family I use, whose scores tend to bunch between about 0.7 and 0.9. So don't read 0.86 as "86% similar". The absolute number means almost nothing; only the ranking is trustworthy: which section is above which.

Running an embedding model on your own machine

Anthropic doesn't offer its own embedding model. The Embeddings page in the Claude documentation says so directly and suggests Voyage AI if you want an API service. For this post I chose another way, running a small model right on your machine: it's free, needs no key, and you can rerun it as often as you like without spending a cent. The model I use is multilingual-e5-small, which supports many languages, converted to a format that the @xenova/transformers library can run in JavaScript.

Create a new folder, put policy.md and the three files split-sections.ts, find-sections.ts and answer.ts from the previous post into src/ (they don't change), add a package.json containing "type": "module", then install two libraries:

npm install --save-exact @anthropic-ai/sdk@1.5.0 @xenova/transformers@2.17.2

I pin the exact versions because every result in this post comes from them. The @anthropic-ai/sdk library is only needed because answer.ts imports it. The first new file is src/embed.ts:

import { pipeline } from "@xenova/transformers";

import type { Section } from "./split-sections.ts";

// A small embedding model (384 dimensions, about 120 MB) that runs on your own machine and understands many languages.
// The first run downloads it once and stores it in the library's cache folder.
const MODEL_NAME = "Xenova/multilingual-e5-small";

let embedder: Awaited<ReturnType<typeof pipeline>> | undefined;

async function embed(texts: string[]): Promise<number[][]> {
  embedder ??= await pipeline("feature-extraction", MODEL_NAME);
  const output = await embedder(texts, { pooling: "mean", normalize: true });
  return output.tolist() as number[][];
}

// The e5 model family is trained with prefixes: "query: " for questions and "passage: " for document sections.
export async function embedQuestion(question: string): Promise<number[]> {
  return (await embed([`query: ${question}`]))[0];
}

// By default only the body of each section is embedded. Set WITH_TITLE=1 to embed the heading too (see the comparison in the post).
export async function embedSections(sections: Section[]): Promise<number[][]> {
  const withTitle = process.env.WITH_TITLE === "1";
  return embed(sections.map((s) => `passage: ${withTitle ? `${s.title}. ` : ""}${s.body}`));
}

// Vectors are normalized to length 1, so the dot product is exactly the similarity (cosine similarity).
export function similarity(a: number[], b: number[]): number {
  return a.reduce((total, x, i) => total + x * b[i], 0);
}

Three details are worth knowing. First, pooling: "mean" combines the vectors of the individual words into one vector for the whole text, and normalize: true scales the length to 1, which is why similarity needs only a dot product. Second, the prefixes query: and passage: aren't decoration. According to the model page for intfloat/multilingual-e5-small, the model was trained with these prefixes, and leaving them out lowers quality. I haven't measured by how much, so I won't quote a number. Third, in try-meaning.ts above I used query: for all three sentences, because there I'm comparing sentence with sentence, not a question with a document.

On the first run, the library downloads the model, about 120 MB, so you need a network connection exactly once. After that it reads from the cache. With Node.js 22.18 or later, you run the .ts files directly.

Finding policy sections by meaning

With the embedding function in place, searching by meaning comes down to three jobs. Embed each policy section once at startup and keep the result, which is the "index". When a question arrives, embed the question. Then rank the sections by similarity to the question and take the first few:

import { embedQuestion, embedSections, similarity } from "./embed.ts";
import type { Section } from "./split-sections.ts";

export type MeaningResult = { section: Section; score: number };

// The index: the vector of each section, computed once at startup.
export async function buildIndex(sections: Section[]): Promise<number[][]> {
  return embedSections(sections);
}

// Search by meaning: the section whose vector is closest to the question's vector comes first.
export async function findByMeaning(question: string, sections: Section[], index: number[][], limit = 2): Promise<MeaningResult[]> {
  const questionVector = await embedQuestion(question);
  return sections
    .map((section, i) => ({ section, score: similarity(questionVector, index[i]) }))
    .sort((a, b) => b.score - a.score) // sort by the unrounded score: the sections score so close that rounding first would create ties
    .slice(0, limit);
}

The comment on the sort line is a practical warning. Scores here sit very close together. For the question "I brought my own thermos, do I get anything off?", the top three sections score 0.832, 0.831 and 0.831: three sections within 0.001 of each other. If you round to three decimals before ranking, two of them tie, and the order then depends on how sort arranges equal elements. So round only when printing, never when ranking.

Is searching by meaning really better?

This is where the 12 sample questions come in. Each question has a needed list of sections that must be present, and this is the whole set, which you can copy as is:

// The set of questions used to grade the search. `needed` lists the policy sections that must be present for the answer to be possible.
export type SampleQuestion = { question: string; needed: string[] };

export const SAMPLE_QUESTIONS: SampleQuestion[] = [
  { question: "From what amount is shipping free?", needed: ["Delivery fee"] },
  { question: "If you bring me the wrong drink, how long until I get my money back?", needed: ["Replacing a wrong or badly made item", "Refunds"] },
  { question: "I cancelled my bank transfer order, how long until the money is back?", needed: ["Cancelling an order", "Refunds"] },
  { question: "My coffee was made wrong and I want the shop to compensate me", needed: ["Replacing a wrong or badly made item"] },
  { question: "I'm allergic to milk, what can I drink?", needed: ["Allergies and ingredients"] },
  { question: "Are you guys open during Tet?", needed: ["Opening hours and holidays"] },
  { question: "I live 8 km from the shop, do you deliver?", needed: ["Delivery fee"] },
  { question: "I brought my own thermos, do I get anything off?", needed: ["Bring your own cup"] },
  { question: "How long do my reward points last?", needed: ["Loyalty points"] },
  { question: "I want to order 30 coffees for my company, how does that work?", needed: ["Bulk orders"] },
  { question: "Does the shop pass my phone number on to anyone else?", needed: ["Customer privacy"] },
  { question: "Can I stack a promo code with my loyalty points?", needed: ["Promotions and discount codes"] },
];

The questions are phrased the way customers write, not the way the policy is written. The policy says "free delivery", "refunds", "Bring your own cup", "Loyalty points" and "Bulk orders"; the customers say "shipping free", "money back", "thermos", "reward points" and "30 coffees". The grading file runs both kinds of search on this set: keyword (findSections from the previous post) and by meaning (findByMeaning). It takes the number of sections to choose as an argument:

// Grades two searches on the same sample questions: keyword (post 6) and by meaning (embeddings).
// A question is "enough" when every section in `needed` is among the sections chosen. First argument: how many sections to choose (default 2).
import { findSections } from "./find-sections.ts";
import { buildIndex, findByMeaning } from "./find-by-meaning.ts";
import { SAMPLE_QUESTIONS } from "./sample-questions.ts";
import { readPolicy } from "./split-sections.ts";

const LIMIT = Number(process.argv[2] ?? 2);
const SECTIONS = readPolicy();
const index = await buildIndex(SECTIONS);

const mark = (enough: boolean) => (enough ? "enough" : "MISSING");
let keywordEnough = 0;
let meaningEnough = 0;

console.log(`Choosing ${LIMIT} sections per question.   keyword | by meaning`);
for (const { question, needed } of SAMPLE_QUESTIONS) {
  const keyword = findSections(question, SECTIONS, LIMIT).map((result) => result.section.title);
  const meaning = (await findByMeaning(question, SECTIONS, index, LIMIT)).map((result) => result.section.title);
  const enough = (chosen: string[]) => needed.every((title) => chosen.includes(title));
  keywordEnough += Number(enough(keyword));
  meaningEnough += Number(enough(meaning));
  console.log(`${mark(enough(keyword)).padEnd(7)} | ${mark(enough(meaning)).padEnd(7)}   ${question}`);
}
console.log(`
Enough information: keyword ${keywordEnough}/${SAMPLE_QUESTIONS.length}, by meaning ${meaningEnough}/${SAMPLE_QUESTIONS.length}`);

Running it with two sections chosen per question, as in the previous post:

Choosing 2 sections per question.   keyword | by meaning
enough  | enough    From what amount is shipping free?
MISSING | enough    If you bring me the wrong drink, how long until I get my money back?
MISSING | enough    I cancelled my bank transfer order, how long until the money is back?
enough  | enough    My coffee was made wrong and I want the shop to compensate me
enough  | enough    I'm allergic to milk, what can I drink?
enough  | enough    Are you guys open during Tet?
enough  | enough    I live 8 km from the shop, do you deliver?
enough  | MISSING   I brought my own thermos, do I get anything off?
enough  | enough    How long do my reward points last?
MISSING | enough    I want to order 30 coffees for my company, how does that work?
enough  | enough    Does the shop pass my phone number on to anyone else?
enough  | enough    Can I stack a promo code with my loyalty points?

Enough information: keyword 9/12, by meaning 11/12

Keyword search has enough information in 9 of 12 questions, search by meaning in 11. Look at the third question, the "money is back" one that left the previous post without the three-working-days figure: search by meaning gets both Cancelling an order and Refunds. And "30 coffees for my company" is the kind of question where the customer uses none of the document's words, which is exactly where word counting gives up. Search by meaning isn't perfect either: the thermos question, where the customer never says "cup", is missed with two sections.

I also tried three sections instead of two (node src/compare.ts 3). Keyword search improves from 9 to 10 out of 12, because it gets one more chance to hit, and search by meaning reaches 12 out of 12. The question about getting the wrong drink delivered, and the one about 30 coffees, are still ones that keyword search never catches, even with three sections. The price of taking more sections is a longer prompt and more noise, so I wouldn't use it to patch up a weak search.

Should you embed the section title too?

When I wrote embedSections, I assumed that since each section has a title, it's worth including, because a title like "Refunds" tells the model what the section is about. I measured it with the WITH_TITLE environment variable:

Enough information: keyword 9/12, by meaning 12/12
Enough information: keyword 10/12, by meaning 12/12

Embedding the title as well lifts search by meaning from 11 to 12 out of 12 when two sections are chosen, and the thermos question is now found. With three sections both versions score 12 out of 12. I don't know for sure why it helps, and I haven't verified my guess: a title like "Bring your own cup" gives the section's embedding a short, strong anchor that matches the customer's "my own thermos". Here is a result worth noting: in the Vietnamese edition of this post, the same experiment went the other way, and including the titles lowered the score. So the effect depends on the document and the language.

That's why I think you should have a set of sample questions before tuning anything in RAG. Without the set, I'd have decided by gut feeling either way. With 12 questions, a difference of one question can still be chance, so treat this as a reason to test on your own documents, not as a rule. I keep the default of body only in the code, because it's the simpler setting and the difference disappears at three sections.

Connect it to the chatbot and see what it sends

Finally, I replace the search step in the previous post's program with search by meaning, taking three sections:

// Full RAG, search-by-meaning version: embed the question, take the 3 closest sections, build the prompt, ask the model.
import { buildIndex, findByMeaning } from "./find-by-meaning.ts";
import { buildPrompt, answer } from "./answer.ts";
import { readPolicy } from "./split-sections.ts";

const SECTIONS = readPolicy();
const QUESTIONS = process.argv.slice(2).length ? [process.argv.slice(2).join(" ")] : [
  "I cancelled my bank transfer order, how long until the money is back?",
  "My coffee was made wrong and I want the shop to compensate me",
];

const index = await buildIndex(SECTIONS);
for (const question of QUESTIONS) {
  const results = await findByMeaning(question, SECTIONS, index, 3);
  const chosen = results.map((result) => result.section);
  console.log(`Customer: ${question}`);
  console.log(`  sent: ${results.map((result) => `${result.section.title} (${result.score.toFixed(3)})`).join(" + ")}`);
  console.log(`  ${buildPrompt(question, chosen).length} characters`);
  console.log(`Mộc: ${await answer(question, chosen)}\n`);
}

Customer: I cancelled my bank transfer order, how long until the money is back?
  sent: Cancelling an order (0.891) + Refunds (0.890) + Ordering and payment (0.873)
  1049 characters
Mộc: A bank-transfer order is made once the shop sees the money arrive in its account, which usually takes under two minutes. A valid cancellation of a bank-transfer order is refunded in full, as described in the Refunds section. [Ordering and payment; Cancelling an order]

Customer: My coffee was made wrong and I want the shop to compensate me
  sent: Replacing a wrong or badly made item (0.866) + Cancelling an order (0.833) + Allergies and ingredients (0.827)
  1197 characters
Mộc: Sorry, Mộc doesn't have that information yet.

Look at the first answer. The Refunds section, which holds the number "three working days", is among the three sections sent. Yet the answer still lacks that number. The reason isn't the search. It's the simulated model I've used since the previous post, which only picks the sentences sharing the most words with the question and reads them back, so it can't know that "money is back" is asking about a refund. I haven't run the real-Claude mode for this post, so I won't claim that a real model would answer correctly, only that it would have all the information in hand. If you have an ANTHROPIC_API_KEY, run the command again and compare.

The second answer shows something else. The right section, Replacing a wrong or badly made item, comes first with 0.866, and the other two sections score 0.833 and 0.827, noise that came along. The simulated model still says the shop has no information, because no sentence in the sections shares two words with the question, since the customer wrote "compensate" and the policy says "refund". Notice the margin: the first section leads the second by 0.033 here, but on the first question the top three are within 0.02 of each other. With gaps that small I wouldn't trust taking exactly one section. Three is much safer. The price is the noise: the prompts above are 1,049 and 1,197 characters instead of the roughly 700 two sections would give.

When should you use embeddings?

In my view they're worth it when your customers don't write in the document's style, which is most cases where real people ask. If the document is short, you can still send all of it in the prompt, as I said in the previous post, and skip the extra component.

A few practical notes if you try this with a real document:

• With 14 sections, embedding at startup is quick. With thousands of sections, store the vectors in a database (Postgres with the pgvector extension, for example) instead of recomputing them on every run.
• A small model running locally is easy to try and free, but not necessarily the best. For higher quality, switch to a larger model or an API service such as Voyage AI, which the Anthropic documentation mentions. The rest of the code doesn't change, because only embed.ts knows which model is running.
• The e5 model truncates input at about 512 tokens. Mộc's policy sections are short, so that's fine. A section several pages long would need to be split into smaller pieces before embedding.

The agent's tools still live inside one application

The Mộc chatbot now finds the right policy by meaning, and it already has tools to look up orders and add items, plus the policy search step. But all those tools sit directly in the chatbot's own code. If the shop wants to use the same tools in another application, say an assistant for staff at the counter, someone has to copy the code across and keep two versions in sync. How to package tools so several applications can share them is the job of later posts.

You can follow the whole series at Build your first AI agent. If you're preparing for interviews, the questions on AI agents are in the interview question bank.