Back to Articles
AI & Search
July 17, 20267 min readAmrit Sapkota

How AI Search Works: Semantic Search, Embeddings & RAG Explained

How AI Search Works: Semantic Search, Embeddings & RAG Explained

If you want to understand how AI search works, it comes down to three ideas working together: embeddings that turn text into numbers, vector search that finds meaning instead of matching keywords, and retrieval-augmented generation (RAG) that feeds those results to a language model so it can answer in plain English. This guide walks through each layer the way I'd explain it to another developer, then shows how to assemble them into a working search pipeline.

AI search vs. keyword search: what actually changed

Traditional search is lexical. Engines like Elasticsearch (using BM25) rank documents by how often your exact words appear, weighted by how rare those words are. It's fast and predictable, but it's literal. Search "how do I stop my app from crashing" and a keyword index will miss a perfect document titled "preventing runtime exceptions" because the words don't overlap.

AI search is semantic. Instead of matching characters, it matches meaning. It understands that "crashing," "runtime exception," and "unhandled error" live in the same neighborhood of concepts. That's the whole shift: from matching strings to matching ideas. The trade-off is that semantic systems are heavier to run and can occasionally surface something topically close but literally wrong, which is why the best production systems combine both approaches (often called hybrid search).

Embeddings: turning text into vectors

An embedding is a list of numbers, a vector, that represents the meaning of a piece of text. An embedding model reads your sentence and outputs something like [0.021, -0.44, 0.87, ...] with hundreds or thousands of dimensions. The key property is that similar meanings produce similar vectors. "Nepal" and "Kathmandu" land close together; "Nepal" and "invoice parsing" land far apart.

You don't compute these by hand. You call a model:

  • Hosted APIs like OpenAI's text-embedding-3 family, Cohere Embed, or Voyage AI. Easy to start with, priced per token.
  • Open-source models like the sentence-transformers library, BGE, or E5, which you can self-host for privacy or cost control.

Two rules matter in practice. First, pick one embedding model and stick with it. You cannot mix vectors from different models, because their coordinate systems are unrelated. Second, embeddings have a fixed dimensionality (for example 1536), and your vector store has to be configured to match it.

Vector search: finding the nearest neighbors

Once your documents are vectors, "search" becomes a geometry problem: given the query vector, find the stored vectors closest to it. Closeness is measured with a distance metric, most commonly cosine similarity (the angle between two vectors) or dot product. Higher similarity means more related meaning.

Comparing the query against every document one by one works for a few thousand items but collapses at scale. That's where Approximate Nearest Neighbor (ANN) algorithms come in. Instead of an exhaustive scan, structures like HNSW (a navigable graph) or IVF (clustered buckets) find the closest matches in milliseconds across millions of vectors, trading a tiny bit of accuracy for enormous speed.

This is the job of a vector database. Common options:

  • pgvector — a Postgres extension, great if you already run Postgres and want vectors next to your relational data.
  • Qdrant, Weaviate, Milvus — purpose-built open-source vector engines.
  • Pinecone — a fully managed hosted service.

For a deeper look at the tools in this space, I compare several in Top 10 AI Tools Every Developer Should Use in 2026.

Semantic ranking and re-ranking

Vector search gives you a shortlist, say the top 50 candidates, ranked by raw similarity. But the top vector match isn't always the best answer, because the embedding compressed each document into a single point and lost nuance. This is where a second stage helps: re-ranking.

A re-ranker (typically a cross-encoder, like Cohere Rerank or an open-source model) takes the query and each candidate together and scores their true relevance. It's slower per comparison, so you only run it on the shortlist, not the whole corpus. The pattern is:

  1. Retrieve a broad candidate set fast with ANN vector search.
  2. Re-rank that small set precisely with a cross-encoder.
  3. Return the top handful.

Adding a re-ranking stage is one of the highest-leverage improvements you can make. Many teams report that it noticeably lifts answer quality with very little extra code.

Retrieval-Augmented Generation (RAG): grounding the answer

Semantic search returns documents. Users usually want an answer. RAG bridges the gap: you retrieve the most relevant chunks, then hand them to a large language model as context and ask it to answer using only that material.

Why bother instead of just asking the model directly? Two reasons. LLMs have a training cutoff and know nothing about your private docs, your codebase, or yesterday's ticket. And when they don't know something, they tend to make it up. RAG fixes both by grounding the model in real, retrieved text, which also lets you cite sources. A typical RAG prompt looks like:

You are a support assistant. Answer using ONLY the context below.
If the answer isn't in the context, say you don't know.

Context:
{retrieved_chunks}

Question: {user_question}

If you're new to wiring models and APIs together, I walk through the fundamentals in Building Your First AI Workflow with LLMs and APIs.

How AI search works end-to-end: building the pipeline

Here's the full flow. It splits cleanly into an offline indexing phase and an online query phase.

1. Chunk your documents

You can't embed a whole 40-page PDF as one vector — meaning gets diluted. Split content into chunks of roughly a few hundred tokens, ideally along natural boundaries like paragraphs or headings, with a small overlap so context isn't cut mid-thought. Chunking quality quietly determines retrieval quality more than almost anything else.

2. Embed and store

Run each chunk through your embedding model and store the vector alongside its source text and metadata (URL, title, date) in your vector database. Keep the original text — you'll need it for the LLM later.

3. Embed the query and retrieve

At query time, embed the user's question with the same model, run an ANN search, and pull back the top candidates. Optionally combine this with keyword search for hybrid retrieval.

4. Re-rank and generate

Re-rank the candidates, take the best few, drop them into your prompt, and call the LLM. Return the answer with links back to the source chunks so users can verify it.

That's the entire system. The reason so many teams are standardizing on this shape is exactly what I cover in Why Dev Teams Are Adopting AI Workflows.

Common mistakes to avoid

  • Mismatched embedding models. Indexing with one model and querying with another produces garbage. Use the same model on both sides.
  • Chunks that are too big or too small. Too big dilutes meaning; too small loses context. Tune this first when results feel off.
  • Skipping re-ranking. Pure vector similarity is a good first pass, not a final ranking.
  • No "I don't know" path. If retrieval returns nothing relevant, your prompt should let the model decline rather than hallucinate.
  • Ignoring metadata filters. Combining vector search with plain filters (date, product, permissions) is often what makes results actually usable.

Frequently Asked Questions

What is the difference between semantic search and keyword search?

Keyword search matches the literal words in your query against words in the documents and ranks by frequency and rarity (BM25 is the classic algorithm). Semantic search matches meaning by comparing embedding vectors, so it can find relevant results that share no exact words with the query. In production, many teams run both together as hybrid search to get precision and recall.

What are embeddings in AI search?

Embeddings are numeric vectors that represent the meaning of text, produced by an embedding model. Text with similar meaning gets vectors that sit close together in vector space, which is what lets a computer measure how related two pieces of text are using a distance metric like cosine similarity.

What is RAG (retrieval-augmented generation)?

RAG is a technique where you first retrieve relevant documents (usually with vector search), then pass them to a large language model as context and ask it to answer using that material. It grounds the model in real, up-to-date, or private data, which reduces hallucination and lets the answer cite its sources.

Do I need a vector database to build AI search?

For anything beyond a few thousand documents, yes, because it provides fast approximate nearest neighbor search, metadata filtering, and persistence. For a small prototype you can hold vectors in memory or use a lightweight library, but a dedicated store (pgvector, Qdrant, Weaviate, Pinecone) becomes necessary as your data grows.

Conclusion

AI search isn't magic once you see the moving parts: embeddings turn text into vectors, vector search finds the nearest neighbors by meaning, re-ranking sharpens the shortlist, and RAG turns retrieved chunks into grounded answers. The practical takeaway is to build it in stages, get clean chunking and consistent embeddings working first, then add re-ranking and generation, measuring quality at each step. If you're thinking about how this affects discoverability, my follow-up on SEO in the Age of AI Search covers how to get your own content cited by these systems.

Thanks for reading!