What Actually Happens Inside a RAG Pipeline (And Why “Banking” Is the Perfect Example)
A plain-English walkthrough of embeddings, vector stores, HNSW, and why Elasticsearch and vector databases are secretly cousins - even if a few degrees removed :)
LLMs are the RAGe and Retrieval-Augmented Generation (RAG) is fascinating (okay bad pun!). Those even remotely familiar with ‘AIspeak’ have heard about RAG. A good initial framing of what it is is as follows → “the AI searches your documents, then answers using what it finds.” This article is an exploration of a few key ideas pertaing to RAGs, Transformers, Attention, Indexes and a quick comparison with the realm of text-based searching via systems like ELASTICSEARCH, SOLR etc.
The article does require reasonable technical familiarity with database concepts (the regular RDBMSs and Document DB ones) are and pivots from there to Vector DBs which make RAG possible.
The essential mental model for RAG systems is that they provide a layer of grounding to the otherwise global LLM responses that one would get from AI (LLM) directly stripped of additional context- what I like to call as ‘Ground Truths’. So, for example, if a user wants to ask some HR related questions to Claude or any AI - they can certainly issue a query and Claude will answer on the basis of the global data it has been trained on. But if the user wants answers from AI grounded in the context of their company’s HR policies then the user must provide additional context in the prompt. RAG is the technique that allows achieving this.
I end by touching over a topic that is gaining traction lately - viz. with all the recent advances in RDBMSs supporting the storage of vector embeddings - like pgVector and so on…are pure play vector DBs like pinecone, weaviate losing ground?
The rest of the article is about how such systems operate under the hood.
Step 1: Chunking
Before any AI touches your documents, they get broken into smaller pieces — usually a paragraph or a fixed window of a few hundred tokens, sometimes with slight overlap so meaning doesn’t get broken mid-thought. This is crucial: search a whole 40-page document and a single relevant paragraph buried inside it gets diluted into irrelevance. Search a paragraph-sized chunk, and precision goes up dramatically.
Step 2: Tokenization
Each chunk gets split into tokens — sub-word pieces, not quite full words. Critically, tokenization is purely mechanical. Take the same word ‘bank’ appearing in two different contexts where the meaning of the word changes.
I am banking on you.
I prefer ICICI as my banking partner.
In these two sentences the word banking means different things and the LLM must be able to tell. Indeed this is the secret sauce of AI - but at the stage of tokenization both these words will tokenize identically.
Step 3: Contextual Embeddings
This is where the magic alluded to earlier actually happens. And this is so critical that it needs a somewhat more detailed exposition than I have done heretofore as understanding this clearly is important.
Any transformer — to be sure there are several architectures around — produces numerical representations for tokens (text), and those representations are what get shaped into embeddings (so called embeddings-vector - a really high dimenstional space in tech-speak, essentially amounting to a large set of numbers).
Two lineages are of importance here. BERT-based models are encoder-only, trained to read a chunk of text in both directions at once, which makes them a natural fit for producing a single meaning-representation — from this lineage arose dedicated embedding models like Sentence-BERT, BGE etc.
GPT-based models are decoder-only, trained to predict the next word while looking only left-to-right; these were built for generation, not for summarizing meaning.
The tokens are processed together using a mechanism called attention, which evaluates each token's representation based on the words around it. So “banking” near “on you” ends up mathematically different from “banking” near “partner” — same string, but different embedding vector, because the model analyzes the context around the word.
Step 4: One Vector Per Chunk
The individual token-level vectors get pooled into a single vector — commonly 1536 numbers if you’re using a popular OpenAI model (though the actual number varies by model: Cohere uses 1024, Google’s models use 768, some lightweight open-source models use as few as 384). This single vector is what represents “the meaning of this chunk,” and it’s what actually gets stored in the Vector DB - explained in the next section.
Step 5: The Vector Store
Picture a simple table:
Chunk ID Vector Text
1 0.01, -0.45, ... ”This is not the end of the world”
2 0.61, -0.66, ... ”This is just the beginning of AI”
At query time, your question (prompt) gets embedded the same way (like the embeddings created earlier when loading the documents into Vector DB), producing a query vector.
On a side note - we should ensure that the same embedding model that produced the Vector store embeddings is utilized for creating the vector embeddings for the query as well.
This query vector is then searched for similarity against the Vector DB using something called cosine similarity — to determine essentially, how closely aligned two vectors’ directions are, which is a proxy for shared meaning. The most similar chunks come back, and their original text gets stuffed into the prompt sent to the LLM.
Step 6: The Vector Indexes
Comparing your query vector against every single stored vector works fine for small number of rows. It falls apart for scaled datasets (>500k rows). So production vector stores build an index — a separate structure sitting alongside it, purpose-built to avoid brute-force comparison through scanning. Recall how indexes are built (like clustered, non clustered indexes in the RDBMS world…it’s a similar indexing concept here as well).
Two dominant approaches are relevant for building indexes in the Vector DB world:
HNSW (Hierarchical Navigable Small World) — a multi-layer index built over the vector store. Layer 0, the bottom layer, holds every vector (for the chunks in the documents). Each layer above it holds a smaller, randomly-selected subset of those same vectors, with the fewest vectors and longest-range links on the top layer. A query starts at the top, hops to whichever linked vector is nearest, drops down a layer, repeats — narrowing in as it descends, until it reaches Layer 0, where the full neighbor set gives the actual nearest matches.
It navigates like a B-tree (descend level by level, check only a few candidates per step) — with the notable exception that a B-tree compares keys (exact), while HNSW compares distances (greedy, approximate) — hence “Approximate” Nearest Neighbor”.
IVF (Inverted File Index) — vectors get pre-clustered, and a query only searches a handful of clusters nearest to it. Similar in approach to “which chapter to search,” in the book.
If you’ve worked with relational databases or big-data systems, both of these should feel familiar:
HNSW/IVF behave like a non-clustered index — a separate structure pointing back to the real data, not reordering the table itself (unlike clustered indexes that actually reorder the entire table on which they are built)
Furthermore, IVF can be understood as something similar to partition pruning in Spark — it allows skipping entire irrelevant clusters/partitions rather than scanning everything.
But bear in mind that classic indexes and partition pruning are always exact. Vector indexes are deliberately approximate — they’ll occasionally miss the true single-best match in exchange for searching billions of vectors in milliseconds. That tradeoff, called Approximate Nearest Neighbor (ANN) search, is the one genuinely new idea vector search introduces that neither of those older systems has.
Meanwhile, a Parallel Universe: Text Search
Elasticsearch, OpenSearch, and Solr solve a related but distinct problem, using none of the above. Instead of an embedding model whats at play here is inverted index: a term-to-document lookup table.
A document gets run through a fixed, rule-based analyzer: tokenize, lowercase, drop stopwords (”the,” “is,” “an”), and stem words to a root form (”terminate”/”termination” → termin). Whatever survives becomes a searchable term, mapped to every document that contains it.
Matches then get ranked with BM25, which scores relevance using three signals: how often a term appears in a document, how rare that term is across the whole collection (the corpus), and how long the document is (so a short document isn’t unfairly outscored by a long one repeating the term more times).
What needs to be appreciated - compared with the AI and Vector DB scenarios is that this whole approach in text-based searches is purely lexical. It has no idea that “banking on you” and “relying on you” mean the same thing - the semantic meaning has no place here — it only knows whether the exact stemmed term is present. That’s why real-world systems increasingly run both — an inverted-index pass and a vector-similarity pass — and merge the results. This is called hybrid search, and it exists precisely because exact-term matching (for example, contract clause numbers, legal terms, SKUs) and conceptual matching (paraphrased questions) each need a different engine to shine.
Why This Matters Right Now
There’s an active industry debate about whether standalone vector databases (Pinecone, Weaviate, etc.) are worth running at all, versus using a Postgres extension like pgvector that bolts an ANN index onto a normal relational database. In my view while the general-purpose option wins on cost and simplicity — purpose-built vector databases would earn their keep at scale - billion-vector scale (as for example in teh case of NVIDIA’s autonomous-vehicle sensor data, or TripAdvisor’s billion-plus review corpus).
It is the fundamentals I goad you to keep intact while you navigate complexity and stay above marketing fizz buzz. So regardless of whether its pinecone of pgvectore- the pipeline - chunk, tokenize, contextually embed, pool, index, retrieve by similarity stays exactly the same. Understanding that pipeline is what actually lets you reason about tradeoffs, instead of just picking a vendor because of a sales deck.
If this was useful, I write more breakdowns like this — takes on how AI infrastructure actually works under the hood, minus the marketing.