Just Enough to Be Dangerous

TECHNICAL REFERENCE

RAG for Car Dealerships: A Technical Reference

Retrieval-Augmented Generation (RAG)QuestionSearch your datayour knowledge baseRelevant factsLLMGroundedanswerThe model answers from your facts, not guesses.

1. The RAG pipeline

A RAG system has several stages:

[Source documents]
   ↓
[Chunking]
   ↓
[Embedding model]
   ↓
[Vector database]
   ↓
[Query]
   ↓
[Retrieval: vector search + optional hybrid]
   ↓
[Re-ranking (optional)]
   ↓
[Context assembly]
   ↓
[LLM with prompt + context]
   ↓
[Response]

Each stage has options and trade-offs.

2. Chunking

The strategy for splitting documents into chunks affects retrieval quality.

Strategies:

  • Fixed-size chunking. Split into N-token chunks with overlap. Simple, predictable.
  • Sentence-based. Split on sentence boundaries. Preserves semantic units.
  • Paragraph-based. Split on paragraph breaks. Coarser chunks.
  • Section-based. Split on document structure (headings, sections). Requires structured documents.
  • Semantic chunking. Use the embedding model to identify natural breakpoints. More expensive, often better.
  • Recursive chunking. Try paragraph, then sentence, then word — recursively until chunks are the right size. Common in LangChain and LlamaIndex.

Chunk size trade-offs:

  • Small chunks (100-300 tokens). More granular, better for specific facts. But may lack context.
  • Medium chunks (300-800 tokens). Good balance. Common default.
  • Large chunks (800-2000 tokens). More context per chunk. But more noise, more tokens per retrieval.

For dealer data:

  • FAQs: small to medium chunks. Each Q+A is a chunk.
  • Service descriptions: medium chunks. One service per chunk.
  • Inventory: structured fields, not free text. Chunks are VIN-level records.
  • Policies: medium to large chunks. Preserves context.
  • Long documents (employee handbooks): semantic or section-based.

Overlap: A small overlap (10-20%) between chunks reduces the chance of important context being split across chunks. Too much overlap wastes storage and tokens.

3. Embedding models

The embedding model converts text to a vector. The vector represents the meaning of the text in a high-dimensional space. Similar meanings have similar vectors.

Popular embedding models:

  • OpenAI. text-embedding-3-small, text-embedding-3-large. Strong general-purpose, well-supported.
  • Anthropic. No first-party embedding model as of writing. Use third-party.
  • Cohere. embed-english-v3.0, embed-multilingual-v3.0. Strong, multilingual.
  • Voyage AI. voyage-3, voyage-large-2. High quality, especially for technical content.
  • Google. text-embedding-005 (via Vertex AI). Integrated with GCP.
  • Open-source. bge-large-en-v1.5, e5-large-v2, gte-large, nomic-embed-text. Self-hostable, no per-token cost.

Trade-offs:

  • Quality. Better embeddings = better retrieval = better answers. The big models (text-embedding-3-large, voyage-large-2) generally outperform the small ones.
  • Cost. Per-token cost. For a dealer’s knowledge base, the cost is usually a few dollars to embed the entire corpus.
  • Latency. Smaller models are faster.
  • Vector dimension. Larger dimensions = more expressive but more storage and slower search. 768-3072 is typical.
  • Multilingual support. If the dealership serves Spanish-speaking customers, multilingual embeddings matter.

For most dealer use cases, OpenAI’s text-embedding-3-small or a comparable model is the right starting point. Upgrade to a larger model if retrieval quality is insufficient.

4. Vector databases

The vector database stores embeddings and supports similarity search.

Managed / cloud:

  • Pinecone. Fully managed, scalable, fast. The most popular.
  • Weaviate Cloud. Open-source + managed offering. Strong feature set.
  • Qdrant Cloud. Open-source + managed. Strong performance.
  • Chroma Cloud. Simple, popular for prototyping.
  • Vertex AI Vector Search. GCP-managed, integrated with Google Cloud.
  • Azure AI Search. Azure-managed.
  • Amazon OpenSearch Service. With k-NN plugin.

Self-hosted:

  • Weaviate, Qdrant, Milvus, Chroma. Open source. Self-hostable.
  • PostgreSQL with pgvector. Use the existing Postgres database.
  • Elasticsearch / OpenSearch. With dense vector support.

Trade-offs:

  • Scale. For a dealer’s knowledge base (thousands to hundreds of thousands of chunks), any of the above work.
  • Operational cost. Managed is easier; self-hosted is more control.
  • Features. Filtering by metadata, hybrid search, re-ranking, multi-tenancy. Different providers offer different features.
  • Cost. Per-query or per-storage. For dealer volumes, costs are typically $0-$500/month.

For a single dealer, a managed vector database (Pinecone Serverless, Qdrant Cloud Free, Weaviate Cloud Sandbox) is the right starting point. For multi-dealer or multi-tenant systems, consider self-hosted or a multi-tenant managed solution.

5. Retrieval strategies

Vector search. Embed the query, search for similar embeddings. Returns the top K most similar chunks. Simple, fast.

Keyword search (BM25). Traditional search. Good for exact matches, part numbers, names. Less semantic.

Hybrid search. Combine vector search and keyword search. Use a weighted combination or a re-ranker to merge the results. Better than either alone for most use cases.

Metadata filtering. Before vector search, filter by metadata (e.g., only “Rav4” documents, only documents newer than 2024-01-01, only “service” category). Reduces noise.

Multi-query retrieval. Generate multiple variations of the query, retrieve for each, merge the results. Improves recall.

HyDE (Hypothetical Document Embeddings). Generate a hypothetical answer, embed that, search. Useful when the user’s question is short and the answer is long.

Re-ranking. Initial retrieval returns a candidate set. A re-ranker (a cross-encoder model, Cohere Rerank, or a smaller LLM) scores the candidates and reorders. Improves precision, especially for the top results.

For dealer RAG, a typical setup:

  • Hybrid search (vector + BM25).
  • Metadata filtering (e.g., by category, by recency).
  • Top K = 5-10 results.
  • Optional re-ranking for the top candidates.
  • LLM context with the top 3-5 chunks.

6. Context assembly

The retrieved chunks are assembled into a prompt for the LLM. Considerations:

  • Token budget. The LLM has a context window. Allocate space for the system prompt, the user’s question, the retrieved context, and the LLM’s response.
  • Order matters. The LLM is more influenced by the first and last pieces of context. Put the most relevant chunks at the top.
  • Citations. Include source metadata (document title, URL, last updated) so the LLM can cite its sources.
  • Deduplication. If multiple chunks are near-duplicates, keep only one.
  • Compression. For long contexts, summarize or compress the chunks before adding to the prompt.

A typical prompt structure:

System: You are a helpful assistant for Smith Ford. Answer the user's question using only the provided context. If the answer isn't in the context, say so. Cite your sources.

Context:
[Source 1: Inventory, updated 2025-06-15]
2024 Ford F-150 XLT, stock #F12345, $52,995, Oxford White, 12,450 miles, available.

[Source 2: Specials, updated 2025-06-01]
June 2025 Special: 0% APR for 60 months on all new 2024 F-150 XLT in stock.

User: Do you have any red F-150s in stock?

Assistant:

The LLM uses the context to answer. If the answer isn’t in the context, it says so.

7. Real-time data sources

The simplest RAG is over static documents. The hardest is real-time data (inventory, pricing, availability).

For real-time data, the pattern is:

  • Don’t embed real-time data. It’s stale the moment it’s embedded.
  • Use function calling / tool use. The LLM calls a function (e.g., search_inventory(make="Ford", model="F-150", color="red")) to fetch real-time data.
  • The function returns structured data. The LLM uses the data to answer.
  • Combine RAG with tools. Documents (policies, FAQs) come from the vector database. Real-time data (inventory, pricing) comes from function calls.

This is the “agentic RAG” pattern. The RAG over documents provides static context; the tool use provides dynamic context.

8. Evaluation

How do you know the RAG is working? The metrics:

Retrieval metrics:

  • Recall@K. Of the relevant chunks, how many are in the top K results?
  • Precision@K. Of the top K results, how many are relevant?
  • MRR (Mean Reciprocal Rank). How high is the first relevant result?
  • NDCG (Normalized Discounted Cumulative Gain). Considers both rank and relevance.

Generation metrics:

  • Faithfulness. Does the LLM’s response match the retrieved context? (i.e., no hallucination outside the context.)
  • Answer relevance. Is the response actually answering the question?
  • Context relevance. Is the retrieved context relevant to the question?

End-to-end metrics:

  • Human evaluation. Sample responses, have humans score them.
  • Task completion. Did the user get what they needed?
  • Override rate. How often does a human need to correct the AI?

Tools:

  • RAGAS. Open-source RAG evaluation framework.
  • TruLens. LLM evaluation with RAG support.
  • LangSmith / LangChain. Tracing and evaluation.
  • Arize Phoenix. Open-source LLM observability.
  • Custom. Build a labeled evaluation set from real dealer interactions.

9. The RAG knowledge base for a dealer

A realistic RAG knowledge base:

Source Update Frequency Chunk Strategy Metadata
Inventory (real-time) Every 15 min Per-VIN record make, model, year, price, status
Specials Daily Per offer date range, model, terms
Service menu Quarterly Per service category, model, interval
FAQs On update Per Q&A category, last reviewed
Policies On update Per policy effective date, applies to
OEM programs On update Per program OEM, model, eligibility
Hours and locations On change Per location rooftop, holiday
Employee handbook On update Section-based department, role
Sales playbooks On update Per script stage, scenario

The metadata is filterable. The chunks are embedded. The vector database holds the embeddings and the original text. The LLM uses the retrieved text as context.

10. Common RAG mistakes

  • Bad chunking. Chunks too small (no context) or too large (too much noise). Iterative tuning needed.
  • Stale knowledge base. Documents not updated. AI gives outdated answers.
  • No metadata filtering. Retrieval returns irrelevant chunks. The AI gets confused.
  • Ignoring the prompt. The LLM doesn’t follow the prompt instructions. Output validation needed.
  • No evaluation. No way to know if the system is improving or degrading.
  • PII in the knowledge base. Customer data embedded. Privacy and compliance risk.
  • Cost overrun. Embeddings generated repeatedly. Vector database growing without bound.
  • Hallucination outside context. LLM invents facts. Prompt enforcement and output validation needed.

11. References

  • “Retrieval-Augmented Generation for Large Language Models: A Survey” by Gao et al.
  • Lewis et al., 2020 — “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” — the original RAG paper.
  • LangChain and LlamaIndex documentation.
  • Pinecone’s learning center.
  • Weaviate’s blog and documentation.
  • RAGAS documentation for RAG evaluation.
  • “Designing Machine Learning Systems” by Chip Huyen — for the production ML perspective.
  • OpenAI, Anthropic, Cohere, Voyage AI documentation.

Part of the Just Enough to Be Dangerous infrastructure series from VCTRS. Prefer the plain-English version? Read What RAG Actually Means for Your Dealership (And How AI Uses Your Own Knowledge).

Agreement