Question: "What's our company's vacation policy?"

ChatGPT response: "Generally, companies offer 
10-20 vacation days per year. Check your 
employee handbook or HR department for 
your specific policy..."

What you actually need: "According to the 
Employee Handbook Section 4.2, full-time 
employees receive 22 vacation days per 
year after their first year. Days are 
accrued monthly at 1.83 days per month..."

Traditional AI:
User question → AI model → Generic answer

RAG-powered AI:
User question → Search your documents → 
Find relevant info → AI model + context → 
Accurate, sourced answer

Question: "What's the approval process 
          for purchases over €5,000?"

RAG process:
1. Search company documents
2. Find: Procurement Policy v3.2, Section 2.4
3. Retrieve relevant text
4. Generate answer with source

Response: "For purchases over €5,000, you need:

1. Submit PR-1000 form with justification
2. Get manager approval (Level 1)
3. Get Finance review for budget confirmation
4. Get VP approval for amounts €5,000-€25,000
5. Get CEO approval for amounts over €25,000

Processing time: 3-5 business days

Source: Procurement Policy v3.2, Section 2.4
Last updated: October 2025"

┌─────────────────────────────────────────┐
│           RAG ARCHITECTURE              │
└─────────────────────────────────────────┘

1. DOCUMENT INGESTION
   ├── Source documents (PDF, Word, web, DB)
   ├── Text extraction & cleaning
   ├── Chunking into segments
   └── Metadata extraction

2. EMBEDDING & INDEXING  
   ├── Convert text to vectors (embeddings)
   ├── Store in vector database
   ├── Create searchable index
   └── Maintain metadata links

3. RETRIEVAL PIPELINE
   ├── User query processing
   ├── Query embedding generation
   ├── Similarity search
   └── Relevance ranking

4. GENERATION PIPELINE
   ├── Context assembly
   ├── Prompt construction
   ├── LLM generation
   └── Response formatting

5. USER INTERFACE
   ├── Chat interface
   ├── API endpoints
   ├── Integration points
   └── Feedback collection

Why chunking matters:
├── LLMs have context limits (8k-128k tokens)
├── Better retrieval with smaller chunks
├── Preserves semantic meaning
└── Enables precise sourcing

Chunking approaches:

Fixed-size chunks:
├── Split every N characters/tokens
├── Simple to implement
├── May break mid-sentence
└── Good for: Homogeneous content

Semantic chunks:
├── Split by paragraphs/sections
├── Respects document structure
├── Variable sizes
└── Good for: Structured documents

Sliding window:
├── Overlapping chunks
├── Captures context across boundaries
├── Higher storage requirements
└── Good for: Dense technical content

Recommended: Semantic with overlap
├── Chunk size: 500-1000 tokens
├── Overlap: 50-100 tokens
└── Preserve paragraph boundaries

Text → Embedding model → Vector (numbers)

Example:
"Company vacation policy" → [0.23, -0.45, 0.12, ...]

Similar concepts = similar vectors:
├── "PTO policy" → [0.21, -0.43, 0.14, ...]
├── "Time off rules" → [0.25, -0.44, 0.11, ...]
├── "Salary structure" → [-0.33, 0.12, 0.67, ...]

Vector similarity enables semantic search:
├── Find documents about "vacation" 
├── Even if they say "PTO" or "time off"
└── Not just keyword matching

OpenAI text-embedding-3-large:
├── Dimensions: 3072
├── Cost: $0.13/million tokens
├── Quality: Excellent
└── Best for: General purpose

Cohere embed-v3:
├── Dimensions: 1024
├── Cost: $0.10/million tokens
├── Quality: Very good
└── Best for: Multi-language

Open source (sentence-transformers):
├── Dimensions: 384-1024
├── Cost: Free (compute only)
├── Quality: Good
└── Best for: Privacy-sensitive, high volume

Vector database role:
├── Store embeddings efficiently
├── Fast similarity search
├── Scale to millions of vectors
└── Filter by metadata

Popular options:

Pinecone (managed):
├── Fully managed, easy setup
├── Good performance
├── Cost: Based on storage + queries
└── Best for: Quick deployment

Weaviate (open source):
├── Self-hosted or cloud
├── Hybrid search (vector + keyword)
├── GraphQL API
└── Best for: Complex search needs

Qdrant (open source):
├── High performance
├── Good filtering
├── Easy to deploy
└── Best for: Performance-critical

Chroma (lightweight):
├── Simple, embedded
├── Good for prototyping
├── Limited scale
└── Best for: Small projects

PostgreSQL + pgvector:
├── Use existing Postgres
├── Combined with relational data
├── Moderate performance
└── Best for: Integrated with existing DB

Identify knowledge sources:
├── Policy documents
├── Product documentation
├── Training materials
├── FAQ content
├── Process guides
├── Historical data
└── External references

Assessment questions:
├── Format (PDF, Word, HTML, DB)?
├── Volume (hundreds or millions)?
├── Update frequency?
├── Access restrictions?
├── Quality and consistency?
└── Language(s)?

For each document:

1. Extract text
   ├── PDF: Use pypdf or pdfplumber
   ├── Word: Use python-docx
   ├── HTML: Use Beautiful Soup
   └── Images/scans: Use OCR (Tesseract)

2. Clean and normalize
   ├── Remove headers/footers
   ├── Fix encoding issues
   ├── Standardize formatting
   └── Remove duplicates

3. Extract metadata
   ├── Document title
   ├── Author/owner
   ├── Date created/modified
   ├── Document type
   └── Access level

4. Chunk content
   ├── Split by sections
   ├── Maintain hierarchy
   ├── Add chunk metadata
   └── Create chunk IDs

For each chunk:

1. Prepare text
   ├── Clean whitespace
   ├── Truncate if too long
   └── Format consistently

2. Generate embedding
   ├── Call embedding API
   ├── Handle rate limits
   └── Implement retries

3. Store with metadata
   ├── Vector (embedding)
   ├── Original text
   ├── Source document ID
   ├── Chunk position
   ├── Document metadata
   └── Creation timestamp

Vector index settings:
├── Metric: Cosine similarity (most common)
├── Index type: HNSW (fast, approximate)
├── Dimensions: Match embedding model
└── Capacity: Plan for growth

Metadata indexes:
├── Document type (filter)
├── Date range (filter)
├── Department (filter)
├── Access level (security)
└── Full-text (hybrid search)

When user asks a question:

1. Preprocess query
   ├── Clean and normalize
   ├── Expand abbreviations
   └── Handle multi-part questions

2. Generate query embedding
   ├── Same model as documents
   └── Single API call

3. Search vector database
   ├── Find top-k similar chunks
   ├── Apply metadata filters
   └── Return with scores

4. Re-rank results
   ├── Score by relevance
   ├── Deduplicate similar chunks
   ├── Apply business rules
   └── Return top N chunks

Key parameters:
├── k (initial results): 10-20
├── Final results: 3-5 chunks
├── Similarity threshold: 0.7-0.8
└── Chunk size for context: 500-1000 tokens

Hybrid search (recommended):
├── Vector search (semantic)
├── + Keyword search (exact matches)
├── + Metadata filters
└── = Better coverage

System prompt:
"You are a helpful assistant that answers 
questions using the provided context. 
Always cite your sources. If the context 
doesn't contain the answer, say so."

User prompt structure:
────────────────────────────────
Context:
[Retrieved chunk 1]
Source: Document A, Section 2.1

[Retrieved chunk 2]
Source: Document B, Page 15

[Retrieved chunk 3]
Source: Document C, FAQ #42
────────────────────────────────

Question: [User's question]

Instructions:
- Answer based only on the context above
- Cite sources for each fact
- If unsure, say "I don't have information about that"
────────────────────────────────

Model selection:
├── GPT-4: Best quality, highest cost
├── GPT-3.5-turbo: Good balance
├── Claude: Strong reasoning
├── Open source: Llama, Mistral

Parameters:
├── Temperature: 0.1-0.3 (factual)
├── Max tokens: 500-1000 (responses)
├── Top-p: 0.9
└── Frequency penalty: 0 (no creativity needed)

Test categories:

Retrieval quality:
├── Does it find relevant documents?
├── Are top results actually useful?
├── Does it miss important sources?
└── Metric: Recall@k, MRR

Generation quality:
├── Are answers accurate?
├── Are sources cited correctly?
├── Does it hallucinate?
├── Is tone appropriate?
└── Metric: Human evaluation, faithfulness

End-to-end testing:
├── Common questions (should answer well)
├── Edge cases (should handle gracefully)
├── Out-of-scope (should decline politely)
└── Adversarial (should not break)

Improve retrieval:
├── Better chunking strategy
├── Query expansion
├── Metadata enrichment
├── Re-ranking models
└── Hybrid search tuning

Improve generation:
├── Prompt engineering
├── Few-shot examples
├── Response templates
├── Citation formatting
└── Error handling

Monitor and iterate:
├── Log all queries and responses
├── Identify failure patterns
├── Add missing content
├── Refine prompts
└── Expand knowledge base

Document preparation:
├── Remove outdated content
├── Fix formatting issues
├── Standardize terminology
├── Add clear headings
├── Update regularly
└── Version control

Metadata hygiene:
├── Consistent categorization
├── Accurate dates
├── Clear ownership
├── Access levels defined
└── Regular audits

Access control:
├── User authentication
├── Document-level permissions
├── Role-based access
├── Audit logging
└── Data encryption

Prevent data leakage:
├── Filter results by user access
├── Don't cache sensitive content
├── Anonymize logs
├── Regular security reviews
└── Compliance monitoring

Latency targets:
├── Retrieval: <100ms
├── Generation: <2s
├── Total: <3s

Optimization strategies:
├── Pre-compute embeddings
├── Cache frequent queries
├── Use async processing
├── Scale vector database
├── Optimize chunk sizes
└── Use faster models when appropriate

Problem: Wrong documents retrieved
Causes:
├── Poor chunking (context lost)
├── Bad embeddings (wrong model)
├── Missing metadata filters
├── Outdated content indexed
Solution: Better preprocessing, hybrid search

Problem: Nothing retrieved
Causes:
├── Query too different from docs
├── Threshold too high
├── Content not indexed
Solution: Query expansion, lower threshold

Problem: AI ignores context
Causes:
├── Context too long
├── Wrong prompt structure
├── Model limitations
Solution: Better prompts, shorter context

Problem: Hallucinations
Causes:
├── Insufficient context
├── Temperature too high
├── Model overconfidence
Solution: Lower temperature, explicit instructions

Before launch:
□ Test with 100+ real questions
□ Verify source citations
□ Check edge cases
□ Test with various users
□ Measure latency
□ Review security
□ Plan content updates
□ Set up monitoring

After launch:
□ Monitor query logs
□ Track failed queries
□ Collect user feedback
□ Update content regularly
□ Retrain as needed
□ Scale infrastructure
□ Iterate on prompts

Good fit if you have:
□ Substantial knowledge base (100+ documents)
□ Frequent information queries
□ Need for accurate, sourced answers
□ Existing document management
□ Technical capacity to maintain

May not need RAG if:
□ Small, static knowledge base
□ Simple FAQ-style queries
□ No document infrastructure
□ Very limited budget