Talk to Your Documents — RAG with Video Avatars

The Challenge

A door and window manufacturer needed their sub-vendors to quickly find product specifications without navigating complex websites. A previous contractor had built a flashy demo but abandoned the project. The task was to rebuild the entire platform from scratch, improving on the original concept.

Technical Deep Dive — RAG Architecture

• Started with an OpenAI custom assistant, but discovered a critical limitation: all documents were lumped into a single database, which meant the AI couldn't distinguish between specs for different products (Window 1 vs Window 2).
• Solution: migrated to a vector database with granular per-product metadata, giving each product its own isolated context.
• Chunking strategy: parent-child chunking with keyword enrichment to preserve semantic meaning and boost similarity scores.
• Vision model integration: product brochures are highly visual, so traditional OCR failed. Used vision models to extract granular product details from images and diagrams.

Video Avatars & WebRTC

• Company AI avatar for face-to-face interaction with vendors
• WebRTC live streaming integration for real-time conversations
• Dual interface: text chatbot or video avatar depending on vendor preference

Key Features

• Vector database with per-product metadata isolation
• Parent-child chunking with keyword enrichment
• Vision model document processing for visual brochures
• PDF reference links shown alongside answers
• Video avatar with WebRTC streaming
• Text chatbot alternative

Results

• Vendors can instantly find specifications across the full product catalog
• PDFs shown as clickable reference links for manual review
• Eliminated the need to navigate multiple vendor websites
• Rebuilt from scratch after contractor abandonment

Tech Stack