Description
Automated context-based document chunking and embedding for enhanced retrieval in RAG pipelines — powered by AI
📚 Say goodbye to rigid splitting — this workflow intelligently segments documents into context-preserving chunks and stores them in Pinecone for semantically rich search and Retrieval-Augmented Generation (RAG).
🧠 What Problem Does It Solve?
Standard chunking in RAG pipelines often loses context, leading to poor retrieval performance. This workflow automates context-aware chunking using section-based logic and AI to retain document-level meaning, dramatically improving LLM accuracy during retrieval.
It provides semantic-ready embeddings with full context on:
✅ Document sections
✅ Cross-referenced metadata
✅ Meaning-preserving chunking
✅ Enhanced semantic embeddings
⚙️ How It Works
📁 Pulls a structured document from Google Drive
📄 Extracts text and detects section boundaries
🧩 Splits text into context-aware chunks using code logic
🔁 Loops through each chunk for individual processing
🤖 Uses OpenRouter + GPT-4.0-mini to generate succinct chunk context
🪄 Prepends AI-generated context to each chunk
🧠 Embeds enriched chunks using Google Gemini (text-embedding-004)
📦 Stores embeddings in Pinecone vector store with metadata
✨ Key Features
📥 Automatically fetches documents from Google Drive
🧠 Uses GPT-4.0-mini via OpenRouter to generate contextual metadata
🧾 Prepends context to boost semantic relevance
🧭 Improves retrieval accuracy in RAG workflows
🧬 Creates AI-enriched vector representations with Google Gemini
🗂️ Stores structured embeddings into Pinecone with traceable metadata
⚙️ Scales across document types and projects
🛠 Built-in error handling and modular design
🧰 What You Need
✅ Google Drive file with structured sections
✅ Pinecone account and index
✅ OpenRouter or OpenAI API access (GPT-4.0-mini)
✅ Google Gemini API key (for embeddings)
✅ n8n setup for automation
✅ (Optional) YouTube link for demo & visualization
🛠 Setup Instructions
🔗 Connect the workflow to your source folder in Google Drive
🔑 Add OpenAI/OpenRouter and Gemini API credentials
🧾 Use structured text markers like [SECTIONEND] for clean chunking
🔁 Loop through sections and enrich with AI-generated context
🧠 Generate embeddings using Gemini's text-embedding-004 model
📦 Store final vectors in Pinecone, including original + enriched context
🧪 Test with a small document before scaling
🔌 Integrations
Google Drive
OpenRouter (GPT-4.0-mini)
Google Gemini
Pinecone Vector Store
n8n
