LLM/09_rag_langchain.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building a Retrieval-Augmented Generation (RAG) System Workshop\n",
    "\n",
    "## Learning Objectives\n",
    "### Remember\n",
    "- Define the key components of a RAG system\n",
    "- Identify the essential libraries required for building a RAG pipeline\n",
    "- List the main steps in the RAG process\n",
    "\n",
    "### Understand\n",
    "- Explain how document embedding works\n",
    "- Describe the role of vector databases in RAG\n",
    "- Interpret the relationship between chunks, embeddings, and retrieval\n",
    "\n",
    "### Apply\n",
    "- Implement a document loading and chunking pipeline\n",
    "- Configure a language model for generation\n",
    "- Set up a vector store for document retrieval\n",
    "\n",
    "### Analyze\n",
    "- Compare different chunking strategies\n",
    "- Examine the impact of various parameter settings\n",
    "- Debug common issues in RAG systems\n",
    "\n",
    "### Evaluate\n",
    "- Assess the quality of generated responses\n",
    "- Test system performance with different configurations\n",
    "- Judge the effectiveness of retrieval strategies\n",
    "\n",
    "### Create\n",
    "- Design a complete RAG pipeline\n",
    "- Develop custom prompt templates\n",
    "- Build an interactive question-answering system\n",
    "\n",
    "## Prerequisites\n",
    "- Basic understanding of Python\n",
    "- Familiarity with machine learning concepts\n",
    "- Understanding of basic NLP terminology\n",
    "\n",
    "Let's begin our journey into building a RAG system!"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is RAG (Retrieval-augmented generation)\n",
    "\n",
    "**Retrieval-augmented generation (RAG)** is a technique for augmenting LLM knowledge with additional, often private or real-time, data. LLMs can reason about wide-ranging topics, but their knowledge is limited to the public data up to a specific point in time that they were trained on. If you want to build AI applications that can reason about private data or data introduced after a model’s cutoff date, you need to augment the knowledge of the model with the specific information it needs. The process of bringing the appropriate information and inserting it into the model prompt is known as Retrieval Augmented Generation (RAG)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup llama.cpp python for Intel CPUs and GPUs\n",
    "The llama.cpp SYCL backend is designed to support Intel GPU. Based on the cross-platform feature of SYCL.\n",
    "\n",
    "We will setup Python environment and corresponding custom kernel for Jupyter Notebook, and we will install/build llama.cpp that will be used for the RAG Application.\n",
    "\n",
    "### Step 1: Create and activate Python environment:\n",
    "\n",
    "Open Terminal, make sure mini-forge is install and create new virtual environment\n",
    "\n",
    "```\n",
    "    conda create -n llm-sycl python=3.11\n",
    "\n",
    "    conda activate llm-sycl\n",
    "\n",
    "```\n",
    "_Note: In case you want to remove the virtual environment, run the following command:_\n",
    "```\n",
    "    [conda remove -n llm-sycl --all]\n",
    "```\n",
    "\n",
    "### Step 2: Setup a custom kernel for Jupyter notebook:\n",
    "\n",
    "Run the following commands in the terminal to setup custom kernel for the Jupyter Notebook.\n",
    "\n",
    "```\n",
    "    conda install -c conda-forge ipykernel\n",
    "\n",
    "    python -m ipykernel install --user --name=llm-sycl\n",
    "```\n",
    "_Note: In case you want to remove the custom kernel from Jupyter, run the following command:_\n",
    "```\n",
    "    [python -m jupyter kernelspec uninstall llm-sycl]\n",
    "```\n",
    "\n",
    "<img src=\"Assets/llm4.png\">\n",
    "\n",
    "### Step 3: Install and Build llama.cpp\n",
    "\n",
    "### For Linux\n",
    "\n",
    "#### 1. Enable oneAPI environment\n",
    "\n",
    "Make sure oneAPI Base Toolkit is installed to use the SYCL compiler for building llama.cpp\n",
    "\n",
    "Run the following commands in terminal to initialize oneAPI environment and check available devices:\n",
    "\n",
    "```\n",
    "    source /opt/intel/oneapi/setvars.sh\n",
    "    sycl-ls\n",
    "```\n",
    "\n",
    "#### 2. Install and build llama.cpp Python\n",
    "\n",
    "Run the following commands in terminal to install and build llama.cpp\n",
    "\n",
    "```\n",
    "    CMAKE_ARGS=\"-DGGML_SYCL=on -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx\" pip install llama-cpp-python==0.3.1\n",
    "```\n",
    "\n",
    "### For Windows\n",
    "\n",
    "#### 1. Enable oneAPI environment\n",
    "\n",
    "Make sure oneAPI Base Toolkit is installed to use the SYCL compiler for building llama.cpp\n",
    "\n",
    "Type oneapi in the windows search and then open the Intel oneAPI command prompt for Intel 64 for Visual Studio 2022 App.\n",
    "\n",
    "Run the following commands to initialize oneAPI environment and check available devices:\n",
    "\n",
    "```\n",
    "    @call \"C:\\Program Files (x86)\\Intel\\oneAPI\\setvars.bat\" intel64 --force\n",
    "    sycl-ls\n",
    "```\n",
    "\n",
    "#### 2. Install build tools\n",
    "\n",
    "* Download & install [cmake for Windows](https://cmake.org/download/):\n",
    "* The new Visual Studio will install Ninja as default. (If not, please install it manually: https://ninja-build.org/)\n",
    "\n",
    "#### 3. Install and build llama.cpp Python\n",
    "\n",
    "* On the oneAPI command line window, step into the llama.cpp main directory and run the following:\n",
    "  \n",
    "```\n",
    "    set CMAKE_GENERATOR=Ninja\n",
    "    set CMAKE_C_COMPILER=cl\n",
    "    set CMAKE_CXX_COMPILER=icx\n",
    "    set CXX=icx\n",
    "    set CC=cl\n",
    "    set CMAKE_ARGS=\"-DGGML_SYCL=ON -DGGML_SYCL_F16=ON -DCMAKE_CXX_COMPILER=icx -DCMAKE_C_COMPILER=cl\"\n",
    "    \n",
    "    pip install llama-cpp-python==0.3.1 -U --force --no-cache-dir --verbose\n",
    "```\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RAG Application Details\n",
    "\n",
    "A typical RAG application has two main components:\n",
    "\n",
    "- **Indexing**: a pipeline for ingesting data from a source and indexing it. This usually happens offline.\n",
    "\n",
    "- **Retrieval and generation**: the actual RAG chain, which takes the user query at run time and retrieves the relevant data from the index, then passes that to the model.\n",
    "\n",
    "The most common full sequence from raw data to answer looks like:\n",
    "\n",
    "**Indexing**\n",
    "\n",
    "1. `Load`: First we need to load our data. We’ll use DocumentLoaders for this.\n",
    "2. `Split`: Text splitters break large Documents into smaller chunks. This is useful both for indexing data and for passing it in to a model, since large chunks are harder to search over and won’t in a model’s finite context window.\n",
    "3. `Store`: We need somewhere to store and index our splits, so that they can later be searched over. This is often done using a VectorStore and Embeddings model.\n",
    "\n",
    "![Indexing pipeline](https://github.com/openvinotoolkit/openvino_notebooks/assets/91237924/dfed2ba3-0c3a-4e0e-a2a7-01638730486a)\n",
    "\n",
    "**Retrieval and generation**\n",
    "\n",
    "1. `Retrieve`: Given a user input, relevant splits are retrieved from storage using a Retriever.\n",
    "2. `Generate`: A LLM produces an answer using a prompt that includes the question and the retrieved data.\n",
    "\n",
    "![Retrieval and generation pipeline](https://github.com/openvinotoolkit/openvino_notebooks/assets/91237924/f0545ddc-c0cd-4569-8c86-9879fdab105a)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Python Modules\n",
    "In Jupyter Lab, select `llm-sycl` as the kernel.\n",
    "\n",
    "You can now proceed installing modules and running python code in Jupyter Notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install -r rag/requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Setting Up the Environment\n",
    "\n",
    "First, let's import all necessary libraries. We'll go through each import and understand its role in our RAG system.\n",
    "\n",
    "Key Components:\n",
    "- LangChain: Framework for developing applications powered by language models\n",
    "- HuggingFace: Platform for accessing pre-trained models and embeddings\n",
    "- Chroma: Vector store for efficient similarity search\n",
    "- LlamaCpp: Efficient C++ implementation of Llama models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import time\n",
    "import warnings\n",
    "import hashlib\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "from langchain_community.document_loaders import WebBaseLoader\n",
    "from langchain_community.document_loaders import DirectoryLoader, TextLoader\n",
    "from langchain_community.document_loaders.pdf import PyPDFLoader\n",
    "from langchain_huggingface import HuggingFaceEmbeddings\n",
    "from langchain_chroma import Chroma\n",
    "from langchain_community.llms import LlamaCpp\n",
    "from langchain import chains, text_splitter, PromptTemplate\n",
    "from huggingface_hub import hf_hub_download\n",
    "from langchain_core.callbacks import BaseCallbackHandler\n",
    "import sys\n",
    "\n",
    "# Setting constants and environment variables\n",
    "VECTOR_DB_DIR = \"vector_dbs\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Understanding Streaming Callbacks\n",
    "\n",
    "In RAG systems, especially when working with large language models, it's important to provide real-time feedback to users. The streaming callback handler allows us to see the model's output token by token, creating a more interactive experience.\n",
    "\n",
    "Key Concepts:\n",
    "- Callbacks in LangChain\n",
    "- Token-by-token streaming\n",
    "- Real-time output handling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class StreamingStdOutCallbackHandler(BaseCallbackHandler):\n",
    "    \"\"\"Callback handler for streaming LLM outputs to standard output.\n",
    "    \n",
    "    This handler intercepts tokens as they're generated by the language model\n",
    "    and prints them to stdout in real-time, creating a streaming effect.\n",
    "    \n",
    "    Attributes:\n",
    "        None\n",
    "        \n",
    "    Methods:\n",
    "        on_llm_new_token: Handles each new token generated by the LLM\n",
    "    \"\"\"\n",
    "    \n",
    "    def on_llm_new_token(self, token: str, **kwargs) -> None:\n",
    "        print(token, end=\"\", flush=True)\n",
    "\n",
    "def get_source_hash(source, source_type):\n",
    "    \"\"\"Generate a unique hash for the source\"\"\"\n",
    "    source_string = f\"{source_type}:{source}\"\n",
    "    return hashlib.md5(source_string.encode()).hexdigest()\n",
    "\n",
    "def get_source_input():\n",
    "    \"\"\"Get source type and location from user input.\"\"\"\n",
    "    while True:\n",
    "        source_type = input(\"Enter source type (url/local) or 'quit' to exit: \").lower()\n",
    "        if source_type == 'quit':\n",
    "            return None, None\n",
    "        if source_type in ['url', 'local']:\n",
    "            break\n",
    "        print(\"Invalid source type. Please enter 'url' or 'local'\")\n",
    "\n",
    "    if source_type == 'url':\n",
    "        source = input(\"Enter the URL: \")\n",
    "    else:\n",
    "        print(\"\\nAvailable files in data folder:\")\n",
    "        data_dir = os.path.join(os.getcwd(), \"data\")\n",
    "        if os.path.exists(data_dir):\n",
    "            files = os.listdir(data_dir)\n",
    "            for file in files:\n",
    "                print(f\"- {file}\")\n",
    "        source = input(\"\\nEnter filename or directory name from data folder: \")\n",
    "\n",
    "    return source, source_type"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Document Loading and Processing\n",
    "\n",
    "A crucial part of any RAG system is how it handles document loading and processing. We'll create functions to:\n",
    "1. Load documents from URLs\n",
    "2. Split documents into manageable chunks\n",
    "3. Initialize our embedding model\n",
    "\n",
    "Understanding these processes is crucial as they directly impact the quality of our retrieval system.\n",
    "\n",
    "* Document loaders in RAG are used to load and preprocess the documents that will be used for retrieval during the question answering process.\n",
    "* Document loaders are responsible for preprocessing the documents. This includes tokenizing the text, converting it to the format expected by the retriever, and creating batches of documents.\n",
    "* Document loaders work in conjunction with the retriever in RAG. The retriever uses the documents loaded by the document loader to find the most relevant documents for a given query.\n",
    "* The WebBaseLoader in Retrieval Augmented Generation (RAG) is a type of document loader that is designed to load documents from the web.\n",
    "* The WebBaseLoader is used when the documents for retrieval are not stored locally or in a Hugging Face dataset, but are instead located on the web. This can be useful when you want to use the most up-to-date information available on the internet for your question answering system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text splitter\n",
    "\n",
    "* RecursiveCharacterTextSplitter is used to split text into smaller pieces recursively at the character level. \n",
    "* split_documents fuctions splits larger documents into smaller chunks, for easier processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_document(source, source_type=\"local\"):\n",
    "    \"\"\"Load a document from a given URL using LangChain's WebBaseLoader.\n",
    "    \n",
    "    This function handles the initial document ingestion process, loading\n",
    "    web content while managing potential network and parsing issues.\n",
    "    \n",
    "    Args:\n",
    "        url (str): The URL of the document to load\n",
    "        \n",
    "    Returns:\n",
    "        list: List of Document objects containing the loaded content\n",
    "    \"\"\"\n",
    "    print(\"Loading document from URL...\")\n",
    "    if source_type == \"url\":\n",
    "        print(\"Loading document from URL...\")\n",
    "        loader = WebBaseLoader(source)\n",
    "        return loader.load()\n",
    "    else:\n",
    "        current_dir = os.getcwd()\n",
    "        data_dir = os.path.join(current_dir, \"data\")\n",
    "        full_path = os.path.join(data_dir, source)\n",
    "        \n",
    "        if not os.path.exists(full_path):\n",
    "            raise FileNotFoundError(f\"File or directory not found: {full_path}\")\n",
    "        \n",
    "        print(f\"Loading document from: {full_path}\")\n",
    "        \n",
    "        if os.path.isdir(full_path):\n",
    "            loader = DirectoryLoader(\n",
    "                full_path,\n",
    "                glob=\"**/*.*\",\n",
    "                loader_cls=TextLoader\n",
    "            )\n",
    "        elif source.endswith('.pdf'):\n",
    "            loader = PyPDFLoader(full_path)\n",
    "        else:\n",
    "            loader = TextLoader(full_path)\n",
    "        \n",
    "        return loader.load()\n",
    "\n",
    "def split_document(text, chunk_size=1000, overlap=100):\n",
    "    \"\"\"Split documents into smaller chunks for processing.\n",
    "    \n",
    "    Uses recursive character text splitting to create overlapping chunks\n",
    "    of text that are small enough to process effectively.\n",
    "    \n",
    "    Args:\n",
    "        text (list): List of Document objects to split\n",
    "        chunk_size (int): Maximum size of each chunk\n",
    "        overlap (int): Number of characters to overlap between chunks\n",
    "        \n",
    "    Returns:\n",
    "        list: List of split Document objects\n",
    "    \"\"\"\n",
    "    print(\"Splitting document into chunks...\")\n",
    "    text_splitter_instance = text_splitter.RecursiveCharacterTextSplitter(\n",
    "        chunk_size=chunk_size,\n",
    "        chunk_overlap=overlap\n",
    "    )\n",
    "    return text_splitter_instance.split_documents(text)\n",
    "\n",
    "def initialize_embedding_fn(model_name=\"sentence-transformers/paraphrase-MiniLM-L3-v2\"):\n",
    "    \"\"\"Initialize the embedding model for converting text to vector representations.\n",
    "    \n",
    "    Uses HuggingFace's sentence transformers to create embeddings that\n",
    "    capture semantic meaning of text chunks.\n",
    "    \n",
    "    Args:\n",
    "        model_name (str): Name of the HuggingFace model to use\n",
    "        \n",
    "    Returns:\n",
    "        HuggingFaceEmbeddings: Initialized embedding model\n",
    "    \"\"\"\n",
    "    \n",
    "    print(f\"Initializing embedding model with {model_name}...\")\n",
    "    return HuggingFaceEmbeddings(model_name=model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Vector Store Management\n",
    "\n",
    "The vector store is a crucial component in RAG systems. It enables efficient similarity search over document embeddings, allowing us to retrieve relevant context for any given query.\n",
    "\n",
    "Key Concepts:\n",
    "- Vector databases\n",
    "- Persistence and caching\n",
    "- Similarity search\n",
    "- Document retrieval\n",
    "- In Retrieval Augmented Generation (RAG) embeddings play a crucial role in the retrieval of relevant documents for a given query.\n",
    "\n",
    "* In RAG, each document in the knowledge base is represented as a dense vector, also known as an embedding. These embeddings are typically generated by a transformer model.\n",
    "* When a query is received, it is also converted into an embedding using the same transformer model. This ensures that the query and the documents are in the same vector space, making it possible to compare them.\n",
    "* Retrieval: The retrieval step in RAG involves finding the documents whose embeddings are most similar to the query embedding. This is typically done using a nearest neighbor search.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_or_create_embeddings(source, source_type, embedding_fn):\n",
    "    \"\"\"Create or load a persistent vector store for document embeddings.\n",
    "    \n",
    "    This function manages the vector database, either loading an existing one\n",
    "    or creating a new one if it doesn't exist. It handles document processing,\n",
    "    embedding generation, and storage.\n",
    "    \n",
    "    Args:\n",
    "        document_url (str): URL of the document to process\n",
    "        embedding_fn: Function to generate embeddings\n",
    "        persist_dir (str): Directory for storing the vector database\n",
    "        \n",
    "    Returns:\n",
    "        Chroma: Initialized vector store with document embeddings\n",
    "    \"\"\"\n",
    "    source_hash = get_source_hash(source, source_type)\n",
    "    persist_dir = os.path.join(VECTOR_DB_DIR, source_hash)\n",
    "\n",
    "    # Check if embeddings already exist\n",
    "    if os.path.exists(persist_dir):\n",
    "        print(\"Loading existing embeddings...\")\n",
    "        return Chroma(persist_directory=persist_dir, embedding_function=embedding_fn)\n",
    "\n",
    "    # Create new embeddings if they don't exist\n",
    "    start_time = time.time()\n",
    "    print(\"Creating new embeddings...\")\n",
    "    document = load_document(source, source_type)\n",
    "    documents = split_document(document)\n",
    "    vector_store = Chroma.from_documents(\n",
    "        documents=documents,\n",
    "        embedding=embedding_fn,\n",
    "        persist_directory=persist_dir\n",
    "    )\n",
    "    \n",
    "    print(f\"Embedding time: {time.time() - start_time:.2f} seconds\")\n",
    "    return vector_store"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Language Model Configuration\n",
    "\n",
    "The language model is the heart of our RAG system. Proper configuration is essential for optimal performance and resource usage.\n",
    "\n",
    "Key Aspects:\n",
    "- Model parameters\n",
    "- GPU optimization\n",
    "- Generation settings\n",
    "- Performance tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_llm(model_path):\n",
    "    \"\"\"Initialize and configure the Llama language model.\n",
    "    \n",
    "    Sets up the language model with optimized parameters for performance\n",
    "    and resource usage. Configures GPU acceleration and generation settings.\n",
    "    \n",
    "    Args:\n",
    "        model_path (str): Path to the model weights\n",
    "        \n",
    "    Returns:\n",
    "        LlamaCpp: Configured language model instance\n",
    "    \"\"\"\n",
    "    return LlamaCpp(\n",
    "        model_path=model_path,\n",
    "        n_gpu_layers=33,\n",
    "        n_batch=256,\n",
    "        n_ctx=4096,\n",
    "        f16_kv=True,\n",
    "        verbose=True,\n",
    "        streaming=True,\n",
    "        temperature=0.7,\n",
    "        max_tokens=512,\n",
    "        top_p=0.95,\n",
    "        repeat_penalty=1.1\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Question-Answering Interface\n",
    "\n",
    "The question-answering interface provides the user interaction layer of our RAG system. It handles:\n",
    "1. User input processing\n",
    "2. Response generation\n",
    "3. Error handling\n",
    "4. Performance monitoring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ask_questions(qachain):\n",
    "    \"\"\"Handle user questions and generate responses using the RAG pipeline.\n",
    "    \n",
    "    Manages the interaction loop with users, processes questions, generates\n",
    "    responses, and handles any errors that occur during generation.\n",
    "    \n",
    "    Args:\n",
    "        qachain: The configured question-answering chain\n",
    "        \n",
    "    Returns:\n",
    "        bool: Whether to continue the interaction loop\n",
    "    \"\"\"\n",
    "    while True:\n",
    "        try:\n",
    "            question = input(\"\\nEnter your question (or 'switch' to change document, 'quit' to exit): \")\n",
    "            if question.lower() == 'quit':\n",
    "                return False\n",
    "            if question.lower() == 'switch':\n",
    "                return True\n",
    "                \n",
    "            start_time = time.time()\n",
    "            response = qachain.invoke(\n",
    "                {\n",
    "                    \"query\": question,\n",
    "                    \"max_tokens\": 512,\n",
    "                    \"temperature\": 0.7\n",
    "                }, \n",
    "                config={\"callbacks\": [StreamingStdOutCallbackHandler()]}\n",
    "            )\n",
    "            print(f\"\\nResponse time: {time.time() - start_time:.2f} seconds\")\n",
    "            \n",
    "        except KeyboardInterrupt:\n",
    "            print(\"\\nGeneration interrupted by user\")\n",
    "            return False\n",
    "        except Exception as e:\n",
    "            print(f\"\\nAn error occurred: {str(e)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Main Application Logic\n",
    "\n",
    "The main function orchestrates all components of our RAG system. It:\n",
    "1. Initializes components\n",
    "2. Sets up the RAG pipeline\n",
    "3. Manages the interaction loop\n",
    "4. Handles errors and cleanup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrievers\n",
    "\n",
    "* Retrievers are responsible for fetching relevant documents from a document store or knowledge base given a query. The retrieved documents are then used by the generator to produce a response.\n",
    "* RetrievalQA is a type of question answering system that uses a retriever to fetch relevant documents given a question, and then uses a reader to extract the answer from the retrieved documents.\n",
    "* RetrievalQA can be seen as a two-step process:\n",
    "    * Retrieval: The retriever fetches relevant documents from the document store given a query.    \n",
    "    * Generation: The generator uses the retrieved documents to generate a response.\n",
    "* This two-step process allows RAG to leverage the strengths of both retrieval-based and generation-based approaches to question answering. The retriever allows RAG to efficiently search a large document store, while the generator allows RAG to generate detailed and coherent responses.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#from langchain_community.vectorstores.chroma import Chroma\n",
    "from langchain_chroma import Chroma\n",
    "def main():\n",
    "    \"\"\"Main function to run the RAG system.\n",
    "    \n",
    "    Orchestrates the entire RAG pipeline, including model initialization,\n",
    "    vector store setup, and the question-answering loop.\n",
    "    \n",
    "    Args:\n",
    "        None\n",
    "        \n",
    "    Returns:\n",
    "        None\n",
    "    \"\"\"\n",
    "    model_name_or_path = \"TheBloke/Llama-2-7B-Chat-GGUF\"\n",
    "    model_basename = \"llama-2-7b-chat.Q4_K_M.gguf\"\n",
    "    print(\"Downloading Model...\\n   \" + model_name_or_path + \"/\" + model_basename)\n",
    "    MODEL_PATH = hf_hub_download(repo_id=model_name_or_path, filename=model_basename)\n",
    "    print(\"Download Complete.\")\n",
    "    \n",
    "    # Initialize components that don't need to be recreated\n",
    "    embedding_fn = initialize_embedding_fn()\n",
    "    chat_model = create_llm(MODEL_PATH)\n",
    "    \n",
    "    # Create base prompt template\n",
    "    prompt_template = \"\"\"\n",
    "    Answer the question based on the context below. Keep your answer concise.\n",
    "    If you don't know, just say \"I don't know.\"\n",
    "\n",
    "    Context: {context}\n",
    "\n",
    "    Question: {question}\n",
    "    Answer:\n",
    "    \"\"\"\n",
    "    prompt = PromptTemplate(template=prompt_template, input_variables=[\"context\", \"question\"])\n",
    "    chain_type_kwargs = {\"prompt\": prompt}\n",
    "\n",
    "    while True:\n",
    "        try:\n",
    "            # Get source input from user\n",
    "            source, source_type = get_source_input()\n",
    "            if source is None:  # User chose to quit\n",
    "                break\n",
    "\n",
    "            # Get or create embeddings for the current document\n",
    "            vector_store = get_or_create_embeddings(source, source_type, embedding_fn)\n",
    "            \n",
    "            # Setup retriever and chain\n",
    "            retriever = vector_store.as_retriever(search_kwargs={\"k\": 4})\n",
    "            qachain = chains.RetrievalQA.from_chain_type(\n",
    "                llm=chat_model,\n",
    "                retriever=retriever,\n",
    "                chain_type=\"stuff\",\n",
    "                chain_type_kwargs=chain_type_kwargs,\n",
    "                return_source_documents=False\n",
    "            )\n",
    "\n",
    "            print(\"\\nModel is ready! Ask your questions.\")\n",
    "            if not ask_questions(qachain):\n",
    "                break\n",
    "                \n",
    "        except Exception as e:\n",
    "            print(f\"An error occurred: {str(e)}\")\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    try:\n",
    "        main()\n",
    "    except KeyboardInterrupt:\n",
    "        print(\"\\nProgram interrupted by user\")\n",
    "    except Exception as e:\n",
    "        print(f\"An error occurred: {str(e)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Workshop Summary\n",
    "\n",
    "In this workshop, we've built a complete RAG system from scratch, covering:\n",
    "\n",
    "### Technical Components\n",
    "1. Document Processing\n",
    "   - Web document loading\n",
    "   - Text chunking strategies\n",
    "   - Embedding generation\n",
    "\n",
    "2. Vector Store Management\n",
    "   - Persistent storage\n",
    "   - Efficient retrieval\n",
    "   - Similarity search\n",
    "\n",
    "3. Language Model Integration\n",
    "   - Model configuration\n",
    "   - Parameter optimization\n",
    "   - Response generation\n",
    "\n",
    "### Key Learnings\n",
    "1. System Architecture\n",
    "   - Understanding RAG pipeline components\n",
    "   - Component interaction\n",
    "   - Error handling\n",
    "\n",
    "2. Performance Optimization\n",
    "   - Memory management\n",
    "   - GPU utilization\n",
    "   - Response time optimization\n",
    "\n",
    "3. Best Practices\n",
    "   - Code organization\n",
    "   - Documentation\n",
    "   - Error handling\n",
    "\n",
    "### Next Steps\n",
    "To further improve the system, consider:\n",
    "1. Implementing different embedding models\n",
    "2. Experimenting with chunk sizes and overlap\n",
    "3. Adding evaluation metrics\n",
    "4. Implementing caching mechanisms\n",
    "5. Adding support for different document types\n",
    "\n",
    "This workshop provides a foundation for building and understanding RAG systems, which you can extend and customize for your specific use cases."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}