notebooks/llm-native-agent-react/llm-native-agent-react.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5fc61857-629d-476c-84fc-927a63a12f0f",
   "metadata": {},
   "source": [
    "# Create a native Agent with OpenVINO\n",
    "\n",
    "LLM are limited to the knowledge on which they have been trained and the additional knowledge provided as context, as a result, if a useful piece of information is missing the provided knowledge, the model cannot “go around” and try to find it in other sources. This is the reason why we need to introduce the concept of Agents.\n",
    "\n",
    "The core idea of agents is to use a language model to choose a sequence of actions to take. In agents, a language model is used as a reasoning engine to determine which actions to take and in which order. Agents can be seen as applications powered by LLMs and integrated with a set of tools like search engines, databases, websites, and so on. Within an agent, the LLM is the reasoning engine that, based on the user input, is able to plan and execute a set of actions that are needed to fulfill the request.\n",
    "\n",
    "![agent](https://github.com/openvinotoolkit/openvino_notebooks/assets/91237924/22fa5396-8381-400f-a78f-97e25d57d807)\n",
    "\n",
    "This example will demonstrate how to create a native agent with OpenVINO.\n",
    "\n",
    "#### Table of contents:\n",
    "\n",
    "- [Prerequisites](#Prerequisites)\n",
    "- [Create LLM as agent](#Create-LLM-as-agent)\n",
    "    - [Download model](#Select-model)\n",
    "    - [Select inference device for LLM](#Select-inference-device-for-LLM)\n",
    "    - [Instantiate LLM using Optimum Intel](#Instantiate-LLM-using-Optimum-Intel)\n",
    "    - [Create text generation method](#Create-text-generation-method)\n",
    "- [Create prompt template](#Create-prompt-template)\n",
    "- [Create parser](#Create-parers)\n",
    "- [Create tools calling](#Create-tool-calling)\n",
    "- [Run agent](#Run-agent)\n",
    "- [Create AI agent demo with Gradio UI](#Create-AI-agent-demo-with-Gradio-UI)\n",
    "\n",
    "### Installation Instructions\n",
    "\n",
    "This is a self-contained example that relies solely on its own code.\n",
    "\n",
    "We recommend  running the notebook in a virtual environment. You only need a Jupyter server to start.\n",
    "For details, please refer to [Installation Guide](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/README.md#-installation-guide).\n",
    "\n",
    "<img referrerpolicy=\"no-referrer-when-downgrade\" src=\"https://static.scarf.sh/a.png?x-pxid=5b5a4db0-7875-4bfb-bdbd-01698b5b1a77&file=notebooks/llm-native-agent-react/llm-native-agent-react.ipynb\" />\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d6e66ddf",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd4d3ed6-01da-477f-a41b-0be41f758e68",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import requests\n",
    "from pathlib import Path\n",
    "\n",
    "if not Path(\"notebook_utils.py\").exists():\n",
    "    r = requests.get(\n",
    "        url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py\",\n",
    "    )\n",
    "    open(\"notebook_utils.py\", \"w\").write(r.text)\n",
    "\n",
    "if not Path(\"pip_helper.py\").exists():\n",
    "    r = requests.get(\n",
    "        url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/pip_helper.py\",\n",
    "    )\n",
    "    open(\"pip_helper.py\", \"w\").write(r.text)\n",
    "\n",
    "os.environ[\"GIT_CLONE_PROTECTION_ACTIVE\"] = \"false\"\n",
    "\n",
    "from pip_helper import pip_install\n",
    "\n",
    "pip_install(\n",
    "    \"-q\",\n",
    "    \"--extra-index-url\",\n",
    "    \"https://download.pytorch.org/whl/cpu\",\n",
    "    \"transformers>=4.43.1\",\n",
    "    \"gradio>=4.19\",\n",
    ")\n",
    "pip_install(\n",
    "    \"-q\",\n",
    "    \"git+https://github.com/huggingface/optimum-intel.git\",\n",
    "    \"git+https://github.com/openvinotoolkit/nncf.git\",\n",
    "    \"datasets\",\n",
    "    \"accelerate\",\n",
    "    \"huggingface-hub>=0.26.5\",\n",
    ")\n",
    "pip_install(\"--pre\", \"-Uq\", \"openvino>=2024.4.0\", \"--extra-index-url\", \"https://storage.openvinotoolkit.org/simple/wheels/nightly\")\n",
    "\n",
    "# Read more about telemetry collection at https://github.com/openvinotoolkit/openvino_notebooks?tab=readme-ov-file#-telemetry\n",
    "from notebook_utils import collect_telemetry\n",
    "\n",
    "collect_telemetry(\"llm-native-agent-react.ipynb\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e04a1a2b-873b-4c05-a9fc-9a762ddeffe7",
   "metadata": {},
   "source": [
    "## Create LLM as agent\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "### Download LLM\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "To run LLM locally, we have to download the model in the first step. It is possible to [export your model](https://github.com/huggingface/optimum-intel?tab=readme-ov-file#export) to the OpenVINO IR format with the CLI, and load the model from local folder.\n",
    "\n",
    "Large Language Models (LLMs) are a core component of agent. LlamaIndex does not serve its own LLMs, but rather provides a standard interface for interacting with many different LLMs. In this example, we can select `Qwen2.5` as LLM in agent pipeline.\n",
    "* **qwen2.5-3b-instruct/qwen2.5-7b-instruct/qwen2.5-14b-instruct** - Qwen2.5 is the latest series of Qwen large language models. Comparing with Qwen2, Qwen2.5 series brings significant improvements in coding, mathematics and general knowledge skills. Additionally, it brings long-context and multiple languages support including Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, Arabic, and more. \n",
    "For more details, please refer to [model_card](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct), [blog](https://qwenlm.github.io/blog/qwen2.5/), [GitHub](https://github.com/QwenLM/Qwen2.5), and [Documentation](https://qwen.readthedocs.io/en/latest/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c0c9e882-5622-454e-b28c-9d799678b8cd",
   "metadata": {
    "test_replace": {
     "Qwen/Qwen2.5-3B-Instruct": "Qwen/Qwen2.5-1.5B-Instruct"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "13f76edf623645bda82412eb5d3195a4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Model:', options=('Qwen/Qwen2.5-3B-Instruct', 'Qwen/Qwen2.5-7B-Instruct', 'Qwen/qwen2.5-…"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import ipywidgets as widgets\n",
    "\n",
    "llm_model_ids = [\"Qwen/Qwen2.5-3B-Instruct\", \"Qwen/Qwen2.5-7B-Instruct\", \"Qwen/qwen2.5-14b-instruct\"]\n",
    "\n",
    "llm_model_id = widgets.Dropdown(\n",
    "    options=llm_model_ids,\n",
    "    value=llm_model_ids[0],\n",
    "    description=\"Model:\",\n",
    "    disabled=False,\n",
    ")\n",
    "\n",
    "llm_model_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "47707d27-9fe6-4bd6-b7c7-087311170a9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "llm_model_path = llm_model_id.value.split(\"/\")[-1]\n",
    "\n",
    "if not Path(llm_model_path).exists():\n",
    "    !optimum-cli export openvino --model {llm_model_id.value} --task text-generation-with-past --trust-remote-code --weight-format int4 --group-size 128 --ratio 1.0 --sym {llm_model_path}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "30dcdf3a",
   "metadata": {},
   "source": [
    "### Select inference device for LLM\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "74be1b36-a56d-41a7-9d1a-d487b22e5dd3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5b8db0163d4e42e396b9772c3d96fcaa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Device:', options=('CPU', 'GPU', 'AUTO'), value='CPU')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from notebook_utils import device_widget\n",
    "\n",
    "llm_device = device_widget(\"CPU\", exclude=[\"NPU\"])\n",
    "\n",
    "llm_device"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0e47dddf",
   "metadata": {},
   "source": [
    "## Instantiate LLM using Optimum Intel\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Optimum Intel can be used to load optimized models from the [Hugging Face Hub](https://huggingface.co/docs/optimum/intel/hf.co/models) and create pipelines to run an inference with OpenVINO Runtime using Hugging Face APIs. The Optimum Inference models are API compatible with Hugging Face Transformers models.  This means we just need to replace `AutoModelForXxx` class with the corresponding `OVModelForXxx` class.\n",
    "\n",
    "Below is an example of the RedPajama model\n",
    "\n",
    "```diff\n",
    "-from transformers import AutoModelForCausalLM\n",
    "+from optimum.intel.openvino import OVModelForCausalLM\n",
    "from transformers import AutoTokenizer, pipeline\n",
    "\n",
    "model_id = \"togethercomputer/RedPajama-INCITE-Chat-3B-v1\"\n",
    "-model = AutoModelForCausalLM.from_pretrained(model_id)\n",
    "+model = OVModelForCausalLM.from_pretrained(model_id, export=True)\n",
    "```\n",
    "\n",
    "Model class initialization starts with calling `from_pretrained` method. When downloading and converting Transformers model, the parameter `export=True` should be added (as we already converted model before, we do not need to provide this parameter). We can save the converted model for the next usage with the `save_pretrained` method.\n",
    "Tokenizer class and pipelines API are compatible with Optimum models.\n",
    "\n",
    "You can find more details about OpenVINO LLM inference using HuggingFace Optimum API in [LLM inference guide](https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "587d43fd-2f20-4ac5-9e5e-5fa0f48bed36",
   "metadata": {},
   "outputs": [],
   "source": [
    "from optimum.intel.openvino import OVModelForCausalLM\n",
    "from transformers import AutoTokenizer, AutoConfig, TextStreamer\n",
    "from transformers.generation import (\n",
    "    StoppingCriteriaList,\n",
    "    StoppingCriteria,\n",
    ")\n",
    "import openvino.properties as props\n",
    "import openvino.properties.hint as hints\n",
    "import openvino.properties.streams as streams\n",
    "\n",
    "import json\n",
    "import json5\n",
    "import torch\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(llm_model_path, trust_remote_code=True)\n",
    "\n",
    "ov_config = {hints.performance_mode(): hints.PerformanceMode.LATENCY, streams.num(): \"1\", props.cache_dir(): \"\"}\n",
    "\n",
    "llm = OVModelForCausalLM.from_pretrained(\n",
    "    llm_model_path,\n",
    "    device=llm_device.value,\n",
    "    ov_config=ov_config,\n",
    "    config=AutoConfig.from_pretrained(llm_model_path, trust_remote_code=True),\n",
    "    trust_remote_code=True,\n",
    ")\n",
    "\n",
    "llm.generation_config.top_k = 1\n",
    "llm.generation_config.max_length = 2000"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0c2bff03",
   "metadata": {},
   "source": [
    "### Create text generation method\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "In this example, we would like to stream the output text though `TextStreamer`, and stop text generation before `Observation` received from tool calling.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c379d83b",
   "metadata": {},
   "outputs": [],
   "source": [
    "class StopSequenceCriteria(StoppingCriteria):\n",
    "    \"\"\"\n",
    "    This class can be used to stop generation whenever a sequence of tokens is encountered.\n",
    "\n",
    "    Args:\n",
    "        stop_sequences (`str` or `List[str]`):\n",
    "            The sequence (or list of sequences) on which to stop execution.\n",
    "        tokenizer:\n",
    "            The tokenizer used to decode the model outputs.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, stop_sequences, tokenizer):\n",
    "        if isinstance(stop_sequences, str):\n",
    "            stop_sequences = [stop_sequences]\n",
    "        self.stop_sequences = stop_sequences\n",
    "        self.tokenizer = tokenizer\n",
    "\n",
    "    def __call__(self, input_ids, scores, **kwargs) -> bool:\n",
    "        decoded_output = self.tokenizer.decode(input_ids.tolist()[0])\n",
    "        return any(decoded_output.endswith(stop_sequence) for stop_sequence in self.stop_sequences)\n",
    "\n",
    "\n",
    "def text_completion(prompt: str, stop_words) -> str:\n",
    "    im_end = \"<|im_end|>\"\n",
    "    if im_end not in stop_words:\n",
    "        stop_words = stop_words + [im_end]\n",
    "    streamer = TextStreamer(tokenizer, timeout=60.0, skip_prompt=True, skip_special_tokens=True)\n",
    "\n",
    "    stopping_criteria = StoppingCriteriaList([StopSequenceCriteria(stop_words, tokenizer)])\n",
    "    input_ids = torch.tensor([tokenizer.encode(prompt)])\n",
    "    generate_kwargs = dict(\n",
    "        input_ids=input_ids,\n",
    "        streamer=streamer,\n",
    "        stopping_criteria=stopping_criteria,\n",
    "    )\n",
    "    output = llm.generate(**generate_kwargs)\n",
    "    output = output.tolist()[0]\n",
    "    output = tokenizer.decode(output, errors=\"ignore\")\n",
    "    assert output.startswith(prompt)\n",
    "    output = output[len(prompt) :].replace(\"<|endoftext|>\", \"\").replace(im_end, \"\")\n",
    "\n",
    "    for stop_str in stop_words:\n",
    "        idx = output.find(stop_str)\n",
    "        if idx != -1:\n",
    "            output = output[: idx + len(stop_str)]\n",
    "    return output"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "05fe6e0c",
   "metadata": {},
   "source": [
    "## Create prompt template\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "A prompt for a language model is a set of instructions or input provided by a user to guide the model's response, helping it understand the context and generate relevant and coherent language-based output, such as answering questions, completing sentences, or engaging in a conversation.\n",
    "\n",
    "Different agents have different prompting styles for reasoning. In this example, we will use [ReAct agent](https://react-lm.github.io/) with its typical prompt template. For a full list of built-in agents see [agent types](https://python.langchain.com/docs/modules/agents/agent_types/).\n",
    "\n",
    "![react](https://github.com/user-attachments/assets/c26432c2-3cf1-4942-ae03-fd8e8ebb4509)\n",
    "\n",
    "A ReAct prompt consists of few-shot task-solving trajectories, with human-written text reasoning traces and actions, as well as environment observations in response to actions. ReAct prompting is intuitive and flexible to design, and achieves state-of-the-art few-shot performances across a variety of tasks, from question answering to online shopping!\n",
    "\n",
    "In an prompt template for agent, `query` is user's query and other parameter should be a sequence of messages that contains the `descriptions` and `parameters` of agent tool."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f614efa9-2c13-45ec-89ad-1dab63f346d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "TOOL_DESC = \"\"\"{name_for_model}: Call this tool to interact with the {name_for_human} API. What is the {name_for_human} API useful for? {description_for_model} Parameters: {parameters}\"\"\"\n",
    "\n",
    "PROMPT_REACT = \"\"\"Answer the following questions as best you can. You have access to the following APIs:\n",
    "\n",
    "{tools_text}\n",
    "\n",
    "Use the following format:\n",
    "\n",
    "Question: the input question you must answer\n",
    "Thought: you should always think about what to do\n",
    "Action: the action to take, should be one of [{tools_name_text}]\n",
    "Action Input: the input to the action\n",
    "Observation: the result of the action\n",
    "... (this Thought/Action/Action Input/Observation can be repeated zero or more times)\n",
    "Thought: I now know the final answer\n",
    "Final Answer: the final answer to the original input question\n",
    "\n",
    "Begin!\n",
    "\n",
    "Question: {query}\"\"\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0548431e",
   "metadata": {},
   "source": [
    "Meanwhile we have to create function for consolidate the tools information and conversation history into the prompt template."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "71d46914",
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_input_text(chat_history, list_of_tool_info) -> str:\n",
    "    tools_text = []\n",
    "    for tool_info in list_of_tool_info:\n",
    "        tool = TOOL_DESC.format(\n",
    "            name_for_model=tool_info[\"name_for_model\"],\n",
    "            name_for_human=tool_info[\"name_for_human\"],\n",
    "            description_for_model=tool_info[\"description_for_model\"],\n",
    "            parameters=json.dumps(tool_info[\"parameters\"], ensure_ascii=False),\n",
    "        )\n",
    "        if tool_info.get(\"args_format\", \"json\") == \"json\":\n",
    "            tool += \" Format the arguments as a JSON object.\"\n",
    "        elif tool_info[\"args_format\"] == \"code\":\n",
    "            tool += \" Enclose the code within triple backticks (`) at the beginning and end of the code.\"\n",
    "        else:\n",
    "            raise NotImplementedError\n",
    "        tools_text.append(tool)\n",
    "    tools_text = \"\\n\\n\".join(tools_text)\n",
    "\n",
    "    tools_name_text = \", \".join([tool_info[\"name_for_model\"] for tool_info in list_of_tool_info])\n",
    "\n",
    "    messages = [{\"role\": \"system\", \"content\": \"You are a helpful assistant.\"}]\n",
    "    for i, (query, response) in enumerate(chat_history):\n",
    "        if list_of_tool_info:\n",
    "            if (len(chat_history) == 1) or (i == len(chat_history) - 2):\n",
    "                query = PROMPT_REACT.format(\n",
    "                    tools_text=tools_text,\n",
    "                    tools_name_text=tools_name_text,\n",
    "                    query=query,\n",
    "                )\n",
    "        if query:\n",
    "            messages.append({\"role\": \"user\", \"content\": query})\n",
    "        if response:\n",
    "            messages.append({\"role\": \"assistant\", \"content\": response})\n",
    "\n",
    "    prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False, return_tensors=\"pt\")\n",
    "\n",
    "    return prompt"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "dbed3668",
   "metadata": {},
   "source": [
    "## Create parser\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "A Parser is used to convert raw output of LLM to the input arguments of tools."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8c9cbc6a-6114-4183-b49d-25d0ae2b5a9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_latest_tool_call(text):\n",
    "    tool_name, tool_args = \"\", \"\"\n",
    "    i = text.rfind(\"\\nAction:\")\n",
    "    j = text.rfind(\"\\nAction Input:\")\n",
    "    k = text.rfind(\"\\nObservation:\")\n",
    "    if 0 <= i < j:  # If the text has `Action` and `Action input`,\n",
    "        if k < j:  # but does not contain `Observation`,\n",
    "            # then it is likely that `Observation` is ommited by the LLM,\n",
    "            # because the output text may have discarded the stop word.\n",
    "            text = text.rstrip() + \"\\nObservation:\"  # Add it back.\n",
    "        k = text.rfind(\"\\nObservation:\")\n",
    "        tool_name = text[i + len(\"\\nAction:\") : j].strip()\n",
    "        tool_args = text[j + len(\"\\nAction Input:\") : k].strip()\n",
    "        text = text[:k]\n",
    "    return tool_name, tool_args, text"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "23b1673b",
   "metadata": {},
   "source": [
    "## Create tools calling\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "In this examples, we will create 2 customized tools for `image generation` and `weather qurey`. A detailed description of these tools should be defined in json format, which will be used as part of prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b7b30a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "tools = [\n",
    "    {\n",
    "        \"name_for_human\": \"get weather\",\n",
    "        \"name_for_model\": \"get_weather\",\n",
    "        \"description_for_model\": \"Get the current weather in a given city name.\",\n",
    "        \"parameters\": [\n",
    "            {\n",
    "                \"name\": \"city_name\",\n",
    "                \"description\": \"City name\",\n",
    "                \"required\": True,\n",
    "                \"schema\": {\"type\": \"string\"},\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"name_for_human\": \"image generation\",\n",
    "        \"name_for_model\": \"image_gen\",\n",
    "        \"description_for_model\": \"AI painting (image generation) service, input text description, and return the image URL drawn based on text information.\",\n",
    "        \"parameters\": [\n",
    "            {\n",
    "                \"name\": \"prompt\",\n",
    "                \"description\": \"describe the image\",\n",
    "                \"required\": True,\n",
    "                \"schema\": {\"type\": \"string\"},\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "975b72e3",
   "metadata": {},
   "source": [
    "Then we should implement these tools with inputs and outputs, and execute them according to the output of LLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "1b544d16",
   "metadata": {},
   "outputs": [],
   "source": [
    "def call_tool(tool_name: str, tool_args: str) -> str:\n",
    "    if tool_name == \"get_weather\":\n",
    "        city_name = json5.loads(tool_args)[\"city_name\"]\n",
    "        key_selection = {\n",
    "            \"current_condition\": [\n",
    "                \"temp_C\",\n",
    "                \"FeelsLikeC\",\n",
    "                \"humidity\",\n",
    "                \"weatherDesc\",\n",
    "                \"observation_time\",\n",
    "            ],\n",
    "        }\n",
    "        resp = requests.get(f\"https://wttr.in/{city_name}?format=j1\")\n",
    "        resp.raise_for_status()\n",
    "        resp = resp.json()\n",
    "        ret = {k: {_v: resp[k][0][_v] for _v in v} for k, v in key_selection.items()}\n",
    "        return str(ret)\n",
    "    elif tool_name == \"image_gen\":\n",
    "        import urllib.parse\n",
    "\n",
    "        tool_args = tool_args.replace(\"(\", \"\").replace(\")\", \"\")\n",
    "        prompt = json5.loads(tool_args)[\"prompt\"]\n",
    "        prompt = urllib.parse.quote(prompt)\n",
    "        return json.dumps(\n",
    "            {\"image_url\": f\"https://image.pollinations.ai/prompt/{prompt}\"},\n",
    "            ensure_ascii=False,\n",
    "        )\n",
    "    else:\n",
    "        raise NotImplementedError\n",
    "\n",
    "\n",
    "def llm_with_tool(prompt: str, history, list_of_tool_info=()):\n",
    "    chat_history = [(x[\"user\"], x[\"bot\"]) for x in history] + [(prompt, \"\")]\n",
    "\n",
    "    planning_prompt = build_input_text(chat_history, list_of_tool_info)\n",
    "    text = \"\"\n",
    "    while True:\n",
    "        output = text_completion(planning_prompt + text, stop_words=[\"Observation:\", \"Observation:\\n\"])\n",
    "        action, action_input, output = parse_latest_tool_call(output)\n",
    "        if action:\n",
    "            observation = call_tool(action, action_input)\n",
    "            output += f\"\\nObservation: = {observation}\\nThought:\"\n",
    "            observation = f\"{observation}\\nThought:\"\n",
    "            print(observation)\n",
    "            text += output\n",
    "        else:\n",
    "            text += output\n",
    "            break\n",
    "\n",
    "    new_history = []\n",
    "    new_history.extend(history)\n",
    "    new_history.append({\"user\": prompt, \"bot\": text})\n",
    "    return text, new_history"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "745969ca",
   "metadata": {},
   "source": [
    "## Run agent\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "560cd122-17b8-4d04-a75b-3d048ac89ae7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Thought: First, I need to use the get_weather API to get the current weather in London.\n",
      "Action: get_weather\n",
      "Action Input: {\"city_name\": \"London\"}\n",
      "Observation:\n",
      "{'current_condition': {'temp_C': '6', 'FeelsLikeC': '3', 'humidity': '75', 'weatherDesc': [{'value': 'Overcast'}], 'observation_time': '12:32 AM'}}\n",
      "Thought:\n",
      " Now that I have the weather information, I will use the image_gen API to generate an image of Big Ben based on the weather conditions.\n",
      "Action: image_gen\n",
      "Action Input: {\"prompt\": \"Big Ben under an overcast sky with a temperature of 6 degrees Celsius and humidity of 75%\"}\n",
      "Observation:\n",
      "{\"image_url\": \"https://image.pollinations.ai/prompt/Big%20Ben%20under%20an%20overcast%20sky%20with%20a%20temperature%20of%206%20degrees%20Celsius%20and%20humidity%20of%2075%25\"}\n",
      "Thought:\n",
      " I now know the final answer.\n",
      "Final Answer: The current weather in London is overcast with a temperature of 6 degrees Celsius and humidity of 75%. Based on this information, here is the generated image of Big Ben under an overcast sky: ![](https://image.pollinations.ai/prompt/Big%20Ben%20under%20an%20overcast%20sky%20with%20a%20temperature%20of%206%20degrees%20Celsius%20and%20humidity%20of%2075%25).\n"
     ]
    }
   ],
   "source": [
    "history = []\n",
    "query = \"get the weather in London, and create a picture of Big Ben based on the weather information\"\n",
    "\n",
    "response, history = llm_with_tool(prompt=query, history=history, list_of_tool_info=tools)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4d7999a8-32a7-48f2-a870-9b51ec47f92f",
   "metadata": {},
   "source": [
    "## Create AI agent demo with Gradio UI\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ae12eeb-e1d8-47ca-9fd3-a30c1687bbb6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import TextIteratorStreamer\n",
    "from threading import Thread\n",
    "\n",
    "\n",
    "def run_chatbot(history):\n",
    "    \"\"\"\n",
    "    callback function for running chatbot on submit button click\n",
    "\n",
    "    Params:\n",
    "      history: conversation history\n",
    "\n",
    "    \"\"\"\n",
    "    chat_history = [(history[-1][0], \"\")]\n",
    "\n",
    "    prompt = build_input_text(chat_history, tools)\n",
    "    text = \"\"\n",
    "    while True:\n",
    "        planning_prompt = prompt + text\n",
    "        im_end = \"<|im_end|>\"\n",
    "        stop_words = [\"Observation:\", \"Observation:\\n\"]\n",
    "        if im_end not in stop_words:\n",
    "            stop_words = stop_words + [im_end]\n",
    "        streamer = TextIteratorStreamer(tokenizer, timeout=60.0, skip_prompt=True, skip_special_tokens=True)\n",
    "\n",
    "        stopping_criteria = StoppingCriteriaList([StopSequenceCriteria(stop_words, tokenizer)])\n",
    "        input_ids = torch.tensor([tokenizer.encode(planning_prompt)])\n",
    "        generate_kwargs = dict(\n",
    "            input_ids=input_ids,\n",
    "            streamer=streamer,\n",
    "            stopping_criteria=stopping_criteria,\n",
    "        )\n",
    "\n",
    "        thread = Thread(target=llm.generate, kwargs=generate_kwargs)\n",
    "        thread.start()\n",
    "        output = \"\"\n",
    "        output_gui = \"\"\n",
    "        show_response = False\n",
    "        for new_text in streamer:\n",
    "            output += new_text\n",
    "            if \"Final\" in new_text:\n",
    "                show_response = True\n",
    "            if show_response:\n",
    "                output_gui += new_text\n",
    "                history[-1][1] = output_gui\n",
    "                yield history\n",
    "\n",
    "        # assert buffer.startswith(prompt)\n",
    "        for stop_str in stop_words:\n",
    "            idx = output.find(stop_str)\n",
    "            if idx != -1:\n",
    "                output = output[: idx + len(stop_str)]\n",
    "        print(output)\n",
    "        action, action_input, output = parse_latest_tool_call(output)\n",
    "        if action:\n",
    "            observation = call_tool(action, action_input)\n",
    "            output += f\"\\nObservation: = {observation}\\nThought:\"\n",
    "            observation = f\"{observation}\\nThought:\"\n",
    "            print(observation)\n",
    "            text += output\n",
    "        else:\n",
    "            text += output\n",
    "            break\n",
    "\n",
    "\n",
    "def request_cancel():\n",
    "    llm.request.cancel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f24d3a8e-58fa-4bc0-be7e-78a6054fcdab",
   "metadata": {},
   "outputs": [],
   "source": [
    "if not Path(\"gradio_helper.py\").exists():\n",
    "    r = requests.get(url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/llm-native-agent-react/gradio_helper.py\")\n",
    "    open(\"gradio_helper.py\", \"w\").write(r.text)\n",
    "\n",
    "from gradio_helper import make_demo\n",
    "\n",
    "examples = [\n",
    "    [\"Based on current weather in Beijing, show me a picture of Great Wall through its URL\"],\n",
    "    [\"Create an image of pink cat and return its URL\"],\n",
    "    [\"What is the weather like in New York now ?\"],\n",
    "]\n",
    "\n",
    "demo = make_demo(run_fn=run_chatbot, stop_fn=request_cancel, examples=examples)\n",
    "\n",
    "try:\n",
    "    demo.launch()\n",
    "except Exception:\n",
    "    demo.launch(share=True)\n",
    "# If you are launching remotely, specify server_name and server_port\n",
    "# EXAMPLE: `demo.launch(server_name='your server name', server_port='server port in int')`\n",
    "# To learn more please refer to the Gradio docs: https://gradio.app/docs/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "efdbf24f-20de-48f1-a5c1-5f381ffe1a17",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Closing server running on port: 5612\n"
     ]
    }
   ],
   "source": [
    "# please uncomment and run this cell for stopping gradio interface\n",
    "# demo.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  },
  "openvino_notebooks": {
   "imageUrl": "https://github.com/openvinotoolkit/openvino_notebooks/assets/91237924/22fa5396-8381-400f-a78f-97e25d57d807",
   "tags": {
    "categories": [
     "Model Demos",
     "AI Trends"
    ],
    "libraries": [],
    "other": [
     "LLM"
    ],
    "tasks": [
     "Text Generation"
    ]
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}