notebooks/minicpm-o-omnimodal-chatbot/minicpm-o-omnimodal-chatbot.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5918b41c-dad7-4f7b-9e39-b3026933dddf",
   "metadata": {},
   "source": [
    "# Omnimodal assistant with MiniCPM-o 2.6 and OpenVINO\n",
    "\n",
    "MiniCPM-o 2.6 is the latest and most capable model in the MiniCPM-o series. The model is built in an end-to-end fashion based on SigLip-400M, Whisper-medium-300M, ChatTTS-200M, and Qwen2.5-7B with a total of 8B parameters. It exhibits a significant performance improvement over MiniCPM-V 2.6, and introduces new features for real-time speech conversation and multimodal live streaming.\n",
    "\n",
    "More details about model can be found in [model card](https://huggingface.co/openbmb/MiniCPM-o-2_6) and original [repo](https://github.com/OpenBMB/MiniCPM-O).\n",
    "\n",
    "In this tutorial we consider how to convert and optimize MiniCPM-o 2.6 model for creating omnimodal chatbot. Additionally, we demonstrate how to apply stateful transformation on LLM part and model optimization techniques like weights compression using [NNCF](https://github.com/openvinotoolkit/nncf)\n",
    "\n",
    "#### Table of contents:\n",
    "\n",
    "- [Prerequisites](#Prerequisites)\n",
    "- [Convert model to OpenVINO Intermediate Representation](#Convert-model-to-OpenVINO-Intermediate-Representation)\n",
    "    - [Compress Language Model Weights to 4 bits](#Compress-Language-Model-Weights-to-4-bits)\n",
    "- [Prepare model inference pipeline](#Prepare-model-inference-pipeline)\n",
    "    - [Select device](#Select-device)\n",
    "    - [Select language model variant](#Select-language-model-variant)\n",
    "- [Run OpenVINO model inference](#Run-OpenVINO-model-inference)\n",
    "    - [Omni mode](#Omni-mode)\n",
    "    - [Vision-Only mode](#Vision-Only-mode)\n",
    "- [Interactive demo](#Interactive-demo)\n",
    "\n",
    "\n",
    "### Installation Instructions\n",
    "\n",
    "This is a self-contained example that relies solely on its own code.\n",
    "\n",
    "We recommend  running the notebook in a virtual environment. You only need a Jupyter server to start.\n",
    "For details, please refer to [Installation Guide](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/README.md#-installation-guide).\n",
    "\n",
    "<img referrerpolicy=\"no-referrer-when-downgrade\" src=\"https://static.scarf.sh/a.png?x-pxid=5b5a4db0-7875-4bfb-bdbd-01698b5b1a77&file=notebooks/minicpm-o-omnimodal-chatbot/minicpm-o-omnimodal-chatbot.ipynb\" />\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8ff80f4e-08df-4bcd-a72a-4f77bdd5768b",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0116846d-da6f-4e81-b6be-0a882a3eb872",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install -q \"torch>=2.3.1\" \"torchvision==0.18.1\" \"librosa==0.9.0\" \"vocos==0.1.0\" \\\n",
    "\"vector-quantize-pytorch==1.18.5\" \"torchaudio==2.3.1\" \"soundfile==0.12.1\" \"timm>=0.9.2\" \"transformers>=4.40\" \"Pillow\" \"gradio>=4.19\" \\\n",
    "\"tqdm\" \"sentencepiece\" \"peft\" \"huggingface-hub>=0.24.0\" \"moviepy==1.0.3\" --no-cache-dir --extra-index-url https://download.pytorch.org/whl/cpu\n",
    "%pip install -q \"openvino>=2024.3.0\" \"nncf>=2.12.0\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1534e378-1b87-4f1b-94e8-09061e960700",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from pathlib import Path\n",
    "\n",
    "if not Path(\"minicpm_o_helper.py\").exists():\n",
    "    r = requests.get(\n",
    "        url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/minicpm-o-multimodal-chatbot/minicpm_o_helper.py\"\n",
    "    )\n",
    "    open(\"minicpm_o_helper.py\", \"w\").write(r.text)\n",
    "\n",
    "\n",
    "if not Path(\"gradio_helper.py\").exists():\n",
    "    r = requests.get(url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/minicpm-o-multimodal-chatbot/gradio_helper.py\")\n",
    "    open(\"gradio_helper.py\", \"w\").write(r.text)\n",
    "\n",
    "if not Path(\"notebook_utils.py\").exists():\n",
    "    r = requests.get(url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py\")\n",
    "    open(\"notebook_utils.py\", \"w\").write(r.text)\n",
    "\n",
    "# Read more about telemetry collection at https://github.com/openvinotoolkit/openvino_notebooks?tab=readme-ov-file#-telemetry\n",
    "from notebook_utils import collect_telemetry\n",
    "\n",
    "collect_telemetry(\"minicpm-o-omnimodal-chatbot.ipynb\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "539aae31-dded-4036-8946-67a4ec9e6034",
   "metadata": {},
   "source": [
    "## Convert model to OpenVINO Intermediate Representation\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "OpenVINO supports PyTorch models via conversion to OpenVINO Intermediate Representation (IR). [OpenVINO model conversion API](https://docs.openvino.ai/2024/openvino-workflow/model-preparation.html#convert-a-model-with-python-convert-model) should be used for these purposes. `ov.convert_model` function accepts original PyTorch model instance and example input for tracing and returns `ov.Model` representing this model in OpenVINO framework. Converted model can be used for saving on disk using `ov.save_model` function or directly loading on device using `core.complie_model`.\n",
    "\n",
    "`minicpm_helper.py` script contains helper function for model conversion, please check its content if you interested in conversion details.\n",
    "\n",
    "<details>\n",
    "  <summary><b>Click here for more detailed explanation of conversion steps</b></summary>\n",
    "MiniCPM-o 2.6 is autoregressive transformer generative model, it means that each next model step depends from model output from previous step. The generation approach is based on the assumption that the probability distribution of a word sequence can be decomposed into the product of conditional next word distributions. In other words, model predicts the next token in the loop guided by previously generated tokens until the stop-condition will be not reached (generated sequence of maximum length or end of string token obtained). The way the next token will be selected over predicted probabilities is driven by the selected decoding methodology. You can find more information about the most popular decoding methods in this [blog](https://huggingface.co/blog/how-to-generate). The entry point for the generation process for models from the Hugging Face Transformers library is the `generate` method. You can find more information about its parameters and configuration in the [documentation](https://huggingface.co/docs/transformers/v4.26.1/en/main_classes/text_generation#transformers.GenerationMixin.generate). To preserve flexibility in the selection decoding methodology, we will convert only model inference for one step.\n",
    "\n",
    "The inference flow has difference on first step and for the next. On the first step, model accept preprocessed input instruction, audio and image, that transformed to the unified embedding space using `text embedding`, `audio encoder` and `image encoder` models, after that `language model`, LLM-based part of model, runs on input embeddings to predict probability of next generated tokens. \n",
    "\n",
    "With increasing model size like in modern LLMs, we also can note an increase in the number of attention blocks and size past key values tensors respectively. The strategy for handling cache state as model inputs and outputs in the inference cycle may become a bottleneck for memory-bounded systems, especially with processing long input sequences, for example in a chatbot scenario. OpenVINO suggests a transformation that removes inputs and corresponding outputs with cache tensors from the model keeping cache handling logic inside the model. Such models are also called stateful. A stateful model is a model that implicitly preserves data between two consecutive inference calls. The tensors saved from one run are kept in an internal memory buffer called a `state` or a `variable` and may be passed to the next run, while never being exposed as model output. Hiding the cache enables storing and updating the cache values in a more device-friendly representation. It helps to reduce memory consumption and additionally optimize model performance. More details about stateful models and working with state can be found in [OpenVINO documentation](https://docs.openvino.ai/2024/openvino-workflow/running-inference/stateful-models.html).\n",
    "\n",
    "In LLMs, `text_embedding` is a part of language model, but for multimodal case, the first step hidden state produced by this model part should be integrated with image embeddings into common embedding space. For ability to reuse this model part and avoid introduction of llm model instance, we will use it separately.\n",
    "\n",
    "To sum up above, model consists of 4 parts:\n",
    "\n",
    "* **Image Encoder** for encoding input images into embedding space. It includes SigLIP model.\n",
    "* **Audio Encoder** for encoding input audio into embedding space. It includes Whisper model.\n",
    "* **Resampler** for compression image representation.\n",
    "* **Text Embedding** for conversion input text tokens into embedding space.\n",
    "* **Language Model** for generation answer based on input embeddings provided by Image Encoder and Input Embedding models.\n",
    "\n",
    "Let's convert each model part.\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ff007127-7421-448e-9668-b3fdb32eca51",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home2/ethan/intel/openvino_notebooks/openvino_venv/lib/python3.10/site-packages/openvino/runtime/__init__.py:10: DeprecationWarning: The `openvino.runtime` module is deprecated and will be removed in the 2026.0 release. Please replace `openvino.runtime` with `openvino`.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "⌛ openbmb/MiniCPM-o-2_6 conversion started. Be patient, it may takes some time.\n",
      "⌛ Load Original model\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a17d2f82e0ab499baa340c4e7558520c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Fetching 40 files:   0%|          | 0/40 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1b6a6cce82fd4694842f57b4e42f19e2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Vocos.pt:   0%|          | 0.00/54.4M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "917d03819bfd4ad295047e5fc3db4af3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "added_tokens.json:   0%|          | 0.00/1.41k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "98cc4045442e4e44b9ed35d7a26d0af3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Skiing.mp4:   0%|          | 0.00/8.53M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "303bf99b04c544489afccc1f5ad32868",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "README.md:   0%|          | 0.00/50.0k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8ce22c22220147b6b7240c9e58fa382a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/7.85k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dda5c20416514014ba76ab6ff32a37a4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/11.0k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "59018b09897f4e639789e601f934cb56",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Trump_WEF_2018_10s.mp3:   0%|          | 0.00/161k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "307d88893596473ea649ab2e1020950d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "assistant_default_female_voice.wav:   0%|          | 0.00/224k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bef63e13f1454f9da1c1c1d247a0535e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "demo.wav:   0%|          | 0.00/1.45M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e9cb0a407f924d1dbfcf35f1a16c47c8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "assistant_female_voice.wav:   0%|          | 0.00/235k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "51c69910756a465cbe09cbad99392c09",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "assistant_male_voice.wav:   0%|          | 0.00/144k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "849e115b3beb42128eb49dd2f1cbb0c3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "audio_understanding.mp3:   0%|          | 0.00/321k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "03ed53148b174eeab69f673d151461ff",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/449k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0c873457647140bca3b69645be12523d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "cxk_original.wav:   0%|          | 0.00/384k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2412d413729d49b98b9aa2c46353bb7d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "indian-accent.wav:   0%|          | 0.00/1.41M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "64910dc6d6e84ceb83166959a49afca2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "exciting-emotion.wav:   0%|          | 0.00/696k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3c5011ac70674782a8147b4274954a48",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "icl_20.wav:   0%|          | 0.00/619k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d57153a76e0a45d9be9274795902b8f3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "configuration_minicpm.py:   0%|          | 0.00/7.55k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3e797bba25b445afa250e15c85a73cef",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "image_processing_minicpmv.py:   0%|          | 0.00/16.7k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "33607657ed3447329b466e7a4962c482",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       ".gitattributes:   0%|          | 0.00/1.74k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3066a67fc20643928d6b16e778728260",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "merges.txt:   0%|          | 0.00/1.67M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "995da2ca9a234146b9d70f05c173ad3d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00002-of-00004.safetensors:   0%|          | 0.00/4.93G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "df08c14a2b31421eb03b5ec56f9815ae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00003-of-00004.safetensors:   0%|          | 0.00/4.33G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "37008bc198d44c42bf94b86cf137b9fe",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00004-of-00004.safetensors:   0%|          | 0.00/3.21G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0704253c566f4221b81f6939357eaa6a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "chi-english-1.wav:   0%|          | 0.00/492k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "67fe108fc63249b5b5b4567adf7088e4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/3.44k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "73c698762f9a4f9cb749d01c61861892",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00001-of-00004.safetensors:   0%|          | 0.00/4.88G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2eea609ac3a742b080e98ecc7805f04a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "fast-pace.wav:   0%|          | 0.00/986k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "53dfa466e9ff44739727ecd6e1639a56",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "modeling_navit_siglip.py:   0%|          | 0.00/42.1k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e3eed53138e2423cb7c6443eda9e40ce",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "modeling_minicpmo.py:   0%|          | 0.00/141k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2f6d3f5689704815a4349013474ac95d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors.index.json:   0%|          | 0.00/133k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bea15e0daaf84755bc2127644a3ca8b9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "preprocessor_config.json:   0%|          | 0.00/714 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b9537901be8245bb977d5b7bbb8683ea",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/5.35k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "20d72087ac094dc7a89010cbe7eced40",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "processing_minicpmo.py:   0%|          | 0.00/20.0k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1163d663fc204a30a542dd6c70afc750",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "resampler.py:   0%|          | 0.00/35.6k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c5b053b2bb874b48a8fe2b977bc72526",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenization_minicpmo_fast.py:   0%|          | 0.00/3.04k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7aecee520f464520a5c2f9896d451a22",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/7.04M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "728f5f9df69e49f1aa2dbb5483a5ea42",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "utils.py:   0%|          | 0.00/7.24k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3936688ff023488683de529290041dde",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.json:   0%|          | 0.00/2.78M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ff05a1e8288845258da42cc025e780e7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/14.1k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f826cec12937443aabf710a4ed09cf70",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/3.44k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "57ebb420c4c94c75b8557f0ea660feb6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "configuration_minicpm.py:   0%|          | 0.00/7.55k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "006091c0cef4479483fd9b7bb26e099e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "modeling_navit_siglip.py:   0%|          | 0.00/42.1k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "A new version of the following files was downloaded from https://huggingface.co/openbmb/MiniCPM-o-2_6:\n",
      "- modeling_navit_siglip.py\n",
      ". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\n",
      "A new version of the following files was downloaded from https://huggingface.co/openbmb/MiniCPM-o-2_6:\n",
      "- configuration_minicpm.py\n",
      "- modeling_navit_siglip.py\n",
      ". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "14c5b3c9d3c64dab940194c75b339efe",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "modeling_minicpmo.py:   0%|          | 0.00/141k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1af13ac2ddfe4906a933c03bad7f0a35",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "processing_minicpmo.py:   0%|          | 0.00/20.0k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4ffbf75a1a7e4cde99627b0983f14338",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "image_processing_minicpmv.py:   0%|          | 0.00/16.7k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "A new version of the following files was downloaded from https://huggingface.co/openbmb/MiniCPM-o-2_6:\n",
      "- image_processing_minicpmv.py\n",
      ". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\n",
      "A new version of the following files was downloaded from https://huggingface.co/openbmb/MiniCPM-o-2_6:\n",
      "- processing_minicpmo.py\n",
      "- image_processing_minicpmv.py\n",
      ". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "79693849cbef41f69e6046daabe9c0c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "utils.py:   0%|          | 0.00/7.24k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "A new version of the following files was downloaded from https://huggingface.co/openbmb/MiniCPM-o-2_6:\n",
      "- utils.py\n",
      ". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b8cd19f323ba417aa7ce830de5286e51",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "resampler.py:   0%|          | 0.00/35.6k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "A new version of the following files was downloaded from https://huggingface.co/openbmb/MiniCPM-o-2_6:\n",
      "- resampler.py\n",
      ". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\n",
      "A new version of the following files was downloaded from https://huggingface.co/openbmb/MiniCPM-o-2_6:\n",
      "- modeling_minicpmo.py\n",
      "- processing_minicpmo.py\n",
      "- utils.py\n",
      "- resampler.py\n",
      ". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bc55df0676b84ce5a84532d753fcb099",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors.index.json:   0%|          | 0.00/133k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "51baa700ed9c4479b00c6aacec2f68c7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading shards:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "909979b4c2344e6885e6475237434f1d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00001-of-00004.safetensors:   0%|          | 0.00/4.88G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "81a83d2ec44b44698d7fdbf99335a230",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00002-of-00004.safetensors:   0%|          | 0.00/4.93G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0aa02db918b74e7da4181da2999c8232",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00003-of-00004.safetensors:   0%|          | 0.00/4.33G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dea9e6e9ff9f4dbf864defaba1498d63",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00004-of-00004.safetensors:   0%|          | 0.00/3.21G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bb4e3ae9bdbe4c209ce22a977a6d8ad9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "preprocessor_config.json:   0%|          | 0.00/714 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home2/ethan/intel/openvino_notebooks/openvino_venv/lib/python3.10/site-packages/transformers/models/auto/image_processing_auto.py:517: FutureWarning: The image_processor_class argument is deprecated and will be removed in v4.42. Please use `slow_image_processor_class`, or `fast_image_processor_class` instead\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a1aab9685ba0427b8e7d265382281c62",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/14.1k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c9798d662f744d05b861893989d7e7f1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenization_minicpmo_fast.py:   0%|          | 0.00/3.04k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cba049be97af480b9cdcf704aec5e24b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.json:   0%|          | 0.00/2.78M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d9d5a0ed9b7e47c5906e8a34627ffa2a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "merges.txt:   0%|          | 0.00/1.67M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4dd04f6fbb44484a9cb5036ab1b5270b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/7.04M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fd3700591af34d89b547c9b97ce03caf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "added_tokens.json:   0%|          | 0.00/1.41k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f586fe0cd3df463ab0fc774fbb662f6f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/5.35k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1dc27c67bf6041d4bdaac59b829cabb3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Original model successfully loaded\n",
      "⌛ ConvertText embedding model\n",
      "✅ Text embedding model successfully converted\n",
      "⌛ Convert Language model\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home2/ethan/intel/openvino_notebooks/openvino_venv/lib/python3.10/site-packages/transformers/modeling_utils.py:4773: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead\n",
      "  warnings.warn(\n",
      "We detected that you are passing `past_key_values` as a tuple of tuples. This is deprecated and will be removed in v4.47. Please convert your cache or use an appropriate `Cache` class (https://huggingface.co/docs/transformers/kv_cache#legacy-cache-format)\n",
      "/home2/ethan/intel/openvino_notebooks/openvino_venv/lib/python3.10/site-packages/transformers/models/qwen2/modeling_qwen2.py:103: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  if sequence_length != 1:\n",
      "Starting from v4.46, the `logits` model output will have the same type as the model (except at train time, where it will always be FP32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Language model successfully converted\n",
      "⌛ Convert Image embedding model\n",
      "✅ Image embedding model successfully converted\n",
      "⌛ Convert Resamler model\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ethan/.cache/huggingface/modules/transformers_modules/openbmb/MiniCPM-o-2_6/df3d4c9d637d5cb32c4fcc256336ff6e3e15c71a/resampler.py:489: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  assert (\n",
      "/home/ethan/.cache/huggingface/modules/transformers_modules/openbmb/MiniCPM-o-2_6/df3d4c9d637d5cb32c4fcc256336ff6e3e15c71a/resampler.py:497: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  assert head_dim * num_heads == embed_dim, f\"embed_dim {embed_dim} not divisible by num_heads {num_heads}\"\n",
      "/home/ethan/.cache/huggingface/modules/transformers_modules/openbmb/MiniCPM-o-2_6/df3d4c9d637d5cb32c4fcc256336ff6e3e15c71a/resampler.py:504: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  assert key.shape == value.shape, f\"key shape {key.shape} does not match value shape {value.shape}\"\n",
      "/home/ethan/.cache/huggingface/modules/transformers_modules/openbmb/MiniCPM-o-2_6/df3d4c9d637d5cb32c4fcc256336ff6e3e15c71a/resampler.py:594: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  assert key_padding_mask.shape == (\n",
      "/home/ethan/.cache/huggingface/modules/transformers_modules/openbmb/MiniCPM-o-2_6/df3d4c9d637d5cb32c4fcc256336ff6e3e15c71a/resampler.py:618: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  q_scaled = q / math.sqrt(E)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Resampler model successfully converted\n",
      "⌛ Convert Audio embedding model\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home2/ethan/intel/openvino_notebooks/openvino_venv/lib/python3.10/site-packages/transformers/models/whisper/modeling_whisper.py:608: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  if attn_output.size() != (bsz, self.num_heads, tgt_len, self.head_dim):\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Audio embedding model successfully converted\n",
      "✅ openbmb/MiniCPM-o-2_6 model sucessfully converted. You can find results in MiniCPM-o-2_6\n"
     ]
    }
   ],
   "source": [
    "from minicpm_o_helper import convert_minicpmo26\n",
    "\n",
    "model_id = \"openbmb/MiniCPM-o-2_6\"\n",
    "\n",
    "model_dir = convert_minicpmo26(model_id)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "82f37188-7434-4dc9-ba3c-a9779cd4cdc8",
   "metadata": {},
   "source": [
    "### Compress Language Model Weights to 4 bits\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "For reducing memory consumption, weights compression optimization can be applied using [NNCF](https://github.com/openvinotoolkit/nncf). \n",
    "\n",
    "<details>\n",
    "    <summary><b>Click here for more details about weight compression</b></summary>\n",
    "Weight compression aims to reduce the memory footprint of a model. It can also lead to significant performance improvement for large memory-bound models, such as Large Language Models (LLMs). LLMs and other models, which require extensive memory to store the weights during inference, can benefit from weight compression in the following ways:\n",
    "\n",
    "* enabling the inference of exceptionally large models that cannot be accommodated in the memory of the device;\n",
    "\n",
    "* improving the inference performance of the models by reducing the latency of the memory access when computing the operations with weights, for example, Linear layers.\n",
    "\n",
    "[Neural Network Compression Framework (NNCF)](https://github.com/openvinotoolkit/nncf) provides 4-bit / 8-bit mixed weight quantization as a compression method primarily designed to optimize LLMs. The main difference between weights compression and full model quantization (post-training quantization) is that activations remain floating-point in the case of weights compression which leads to a better accuracy. Weight compression for LLMs provides a solid inference performance improvement which is on par with the performance of the full model quantization. In addition, weight compression is data-free and does not require a calibration dataset, making it easy to use.\n",
    "\n",
    "`nncf.compress_weights` function can be used for performing weights compression. The function accepts an OpenVINO model and other compression parameters. Compared to INT8 compression, INT4 compression improves performance even more, but introduces a minor drop in prediction quality.\n",
    "\n",
    "More details about weights compression, can be found in [OpenVINO documentation](https://docs.openvino.ai/2024/openvino-workflow/model-optimization-guide/weight-compression.html).\n",
    "\n",
    "</details>\n",
    "\n",
    "> **Note:** weights compression process may require additional time and memory for performing. You can disable it using widget below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "da5cae73-0ae2-4683-a4d1-c85c622f7f84",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2ce308aeeca84d228e666f8b5ce32d48",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Checkbox(value=True, description='Weights Compression')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from minicpm_o_helper import compression_widget\n",
    "\n",
    "to_compress_weights = compression_widget()\n",
    "\n",
    "to_compress_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6e938ae8-7e49-4c61-88b9-0b73c8fa8407",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:nncf:NNCF provides best results with torch==2.6.*, while current torch version is 2.3.1+cpu. If you encounter issues, consider switching to torch==2.6.*\n",
      "INFO:nncf:Statistics of the bitwidth distribution:\n",
      "┍━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑\n",
      "│ Weight compression mode   │ % all parameters (layers)   │ % ratio-defining parameters (layers)   │\n",
      "┝━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥\n",
      "│ int4_sym                  │ 100% (197 / 197)            │ 100% (197 / 197)                       │\n",
      "┕━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ebee6acc70474edc9fba13586095eac5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import nncf\n",
    "import gc\n",
    "import openvino as ov\n",
    "\n",
    "from minicpm_o_helper import llm_path, copy_llm_files\n",
    "\n",
    "\n",
    "compression_configuration = {\"mode\": nncf.CompressWeightsMode.INT4_SYM, \"group_size\": 64, \"ratio\": 1.0, \"all_layers\": True}\n",
    "\n",
    "\n",
    "core = ov.Core()\n",
    "llm_int4_path = Path(\"language_model_int4\") / llm_path.name\n",
    "if to_compress_weights.value and not (model_dir / llm_int4_path).exists():\n",
    "    ov_model = core.read_model(model_dir / llm_path)\n",
    "    ov_compressed_model = nncf.compress_weights(ov_model, **compression_configuration)\n",
    "    ov.save_model(ov_compressed_model, model_dir / llm_int4_path)\n",
    "    del ov_compressed_model\n",
    "    del ov_model\n",
    "    gc.collect()\n",
    "    copy_llm_files(model_dir, llm_int4_path.parent)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "43855dca-3ef2-4cea-8fc2-3700f769dd06",
   "metadata": {},
   "source": [
    "## Prepare model inference pipeline\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "As discussed, the model comprises Image Encoder and LLM (with separated text embedding part) that generates answer. In `minicpm_helper.py` we defined LLM inference class `OvModelForCausalLMWithEmb` that will represent generation cycle, It is based on [HuggingFace Transformers `GenerationMixin`](https://huggingface.co/docs/transformers/main_classes/text_generation) and looks similar to [Optimum Intel](https://huggingface.co/docs/optimum/intel/index) `OVModelForCausalLM`that is used for LLM inference with only difference that it can accept input embedding. In own turn, general multimodal model class `OvMiniCPMVModel` handles chatbot functionality including image processing and answer generation using LLM."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "06eada4a-cac9-4501-bed0-059c4e585d57",
   "metadata": {},
   "source": [
    "### Select device\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e56db20f-7cf0-4ead-b6af-8e048e61b059",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e8008d5cfb9f441c9cd1195444f65c86",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Device:', index=2, options=('CPU', 'GPU', 'AUTO'), value='AUTO')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from notebook_utils import device_widget\n",
    "\n",
    "device = device_widget(default=\"AUTO\", exclude=[\"NPU\"])\n",
    "\n",
    "device"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "523fdc7d-7c4c-4f91-9c48-42fc2e39210a",
   "metadata": {},
   "source": [
    "### Select language model variant\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "929f25e9-f793-488b-8be3-6622b587cc01",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f0350c4c99974ab98380400c0292e726",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Checkbox(value=True, description='INT4 language model')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from minicpm_o_helper import lm_variant_selector, init_model\n",
    "\n",
    "\n",
    "use_int4_lang_model = lm_variant_selector(model_dir / llm_int4_path)\n",
    "\n",
    "use_int4_lang_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8d1cbb31",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "applied slice for lm head\n"
     ]
    }
   ],
   "source": [
    "ov_model = init_model(model_dir, llm_path.parent if not use_int4_lang_model.value else llm_int4_path.parent, device.value)\n",
    "tokenizer = ov_model.processor.tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8328887c",
   "metadata": {},
   "source": [
    "## Run model inference\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Let's explore model capabilities using multimodal input"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f44fb4ca",
   "metadata": {},
   "source": [
    "### Omni mode\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "This is an example for real-time video understanding, omni-source (video & audio) understanding, and multimodal contextual understanding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8db596aa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "video_duration: 9.41\n",
      "MoviePy - Writing audio in /tmp/tmpk7tr7kzm.wav\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                                                                                                                                                                                                                                                                               "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MoviePy - Done.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": []
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The person in the picture is skiing down a snowy slope.\n"
     ]
    }
   ],
   "source": [
    "import math\n",
    "import numpy as np\n",
    "from PIL import Image\n",
    "from moviepy.editor import VideoFileClip\n",
    "import tempfile\n",
    "import librosa\n",
    "\n",
    "\n",
    "def get_video_chunk_content(video_path, flatten=True):\n",
    "    video = VideoFileClip(video_path)\n",
    "    print(\"video_duration:\", video.duration)\n",
    "\n",
    "    with tempfile.NamedTemporaryFile(suffix=\".wav\", delete=True) as temp_audio_file:\n",
    "        temp_audio_file_path = temp_audio_file.name\n",
    "        video.audio.write_audiofile(temp_audio_file_path, codec=\"pcm_s16le\", fps=16000)\n",
    "        audio_np, sr = librosa.load(temp_audio_file_path, sr=16000, mono=True)\n",
    "    num_units = math.ceil(video.duration)\n",
    "\n",
    "    # 1 frame + 1s audio chunk\n",
    "    contents = []\n",
    "    for i in range(num_units):\n",
    "        frame = video.get_frame(i + 1)\n",
    "        image = Image.fromarray((frame).astype(np.uint8))\n",
    "        audio = audio_np[sr * i : sr * (i + 1)]\n",
    "        if flatten:\n",
    "            contents.extend([\"<unit>\", image, audio])\n",
    "        else:\n",
    "            contents.append([\"<unit>\", image, audio])\n",
    "\n",
    "    return contents\n",
    "\n",
    "\n",
    "video_path = \"MiniCPM-o-2_6/ckpt/assets/Skiing.mp4\"\n",
    "# if use voice clone prompt, please set ref_audio\n",
    "ref_audio_path = \"MiniCPM-o-2_6/ckpt/assets/demo.wav\"\n",
    "ref_audio, _ = librosa.load(ref_audio_path, sr=16000, mono=True)\n",
    "sys_msg = ov_model.get_sys_prompt(ref_audio=ref_audio, mode=\"omni\", language=\"en\")\n",
    "# or use default prompt\n",
    "# sys_msg = model.get_sys_prompt(mode='omni', language='en')\n",
    "\n",
    "contents = get_video_chunk_content(video_path)\n",
    "msg = {\"role\": \"user\", \"content\": contents}\n",
    "msgs = [sys_msg, msg]\n",
    "\n",
    "# please set generate_audio=True and output_audio_path to save the tts result\n",
    "generate_audio = False\n",
    "output_audio_path = \"output.wav\"\n",
    "\n",
    "res = ov_model.chat(\n",
    "    msgs=msgs,\n",
    "    tokenizer=tokenizer,\n",
    "    sampling=True,\n",
    "    temperature=0.5,\n",
    "    max_new_tokens=4096,\n",
    "    omni_input=True,  # please set omni_input=True when omni inference\n",
    "    use_tts_template=True,\n",
    "    generate_audio=generate_audio,\n",
    "    output_audio_path=output_audio_path,\n",
    "    max_slice_nums=1,\n",
    "    use_image_id=False,\n",
    "    return_dict=True,\n",
    ")\n",
    "print(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efe8926e",
   "metadata": {},
   "source": [
    "### Vision-Only mode\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "In Vision-Only mode, MiniCPM-o-2_6 has the same inference methods as MiniCPM-V-2_6."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "974f8202",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The image features a cat relaxing inside an open cardboard box."
     ]
    }
   ],
   "source": [
    "import requests\n",
    "from io import BytesIO\n",
    "\n",
    "image_path = Path(\"cat.png\")\n",
    "\n",
    "if not image_path.exists():\n",
    "    url = \"https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/d5fbbd1a-d484-415c-88cb-9986625b7b11\"\n",
    "    image = Image.open(BytesIO(requests.get(url).content)).convert(\"RGB\")\n",
    "    image.save(image_path)\n",
    "else:\n",
    "    image = Image.open(image_path).convert(\"RGB\")\n",
    "question = \"What is in the image?\"\n",
    "msgs = [{\"role\": \"user\", \"content\": [image, question]}]\n",
    "## if you want to use streaming, please make sure sampling=True and stream=True\n",
    "## the model.chat will return a generator\n",
    "res = ov_model.chat(msgs=msgs, tokenizer=tokenizer, sampling=True, stream=True)\n",
    "generated_text = \"\"\n",
    "for new_text in res:\n",
    "    generated_text += new_text\n",
    "    print(new_text, flush=True, end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ff8cc97",
   "metadata": {},
   "source": [
    "## Interactive demo\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e01645bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from gradio_helper import make_demo\n",
    "\n",
    "demo = make_demo(ov_model, ov_model.processor)\n",
    "\n",
    "try:\n",
    "    demo.launch(debug=True)\n",
    "except Exception:\n",
    "    demo.launch(debug=True, share=True)\n",
    "# if you are launching remotely, specify server_name and server_port\n",
    "# demo.launch(server_name='your server name', server_port='server port in int')\n",
    "# Read more in the docs: https://gradio.app/docs/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  },
  "openvino_notebooks": {
   "imageUrl": "https://github.com/user-attachments/assets/e81ecec6-35e3-4c87-827c-ffd49958b2ff",
   "tags": {
    "categories": [
     "Model Demos",
     "AI Trends"
    ],
    "libraries": [],
    "other": [],
    "tasks": [
     "Visual Question Answering",
     "Image-to-Text",
     "Text Generation"
    ]
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "00d0588f60b440f9a9ca54d9158a1190": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "0a03d20cd008493cb08835a0323d4b68": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "FloatProgressModel",
      "state": {
       "bar_style": "success",
       "layout": "IPY_MODEL_8e7f11536c114af89d491f0568df127b",
       "max": 4,
       "style": "IPY_MODEL_e03d3948ed9844c9864d0c24f7b4d7d9",
       "value": 4
      }
     },
     "20e857999d2642e3ac8f032f5c0c33f0": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "CheckboxStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "21081ab3eaff400699d5551fbc3499fa": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_de6e49dc66664d9bb35c5113eca1f80b",
       "style": "IPY_MODEL_27ac333ee98e4a4c83a63f6f96e47c32",
       "value": "Loading checkpoint shards: 100%"
      }
     },
     "24d89d3e3cbd4209a1de15ffe5805f1a": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "CheckboxModel",
      "state": {
       "description": "INT4 language model",
       "disabled": false,
       "layout": "IPY_MODEL_9b84b8a979914ce2b1d0e66bdf018dcf",
       "style": "IPY_MODEL_f57bb89d44c244ff8674f3c668d0eb0d",
       "value": true
      }
     },
     "27aa952cb21f41eabc45bbe252eaeb0e": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "27ac333ee98e4a4c83a63f6f96e47c32": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLStyleModel",
      "state": {
       "description_width": "",
       "font_size": null,
       "text_color": null
      }
     },
     "4aeb4850b551482a9ffd206c832151eb": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLStyleModel",
      "state": {
       "description_width": "",
       "font_size": null,
       "text_color": null
      }
     },
     "51acb00a08d6403d9096d43d0dcc777c": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "71d979e4987c4c79a8928ab24555af8d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "CheckboxModel",
      "state": {
       "description": "Weights Compression",
       "disabled": false,
       "layout": "IPY_MODEL_00d0588f60b440f9a9ca54d9158a1190",
       "style": "IPY_MODEL_20e857999d2642e3ac8f032f5c0c33f0",
       "value": true
      }
     },
     "8e7f11536c114af89d491f0568df127b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "90e626116e9e44b7af052efd61920727": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "9b84b8a979914ce2b1d0e66bdf018dcf": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "9ca30303287f4554a5a4cdc351bc10f6": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_27aa952cb21f41eabc45bbe252eaeb0e",
       "style": "IPY_MODEL_4aeb4850b551482a9ffd206c832151eb",
       "value": " 4/4 [00:02&lt;00:00,  1.47it/s]"
      }
     },
     "ac7cb12325d04443ae7808f9be4e9dc0": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "b37d56f23d7a45f5bd6a35f745f687b9": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "DropdownModel",
      "state": {
       "_options_labels": [
        "CPU",
        "AUTO"
       ],
       "description": "Device:",
       "index": 1,
       "layout": "IPY_MODEL_90e626116e9e44b7af052efd61920727",
       "style": "IPY_MODEL_ac7cb12325d04443ae7808f9be4e9dc0"
      }
     },
     "c6b2d6c0742a4b35b4c12f1229f5bd0c": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_21081ab3eaff400699d5551fbc3499fa",
        "IPY_MODEL_0a03d20cd008493cb08835a0323d4b68",
        "IPY_MODEL_9ca30303287f4554a5a4cdc351bc10f6"
       ],
       "layout": "IPY_MODEL_51acb00a08d6403d9096d43d0dcc777c"
      }
     },
     "de6e49dc66664d9bb35c5113eca1f80b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "e03d3948ed9844c9864d0c24f7b4d7d9": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "f57bb89d44c244ff8674f3c668d0eb0d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "CheckboxStyleModel",
      "state": {
       "description_width": ""
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}