Dimesionality_Reduction_TSNE_MNIST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "In this post I will be using Kaggle's famous **Digit Recognizer Dataset** (also known as MNIST data) to implement Dimensionality Reduction with PCA (Principle Component Analysis).\n",
    "\n",
    "### [Link to the Kaggle Digit Recognizer Dataset](https://www.kaggle.com/c/digit-recognizer/data)\n",
    "\n",
    "In many practical applications, although the data reside in a high-dimensional space, the true dimensionality, known as intrinsic dimensionality, can be of a much lower value.\n",
    "\n",
    "For example, in a three-dimensional space, the data may cluster around a straight line, or around the circumference of a circle or the graph of a parabola, arbitrarily placed in R^3. In all previous cases, the intrinsic dimensionality of the data is equal to one, as any of these curves can equivalently\n",
    "be described in terms of a single parameter.\n",
    "\n",
    "Below figure illustrates the three cases. Learning the lower-dimensional structure associated with a given set of data is gaining in importance in the context of big data processing and analysis. Some typical examples are the disciplines of computer vision, robotics, medical imaging, and computational neuroscience.\n",
    "\n",
    "![img](https://i.imgur.com/IkFGd65.png)\n",
    "\n",
    "The data reside close to (A) a straight line, (B) the circumference of a circle, and (C) the graph of a parabola in the three-dimensional space. In all three cases, the intrinsic dimensionality of the data is equal to one. In (A) the data are clustered around a (translated/affine) linear subspace and in (B) and (C) around one-dimensional manifolds.\n",
    "\n",
    "\n",
    "### Now here is broadly the various methods for Dimensionality Reductions\n",
    "\n",
    "![Imgur](https://imgur.com/hfWeZUL.png)\n",
    "\n",
    "---\n",
    "\n",
    "## What is t-SNE?\n",
    "\n",
    "t-Distributed Stochastic Neighbor Embedding (t-SNE) is an unsupervised, non-linear technique developed by Laurens van der Maaten and Geoffrey Hinton in 2008.\n",
    "\n",
    " To keep things simple, here’s a brief overview of working of t-SNE:\n",
    "\n",
    "- The algorithms starts by calculating the probability of similarity of points in high-dimensional space and calculating the probability of similarity of points in the corresponding low-dimensional space. So basicaly t-SNE constructs a probability distribution on pairs in higher dimensions such that similar objects are assigned a higher probability and dissimilar objects are assigned lower probability.\n",
    "\n",
    "- The similarity of points is calculated as the conditional probability that a point A would choose point B as its neighbor if neighbors were picked in proportion to their probability density under a Gaussian (normal distribution) centered at A.\n",
    "\n",
    "- It then tries to minimize the difference between these conditional probabilities (or similarities) in higher-dimensional and lower-dimensional space for a perfect representation of data points in lower-dimensional space.\n",
    "\n",
    "- To measure the minimization of the sum of difference of conditional probability t-SNE minimizes the sum of Kullback-Leibler divergence of overall data points using a gradient descent method.\n",
    "Note Kullback-Leibler divergence or KL divergence is is a measure of how one probability distribution diverges from a second, expected probability distribution.\n",
    "\n",
    "In simpler terms, t-Distributed stochastic neighbor embedding (t-SNE) minimizes the divergence between two distributions: a distribution that measures pairwise similarities of the input objects and a distribution that measures pairwise similarities of the corresponding low-dimensional points in the embedding.\n",
    "\n",
    "In this way, t-SNE maps the multi-dimensional data to a lower dimensional space and attempts to find patterns in the data by identifying observed clusters based on similarity of data points with multiple features. However, after this process, the input features are no longer identifiable, and you cannot make any inference based only on the output of t-SNE. Hence it is mainly a data exploration and visualization technique.\n",
    "\n",
    "---\n",
    "\n",
    "### Why not PCA? Why t-SNE?\n",
    "\n",
    "When it comes to Dimensionality Reduction, PCA is famous because it is simple, fast, easy to use, and retains the overall variance of the dataset. For more about PCA at a glance, check this.\n",
    "Though PCA is great, it does have some drawbacks. One of the major drawbacks of PCA is that it does not retain non-linear variance. This means PCA will not be able to get results for figures like this.\n",
    "\n",
    "![Imgur](https://imgur.com/uUe7eaZ.png)\n",
    "\n",
    "---\n",
    "\n",
    "#### The t-SNE algorithm has two steps:\n",
    "\n",
    "We initially construct a probability distribution in such a way that objects with a higher similarity have a higher probability to be grouped together than objects with lower probability. This is done over pairs of higher-dimensional objects.\n",
    "\n",
    "We then construct a similar probability distribution over the lower-dimensional map so that the [Kullback–Leibler divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence) between the two distributions, with respect to their location on the map, is minimized.\n",
    "\n",
    "Usually, the algorithm uses Euclidean distance as the base metric but it can be changed to fit the use of the programmer. These t-SNE clusters are dependent on chosen parameters and sometimes may show a cluster in non-clustered data. However, t-SNE is able to recover well-\n",
    "separated clusters when the correct parameters are chosen.\n",
    "\n",
    "Maaten and Hinton proposed the t-distributed SNE (t-SNE) in which the joint distribution of two samples, zn and zm , in low-dimensional space is modeled by a Student’s t-distribution:\n",
    "\n",
    "\n",
    "![img](https://i.imgur.com/ihkO2Iu.png)\n",
    "\n",
    "In above **ν** means the degree of freedom. t-SNE can alleviate the crowding problem in SNE.\n",
    "\n",
    "---\n",
    "\n",
    "### Here are a few limitations of t-SNE:\n",
    "\n",
    "- Unlike PCA, the cost function of t-SNE is non-convex, meaning there is a possibility that we would be stuck in a local minima.\n",
    "\n",
    "- Similar to other dimensionality reduction techniques, the meaning of the compressed dimensions as well as the transformed features becomes less interpretable.\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "## Now implementing t-SNE with scikit-learn's built-in function\n",
    "\n",
    "### Our Input MNIST Data Description:\n",
    "\n",
    "- The data file contains grey-scale images of hand-drawn digits, from zero through nine.\n",
    "Each image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255, inclusive.\n",
    "\n",
    "- The training data set, (train.csv), has 785 columns. The first column, called “label”, is the digit that was drawn by the user. The rest of the columns contain the pixel-values of the associated image.\n",
    "\n",
    "- Each pixel column in the data set has a name like pixelx, where x is an integer between 0 and 783, inclusive. To locate this pixel on the image, suppose that we have decomposed x as x = i * 28 + j, where i and j are integers between 0 and 27, inclusive. Then pixelx is located on row i and column j of a 28 x 28 matrix, (indexing by zero).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.manifold import TSNE\n",
    "\n",
    "# train_df_org = pd.read_csv('kaggle/input/digit-recognizer/train.csv',  nrows=1000)\n",
    "train_df_org = pd.read_csv('kaggle/input/digit-recognizer/train.csv',  nrows=10)\n",
    "\n",
    "train_df_for_tsne = StandardScaler().fit_transform(train_df_org)\n",
    "train_df_org_label = train_df_org['label']\n",
    "\n",
    "\n",
    "train_df_org.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Dim_1</th>\n",
       "      <th>Dim_2</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>32.420952</td>\n",
       "      <td>8.115230</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.648955</td>\n",
       "      <td>71.707100</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>71.019684</td>\n",
       "      <td>9.971006</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.785981</td>\n",
       "      <td>33.185135</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-42.666500</td>\n",
       "      <td>-9.222070</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Dim_1      Dim_2  Label\n",
       "0  32.420952   8.115230    1.0\n",
       "1   0.648955  71.707100    0.0\n",
       "2  71.019684   9.971006    1.0\n",
       "3   3.785981  33.185135    4.0\n",
       "4 -42.666500  -9.222070    0.0"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tsne = TSNE(n_components=2, random_state=0)\n",
    "mnist_tsne = tsne.fit_transform(train_df_for_tsne)\n",
    "\n",
    "mnist_tsne_1 = np.vstack((mnist_tsne.T, train_df_org_label)).T\n",
    "tsne_df = pd.DataFrame(data=mnist_tsne_1, columns=('Dim_1', 'Dim_2', 'Label'))\n",
    "\n",
    "tsne_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/paul/.local/lib/python3.8/site-packages/seaborn/axisgrid.py:316: UserWarning: The `size` parameter has been renamed to `height`; please update your code.\n",
      "  warnings.warn(msg, UserWarning)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdwAAAGoCAYAAADsEFQiAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAf50lEQVR4nO3de7BdZZnn8e9jEkhEJXINEGxQEaEVJXMGoWhtBQ3hGmrKcrBbwcsMZQ3aQW3U4DQdnbZbixkgDF46QjswjQ10GiQYJSDGabuVaCSaiCESI0pCIkFNFAyXhGf+2OuEnZOT5Fz2ftc+e38/VafOXu9aZ51n1VF+We9+93oiM5EkSe31vLoLkCSpFxi4kiQVYOBKklSAgStJUgEGriRJBYyvu4BWmTFjRt555511lyFJGp6ou4BSuuYO97HHHqu7BEmSdqlrAleSpE5m4EqSVICBK0lSAQauJEkFGLiSJBVg4EqSVICBK0lSAQauJEkFGLiSJBVg4EqSVICBK0lSAQauJEkFGLiSJBXQNe35pJH4yrJ1XL5oFY9s2sKhkydxyWlHc+7xh9VdlqQuZOCqZ31l2Tpm37qCLc9sA2Ddpi3MvnUFgKErqeWcUlbPunzRqu1h22/LM9u4fNGqmiqS1M0MXPWsRzZtGda4JI2GgauedejkScMal6TRMHDVsy457WgmTRi3w9ikCeO45LSja6pIUjdz0ZR6Vv/CKFcpSyrBwFVPO/f4wwxYSUU4pSxJUgEGriRJBRi4kiQVYOBKklSAgStJUgEGriRJBRi4kiQVYOBKklSAgStJUgEGriRJBRi4kiQVYOBKklSAgStJUgEGriRJBRi4kiQVYOBKklSAgStJUgEGriRJBRi4kiQVYOBKklSAgStJUgEGriRJBRi4kiQVYOBKklSAgStJUgEGriRJBdQeuBExOSLmR8QDEbEyIk6KiP0i4u6IeLD6/uK665QkaTRqD1xgLnBnZr4SeA2wEvgYcE9mHgXcU21LkjRm1Rq4EbEv8AbgOoDMfDozNwEzgeurw64Hzq2jPkmSWqXuO9wjgY3AlyJiWURcGxH7AAdn5vrqmA3AwbVVKElSC9QduOOBacDnM/N44AkGTB9nZgI52A9HxIURsTQilm7cuLHtxUqSNFJ1B+5aYG1mLqm259MI4F9FxCEA1fdHB/vhzJyXmX2Z2XfggQcWKViSpJGoNXAzcwPwcEQcXQ2dCvwEWABcUI1dANxeQ3mSJLXM+LoLAD4A3BgRewFrgHfT+IfALRHxXuAXwNtqrE+SpFGrPXAz84dA3yC7Ti1ciiRJbVP3e7iSJPUEA1eSpAIMXEmSCjBwJUkqwMCVJKkAA1eSpAIMXEmSCjBwJUkqwMCVJKkAA1eSpAIMXEmSCjBwJUkqwMCVJKkAA1eSpAIMXEmSCjBwJUkqwMCVJKkAA1eSpAIMXEmSCjBwJUkqYHzdBUhSXX66ZAPfvf1nPP6bp3jBfntz0syX8YrXTam7LHUpA1dST/rpkg0svvEBtj79LACP/+YpFt/4AIChq7ZwSllST/ru7T/bHrb9tj79LN+9/Wc1VaRuZ+BK6kmP/+apYY1Lo2XgSupJL9hv72GNS6Nl4ErqSSfNfBnj99rxP4Hj93oeJ818WU0Vqdu5aEpST+pfGOUqZZVi4ErqWa943RQDVsU4pSxJUgEGriRJBRi4kiQVYOBKklSAgStJUgEGriRJBRi4kiQVYOBKklSAgStJUgEGriRJBRi4kiQVYOBKklSAgStJUgEGriRJBRi4kiQVYOBKklSAgStJUgEGriRJBRi4kiQVYOBKklSAgStJUgEGriRJBRi4kiQV0BGBGxHjImJZRHy12j4yIpZExOqIuDki9qq7RkmSRqMjAheYBaxs2v4McGVmvhz4LfDeWqqSJKlFag/ciJgKnAlcW20HcAowvzrkeuDcWoqTJKlFag9c4CrgI8Cz1fb+wKbM3FptrwUOG+wHI+LCiFgaEUs3btzY9kIlSRqpWgM3Is4CHs3MH4zk5zNzXmb2ZWbfgQce2OLqJElqnfE1//6TgXMi4gxgIvAiYC4wOSLGV3e5U4F1NdYoSdKo1XqHm5mzM3NqZh4BnAd8MzP/HFgMvLU67ALg9ppKlCSpJTrhPdzBfBT4UESspvGe7nU11yNJ0qjUPaW8XWZ+C/hW9XoNcEKd9UiS1EqdeocrSVJXMXAlSSrAwJUkqQADV5KkAgzcFlq4ZiHT50/nuOuPY/r86Sxcs7DukiRJHaJjVimPdQvXLGTOd+bw5LYnAVj/xHrmfGcOAGe+9MwaK5MkdQLvcFtk7n1zt4dtvye3Pcnc++bWVJEkqZMYuC2y4YkNwxqXJPUWA7dFpuwzZVjjkqTeYuC2yKxps5g4buIOYxPHTWTWtFk1VSRJ6iQummqR/oVRc++by4YnNjBlnynMmjbLBVOSJAAiM+uuoSX6+vpy6dKldZchSRqeqLuAUpxSlvZg8x138OApp7LymGN58JRT2XzHHXWXJGkMckpZ2o3Nd9zB+r+6jHyy8ZGvrY88wvq/ugyAfc8+u87SJI0x3uFKu/HolVdtD9t++eSTPHrlVfUUJGnMMnCl3di6fv2wxiVpVwxcaTfGH3LIsMYlaVcMXGk3DvrgxcTEHT9fHRMnctAHL66nIEljloumpN3oXxj16JVXsXX9esYfcggHffBiF0xJGjY/hytJqpOfw5UkSa1j4EqSVICBK0lSAQauJEkFGLiSJBVg4EqSVICBK0lSAQauJEkFGLiSJBVg4EqSVICBK0lSAQauJEkFGLiSJBVg4EqSVICBK0lSAQauJEkFGLiSJBVg4EqSVICBK0kasyLi8WEcOyci/rJd598TA1eSpAIMXElSV4mIsyNiSUQsi4hvRMTBTbtfExHfjYgHI+K/Nv3MJRHx/YhYHhGfaEddBq4kqdv8G3BiZh4P3AR8pGnfccApwEnAZRFxaERMB44CTgBeC/yHiHhDq4sa3+oTSpJUs6nAzRFxCLAX8POmfbdn5hZgS0QsphGyfwJMB5ZVx7yARgD/ayuLMnAlSd3mfwNXZOaCiHgjMKdpXw44NoEA/i4z/76dRTmlLEnqNvsC66rXFwzYNzMiJkbE/sAbge8Di4D3RMQLACLisIg4qNVFeYcrSRrLnh8Ra5u2r6BxR/vPEfFb4JvAkU37lwOLgQOA/5GZjwCPRMQxwHcjAuBx4B3Ao60sNDIH3l2PTX19fbl06dK6y5AkDU/UXUApTilLklSAgStJUgEGriRJBRi4kiQVUGvgRsThEbE4In4SEfdHxKxqfL+IuLt69NbdEfHiOuuUJGm06r7D3Qp8ODOPBU4ELoqIY4GPAfdk5lHAPdW2JEljVq2Bm5nrM/O+6vXvgZXAYcBM4PrqsOuBc2spUJLUUyJiRkSsiojVEbHTzV5E7B0RN1f7l0TEEUM9d913uNtVRR8PLAEOzsz11a4NwMG7+JkLI2JpRCzduHFjmUIlSV0pIsYBnwVOB44F3l7NujZ7L/DbzHw5cCXwmaGevyOeNFU9TutfgIsz83fVkz4AyMyMiEGfzpGZ84B50HjwRYlaJUn1O+JjC/8M+FvgJcAvgUsf+vSZXx7laU8AVmfmGoCIuInGjOtPmo6ZyXPPZp4PXBMRkUN4ilTtd7gRMYFG2N6YmbdWw7+qujxQfW/p47UkSWNXFbZfBP6IxpOq/gj4YjU+GocBDzdtr63GBj0mM7cCm4H9h3LyulcpB3AdsDIzr2jatYDnHjh9AXB76dokSR3rb4HnDxh7fjXeseq+wz0ZeCdwSkT8sPo6A/g08JaIeBB4c7UtSRI0ppGHMz5U64DDm7an8lzXoZ2OiYjxNDoT/XooJ9/je7jVm8j/pfrFd2bmvzft+++Z+TdD+UWDycx/Y9cPrj51pOeVJHW1X9KYRh5sfDS+DxwVEUfSCNbzgIHT1P0zsN8F3gp8cyjv38LQ7nD/HvhTGgl+dUQ0T/3+p6H8EkmSWuhS4A8Dxv5QjY9Y9Z7s+2n0x10J3JKZ90fEJyPinOqw64D9I2I18CGG8ZyIPbbni4jlmXlc9Xo88DkafQTfDtybmccP85rawvZ8kjQmjag9X5tWKbfVUAL3gcx85YCxy4DTgIOqp0HVzsCVpDHJfrhNlkbEjOaBzPwk8CXgiHYUJUlSt9lj4GbmOzLzzkHGr83MCf3bEfGWVhcnSVK3aOXHgob8eCtJknpNKwO3Z+bhJUkarlYGrs8yliRpF+p+0pQkSR0hIv4hIh6NiB/vYn9ExNVVa77lETFtOOdvZeA+1MJzSZJU2v8BZuxm/+nAUdXXhcDnh3PyIbfnqx7xeCaNjwJt/7n+pgOZ6VOnJEllzNl3pwdfMGfzqB58kZn/uoeG8jOBG6pHOd4bEZMj4pCm/u27NZw73DuAd9FoQ/TCpi9JkspphO1O7fmq8XYaSvu+XRpOA/qp/Y94lCSpRrtrz9exj3cczh3u1yNietsqkSRpaNrVnm9PhtK+b5eGE7j3ArdFxJaI+F1E/D4ifjeMn5ckqRV21YZvtO359mQBcH61WvlEYPNQ37+F4U0pXwGcBKwYau8/SZLa4FIa7+E2TyuPuj1fRPwT8EbggIhYC/w1MAEgM78AfA04A1hd/b53D+f8wwnch4EfG7aSpFrN2fxl5uwLrV+l/PY97E/gopGefziBuwb4VkR8HXiqqYArdv0jkiS1QSNcO3aB1GCGE7g/r772qr4kSdIQDTlwM/MT7SxEkqRutsfAjYirMvPiiLiDQRoUZOY5balMkqQuMpQ73P9bff+f7SxEkjRGLL8F7vkkbF4L+06FUy+D495Wd1Udb4+Bm5k/qL7/v4g4sHq9sd2FSZI60PJb4I6/gGe2NLY3P9zYBkN3D4b04IuImBMRjwGrgJ9GxMaIuKy9pUmSOs49n3wubPs9s6UxPsZFxMSI+F5E/Cgi7o+IndYuRcTeEXFz1aJvyR6aHexgj4EbER8CTgb+Y2bul5kvBl4HnBwRHxzGtUiSxrrNa4c3PrY8BZySma8BXgvMqJ4o1ey9wG8z8+XAlcBnhnryobyH+07gLZn5WP9AZq6JiHcAd1W/UJLUC/ad2phGHmy8oFdf/+qd2vOtuGDFaB98kcDj1eaE6mvgYuGZwJzq9XzgmoiIoTwUaihTyhOaw7apsI1VMZI0dMtvgStfBXMmN74vv6XuijQcp14GEybtODZhUmO8kCpsd2rPV42PSkSMi4gfAo8Cd2fmkgGHbG/Rl5lbgc002tbu0VAC9+kR7pOkHfUvuNn8MJDPLbgxdMeO494GZ18N+x4OROP72VeXXjC1u/Z8o5KZ2zLztTQ6AZ0QEa8a7Tn7DWVK+TW76AoUwMRWFSKpB+xuwY0rXMeO495W99+r7e35MnNTRCwGZgA/btrV36JvbUSMB/YFfj2Uc+7xDjczx2Xmiwb5emFmOqUsaei6e8GNymlLe76IODAiJlevJwFvAR4YcNgC4ILq9VuBbw61qc9w+uFK0ujsamFN4QU3GvMupdEer9mo2/MBhwCLI2I58H0a7+F+NSI+GRH9T1W8Dtg/IlYDHwI+NtSTR7d02+vr68ulS5fWXYak3Rn40ARoLLgp/x6gOkeM5IfasUq53QxcSWX5WEDtaESBOxYNpz2fJI1e/QtupFr4Hq4kSQUYuJIkFWDgSpJUgIErSVIBBq4kSU2q5ykvi4ivDrKvfe35JEnqMbOAlbvYN+L2fAauJGnMWfnKY/5s5SuPeWjlK495tvo+6k5BABExFTgTuHYXh8wErq9ezwdOjYghfZbYwJUkjSlVuO7Unq9FoXsV8BHg2V3sb2t7PkmSOklb2vNFxFnAo5n5g9GcZ1cMXEnSWNOu9nwnA+dExEPATcApEfGPA47pb89Hy9vzSZLUYdrSni8zZ2fm1Mw8AjiPRuu9dww4zPZ8kqSe0a72fIOyPd8AdguSpDFpRN2CqgVSO7TnO+aBlR3dns9uQZKkMacK144O2IGcUpYkqQADV5KkAgxcSZIK6NjAjYgZEbGqekD0kFeBSZLUiToycCNiHPBZ4HTgWODtEXFsvVVJkjRyHRm4wAnA6sxck5lP03jix8yaa5IkdbmIeCgiVkTEDyNip8+aRsPV1ezr8oiYNtRzd+rHgrY/HLqyFnjdwIMi4kLgQoCXvGS0T/SSJAmAN2XmY7vYdzpwVPX1OuDzDJJPg+nUwB2SzJwHzIPGgy9qLkeSVMhn3/fNnR58cdEXTinxudyZwA3V4xzvjYjJEXFIZq7f0w926pTy9odDV6ZWY5KkHleF7U7t+arx0Urgroj4QTWLOtBgM7CHDeXEnRq43weOiogjI2IvGg+RXlBzTZKkztCW9nyVP8nMaTSmji+KiDe04JxAhwZu1dT3/cAiYCVwS2beX29VkqQO0a72fGTmuur7o8BtNBbxNhvxDGxHBi5AZn4tM1+RmS/LzE/VXY8kqWO0pT1fROwTES/sfw1MB3484LAFwPnVauUTgc1Def8WxviiKUlST7qUxnu4zdPKrWjPdzBwW0RAIx+/nJl3RsT7ADLzC8DXgDOA1dXvfPdQT257PklSnUbUnq/GVcojZuBKkuo0osAdi5xSVtdY+e3FfPumG/j9rx/jhfsfwOvPO59jXv+musuSJMDAVZdY+e3F3DXvGrY+/RQAv39sI3fNuwbA0JXUETp2lbI0HN++6YbtYdtv69NP8e2bbqipIknakYGrrvD7Xw/+2NNdjUtSaQauusIL9z9gWOOSVJqBq67w+vPOZ/xee+8wNn6vvXn9eefXVJGksSYijq7a8vV//S4iLh5wTNe155OGpX9hlKuUJY1UZq4CXgsQEeNoPLLxtgGH9WZ7PqnZMa9/kwEr9Yj/9Z/P2unBFx+++autfPDFqcDPMvMXA8a7rj2fJEmDqsJ2p/Z81XirnAf80yDjXdeeT5KkXWlnez6qtrDnAP/civP1M3AlSWNN29rzVU4H7svMXw2yr/va80mStAttac/X5O0MPp0Mo2jPZ+BKksaaS2m0xmvWivZ8/X1w3wLc2jT2vv4WfTTa862h0Z7vi8B/G/K57RYkSarRiLoFFVil3HIGriSpTrbn6yVfWbaOyxet4pFNWzh08iQuOe1ozj1+SKu8JUkakp4P3K8sW8fsW1ew5ZltAKzbtIXZt64AMHQlSS3T84umLl+0anvY9tvyzDYuX7SqpookSd2o5wP3kU1bhjUuSdJI9HzgHjp50rDGJUkaiZ4P3EtOO5pJE8btMDZpwjguOe3omiqSJHWjnl801b8wylXKkqR26vnAhUboGrCSpHbq+SllSZJKMHAlSSrAwJUkqQADV5KkAgxcSZIKMHAlSSrAwJUkqQADV5KkAgxcSZIKMHAlSSrAwJUkqQADV5KkAgxcSZIKMHAlSSrAwJUkqQADV5KkAgxcSZIKMHAlSSrAwJUkqQADV5KkAgxcSZIKMHAlSSrAwJUkqQADV5KkAmoL3Ii4PCIeiIjlEXFbRExu2jc7IlZHxKqIOK2uGiVJapU673DvBl6VmccBPwVmA0TEscB5wB8DM4DPRcS42qqUJKkFagvczLwrM7dWm/cCU6vXM4GbMvOpzPw5sBo4oY4aJUlqlU55D/c9wNer14cBDzftW1uN7SQiLoyIpRGxdOPGjW0uUZKkkRvfzpNHxDeAKYPs+nhm3l4d83FgK3DjcM+fmfOAeQB9fX05ilIlSWqrtgZuZr55d/sj4l3AWcCpmdkfmOuAw5sOm1qNSZI0ZtW5SnkG8BHgnMz8Q9OuBcB5EbF3RBwJHAV8r44aJUlqlbbe4e7BNcDewN0RAXBvZr4vM++PiFuAn9CYar4oM7fVWKckSaNWW+Bm5st3s+9TwKcKliNJUlt1yiplSZK6moErSVIBBq4kSQUYuJIkFWDgSpJUgIErSVIBBq4kSQUYuJIkFWDgSpJUgIErSVIBBq4kSQUYuJIkFWDgSpJUgIErSVIBBq4kSQUYuJIkFWDgSpJUgIErSVIBBq4kSQUYuJIkFWDgSpJUgIErSVIBBq4kSQUYuJIkFWDgSpJUgIErSVIBBq6krrFwzUKmz5/Ocdcfx/T501m4ZmHdJUnbja+7AElqhYVrFjLnO3N4ctuTAKx/Yj1zvjMHgDNfemaNlUkN3uFK6gpz75u7PWz7PbntSebeN7emiqQdGbiSusKGJzYMa1wqzcCV1BWm7DNlWONSaQaupK4wa9osJo6buMPYxHETmTVtVk0VSTty0ZSkrtC/MGrufXPZ8MQGpuwzhVnTZrlgSh0jMrPuGlqir68vly5dWncZkqThiboLKMUpZUmSCjBwJUkqwMCVJKkAA1eSpAIMXEmSCjBwJUkqwMCVJKkAA1eSpAIMXEmSCjBwJUkqwMCVJKkAA1eSpAIMXEmSCjBwJUkqwMCVJKkAA1eSpAIMXEmSCqg9cCPiwxGREXFAtR0RcXVErI6I5RExre4aJUkarVoDNyIOB6YDv2waPh04qvq6EPh8DaVJktRSdd/hXgl8BMimsZnADdlwLzA5Ig6ppTpJklqktsCNiJnAusz80YBdhwEPN22vrcYGO8eFEbE0IpZu3LixTZVKkjR649t58oj4BjBlkF0fBy6lMZ08Ypk5D5gH0NfXl3s4XJKk2rQ1cDPzzYONR8SrgSOBH0UEwFTgvog4AVgHHN50+NRqTJKkMauWKeXMXJGZB2XmEZl5BI1p42mZuQFYAJxfrVY+EdicmevrqFOSpFZp6x3uCH0NOANYDfwBeHe95UiSNHodEbjVXW7/6wQuqq8aSZJar+6PBUmS1BMMXEmSCuiIKWWpFb6ybB2XL1rFI5u2cOjkSVxy2tGce/ygH+GWpOIMXHWFryxbx+xbV7DlmW0ArNu0hdm3rgAwdCV1BKeU1RUuX7Rqe9j22/LMNi5ftKqmiiRpRwauusIjm7YMa1ySSjNw1RUOnTxpWOOSVJqBq65wyWlHM2nCuB3GJk0YxyWnHV1TRZK0IxdNqSv0L4xylbKkTmXgqmuce/xhBqykjuWUsiRJBRi4kiQVYOBKklSAgStJUgEGriRJBRi4kiQVYOBKklSAgStJUgEGriRJBRi4kiQVYOBKklSAgStJUgGRmXXX0BIRsRH4RZt/zQHAY23+HZ2gV64TvNZu1CvXCd1xrY9l5oy6iyihawK3hIhYmpl9ddfRbr1yneC1dqNeuU7orWvtBk4pS5JUgIErSVIBBu7wzKu7gEJ65TrBa+1GvXKd0FvXOub5Hq4kSQV4hytJUgEGriRJBRi4QxQRH46IjIgDqu2IiKsjYnVELI+IaXXXOFoRcXlEPFBdz20RMblp3+zqWldFxGk1ltkyETGjup7VEfGxuutplYg4PCIWR8RPIuL+iJhVje8XEXdHxIPV9xfXXWurRMS4iFgWEV+tto+MiCXV3/bmiNir7hpbISImR8T86v+nKyPipG7+u3YbA3cIIuJwYDrwy6bh04Gjqq8Lgc/XUFqr3Q28KjOPA34KzAaIiGOB84A/BmYAn4uIcbVV2QJV/Z+l8Xc8Fnh7dZ3dYCvw4cw8FjgRuKi6to8B92TmUcA91Xa3mAWsbNr+DHBlZr4c+C3w3lqqar25wJ2Z+UrgNTSuuZv/rl3FwB2aK4GPAM0rzGYCN2TDvcDkiDiklupaJDPvysyt1ea9wNTq9Uzgpsx8KjN/DqwGTqijxhY6AVidmWsy82ngJhrXOeZl5vrMvK96/Xsa/1E+jMb1XV8ddj1wbi0FtlhETAXOBK6ttgM4BZhfHdIV1xoR+wJvAK4DyMynM3MTXfp37UYG7h5ExExgXWb+aMCuw4CHm7bXVmPd4j3A16vX3Xit3XhNO4mII4DjgSXAwZm5vtq1ATi4rrpa7Coa/yB+ttreH9jU9I/HbvnbHglsBL5UTZ9fGxH70L1/164zvu4COkFEfAOYMsiujwOX0phO7gq7u9bMvL065uM0piVvLFmbWisiXgD8C3BxZv6ucePXkJkZEWP+M4ERcRbwaGb+ICLeWHM57TYemAZ8IDOXRMRcBkwfd8vftVsZuEBmvnmw8Yh4NY1/Vf6o+o/VVOC+iDgBWAcc3nT41Gqso+3qWvtFxLuAs4BT87kPaY/Ja92Dbrym7SJiAo2wvTEzb62GfxURh2Tm+urtj0frq7BlTgbOiYgzgInAi2i8zzk5IsZXd7nd8rddC6zNzCXV9nwagduNf9eu5JTybmTmisw8KDOPyMwjaPwPflpmbgAWAOdXq5VPBDY3TeuMSRExg8bU3DmZ+YemXQuA8yJi74g4ksZCse/VUWMLfR84qlrNuheNRWELaq6pJar3MK8DVmbmFU27FgAXVK8vAG4vXVurZebszJxa/f/zPOCbmfnnwGLgrdVh3XKtG4CHI+LoauhU4Cd04d+1W3mHO3JfA86gsYDoD8C76y2nJa4B9gburu7o783M92Xm/RFxC43/c28FLsrMbTXWOWqZuTUi3g8sAsYB/5CZ99dcVqucDLwTWBERP6zGLgU+DdwSEe+l0crybfWUV8RHgZsi4m+AZVQLjbrAB4Abq38krqHx353n0Tt/1zHNRztKklSAU8qSJBVg4EqSVICBK0lSAQauJEkFGLiSJBVg4EqSVICBKxUSEdsi4odVy7wfVS0fn1ft64uIq1vwO95ftaTb3kpSUmfwc7hSIRHxeGa+oHp9EPBl4N8z869b+DuOp9GO7ltAX2Y+1qpzSxodA1cqpDlwq+2X0njE5AHAnwJ/mZlnRcQcGs/wfinwEuCDNPrank7jmcBnZ+Yze/hdD2HgSh3FKWWpJpm5hsZjJQ8aZPfLaPR0PQf4R2BxZr4a2EKj96ukMcbAlTrT16u72BU0QvnOanwFcERdRUkaOQNXqkk1pbyNwdupPQWQmc8CzzS1SnwWm45IY5KBK9UgIg4EvgBcky6kkHqCgSuVM6n/Y0HAN4C7gE+08hdExF9ExFoaTdeXR8S1rTy/pJFzlbIkSQV4hytJUgEuvpDGoIi4jcZndZt9NDMX1VGPpD1zSlmSpAKcUpYkqQADV5KkAgxcSZIKMHAlSSrg/wN5iNXJ7X1aLQAAAABJRU5ErkJggg==",
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg height=\"423.528125pt\" version=\"1.1\" viewBox=\"0 0 475.180586 423.528125\" width=\"475.180586pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n <metadata>\n  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2021-08-16T02:23:20.694432</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.4.2, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 423.528125 \nL 475.180586 423.528125 \nL 475.180586 0 \nL 0 0 \nz\n\" style=\"fill:none;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 49.260938 385.69375 \nL 416.266836 385.69375 \nL 416.266836 7.2 \nL 49.260938 7.2 \nz\n\" style=\"fill:#ffffff;\"/>\n   </g>\n   <g id=\"PathCollection_1\">\n    <defs>\n     <path d=\"M 0 3 \nC 0.795609 3 1.55874 2.683901 2.12132 2.12132 \nC 2.683901 1.55874 3 0.795609 3 0 \nC 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \nC 1.55874 -2.683901 0.795609 -3 0 -3 \nC -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \nC -2.683901 -1.55874 -3 -0.795609 -3 0 \nC -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \nC -1.55874 2.683901 -0.795609 3 0 3 \nz\n\" id=\"m1aa569d267\" style=\"stroke:#1f77b4;\"/>\n    </defs>\n    <g clip-path=\"url(#p95382033cf)\">\n     <use style=\"fill:#1f77b4;stroke:#1f77b4;\" x=\"193.063513\" xlink:href=\"#m1aa569d267\" y=\"24.404261\"/>\n     <use style=\"fill:#1f77b4;stroke:#1f77b4;\" x=\"65.943024\" xlink:href=\"#m1aa569d267\" y=\"266.057112\"/>\n     <use style=\"fill:#1f77b4;stroke:#1f77b4;\" x=\"180.530194\" xlink:href=\"#m1aa569d267\" y=\"368.489489\"/>\n    </g>\n   </g>\n   <g id=\"PathCollection_2\">\n    <defs>\n     <path d=\"M 0 3 \nC 0.795609 3 1.55874 2.683901 2.12132 2.12132 \nC 2.683901 1.55874 3 0.795609 3 0 \nC 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \nC 1.55874 -2.683901 0.795609 -3 0 -3 \nC -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \nC -2.683901 -1.55874 -3 -0.795609 -3 0 \nC -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \nC -1.55874 2.683901 -0.795609 3 0 3 \nz\n\" id=\"m03a64800d2\" style=\"stroke:#ff7f0e;\"/>\n    </defs>\n    <g clip-path=\"url(#p95382033cf)\">\n     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"286.306715\" xlink:href=\"#m03a64800d2\" y=\"214.288289\"/>\n     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"399.58475\" xlink:href=\"#m03a64800d2\" y=\"208.746978\"/>\n    </g>\n   </g>\n   <g id=\"PathCollection_3\">\n    <defs>\n     <path d=\"M 0 3 \nC 0.795609 3 1.55874 2.683901 2.12132 2.12132 \nC 2.683901 1.55874 3 0.795609 3 0 \nC 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \nC 1.55874 -2.683901 0.795609 -3 0 -3 \nC -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \nC -2.683901 -1.55874 -3 -0.795609 -3 0 \nC -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \nC -1.55874 2.683901 -0.795609 3 0 3 \nz\n\" id=\"m7140d0ae91\" style=\"stroke:#2ca02c;\"/>\n    </defs>\n    <g clip-path=\"url(#p95382033cf)\">\n     <use style=\"fill:#2ca02c;stroke:#2ca02c;\" x=\"83.792531\" xlink:href=\"#m7140d0ae91\" y=\"131.608953\"/>\n     <use style=\"fill:#2ca02c;stroke:#2ca02c;\" x=\"309.24145\" xlink:href=\"#m7140d0ae91\" y=\"332.852268\"/>\n    </g>\n   </g>\n   <g id=\"PathCollection_4\">\n    <defs>\n     <path d=\"M 0 3 \nC 0.795609 3 1.55874 2.683901 2.12132 2.12132 \nC 2.683901 1.55874 3 0.795609 3 0 \nC 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \nC 1.55874 -2.683901 0.795609 -3 0 -3 \nC -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \nC -2.683901 -1.55874 -3 -0.795609 -3 0 \nC -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \nC -1.55874 2.683901 -0.795609 3 0 3 \nz\n\" id=\"ma62e2a72c0\" style=\"stroke:#d62728;\"/>\n    </defs>\n    <g clip-path=\"url(#p95382033cf)\">\n     <use style=\"fill:#d62728;stroke:#d62728;\" x=\"202.269933\" xlink:href=\"#ma62e2a72c0\" y=\"139.430063\"/>\n    </g>\n   </g>\n   <g id=\"PathCollection_5\">\n    <defs>\n     <path d=\"M 0 3 \nC 0.795609 3 1.55874 2.683901 2.12132 2.12132 \nC 2.683901 1.55874 3 0.795609 3 0 \nC 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \nC 1.55874 -2.683901 0.795609 -3 0 -3 \nC -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \nC -2.683901 -1.55874 -3 -0.795609 -3 0 \nC -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \nC -1.55874 2.683901 -0.795609 3 0 3 \nz\n\" id=\"m874d50f182\" style=\"stroke:#9467bd;\"/>\n    </defs>\n    <g clip-path=\"url(#p95382033cf)\">\n     <use style=\"fill:#9467bd;stroke:#9467bd;\" x=\"321.437056\" xlink:href=\"#m874d50f182\" y=\"87.383698\"/>\n    </g>\n   </g>\n   <g id=\"PathCollection_6\">\n    <defs>\n     <path d=\"M 0 3 \nC 0.795609 3 1.55874 2.683901 2.12132 2.12132 \nC 2.683901 1.55874 3 0.795609 3 0 \nC 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \nC 1.55874 -2.683901 0.795609 -3 0 -3 \nC -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \nC -2.683901 -1.55874 -3 -0.795609 -3 0 \nC -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \nC -1.55874 2.683901 -0.795609 3 0 3 \nz\n\" id=\"m895bd6bb99\" style=\"stroke:#8c564b;\"/>\n    </defs>\n    <g clip-path=\"url(#p95382033cf)\">\n     <use style=\"fill:#8c564b;stroke:#8c564b;\" x=\"180.498755\" xlink:href=\"#m895bd6bb99\" y=\"250.976833\"/>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <defs>\n       <path d=\"M 0 0 \nL 0 3.5 \n\" id=\"m0d7a0ad886\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"73.768563\" xlink:href=\"#m0d7a0ad886\" y=\"385.69375\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- −40 -->\n      <g transform=\"translate(63.216219 400.292188)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 678 2272 \nL 4684 2272 \nL 4684 1741 \nL 678 1741 \nL 678 2272 \nz\n\" id=\"DejaVuSans-2212\" transform=\"scale(0.015625)\"/>\n        <path d=\"M 2419 4116 \nL 825 1625 \nL 2419 1625 \nL 2419 4116 \nz\nM 2253 4666 \nL 3047 4666 \nL 3047 1625 \nL 3713 1625 \nL 3713 1100 \nL 3047 1100 \nL 3047 0 \nL 2419 0 \nL 2419 1100 \nL 313 1100 \nL 313 1709 \nL 2253 4666 \nz\n\" id=\"DejaVuSans-34\" transform=\"scale(0.015625)\"/>\n        <path d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" id=\"DejaVuSans-30\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-2212\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-34\"/>\n       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"132.463774\" xlink:href=\"#m0d7a0ad886\" y=\"385.69375\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- −20 -->\n      <g transform=\"translate(121.91143 400.292188)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" id=\"DejaVuSans-32\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-2212\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\n       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"191.158985\" xlink:href=\"#m0d7a0ad886\" y=\"385.69375\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 0 -->\n      <g transform=\"translate(187.977735 400.292188)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_4\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"249.854195\" xlink:href=\"#m0d7a0ad886\" y=\"385.69375\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 20 -->\n      <g transform=\"translate(243.491695 400.292188)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-32\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_5\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"308.549406\" xlink:href=\"#m0d7a0ad886\" y=\"385.69375\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 40 -->\n      <g transform=\"translate(302.186906 400.292188)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-34\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_6\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"367.244617\" xlink:href=\"#m0d7a0ad886\" y=\"385.69375\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 60 -->\n      <g transform=\"translate(360.882117 400.292188)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 2113 2584 \nQ 1688 2584 1439 2293 \nQ 1191 2003 1191 1497 \nQ 1191 994 1439 701 \nQ 1688 409 2113 409 \nQ 2538 409 2786 701 \nQ 3034 994 3034 1497 \nQ 3034 2003 2786 2293 \nQ 2538 2584 2113 2584 \nz\nM 3366 4563 \nL 3366 3988 \nQ 3128 4100 2886 4159 \nQ 2644 4219 2406 4219 \nQ 1781 4219 1451 3797 \nQ 1122 3375 1075 2522 \nQ 1259 2794 1537 2939 \nQ 1816 3084 2150 3084 \nQ 2853 3084 3261 2657 \nQ 3669 2231 3669 1497 \nQ 3669 778 3244 343 \nQ 2819 -91 2113 -91 \nQ 1303 -91 875 529 \nQ 447 1150 447 2328 \nQ 447 3434 972 4092 \nQ 1497 4750 2381 4750 \nQ 2619 4750 2861 4703 \nQ 3103 4656 3366 4563 \nz\n\" id=\"DejaVuSans-36\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-36\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_7\">\n     <!-- Dim_1 -->\n     <g transform=\"translate(216.973262 413.970312)scale(0.1 -0.1)\">\n      <defs>\n       <path d=\"M 1259 4147 \nL 1259 519 \nL 2022 519 \nQ 2988 519 3436 956 \nQ 3884 1394 3884 2338 \nQ 3884 3275 3436 3711 \nQ 2988 4147 2022 4147 \nL 1259 4147 \nz\nM 628 4666 \nL 1925 4666 \nQ 3281 4666 3915 4102 \nQ 4550 3538 4550 2338 \nQ 4550 1131 3912 565 \nQ 3275 0 1925 0 \nL 628 0 \nL 628 4666 \nz\n\" id=\"DejaVuSans-44\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 603 3500 \nL 1178 3500 \nL 1178 0 \nL 603 0 \nL 603 3500 \nz\nM 603 4863 \nL 1178 4863 \nL 1178 4134 \nL 603 4134 \nL 603 4863 \nz\n\" id=\"DejaVuSans-69\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 3328 2828 \nQ 3544 3216 3844 3400 \nQ 4144 3584 4550 3584 \nQ 5097 3584 5394 3201 \nQ 5691 2819 5691 2113 \nL 5691 0 \nL 5113 0 \nL 5113 2094 \nQ 5113 2597 4934 2840 \nQ 4756 3084 4391 3084 \nQ 3944 3084 3684 2787 \nQ 3425 2491 3425 1978 \nL 3425 0 \nL 2847 0 \nL 2847 2094 \nQ 2847 2600 2669 2842 \nQ 2491 3084 2119 3084 \nQ 1678 3084 1418 2786 \nQ 1159 2488 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1356 3278 1631 3431 \nQ 1906 3584 2284 3584 \nQ 2666 3584 2933 3390 \nQ 3200 3197 3328 2828 \nz\n\" id=\"DejaVuSans-6d\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 3263 -1063 \nL 3263 -1509 \nL -63 -1509 \nL -63 -1063 \nL 3263 -1063 \nz\n\" id=\"DejaVuSans-5f\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 794 531 \nL 1825 531 \nL 1825 4091 \nL 703 3866 \nL 703 4441 \nL 1819 4666 \nL 2450 4666 \nL 2450 531 \nL 3481 531 \nL 3481 0 \nL 794 0 \nL 794 531 \nz\n\" id=\"DejaVuSans-31\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-44\"/>\n      <use x=\"77.001953\" xlink:href=\"#DejaVuSans-69\"/>\n      <use x=\"104.785156\" xlink:href=\"#DejaVuSans-6d\"/>\n      <use x=\"202.197266\" xlink:href=\"#DejaVuSans-5f\"/>\n      <use x=\"252.197266\" xlink:href=\"#DejaVuSans-31\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_7\">\n      <defs>\n       <path d=\"M 0 0 \nL -3.5 0 \n\" id=\"macf7cbc4c2\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"49.260938\" xlink:href=\"#macf7cbc4c2\" y=\"357.959384\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- −40 -->\n      <g transform=\"translate(21.15625 361.758603)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-2212\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-34\"/>\n       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_8\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"49.260938\" xlink:href=\"#macf7cbc4c2\" y=\"298.239792\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- −20 -->\n      <g transform=\"translate(21.15625 302.03901)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-2212\"/>\n       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\n       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_9\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"49.260938\" xlink:href=\"#macf7cbc4c2\" y=\"238.5202\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- 0 -->\n      <g transform=\"translate(35.898438 242.319418)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_10\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"49.260938\" xlink:href=\"#macf7cbc4c2\" y=\"178.800607\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- 20 -->\n      <g transform=\"translate(29.535938 182.599826)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-32\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_11\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"49.260938\" xlink:href=\"#macf7cbc4c2\" y=\"119.081015\"/>\n      </g>\n     </g>\n     <g id=\"text_12\">\n      <!-- 40 -->\n      <g transform=\"translate(29.535938 122.880234)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-34\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_12\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"49.260938\" xlink:href=\"#macf7cbc4c2\" y=\"59.361423\"/>\n      </g>\n     </g>\n     <g id=\"text_13\">\n      <!-- 60 -->\n      <g transform=\"translate(29.535938 63.160642)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-36\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_14\">\n     <!-- Dim_2 -->\n     <g transform=\"translate(14.798438 212.2375)rotate(-90)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-44\"/>\n      <use x=\"77.001953\" xlink:href=\"#DejaVuSans-69\"/>\n      <use x=\"104.785156\" xlink:href=\"#DejaVuSans-6d\"/>\n      <use x=\"202.197266\" xlink:href=\"#DejaVuSans-5f\"/>\n      <use x=\"252.197266\" xlink:href=\"#DejaVuSans-32\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 49.260938 385.69375 \nL 49.260938 7.2 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 49.260938 385.69375 \nL 416.266836 385.69375 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n  </g>\n  <g id=\"legend_1\">\n   <g id=\"text_15\">\n    <!-- Label -->\n    <g transform=\"translate(430.73918 170.489062)scale(0.1 -0.1)\">\n     <defs>\n      <path d=\"M 628 4666 \nL 1259 4666 \nL 1259 531 \nL 3531 531 \nL 3531 0 \nL 628 0 \nL 628 4666 \nz\n\" id=\"DejaVuSans-4c\" transform=\"scale(0.015625)\"/>\n      <path d=\"M 2194 1759 \nQ 1497 1759 1228 1600 \nQ 959 1441 959 1056 \nQ 959 750 1161 570 \nQ 1363 391 1709 391 \nQ 2188 391 2477 730 \nQ 2766 1069 2766 1631 \nL 2766 1759 \nL 2194 1759 \nz\nM 3341 1997 \nL 3341 0 \nL 2766 0 \nL 2766 531 \nQ 2569 213 2275 61 \nQ 1981 -91 1556 -91 \nQ 1019 -91 701 211 \nQ 384 513 384 1019 \nQ 384 1609 779 1909 \nQ 1175 2209 1959 2209 \nL 2766 2209 \nL 2766 2266 \nQ 2766 2663 2505 2880 \nQ 2244 3097 1772 3097 \nQ 1472 3097 1187 3025 \nQ 903 2953 641 2809 \nL 641 3341 \nQ 956 3463 1253 3523 \nQ 1550 3584 1831 3584 \nQ 2591 3584 2966 3190 \nQ 3341 2797 3341 1997 \nz\n\" id=\"DejaVuSans-61\" transform=\"scale(0.015625)\"/>\n      <path d=\"M 3116 1747 \nQ 3116 2381 2855 2742 \nQ 2594 3103 2138 3103 \nQ 1681 3103 1420 2742 \nQ 1159 2381 1159 1747 \nQ 1159 1113 1420 752 \nQ 1681 391 2138 391 \nQ 2594 391 2855 752 \nQ 3116 1113 3116 1747 \nz\nM 1159 2969 \nQ 1341 3281 1617 3432 \nQ 1894 3584 2278 3584 \nQ 2916 3584 3314 3078 \nQ 3713 2572 3713 1747 \nQ 3713 922 3314 415 \nQ 2916 -91 2278 -91 \nQ 1894 -91 1617 61 \nQ 1341 213 1159 525 \nL 1159 0 \nL 581 0 \nL 581 4863 \nL 1159 4863 \nL 1159 2969 \nz\n\" id=\"DejaVuSans-62\" transform=\"scale(0.015625)\"/>\n      <path d=\"M 3597 1894 \nL 3597 1613 \nL 953 1613 \nQ 991 1019 1311 708 \nQ 1631 397 2203 397 \nQ 2534 397 2845 478 \nQ 3156 559 3463 722 \nL 3463 178 \nQ 3153 47 2828 -22 \nQ 2503 -91 2169 -91 \nQ 1331 -91 842 396 \nQ 353 884 353 1716 \nQ 353 2575 817 3079 \nQ 1281 3584 2069 3584 \nQ 2775 3584 3186 3129 \nQ 3597 2675 3597 1894 \nz\nM 3022 2063 \nQ 3016 2534 2758 2815 \nQ 2500 3097 2075 3097 \nQ 1594 3097 1305 2825 \nQ 1016 2553 972 2059 \nL 3022 2063 \nz\n\" id=\"DejaVuSans-65\" transform=\"scale(0.015625)\"/>\n      <path d=\"M 603 4863 \nL 1178 4863 \nL 1178 0 \nL 603 0 \nL 603 4863 \nz\n\" id=\"DejaVuSans-6c\" transform=\"scale(0.015625)\"/>\n     </defs>\n     <use xlink:href=\"#DejaVuSans-4c\"/>\n     <use x=\"55.712891\" xlink:href=\"#DejaVuSans-61\"/>\n     <use x=\"116.992188\" xlink:href=\"#DejaVuSans-62\"/>\n     <use x=\"180.46875\" xlink:href=\"#DejaVuSans-65\"/>\n     <use x=\"241.992188\" xlink:href=\"#DejaVuSans-6c\"/>\n    </g>\n   </g>\n   <g id=\"PathCollection_7\">\n    <g>\n     <use style=\"fill:#1f77b4;stroke:#1f77b4;\" x=\"432.277461\" xlink:href=\"#m1aa569d267\" y=\"182.542188\"/>\n    </g>\n   </g>\n   <g id=\"text_16\">\n    <!-- 0.0 -->\n    <g transform=\"translate(450.277461 185.167188)scale(0.1 -0.1)\">\n     <defs>\n      <path d=\"M 684 794 \nL 1344 794 \nL 1344 0 \nL 684 0 \nL 684 794 \nz\n\" id=\"DejaVuSans-2e\" transform=\"scale(0.015625)\"/>\n     </defs>\n     <use xlink:href=\"#DejaVuSans-30\"/>\n     <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n     <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n    </g>\n   </g>\n   <g id=\"PathCollection_8\">\n    <g>\n     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"432.277461\" xlink:href=\"#m03a64800d2\" y=\"197.220313\"/>\n    </g>\n   </g>\n   <g id=\"text_17\">\n    <!-- 1.0 -->\n    <g transform=\"translate(450.277461 199.845313)scale(0.1 -0.1)\">\n     <use xlink:href=\"#DejaVuSans-31\"/>\n     <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n     <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n    </g>\n   </g>\n   <g id=\"PathCollection_9\">\n    <g>\n     <use style=\"fill:#2ca02c;stroke:#2ca02c;\" x=\"432.277461\" xlink:href=\"#m7140d0ae91\" y=\"211.898438\"/>\n    </g>\n   </g>\n   <g id=\"text_18\">\n    <!-- 3.0 -->\n    <g transform=\"translate(450.277461 214.523438)scale(0.1 -0.1)\">\n     <defs>\n      <path d=\"M 2597 2516 \nQ 3050 2419 3304 2112 \nQ 3559 1806 3559 1356 \nQ 3559 666 3084 287 \nQ 2609 -91 1734 -91 \nQ 1441 -91 1130 -33 \nQ 819 25 488 141 \nL 488 750 \nQ 750 597 1062 519 \nQ 1375 441 1716 441 \nQ 2309 441 2620 675 \nQ 2931 909 2931 1356 \nQ 2931 1769 2642 2001 \nQ 2353 2234 1838 2234 \nL 1294 2234 \nL 1294 2753 \nL 1863 2753 \nQ 2328 2753 2575 2939 \nQ 2822 3125 2822 3475 \nQ 2822 3834 2567 4026 \nQ 2313 4219 1838 4219 \nQ 1578 4219 1281 4162 \nQ 984 4106 628 3988 \nL 628 4550 \nQ 988 4650 1302 4700 \nQ 1616 4750 1894 4750 \nQ 2613 4750 3031 4423 \nQ 3450 4097 3450 3541 \nQ 3450 3153 3228 2886 \nQ 3006 2619 2597 2516 \nz\n\" id=\"DejaVuSans-33\" transform=\"scale(0.015625)\"/>\n     </defs>\n     <use xlink:href=\"#DejaVuSans-33\"/>\n     <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n     <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n    </g>\n   </g>\n   <g id=\"PathCollection_10\">\n    <g>\n     <use style=\"fill:#d62728;stroke:#d62728;\" x=\"432.277461\" xlink:href=\"#ma62e2a72c0\" y=\"226.576563\"/>\n    </g>\n   </g>\n   <g id=\"text_19\">\n    <!-- 4.0 -->\n    <g transform=\"translate(450.277461 229.201563)scale(0.1 -0.1)\">\n     <use xlink:href=\"#DejaVuSans-34\"/>\n     <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n     <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n    </g>\n   </g>\n   <g id=\"PathCollection_11\">\n    <g>\n     <use style=\"fill:#9467bd;stroke:#9467bd;\" x=\"432.277461\" xlink:href=\"#m874d50f182\" y=\"241.254688\"/>\n    </g>\n   </g>\n   <g id=\"text_20\">\n    <!-- 5.0 -->\n    <g transform=\"translate(450.277461 243.879688)scale(0.1 -0.1)\">\n     <defs>\n      <path d=\"M 691 4666 \nL 3169 4666 \nL 3169 4134 \nL 1269 4134 \nL 1269 2991 \nQ 1406 3038 1543 3061 \nQ 1681 3084 1819 3084 \nQ 2600 3084 3056 2656 \nQ 3513 2228 3513 1497 \nQ 3513 744 3044 326 \nQ 2575 -91 1722 -91 \nQ 1428 -91 1123 -41 \nQ 819 9 494 109 \nL 494 744 \nQ 775 591 1075 516 \nQ 1375 441 1709 441 \nQ 2250 441 2565 725 \nQ 2881 1009 2881 1497 \nQ 2881 1984 2565 2268 \nQ 2250 2553 1709 2553 \nQ 1456 2553 1204 2497 \nQ 953 2441 691 2322 \nL 691 4666 \nz\n\" id=\"DejaVuSans-35\" transform=\"scale(0.015625)\"/>\n     </defs>\n     <use xlink:href=\"#DejaVuSans-35\"/>\n     <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n     <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n    </g>\n   </g>\n   <g id=\"PathCollection_12\">\n    <g>\n     <use style=\"fill:#8c564b;stroke:#8c564b;\" x=\"432.277461\" xlink:href=\"#m895bd6bb99\" y=\"255.932813\"/>\n    </g>\n   </g>\n   <g id=\"text_21\">\n    <!-- 7.0 -->\n    <g transform=\"translate(450.277461 258.557813)scale(0.1 -0.1)\">\n     <defs>\n      <path d=\"M 525 4666 \nL 3525 4666 \nL 3525 4397 \nL 1831 0 \nL 1172 0 \nL 2766 4134 \nL 525 4134 \nL 525 4666 \nz\n\" id=\"DejaVuSans-37\" transform=\"scale(0.015625)\"/>\n     </defs>\n     <use xlink:href=\"#DejaVuSans-37\"/>\n     <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n     <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p95382033cf\">\n   <rect height=\"378.49375\" width=\"367.005899\" x=\"49.260938\" y=\"7.2\"/>\n  </clipPath>\n </defs>\n</svg>\n",
      "text/plain": [
       "<Figure size 483.875x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.FacetGrid(tsne_df, hue='Label', size=6).map(plt.scatter, 'Dim_1', 'Dim_2').add_legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Now will implement the same t-sne graph in a slightly different way\n",
    "\n",
    "Below code is from - https://www.kaggle.com/subinium/dimension-reduction-3-t-sne\n",
    "This produces a better looking color graph\n",
    "But it super-SLOW, never ran in my local JupyterLab\n",
    "Also in Kaggle it took more than 1 hour to run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "import plotly.graph_objects as go\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "color = ['#6388b4', '#ffae34', '#ef6f6a', '#8cc2ca', '#55ad89', '#c3bc3f', '#bb7693', '#baa094', '#a9b5ae', '#767676']\n",
    "\n",
    "mnist_df_without_label = train_df_org.drop(['label'], axis=1)\n",
    "mnist_tsne_2 = tsne.fit_transform(mnist_df_without_label, train_df_org_label)\n",
    "\n",
    "fig = go.Figure()\n",
    "\n",
    "for idx in range(10):\n",
    "    fig.add_trace(go.Scatter(\n",
    "        x = mnist_tsne_2[:, 0][train_df_org_label==idx],\n",
    "        y = mnist_tsne_2[:, 1][train_df_org_label == idx],\n",
    "        name=str(idx),\n",
    "        opacity=0.6,\n",
    "        mode='markers',\n",
    "        marker=dict(color=color[idx])\n",
    "    ))\n",
    "\n",
    "fig.update_layout(\n",
    "    width=800,\n",
    "    height=800,\n",
    "    title='T-SNE on MNIST',\n",
    "    yaxis = dict(\n",
    "        scaleanchor = 'x',\n",
    "        scaleratio = 1\n",
    "    ),\n",
    "    legend=dict(\n",
    "        orientation='h',\n",
    "        yanchor='bottom',\n",
    "        y=1.02,\n",
    "        xanchor='right',\n",
    "        x=1\n",
    "    )\n",
    ")\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "---\n",
    "\n",
    "### The main motivations for dimensionality reduction are:\n",
    "\n",
    "• To speed up a subsequent training algorithm (in some cases it may even remove noise and redundant features, making the training algorithm perform better)\n",
    "• To visualize the data and gain insights on the most important features\n",
    "• To save space (compression)\n",
    "\n",
    "### The main drawbacks are:\n",
    "• Some information is lost, possibly degrading the performance of subsequent training algorithms.\n",
    "• It can be computationally intensive.\n",
    "• Transformed features are often hard to interpret.\n",
    "\n",
    "---\n",
    "\n",
    "## Data preparation for dimensionality reduction\n",
    "\n",
    "There is no best technique for dimensionality reduction and no mapping of techniques to problems.\n",
    "\n",
    "Instead, the best approach is to use systematic controlled experiments to discover what dimensionality reduction techniques, when paired with your model of choice, result in the best performance on your dataset.\n",
    "\n",
    "Typically, linear algebra and manifold learning methods assume that all input features have the same scale or distribution. This suggests that it is good practice to either normalize or standardize data prior to using these methods if the input variables have differing scales or units.\n",
    "\n",
    "---\n",
    "\n",
    "## Strategies and common questions around implementations of various Dimensionality Reduction techniques (mainly t-SNE and Principal Component Analysis).\n",
    "\n",
    "Think of PCA as an exploratory technique to investigate and study your system before doing other things. Dimensionality reduction, by nature, loses some information, just like image compression for instance. Consequently, it will often reduce the prediction quality of your model, but depending on the data it could also leave it unchanged or even improve it in some cases (very noisy data). In general, its main benefit will be to speed up training.\n",
    "\n",
    "Besides speed of training, feature reduction also help with multicollinearity in some cases. Which is mainly an issue if you are interested in parameters estimation, like in causal analysis. For instance, if you use a multiple linear regression model to estimate the effect of some regressors on a dependent variable, multicollinearity will prevent proper parameters estimation because you won't be able to identify the effect of each regressor.\n",
    "\n",
    "However, in many other ML applications, we don't really care about parameters identification, we care about how a set of variables can be used to predict another variable. If some variables are highly correlated, then there will be some redundant information in your data, but this shouldn't be problematic in terms of prediction quality.\n",
    "\n",
    "In my experience, in couple of Kaggle Challenges, after doing some PCA, the accuracy has actually went down. After googling found out that some infer that PCA assumed that the variance could translate to high differentiable information which classifier could use. But at the same time, it threw some information away(not just noise), and these information could play a huge role in classification, although the variance is low.\n",
    "\n",
    "Indeed, the curse of dimensionality refers to the fact that many problems that do not exist in low-dimensional space arise in high-dimensional space. In Machine Learning, one common manifestation is the fact that randomly sampled high-dimensional vectors are generally very sparse, increasing the risk of overfitting and making it very difficult to identify patterns in the data without having plenty of training data.\n",
    "\n",
    "Indeed, it often make any sense to chain two different dimensionality reduction algorithms. A common example is using PCA to quickly get rid of a large number of useless dimensions, then applying another much slower dimensionality reduction algorithm, such as LLE. This two-step approach will likely yield the same performance as using LLE only, but in a fraction of the time.\n",
    "\n",
    "\n",
    "#### Once a dataset’s dimensionality has been reduced, is it possible to reverse the operation?\n",
    "\n",
    "Once a dataset’s dimensionality has been reduced using one of the available algorithms, it is almost always impossible to perfectly reverse the operation, because some information gets lost during dimensionality reduction. Moreover, while some algorithms (such as PCA) have a simple reverse transformation procedure that can reconstruct a dataset relatively similar to the original, other algorithms (such as T-SNE) do not.\n",
    "\n",
    "#### Can PCA be used to reduce the dimensionality of a highly nonlinear dataset?\n",
    "\n",
    "PCA (Principal component analysis) can be used to significantly reduce the dimensionality of most datasets, even if they are highly nonlinear, because it can at least get rid of useless dimensions. However, if there are no useless dimensions—as in a [Swiss roll dataset](http://people.cs.uchicago.edu/~dinoj/manifold/swissroll.html)—then reducing dimensionality with PCA will lose too much information. You want to unroll the Swiss roll, not squash it.\n",
    "\n",
    "#### If I perform PCA on a 1,000-dimensional dataset, setting the explained variance ratio to 95%. How many dimensions will the resulting dataset have ?\n",
    "\n",
    "It depends on the dataset. Let’s look at two extreme examples. First, suppose the dataset is composed of points that are almost perfectly aligned. In this case, PCA can reduce the dataset down to just one dimension while still preserving 95% of the variance. Now imagine that the dataset is composed of perfectly random points, scattered all around the 1,000 dimensions. In this case roughly 950 dimensions are required to preserve 95% of the variance. So the answer is, it depends on the dataset, and it could be any number between 1\n",
    "and 950. Plotting the explained variance as a function of the number of dimensions is one way to get a rough idea of the dataset’s intrinsic dimensionality.\n",
    "\n",
    "#### In what cases would we use vanilla PCA, Incremental PCA, Randomized PCA, or Kernel PCA?\n",
    "\n",
    "Regular PCA is the default, but it works only if the dataset fits in memory. Incremental PCA is useful for large datasets that don’t fit in memory, but it is slower than regular PCA, so if the dataset fits in memory we should prefer regular PCA. Incremental PCA is also useful for online tasks, when we need to apply PCA on the fly, every time a new instance arrives. Randomized PCA is useful when we want to considerably reduce dimensionality and the dataset fits in memory; in this case, it is much faster than regular PCA. Finally, Kernel PCA is useful for nonlinear datasets.\n",
    "\n",
    "#### How to evaluate the performance of a dimensionality reduction algorithm on our dataset?\n",
    "\n",
    "Intuitively, a dimensionality reduction algorithm performs well if it eliminates a lot of dimensions from the dataset without losing too much information. One way to measure this is to apply the reverse transformation and measure the reconstruction error. However, not all dimensionality reduction algorithms provide a reverse transformation. Alternatively, if we are using dimensionality reduction as a preprocessing step before another Machine Learning algorithm (e.g., a Random Forest classifier), then we can simply measure the performance\n",
    "of that second algorithm; if dimensionality reduction did not lose too much information, then the algorithm should perform just as well as when using the original dataset."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "36cf16204b8548560b1c020c4e8fb5b57f0e4c58016f52f2d4be01e192833930"
  },
  "kernelspec": {
   "display_name": "Python 3.9.6 64-bit",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}