PlattScaling_From_Scratch.ipynb

{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Platt Scaling (PS) is probably the most prevailing parametric calibration method. It aims to train a sigmoid function to map the original outputs from a classifier to calibrated probabilities.\n",
                "\n",
                "So its simply is a form of Probability Calibration and is a way of transforming classification output into a probability distribution. For example: If you’ve got the dependent variable as 0 & 1 in the train data set, using this method you can convert it into probability.\n",
                "\n",
                "Platt Scaling is a parametric method. It was originally built to calibrate the support vector machine model and is now also applied to other classifiers. Platt Scaling uses a sigmoid function to map the outputs of a binary classifier to calibrated probabilities. The sigmoidal function of this method is defined as follows:\n",
                "\n",
                "![](https://imgur.com/SLTK19Y.png)\n",
                "\n",
                "\n",
                "### Why Probability Calibration\n",
                "\n",
                "The predictions made by a predictive model sometimes need to be calibrated. Calibrated predictions may (or may not) result in an improved calibration on a reliability diagram.\n",
                "\n",
                "Some algorithms are fit in such a way that their predicted probabilities are already calibrated, such as in logistic regression.\n",
                "\n",
                "Other algorithms do not directly produce predictions of probabilities (most likely be uncalibrated), and instead, a prediction of probabilities must be approximated. Examples include neural networks, support vector machines, and decision trees. So the result from these algos may benefit from being modified via calibration.\n",
                "\n",
                "Calibration of prediction probabilities is a rescaling operation that is applied after the predictions have been made by a predictive model.\n",
                "\n",
                "There are two popular approaches to calibrating probabilities; they are the Platt Scaling and Isotonic Regression.\n",
                "\n",
                "Platt Scaling is simpler and is suitable for reliability diagrams with the S-shape. Isotonic Regression is more complex, requires a lot more data (otherwise it may overfit), but can support reliability diagrams with different shapes (is nonparametric).\n",
                "\n",
                "> _Platt Scaling is most effective when the distortion in the predicted probabilities is sigmoid-shaped. Isotonic Regression is a more powerful calibration method that can correct any monotonic distortion. Unfortunately, this extra power comes at a price. A learning curve analysis shows that Isotonic Regression is more prone to overfitting, and thus performs worse than Platt Scaling, when data is scarce._\n",
                "\n",
                "— [Predicting Good Probabilities With Supervised Learning](https://www.cs.cornell.edu/~alexn/papers/calibration.icml05.crc.rev3.pdf), 2005.\n",
                "\n",
                "#### More interpretations of Platt Calibration\n",
                "\n",
                "In the case of a binary dependent variable, a logistic regression maps the predictors to the probability of occurrence of the dependent variable. Without any transformation, the probability used for training the model is either 1 (if y is positive in the training set) or 0 (if y is negative).\n",
                "\n",
                "So: Instead of using the absolute values 1 for positive class and 0 for negative class when fitting\n",
                "\n",
                "![](https://imgur.com/JtPoAXv.png)\n",
                "\n",
                "\n",
                "Platt suggests to use the mentioned transformation to allow the opposite label to appear with some probability. In this way, some regularization is introduced. When the size of the dataset reaches infinity, _y_+will become 1, and _y_− will become zero.\n",
                "\n",
                "#### Methods of Platt Scaling\n",
                "\n",
                "Refer this [**Paper**](https://drive.google.com/file/d/133odBinMOIVb_rh_GQxxsyMRyW-Zts7a/view)\n",
                "\n",
                "This paper shows that maximum margin methods, such as boosted trees and boosted stumps push probability mass away from 0 and 1 yielding a characteristic sigmoid shaped distortion in the predicted probabilities.\n",
                "\n",
                "Models such as Naive Bayes, which make unrealistic independence assumptions, push probabilities toward 0 and 1. Other models such as neural nets and bagged trees do not have these biases and predict well-calibrated probabilities. The Paper experiments with two ways of correcting the biased probabilities predicted by some learning methods: Platt Scaling and Isotonic Regression. They qualitatively examine what kinds of distortions these calibration methods are suitable for and quantitatively examine how much data they need to be effective. The empirical results show that after calibration boosted trees, random forests, and SVMs predict the best probabilities.\n",
                "\n",
                "From this [Paper](https://drive.google.com/file/d/133odBinMOIVb_rh_GQxxsyMRyW-Zts7a/view)\n",
                "\n",
                "![](https://imgur.com/QHh5CEw.png)\n",
                "\n",
                "\n",
                "> Two questions arise: where does the sigmoid train set come from? and how to avoid overfitting to this training set? If we use the same data set that was used to train the model\n",
                "\n",
                "> we want to calibrate, we introduce unwanted bias. For example, if the model learns to discriminate the train set perfectly and orders all the negative examples before the positive examples, then the sigmoid transformation will output just a 0,1 function. So we need to use an independent calibration set in order to get good posterior probabilities. This, however, is not a draw back, since the same set can be used for model and parameter selection.\n",
                "\n",
                "[Source](https://drive.google.com/file/d/133odBinMOIVb_rh_GQxxsyMRyW-Zts7a/view)\n",
                "\n",
                "![](https://imgur.com/7oFGm1L.png)\n",
                "\n",
                "\n",
                "**The MOST IMPORTANT Point above is — if Y[i] is 1, it will be replaced with y+ value else it will be replaced with y- value**\n",
                "\n",
                "Hence, we have to change the values of y_test as mentioned in the above image. we will calculate y+, y- based on data points in train data\n",
                "\n",
                "#### Implementation from Scratch"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "import numpy as np\n",
                "import pandas as pd\n",
                "from sklearn.datasets import make_classification\n",
                "import numpy as np\n",
                "from sklearn.svm import SVC\n",
                "from sklearn.svm import LinearSVC\n",
                "from sklearn.model_selection import train_test_split\n",
                "import math\n",
                "import matplotlib.pyplot as plt\n",
                "\n",
                "X, y = make_classification(n_samples=5000, n_features=5, n_redundant=2,\n",
                "                           n_classes=2, weights=[0.7], class_sep=0.7, random_state=15)\n",
                "\n",
                "x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2, train_size=0.8)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 4,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Randomly initializing weights (w) and intercept values (b)\n",
                "# dim — size of the w vector we want (or number of features or parameters in this case)\n",
                "def initialize_weights(dim):\n",
                "    ''' In this function, we will initialize our weights and bias\n",
                "    w — weights, a numpy array of size\n",
                "    b — bias, a scalar\n",
                "    '''\n",
                "    #initialize the weights to zeros array of (1,dim) dimensions\n",
                "    #you use zeros_like function to initialize zero, check this link https://docs.scipy.org/doc/numpy/reference/generated/numpy.zeros_like.html\n",
                "    #initialize bias to zero\n",
                "    w = np.zeros_like(dim)\n",
                "    # Above will initialize all w with 0.\n",
                "    b = 0\n",
                "    return w,b\n",
                "  \n",
                "def sigmoid(z):\n",
                "    return 1.0/(1 + np.exp(-z)) \n",
                "\n",
                "\n",
                "\n",
                "def logloss(y_true, y_pred):\n",
                "    \n",
                "    len_y_true = len(y_true)\n",
                "    \n",
                "    number_of_plus = np.count_nonzero(y_true == 1)\n",
                "    number_of_minus = np.count_nonzero(y_true == 0)\n",
                "    \n",
                "    # Platt scaling\n",
                "    # we will calculate y+, y- based on data points in train data\n",
                "    y_plus = (number_of_plus+1)/(number_of_minus+2)\n",
                "    y_minus = 1/(number_of_minus+2)\n",
                "    \n",
                "    sum_of_loss = 0\n",
                "    \n",
                "    for i in range(0, len_y_true):\n",
                "      if (y_true[i] == 1):\n",
                "        sum_of_loss += ((y_plus * np.log10(y_pred[i])) + ((1- y_plus) * np.log10(1-y_pred[i])))\n",
                "      else:\n",
                "        sum_of_loss += ((y_minus * np.log10(y_pred[i])) + ((1 - y_minus) * np.log10(1-y_pred[i])))\n",
                "        \n",
                "    loss = (-1/len_y_true) * sum_of_loss  \n",
                "    return loss\n",
                "    \n",
                "    \n",
                "  \n",
                "def gradient_dw(x,y,w,b,alpha,N):\n",
                "    '''In this function, we will compute the gardient w.r.to w '''\n",
                "    z = np.dot(w, x) + b\n",
                "    dw = x*(y - sigmoid(z)) - ((alpha)*(1/N) * w)\n",
                "    return dw\n",
                "  \n",
                "  \n",
                "def gradient_db(x,y,w,b):\n",
                "    z = np.dot(w, x) + b\n",
                "    db = y - sigmoid(z)\n",
                "\n",
                "    return db"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 5,
            "metadata": {},
            "outputs": [],
            "source": [
                "def train(X_train, y_train, X_test, y_test, epochs, alpha, eta0, tol=1e-3):\n",
                "    \"\"\" In this function, we will implement logistic regression\"\"\"\n",
                "    # Here eta0 is learning rate\n",
                "    # implement the code as follows\n",
                "    # initialize the weights (call the initialize_weights(X_train[0]) function)\n",
                "    w, b = initialize_weights(X_train[0])\n",
                "    # for every epoch\n",
                "    train_loss = []\n",
                "    test_loss = []\n",
                "    N = len(X_train)\n",
                "\n",
                "    loss_threshold = 0.0001\n",
                "\n",
                "    for epoch in range(epochs):\n",
                "        # for every data point(X_train,y_train)\n",
                "        for row in range(N - 1):\n",
                "            # compute gradient w.r.to w (call the gradient_dw() function)\n",
                "            delta_weights = gradient_dw(\n",
                "                X_train[row], y_train[row], w, b, alpha, len(X_train)\n",
                "            )\n",
                "\n",
                "            # compute gradient w.r.to b (call the gradient_db() function)\n",
                "            delta_bias = gradient_db(X_train[row], y_train[row], w, b)\n",
                "\n",
                "            # update w, b\n",
                "            w = w + eta0 * delta_weights\n",
                "            b = b + eta0 * delta_bias\n",
                "\n",
                "        # predict the output of x_train[for all data points in X_train] using w,b\n",
                "        # y_prediction_train is a numpy array (vector) containing all predictions (0/1) for the examples in X\n",
                "        y_prediction_train = [  \n",
                "        sigmoid(np.dot(w, x_row) + b) for x_row in X_train\n",
                "        ]\n",
                "\n",
                "        # compute the loss between predicted and actual values (call the loss function)\n",
                "        # store all the train loss values in a list\n",
                "        train_loss.append(logloss(y_train, y_prediction_train))\n",
                "\n",
                "        # predict the output of x_test[for all data points in X_test] using w,b\n",
                "        y_prediction_test = [\n",
                "            sigmoid(np.dot(w, x_row) + b) for x_row in X_test\n",
                "        ]\n",
                "\n",
                "        print(\n",
                "            f\"For EPOCH No : {epoch} Train Loss is : {logloss(y_train, y_prediction_train)} and Test Loss is : {logloss(y_test, y_prediction_test)}\"\n",
                "        )\n",
                "\n",
                "        # compute the loss between predicted and actual values (call the loss function)\n",
                "        test_loss.append(logloss(y_test, y_prediction_test))\n",
                "\n",
                "      \n",
                "\n",
                "    return w, b, train_loss, test_loss"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 6,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "For EPOCH No : 0 Train Loss is : 0.2744419344800745 and Test Loss is : 0.27396804375413747\n",
                        "For EPOCH No : 1 Train Loss is : 0.25388270599770685 and Test Loss is : 0.2530470672889054\n",
                        "For EPOCH No : 2 Train Loss is : 0.2378933473945702 and Test Loss is : 0.23678528686907455\n",
                        "For EPOCH No : 3 Train Loss is : 0.22535914670131452 and Test Loss is : 0.22404811138173356\n",
                        "For EPOCH No : 4 Train Loss is : 0.2154510564869265 and Test Loss is : 0.2139901088303104\n",
                        "For EPOCH No : 5 Train Loss is : 0.2075576774367939 and Test Loss is : 0.2059869660056321\n",
                        "For EPOCH No : 6 Train Loss is : 0.20122721364761822 and Test Loss is : 0.1995769993734065\n",
                        "For EPOCH No : 7 Train Loss is : 0.1961232499884324 and Test Loss is : 0.19441634384670936\n",
                        "For EPOCH No : 8 Train Loss is : 0.19199265848161423 and Test Loss is : 0.19024629761068107\n",
                        "For EPOCH No : 9 Train Loss is : 0.18864275625360785 and Test Loss is : 0.1868700126719687\n",
                        "For EPOCH No : 10 Train Loss is : 0.18592512476594525 and Test Loss is : 0.18413595061839877\n",
                        "For EPOCH No : 11 Train Loss is : 0.18372411161637026 and Test Loss is : 0.1819261115810273\n",
                        "For EPOCH No : 12 Train Loss is : 0.1819485990950416 and Test Loss is : 0.18014760158224546\n",
                        "For EPOCH No : 13 Train Loss is : 0.18052605425104204 and Test Loss is : 0.17872653533946092\n",
                        "For EPOCH No : 14 Train Loss is : 0.1793981817222153 and Test Loss is : 0.17760358147778926\n",
                        "For EPOCH No : 15 Train Loss is : 0.1785177120798343 and Test Loss is : 0.17673067205968207\n",
                        "For EPOCH No : 16 Train Loss is : 0.1778460026110075 and Test Loss is : 0.1760685453916722\n",
                        "For EPOCH No : 17 Train Loss is : 0.17735122550615595 and Test Loss is : 0.17558489132073285\n",
                        "For EPOCH No : 18 Train Loss is : 0.17700698530272246 and Test Loss is : 0.17525293674068224\n",
                        "For EPOCH No : 19 Train Loss is : 0.17679125334954554 and Test Loss is : 0.1750503561125544\n",
                        "For EPOCH No : 20 Train Loss is : 0.17668553882952096 and Test Loss is : 0.1749584244106994\n",
                        "For EPOCH No : 21 Train Loss is : 0.1766742380660308 and Test Loss is : 0.17496135268769816\n",
                        "For EPOCH No : 22 Train Loss is : 0.176744119481736 and Test Loss is : 0.1750457625162668\n",
                        "For EPOCH No : 23 Train Loss is : 0.1768839127150246 and Test Loss is : 0.17520026700400704\n",
                        "For EPOCH No : 24 Train Loss is : 0.1770839784052766 and Test Loss is : 0.17541513429830508\n",
                        "For EPOCH No : 25 Train Loss is : 0.17733604096850783 and Test Loss is : 0.17568201546436332\n",
                        "For EPOCH No : 26 Train Loss is : 0.17763297094141742 and Test Loss is : 0.17599372298829039\n",
                        "For EPOCH No : 27 Train Loss is : 0.17796860661825117 and Test Loss is : 0.1763440493855824\n",
                        "For EPOCH No : 28 Train Loss is : 0.17833760705097962 and Test Loss is : 0.17672761780159452\n",
                        "For EPOCH No : 29 Train Loss is : 0.1787353302473546 and Test Loss is : 0.1771397582991189\n",
                        "For EPOCH No : 30 Train Loss is : 0.1791577317384783 and Test Loss is : 0.17757640489830942\n",
                        "For EPOCH No : 31 Train Loss is : 0.17960127970877984 and Test Loss is : 0.17803400948016282\n",
                        "For EPOCH No : 32 Train Loss is : 0.18006288366703746 and Test Loss is : 0.17850946946913274\n",
                        "For EPOCH No : 33 Train Loss is : 0.1805398342458837 and Test Loss is : 0.1790000668333707\n",
                        "For EPOCH No : 34 Train Loss is : 0.18102975219211018 and Test Loss is : 0.17950341642667067\n",
                        "For EPOCH No : 35 Train Loss is : 0.18153054498283194 and Test Loss is : 0.18001742207720206\n",
                        "For EPOCH No : 36 Train Loss is : 0.1820403697969484 and Test Loss is : 0.18054023912883696\n",
                        "For EPOCH No : 37 Train Loss is : 0.18255760180519734 and Test Loss is : 0.18107024237962213\n",
                        "For EPOCH No : 38 Train Loss is : 0.18308080692883688 and Test Loss is : 0.18160599855255646\n",
                        "For EPOCH No : 39 Train Loss is : 0.18360871836699472 and Test Loss is : 0.18214624258679252\n",
                        "For EPOCH No : 40 Train Loss is : 0.18414021631372238 and Test Loss is : 0.1826898571607575\n",
                        "For EPOCH No : 41 Train Loss is : 0.18467431038392706 and Test Loss is : 0.1832358549586729\n",
                        "For EPOCH No : 42 Train Loss is : 0.18521012434730919 and Test Loss is : 0.18378336327338765\n",
                        "For EPOCH No : 43 Train Loss is : 0.18574688283478905 and Test Loss is : 0.1843316106049731\n",
                        "For EPOCH No : 44 Train Loss is : 0.18628389973569792 and Test Loss is : 0.184879914969241\n",
                        "For EPOCH No : 45 Train Loss is : 0.18682056804827266 and Test Loss is : 0.18542767367538845\n",
                        "For EPOCH No : 46 Train Loss is : 0.1873563509827245 and Test Loss is : 0.18597435436929083\n",
                        "For EPOCH No : 47 Train Loss is : 0.18789077414663075 and Test Loss is : 0.1865194871699588\n",
                        "For EPOCH No : 48 Train Loss is : 0.1884234186678564 and Test Loss is : 0.1870626577524927\n",
                        "For EPOCH No : 49 Train Loss is : 0.18895391513146734 and Test Loss is : 0.187603501252488\n",
                        "w_coef  [ 0.04136468 -0.17919617 -0.73899583 -0.93469074  2.12216676]\n",
                        "intercept b  -0.6280337633128319\n"
                    ]
                }
            ],
            "source": [
                "alpha=0.0001\n",
                "eta0=0.0001\n",
                "# N=len(X_train)\n",
                "epochs=50\n",
                "\n",
                "w, b, train_log_loss, test_loss = train(x_train, y_train, x_test, y_test, epochs, alpha, eta0)\n",
                "print('w_coef ', w)\n",
                "print('intercept b ', b)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 8,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAr+0lEQVR4nO3deZhU5Zn38e/d0OwIrRBUmkVAkGZtQTRuNIYQjAZcIoGAo46OGkeNr5OZkOiYSJJxS1wS9Y2OcUxGIyITjYkYNYYWfTNGQGURZBWlIQgoIM1O9/3+8Zy2i7a6Kbq7OF1Vv891nauqzlJ1P9jWr855znmOuTsiIiI15cVdgIiINE0KCBERSUoBISIiSSkgREQkKQWEiIgkpYAQEZGkFBAiOcLMLjWz1+OuQzKHAkIylpmtMbPRcddRH2ZWYmaVZlZeY/pi3LWJVGkedwEiOWy9uxfGXYRIbbQHIVnHzFqa2b1mtj6a7jWzltGyTmb2RzPbamafmNlrZpYXLfuuma0zs+1mtszMvpTkvU82sw1m1ixh3vlmtjB6PsLM5pnZp2b2kZndXc82lJrZbWb2ZvRevzezIxOWjzOzd6N2lJpZ/4Rl3czsd2a2ycw+NrP7a7z3T81si5m9b2Zn16c+yQ0KCMlGNwGnAEOBIcAI4OZo2b8AZUBnoAvwfcDNrB9wLXCSu7cHvgKsqfnG7v43YAdwVsLsbwK/jZ7fB9zn7kcAvYEZDWjHPwD/CBwD7Ad+DmBmfYEngRuidswC/mBmLaLg+iPwAdAT6ApMT3jPk4FlQCfgTuBXZmYNqFGymAJCstFkYJq7b3T3TcCtwMXRsn2EL9we7r7P3V/zMCBZBdASKDKzfHdf4+6rann/J4FJAGbWHvhqNK/q/fuYWSd3L3f3N+qo89hoDyBxapuw/L/dfbG77wD+HZgQBcA3gOfd/WV33wf8FGgNnEoIw2OBf3X3He6+290TO6Y/cPf/dPcK4NfRv0WXOv81JWcpICQbHUv4BV3lg2gewF3ASuAlM1ttZlMB3H0l4Rf5D4GNZjbdzI4lud8CF0SHrS4A3nL3qs+7HOgLvGdmc83s3DrqXO/uHWtMOxKWr63RhnzCL/8D2ufuldG6XYFuhBDYX8tnbkjYbmf0tF0dNUoOU0BINloP9Eh43T2ah7tvd/d/cfdewDjgxqq+Bnf/rbufHm3rwB3J3tzdlxC+oM/mwMNLuPsKd58EfCHafmaNvYJD0a1GG/YBm2u2LzpE1A1YRwiK7mamE1CkwRQQkunyzaxVwtSccLjnZjPrbGadgFuAxwHM7Fwz6xN9qW4jHFqqNLN+ZnZWtFewG9gFVNbxub8Fvg2cCTxdNdPMpphZ5+hX/dZodl3vU5cpZlZkZm2AacDM6NDQDOAcM/uSmeUT+lX2AH8F3gT+DtxuZm2jf5PT6vn5kuMUEJLpZhG+zKumHwI/BuYBC4FFwFvRPIDjgT8D5cD/Ag+6+2xC/8PthF/oGwh7AN+r43OfBEYCf3H3zQnzxwLvmlk5ocN6orvvquU9jk1yHcSFCcv/G3gsqqcVcD2Auy8DpgC/iOr9GvA1d98bBcjXgD7Ah4QO+W/U0Q6RWpluGCTS9JhZKfC4uz8Sdy2Su7QHISIiSSkgREQkKR1iEhGRpLQHISIiSWXNudKdOnXynj171nv7HTt20LZtfU9Xz1xqd25Ru3NLKu2eP3/+ZnfvnGxZ1gREz549mTdvXr23Ly0tpaSkpPEKyhBqd25Ru3NLKu02sw9qW6ZDTCIikpQCQkREklJAiIhIUlnTByEimWvfvn2UlZWxe/futLx/hw4dWLp0aVreuylLbHerVq0oLCwkPz8/5e0VECISu7KyMtq3b0/Pnj1Jx/2Ltm/fTvv27Rv9fZu6qna7Ox9//DFlZWUcd9xxKW+vQ0wiErvdu3dz1FFHpSUcBMyMo4466pD30BQQItIkKBzSqz7/vjkfEFu2wLRp8N57ubf7KSJSl5wPiGbN4Ac/gLff7hh3KSISow0bNjBx4kR69+7NsGHD+OpXv8ry5cvp1asXy5YtO2DdG264gTvuOPCGg2vWrGHgwIGNUktJSUmDLvxtLDkfEEccAUcfDWvXtom7FBGJibtz/vnnU1JSwqpVq5g/fz633XYbH330ERMnTmT69OmfrVtZWcnMmTOZOHFijBUfHjkfEAB9+yogRHLZ7Nmzyc/P5+qrr/5s3pAhQzjjjDOYNGkSTz311Gfz58yZQ48ePejRo0eytwJCp/tll13GoEGDKC4uZvbs2QDs3LmTCRMmUFRUxPnnn8/JJ5980D2FJ598kkGDBjFw4EC++93vAlBRUcGll17KwIEDGTRoEPfccw8AP//5zykqKmLw4MGNEmA6zZUQEDNnto67DBEBbrgB3nmncd+zqKglDz5Y+/LFixczbNiwpMsGDRpEXl4eCxYsYMiQIUyfPp1JkybV+XkPPPAAZsaiRYt47733GDNmDMuXL+fBBx+koKCAJUuWsHjxYoYOHVrn+6xfv57vfve7zJ8/n4KCAsaMGcOzzz5Lt27dWLduHYsXLwZg69atANx+++28//77tGzZ8rN5DaE9CEJAbN3agi1b4q5ERJqiSZMmMX36dPbv38+zzz7LRRddVOf6r7/+OlOmTAHghBNOoEePHixfvpzXX3/9s1/2AwcOZPDgwXW+z9y5cykpKaFz5840b96cyZMnM2fOHHr16sXq1au57rrr+NOf/sQRRxwBwODBg5k8eTKPP/44zZs3/Pe/9iCAfv3C44oVMGJEvLWI5Lp7723899y+fQ/QotblAwYMYObMmbUunzhxImPGjGHkyJEMHjyYLl26NH6Rh6CgoIAFCxbw4osv8stf/pIZM2bw6KOP8vzzzzNnzhz+8Ic/8JOf/IS//vWvDfoc7UEQ9iAAapyoICI54qyzzmLPnj08/PDDn81buHAhr732GgC9e/emU6dOTJ069aCHlwDOOOMMnnjiCQCWL1/Ohx9+SL9+/TjttNOYMWMGAEuWLGHRokV1vs+IESN49dVX2bx5MxUVFTz55JOMHDmSzZs3U1lZyYUXXsiPf/xj3nrrLSorK1m7di2jRo3ijjvuYNu2bZSXl9f3nwTQHgQAvXpBXp6zfLku1BHJRWbGM88889npq61ataJnz57cm7A7M2nSJKZOncoFF1xw0Pe75ppr+Na3vsWgQYNo3rw5jz32GC1btuSaa67hkksuoaioiBNOOIEBAwbQoUOHWt/nmGOO4fbbb2fUqFG4O+eccw7jx49nwYIFXHbZZVRWVgJw2223UVFRwZQpU9i2bRvuzvXXX0/Hjh0b9u+SLfekHj58uDfkvOGuXXdx+umtSThZISfoRiq5pam2e+nSpfTv3z9t799UxmKqqKhg3759tGrVilWrVjF69GiWLVtGixa1H/5qiJrtTvbvbGbz3X14su21BxHp1m0ny5frTCYRSZ+dO3cyatQo9u3bh7vz4IMPpi0cGoMCItKt205mzTqKykrIU8+MiKRB+/btm8QV0qnSV2GksHAXO3fC+vVxVyKSm7LlcHdTVZ9/XwVEpLBwJwDLl8dciEgOatWqFR9//LFCIk2q7gfRqlWrQ9pOh5gi3bvvAkJAnHVWzMWI5JjCwkLKysrYtGlTWt5/9+7dh/zlmA0S2111R7lDoYCIHHXUHtq00bUQInHIz88/pDudHarS0lKKi4vT9v5NVUPbrUNMkbw8OP54HWISEamigEjQt68CQkSkigIiQb9+8P77sHdv3JWIiMRPAZGgb1+oqIDVq+OuREQkfgqIBFWD9ukwk4iIAuIACggRkWoKiAQFBdC5swJCRAQUEJ/Tt6+uhRARAQXE5+hUVxGRQAFRQ9++sGEDfPpp3JWIiMRLAVFD4v2pRURyWVoDwszGmtkyM1tpZlOTLL/RzJaY2UIze8XMeiQs625mL5nZ0midnumstYruTy0iEqQtIMysGfAAcDZQBEwys6Iaq70NDHf3wcBM4M6EZb8B7nL3/sAIYGO6ak3UuzeYqR9CRCSdexAjgJXuvtrd9wLTgfGJK7j7bHffGb18AygEiIKkubu/HK1XnrBeWrVqBT16KCBERNI53HdXYG3C6zLg5DrWvxx4IXreF9hqZr8DjgP+DEx194rEDczsSuBKgC5dulBaWlrvYsvLyz/bvnPnwcybl09p6fx6v1+mSGx3LlG7c4vaXU/unpYJ+DrwSMLri4H7a1l3CmEPomXCttuAXoQQ+x/g8ro+b9iwYd4Qs2fP/uz5dde5t2vnXlnZoLfMCIntziVqd25Ru2sHzPNavlfTeYhpHdAt4XVhNO8AZjYauAkY5+57otllwDseDk/tB54FTkxjrQfo2xfKy8PpriIiuSqdATEXON7MjjOzFsBE4LnEFcysGHiIEA4ba2zb0cw6R6/PApaksdYDaEwmEZE0BkT0y/9a4EVgKTDD3d81s2lmNi5a7S6gHfC0mb1jZs9F21YA3wFeMbNFgAH/ma5aa6q6FkKnuopILkvrPandfRYwq8a8WxKej65j25eBwemrrnbdukHLltqDEJHcpiupk9D9qUVEFBC10qB9IpLrFBC16NcPVq2CffvirkREJB4KiFr07Qv798OaNXFXIiISDwVELXSqq4jkOgVELRQQIpLrFBC16NQJjjwS3nsv7kpEROKhgKjDwIGwcGHcVYiIxEMBUYfi4hAQFRUHX1dEJNsoIOpQXAw7d6ofQkRykwKiDsXF4fGtt+KtQ0QkDgqIOvTvH8ZkevvtuCsRETn8FBB1yM+HQYMUECKSmxQQB1FcHAIi3OhORCR3KCAOorgYtmyBDz+MuxIRkcNLAXEQVR3VOswkIrlGAXEQgweH+0PoTCYRyTUKiINo0wZOOEF7ECKSexQQKajqqBYRySUKiBQUF8O6dbBpU9yViIgcPgqIFKijWkRykQIiBUOHhkcFhIjkEgVECo48Enr21JlMIpJbFBApUke1iOQaBUSKiothxQrYvj3uSkREDg8FRIqqOqoXLIi3DhGRw0UBkSKdySQiuUYBkaJjj4UvfEEBISK5QwGRIrOwF6EzmUQkVyggDkFxMbz7LuzZE3clIiLpp4A4BMXFsH9/CAkRkWyngDgE6qgWkVyS1oAws7FmtszMVprZ1CTLbzSzJWa20MxeMbMeNZYfYWZlZnZ/OutMVe/e0L69AkJEckPaAsLMmgEPAGcDRcAkMyuqsdrbwHB3HwzMBO6ssfxHwJx01Xio8vLCuEwKCBHJBencgxgBrHT31e6+F5gOjE9cwd1nu/vO6OUbQGHVMjMbBnQBXkpjjYesuBjeeQcqKuKuREQkvZqn8b27AmsTXpcBJ9ex/uXACwBmlgf8DJgCjK5tAzO7ErgSoEuXLpSWlta72PLy8pS2b936aHbuPIEnnniT7t13HnT9pi7VdmcbtTu3qN31k86ASJmZTQGGAyOjWdcAs9y9zMxq3c7dHwYeBhg+fLiXlJTUu4bS0lJS2b6gAO64A/LzR9CAj2syUm13tlG7c4vaXT/pPMS0DuiW8LowmncAMxsN3ASMc/eqKwy+CFxrZmuAnwL/YGa3p7HWlBUVQYsW6ocQkeyXzj2IucDxZnYcIRgmAt9MXMHMioGHgLHuvrFqvrtPTljnUkJH9ufOgopDfj4MGqSAEJHsl7Y9CHffD1wLvAgsBWa4+7tmNs3MxkWr3QW0A542s3fM7Ll01dOYhg+HN99UR7WIZLe09kG4+yxgVo15tyQ8r7UDOmGdx4DHGru2hjjzTHjooTD094knxl2NiEh66ErqehgZdaW/+mq8dYiIpJMCoh66dg1XVSsgRCSbKSDqaeRIeO01qKyMuxIRkfRQQNTTmWfCJ59oZFcRyV4KiHpSP4SIZLtDCggzKzCzwekqJpP07AnduysgRCR7HTQgzKw0Gnb7SOAt4D/N7O70l9b0nXkmzJkD7nFXIiLS+FLZg+jg7p8CFwC/cfeTqWMAvVwyciRs3AjLlsVdiYhI40slIJqb2THABOCPaa4no6gfQkSyWSoBMY0wXMZKd59rZr2AFektKzP06QPHHKOAEJHsdNChNtz9aeDphNergQvTWVSmMAv9EK++Gvoh6hiZXEQk46TSSX1n1EmdH903elN0/wYhHGZavx5Wr467EhGRxpXKIaYxUSf1ucAaoA/wr+ksKpOoH0JEslVKndTR4znA0+6+LY31ZJz+/aFTJwWEiGSfVIb7/qOZvQfsAr5lZp2B3ektK3Mk9kOIiGSTg+5BRHdyO5VwV7d9wA5gfLoLyyQjR8IHH4RJRCRbpNJJnQ9MAZ4ys5nA5cDH6S4sk1T1Q8yZE28dIiKNKZU+iP8LDAMejKYTo3kSGTgQOnbUYSYRyS6p9EGc5O5DEl7/xcwWpKugTNSsGZxxhgJCRLJLKnsQFWbWu+pFdCV1RfpKykwjR8LKleGaCBGRbJBKQPwrMDsa1fVV4C/Av6S3rMxz5pnhUf0QIpItUjmL6RXgeOB64DqgH3BkmuvKOMXF0L69DjOJSPZI6YZB7r7H3RdG0x7gnjTXlXGaN4fTTlNAiEj2qO8tRzUsXRIlJbB0qfohRCQ71DcgdA+1JM45Jzw+91y8dYiINIZaT3M1s0UkDwIDuqStogw2YAD06gW//z1cfXXc1YiINExd10Gce9iqyBJmcN55cP/98OmncMQRcVckIlJ/tR5icvcP6poOZ5GZZPx42LsX/vSnuCsREWmY+vZBSC1OPTUM//3738ddiYhIwyggGlnz5nDuufD887BvX9zViIjUnwIiDcaPh23bdE2EiGS2VIb7XmRmC2tMr5nZPWZ21EG2HWtmy8xspZlNTbL8RjNbEr3nK2bWI5o/1Mz+18zejZZ9o/5NPPzGjIHWrXWYSUQyWyp7EC8AzwOTo+kPwDxgA/BYbRuZWTPgAeBsoAiYZGZFNVZ7m3AjosHATODOaP5O4B/cfQAwFrjXzDqm1qT4tWkDX/5yCAjXFSMikqFSCYjR7v49d18UTTcBI939DqBnHduNAFa6+2p33wtMp8ad6Nx9trvvjF6+ARRG85e7+4ro+XpgI9D5UBoWt/POg7Vr4e23465ERKR+UrkfRDMzG+HubwKY2UlAs2jZ/jq26wqsTXhdBpxcx/qXE/ZWDmBmI4AWwKoky64ErgTo0qULpaWldbx93crLyxu0fU0FBfnk5Z3Kffd9wGWXrWm0921sjd3uTKF25xa1u57cvc4JOAlYBLwPrAEWRvPaAhPq2O7rwCMJry8G7q9l3SmEPYiWNeYfAywDTjlYncOGDfOGmD17doO2T+b0092HDGn0t21U6Wh3JlC7c4vaXTtgntfyvZrKcN9z3X0QMBQY4u6Do3k73H1GHZuuA7olvC6M5h3AzEYDNwHjPIwUWzX/CELfx03u/sbB6myKzjsPFiyA99+PuxIRkUOXyllMHczsbuAV4BUz+5mZdUjhvecCx5vZcWbWApgIHDCMnZkVAw8RwmFjwvwWwDPAb9x9ZurNaVrGRz0uGrxPRDJRKp3UjwLbgQnR9CnwXwfbyN33A9cCLwJLgRnu/q6ZTTOzcdFqdwHtgKfN7B0zq/oqnQCcCVwazX/HzIYeQruahD59oKhIp7uKSGZKpZO6t7tfmPD6VjN7J5U3d/dZwKwa825JeD66lu0eBx5P5TOauvPOgzvugE8+gSN1Hz4RySCp7EHsMrPTq16Y2WnArvSVlF3Gj4eKijD0hohIJkklIK4GHjCzNWa2BrgfuCqtVWWR4cPh2GN1mElEMk8qZzEtcPchwGBgsLsXA2elvbIskZcH48aF4b937467GhGR1KU8WJ+7f+run0Yvb0xTPVnpvPNgxw544XOXAYqINF31Hc3VGrWKLPelL8Exx8Cjj8ZdiYhI6uobEBqC7hA0bw6XXQazZsG6z10qKCLSNNUaEGa23cw+TTJtB449jDVmhX/8R6ishMcei7sSEZHU1HVP6vbufkSSqb27p3L9hCTo3RtGjQqHmSor465GROTgdEe5w+iKK2D1asjBQSVFJAMpIA6j88+Hjh3hkUfirkRE5OAUEIdR69YwZQr87ndh6A0RkaZMAXGYXXEF7NkDTzwRdyUiInVTQBxmQ4bAsGHhMJPuVy0iTZkCIgZXXAELF8L8+XFXIiJSOwVEDCZNCv0R6qwWkaZMARGDDh3goovgySfDGE0iIk2RAiImV1wBn34KMzP2hqoiku0UEDE5/XTo21eHmUSk6VJAxMQMLr8cXn8dli2LuxoRkc9TQMTokkvCSK/33x93JSIin6eAiFGXLiEkHnkENmyIuxoRkQMpIGI2dSrs3Qt33x13JSIiB1JAxKxPn3BdxIMPwubNcVcjIlJNAdEEfP/74XqI++6LuxIRkWoKiCagqAguvBB+/nPYujXuakREAgVEE3HzzeHCOZ3RJCJNhQKiiRg6FM49F+65B8rL465GREQB0aTcdFO4kdAvfxl3JSIiCogm5ZRTYPRo+OlPYdeuuKsRkVyngGhibr4ZPvpIYzSJSPwUEE3MmWeGgfzuvDPcmlREJC4KiCbGDP7936GsDH7967irEZFcltaAMLOxZrbMzFaa2dQky280syVmttDMXjGzHgnLLjGzFdF0STrrbGq+/GU4+WT44Q/Dqa8iInFIW0CYWTPgAeBsoAiYZGZFNVZ7Gxju7oOBmcCd0bZHAj8ATgZGAD8ws4J01drUmIWL5jZsgFtvjbsaEclV6dyDGAGsdPfV7r4XmA6MT1zB3We7+87o5RtAYfT8K8DL7v6Ju28BXgbGprHWJmfEiHDXufvug8WL465GRHJR8zS+d1dgbcLrMsIeQW0uB16oY9uuNTcwsyuBKwG6dOlCaWlpvYstLy9v0PbpcM45+Tz11AgmT97Bvfe+g1njf0ZTbPfhoHbnFrW7ftIZECkzsynAcGDkoWzn7g8DDwMMHz7cS0pK6l1DaWkpDdk+Xe66C666qiPr15cweXLjv39TbXe6qd25Re2un3QeYloHdEt4XRjNO4CZjQZuAsa5+55D2TYXXHFFONz0ne/Atm1xVyMiuSSdATEXON7MjjOzFsBE4LnEFcysGHiIEA4bExa9CIwxs4Koc3pMNC/n5OXBAw+Ei+d++MO4qxGRXJK2gHD3/cC1hC/2pcAMd3/XzKaZ2bhotbuAdsDTZvaOmT0XbfsJ8CNCyMwFpkXzctLw4XDVVfCLX8DChXFXIyK5Iq19EO4+C5hVY94tCc9H17Hto8Cj6asus/zkJ/D00/DP/wxz5pCWDmsRkUS6kjpDHHkk3HEHvP46PP543NWISC5QQGSQyy4LI77eeCOsXx93NSKS7RQQGSQvDx59FHbuhMmToaIi7opEJJspIDJM//7w4INQWgo/+lHc1YhINlNAZKBLLgnTtGnwl7/EXY2IZCsFRIa6/37o1y8cavroo7irEZFspIDIUO3awYwZsHUrXHwxVFbGXZGIZBsFRAYbNCgMC/7yy3D77XFXIyLZRgGR4a64AiZODHehe+21uKsRkWyigMhwZvDQQ9CrF0yapOsjRKTxKCCywBFHhGE4tm2Dr3wFtmyJuyIRyQYKiCwxdCg8+ywsXw5f+1q4mE5EpCEUEFnkS1+CJ56Av/4VJkyAffvirkhEMpkCIst8/evwy1/C88/D5Zfr9FcRqb8mcctRaVxXXgmbNsHNN0OnTvCzn2l4cBE5dAqILPX978PGjXDPPfCFL8DUqXFXJCKNZcsWWLy4eurYMdwzprEpILKUWQiHjz+G730vvP63f9OehEgm2bkTli6FRYsODIR166rX6dABxo5Nz+crILJYXh7813+FYcGnToW1a+G++6BZs7grE5FE+/fDihXhy78qDBYtglWrwD2s06oVFBWFk1EGDgzToEHQtWv6fvgpILJcfn44s6mwEH760/DL47e/hdat465MJPe4hx9qNYNg6VLYuzesk5cHxx8fTl2fMqU6CHr3Pvw/7hQQOSAvD+66C7p1gxtuCL9AnnsudGCLSHps2RK+/BODYPHicEFrlcLCEABjxlQHQf/+YW+hKVBA5JDrrw+7o5Mnw2mnwQsvxF2RSObbtQvee686DKqmxGFvOnYMX/7f/GZ4rDpEVFAQW9kpUUDkmAsvhKOPhnHj4ItfhFtuOYKSkrirEmn6KipCn0DNPYIVK6qvN2rZsrqfYNCg6jBIZz9BOikgctBpp8H/+3/w1a/C9dcXs3Vr6MRW57VI6CfYsOHAvYHFi2HJkrC3AOHLvnfvEAATJlSHQZ8+0DyLvlWzqClyKE44Ad5+Gy64YCM339yFl16C//5v6N497spEDp9PP60+dTQxED75pHqdLl3Cl//VV1f3ExQVQdu28dV9uCggcliHDnDzzUu59NIuXHMNDBkCDz8MF10Ud2UijWvvXmPhws8fHvrgg+p12rULAXDhhdVBMHAgdO4cX91xU0DkOLNwy9JTTw2d1xMmwGWXhTvVtWsXd3Uih6aiAt5///MXlr333pmf9RPk54c96FNPDcPSVB0e6t49nPEn1RQQAoTjqa+9BrfeCv/xH/DKK+HU2IsuyszONclu7uGansQQqNlPAOFGWoMGwYknfsi55/Zg4EDo2zeEhBycAkI+k58PP/5xuGz/2mvhG9+A++8PV18XF8ddneSqjRvh3XerQ6DqeeL1BMccEw4HXXVV9aGhoqLqveDS0vcpKekRTwMymAJCPuf002H+fPjVr+Cmm2DYsHDv6x//OAz8J5IOn3wSvvyrpqow2LSpep2CAhgwIFxPUHUtwYABcNRR8dWdzRQQklSzZuH47IQJMG0a/OIX8NRTYQjxb31L/RNSf1u2hC/+JUsODIQNG6rXad8+7AGMGxcCoCoMjj5ahzwPJwWE1KljR7j77hAWN94YRoS97Ta45ppwZbb2KKQ2mzaFEFi6NDxWBUJiELRtG4Jg7NgQBFVTt24KgqYgrQFhZmOB+4BmwCPufnuN5WcC9wKDgYnuPjNh2Z3AOYS73r0MfNu9alxDOdxOOAFmzYI33oA77wwd2T/7GVx6KXznO6GTW3JP1eBzS5eG4SaWLq0OhM2bq9dr1y4Ewdlnh8cBA8Jjt246c6gpS1tAmFkz4AHgy0AZMNfMnnP3JQmrfQhcCnynxranAqcRggPgdWAkUJqueiU1p5wCv/sdLFsWAuLRR8O1ExdcEPopRo/WFdnZaPduWLkyhEDNaceO6vUKCsIX//nnh8eqKVOHmsh16dyDGAGsdPfVAGY2HRgPfBYQ7r4mWlbzzskOtAJaAAbkAx+lsVY5RP36hWC49dZwzcRDD8HMmeFskilTwrUVgwbFXaUcCvcwwNzy5eEHQNXjsmXh2oLE+5t36xZGHb3iirB32b9/mDp3VhBkE0vXURsz+zow1t2viF5fDJzs7tcmWfcx4I81DjH9FLiCEBD3u/tNSba7ErgSoEuXLsOmT59e73rLy8tpl4M9r43V7r17jf/936N46aWj+dvfjqSiIo8+fbYzZsxHnHHGZo4+encjVNt4cvW/9/bt5VRUFFBW1pp168JUVtaGsrLWrF3bht27q3f/WrasoLBwF9277/xs6tZtJ4WFO2nduuZvuqYtV/97p9LuUaNGzXf34cmWNcmAMLM+hL6Lb0SrvAz8m7u/VtvnDR8+3OfNm1fvektLSynJwWFN09HuTZtg+nT4zW+g6j/JCSeEwQHPPhvOOCOMehmnbP7vXVkZ9gRWrTpwCoeI9rNjR/WBg7w86Nkz3KCmX78w9e0bHrt2zZ7+gWz+712XVNptZrUGRDoPMa0DuiW8LozmpeJ84A13LwcwsxeALwK1BoQ0HZ07w3XXhWn58tC5/cIL4aK7u+8OZ6586UswalTo0ygujj8wMol7uGbggw/CoZ/334fVq6ufr1kDe/ZUr9+sWQiB3r1h9OiPKCnpyvHHh1Do2RNatIipIdLkpTMg5gLHm9lxhGCYCHwzxW0/BP7JzG4jHGIaSTjbSTJM375huuGG0Jk5e3Z1YDz3XFinRYsQEqecEqZhw8IQCbna2b1zZxhGoqwsPK5dG8KgavrwwwM7hiGcjtyrV7hW4GtfC8979w7DT3fvXj0EdWnpCkpKuh72NklmSltAuPt+M7sWeJFwmuuj7v6umU0D5rn7c2Z2EvAMUAB8zcxudfcBwEzgLGARocP6T+7+h3TVKodH27Zw7rlhgvDl97e/hVNn33gjdHrfd19Y1qJFOMxR1fnZv3943b17OFMm0zpC3cPQ0ps2wUcfwd//Xj1t2BAe168PoZA41HSVTp2gR49wqO4rXwnPu3eH444LU8eOh71JkgPSeh2Eu88CZtWYd0vC87mEQ081t6sArkpnbRK/rl3D6bEXXBBe79sXRuFcsKD6fPr58+Hpp8MXbJU2bcK9fLt1C4+FheEL9Mgjq6eCgvDYti20bt3wvRH3cNimvDz8ei8vD9P27bB1a5i2bKl+3LIlXAewaVOYNm+uvil9oubNw/0GjjkmfOmffnp1m6qmrl1Dm0UON11JLU1Gfj6ceGKYEu3aFfoyVqwIv7DXrq2e/vzn8Ou78iAn1eTnh6Bo1So87t8/gjZtqvdEzMLkHr7I9+4NgVX1fM+eMJT0wTRrFn7NFxSE0OrePRwy69y5eqoKhKOPDutkS0ewZB8FhDR5rVuHmxkNGZJ8eWVlGNnzk08+P+3cGQKm5rRu3Xa+8IU2n+2ZuIfJLHSYt2gRpvz88NiyZbgauG3bzz8WFFSHQrt2mXf4S6Q2CgjJeHl54cu5oCD1IT9KS5dSUtIlvYWJZDjt3IqISFIKCBERSUoBISIiSSkgREQkKQWEiIgkpYAQEZGkFBAiIpKUAkJERJJK2/0gDjcz2wR80IC36ARsPuha2Uftzi1qd25Jpd093L1zsgVZExANZWbzartpRjZTu3OL2p1bGtpuHWISEZGkFBAiIpKUAqLaw3EXEBO1O7eo3bmlQe1WH4SIiCSlPQgREUlKASEiIknlfECY2VgzW2ZmK81satz1pJOZPWpmG81sccK8I83sZTNbET0WxFljYzOzbmY228yWmNm7ZvbtaH62t7uVmb1pZguidt8azT/OzP4W/b0/ZWYt4q41HcysmZm9bWZ/jF7nSrvXmNkiM3vHzOZF8+r9t57TAWFmzYAHgLOBImCSmRXFW1VaPQaMrTFvKvCKux8PvBK9zib7gX9x9yLgFOCfo//G2d7uPcBZ7j4EGAqMNbNTgDuAe9y9D7AFuDy+EtPq28DShNe50m6AUe4+NOH6h3r/red0QAAjgJXuvtrd9wLTgfEx15Q27j4H+KTG7PHAr6PnvwbOO5w1pZu7/93d34qebyd8aXQl+9vt7l4evcyPJgfOAmZG87Ou3QBmVgicAzwSvTZyoN11qPffeq4HRFdgbcLrsmheLuni7n+Pnm8AsvZGzWbWEygG/kYOtDs6zPIOsBF4GVgFbHX3/dEq2fr3fi/wb0Bl9PoocqPdEH4EvGRm883symhevf/Wmzd2dZK53N3NLCvPezazdsD/ADe4+6fhR2WQre129wpgqJl1BJ4BToi3ovQzs3OBje4+38xKYi4nDqe7+zoz+wLwspm9l7jwUP/Wc30PYh3QLeF1YTQvl3xkZscARI8bY66n0ZlZPiEcnnD330Wzs77dVdx9KzAb+CLQ0cyqfhhm49/7acA4M1tDOGR8FnAf2d9uANx9XfS4kfCjYAQN+FvP9YCYCxwfneHQApgIPBdzTYfbc8Al0fNLgN/HWEuji44//wpY6u53JyzK9nZ3jvYcMLPWwJcJ/S+zga9Hq2Vdu939e+5e6O49Cf8//8XdJ5Pl7QYws7Zm1r7qOTAGWEwD/tZz/kpqM/sq4ZhlM+BRd/9JvBWlj5k9CZQQhgD+CPgB8CwwA+hOGC59grvX7MjOWGZ2OvAasIjqY9LfJ/RDZHO7BxM6JJsRfgjOcPdpZtaL8Mv6SOBtYIq774mv0vSJDjF9x93PzYV2R218JnrZHPitu//EzI6inn/rOR8QIiKSXK4fYhIRkVooIEREJCkFhIiIJKWAEBGRpBQQIiKSlAJCMpqZuZn9LOH1d8zsh4303o+Z2dcPvmaDP+ciM1tqZrNrzO9pZruikTmrpn9oxM8tqRrtVCQZDbUhmW4PcIGZ3ebum+MupoqZNU8Y++dgLgf+yd1fT7JslbsPbbzKRFKnPQjJdPsJ9939PzUX1NwDMLPy6LHEzF41s9+b2Wozu93MJkf3T1hkZr0T3ma0mc0zs+XROD9Vg+DdZWZzzWyhmV2V8L6vmdlzwJIk9UyK3n+xmd0RzbsFOB34lZndlWqjzazczO6xcK+HV8ysczR/qJm9EdX1TNXY/2bWx8z+bOH+EG8ltLGdmc00s/fM7AlLHKRKcp4CQrLBA8BkM+twCNsMAa4G+gMXA33dfQRhiOjrEtbrSRjP5hzgl2bWivCLf5u7nwScBPyTmR0XrX8i8G1375v4YWZ2LOGeBGcR7s9wkpmd5+7TgHnAZHf/1yR19q5xiOmMaH5bYJ67DwBeJVwVD/Ab4LvuPphw9XjV/CeAB6L7Q5wKVI3uWQzcQLgfSi/CWEYigA4xSRaIRmf9DXA9sCvFzeZWDYFsZquAl6L5i4BRCevNcPdKYIWZrSaMiDoGGJywd9IBOB7YC7zp7u8n+byTgFJ33xR95hPAmYShTupS2yGmSuCp6PnjwO+igOzo7q9G838NPB2Nz9PV3Z8BcPfdUQ1E9ZZFr98hBGKyQ12SgxQQki3uBd4C/ith3n6ivWQzywMSbzOZOA5PZcLrSg78/6LmWDQOGHCdu7+YuCAa+2dHfYpvBPUdMyfx36ECfSdIAh1ikqwQDT42gwNvJbkGGBY9H0e4q9qhusjM8qJj9r2AZcCLwLeiYcQxs77R6Jl1eRMYaWadLNzqdhLh0FB95VE9Ouk3gdfdfRuwJeEw1MXAq9Gd9MrM7Lyo3pZm1qYBny05Qr8WJJv8DLg24fV/Ar83swXAn6jfr/sPCV/uRwBXu/tuM3uEcCjmrahTdxMHuY2ju//dzKYShp024Hl3T2XY5d7RoZ8qj7r7zwltGWFmNxPG9/9GtPwSQl9JG2A1cFk0/2LgITObBuwDLkrhsyXHaTRXkQxkZuXu3i7uOiS76RCTiIgkpT0IERFJSnsQIiKSlAJCRESSUkCIiEhSCggREUlKASEiIkn9f+EJQOqHQ6xAAAAAAElFTkSuQmCC",
                        "text/plain": [
                            "<Figure size 432x288 with 1 Axes>"
                        ]
                    },
                    "metadata": {
                        "needs_background": "light"
                    },
                    "output_type": "display_data"
                }
            ],
            "source": [
                "x = np.array([i for i in range(0, 50)])\n",
                "\n",
                "train_log_loss_arr = np.array(train_log_loss)\n",
                "\n",
                "plt.plot(x, train_log_loss_arr, \"-b\", label = 'Train log loss')\n",
                "\n",
                "plt.legend(loc=\"upper right\")\n",
                "plt.grid()\n",
                "\n",
                "plt.xlabel('Number of Epoch')\n",
                "plt.ylabel('Log Loss ')\n",
                "\n",
                "plt.title('Loss vs Epoch ')\n",
                "plt.show()"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### More References\n",
                "\n",
                "https://stats.stackexchange.com/questions/5196/why-use-platts-scaling\n",
                "\n",
                "https://drive.google.com/file/d/133odBinMOIVb_rh_GQxxsyMRyW-Zts7a/view\n",
                "\n",
                "[2007 Paper](https://link.springer.com/content/pdf/10.1007/s10994-007-5018-6.pdf)"
            ]
        }
    ],
    "metadata": {
        "interpreter": {
            "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
        },
        "kernelspec": {
            "display_name": "Python 3.8.10 64-bit",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "name": "python",
            "version": "3.8.10"
        },
        "orig_nbformat": 4
    },
    "nbformat": 4,
    "nbformat_minor": 2
}