Decision_Function_SVM_RBF_Kernel_From_Scratch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4LuKrFzC4ZxV"
   },
   "source": [
    "# Implementing Decision Function of SVM RBF Kernel\n",
    "\n",
    "\n",
    "### What category of algorithms does Support Vector Machines classification belong to?\n",
    "\n",
    "Support Vector Machines (SVMs) are most frequently used for solving classification problems, which fall under the supervised machine learning category. However, with small adaptations, SVMs can also be used for other types of problems such as:\n",
    "\n",
    "- **Clustering (unsupervised learning)** through the use of Support Vector Clustering algorithm\n",
    "\n",
    "- **Regression (supervised learning)** through the use of Support Vector Regression algorithm (SVR)\n",
    "\n",
    "---\n",
    "\n",
    "### SVM classification algorithm — a brief explanation\n",
    "\n",
    "Let’s assume we have a set of points that belong to two separate classes. We want to separate those two classes in a way that allows us to correctly assign any future new points to one class or the other.\n",
    "\n",
    "SVM algorithm attempts to find a hyperplane that separates these two classes with the highest possible margin. If classes are fully linearly separable, a hard-margin can be used. Otherwise, it requires a soft-margin.\n",
    "\n",
    "Note, the points that end up on the margins are known as support vectors.\n",
    "\n",
    "![Imgur](https://imgur.com/gVpvenz.png)\n",
    "\n",
    "\n",
    "Hyperplane called “H1” cannot accurately separate the two classes; hence, it is not a viable solution to our problem.\n",
    "\n",
    "The “H2” hyperplane separates classes correctly. However, the margin between the hyperplane and the nearest blue and green points is tiny. Hence, there is a high chance of incorrectly classifying any future new points. E.g., the new grey point (x1=3, x2=3.6) would be assigned to the green class by the algorithm when it is obvious that it should belong to the blue class instead.\n",
    "\n",
    "Finally, the “H3” hyperplane separates the two classes correctly and with the highest possible margin (yellow shaded area). Solution found!\n",
    "Note, finding the largest possible margin allows more accurate classification of new points, making the model a lot more robust. You can see that the new grey point would be assigned correctly to the blue class when using the “H3” hyperplane.\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "### Radial Basis Function (RBF) kernel and Python examples\n",
    "\n",
    "RBF is the default kernel used within the sklearn’s SVM classification algorithm and can be described with the following formula:\n",
    "\n",
    "\n",
    "![Imgur](https://imgur.com/eLqLPD2.png)\n",
    "\n",
    "\n",
    "$||x - x'||²$ is the squared Euclidean distance between two feature vectors (2 points).\n",
    "\n",
    "Gamma is a scalar that defines how much influence a single training example (point) has. \n",
    "\n",
    "---\n",
    "\n",
    "### General Description of Decision Function in Machine Learning \n",
    "\n",
    "[Source](https://www.geeksforgeeks.org/ml-decision-function/)\n",
    "\n",
    "Decision function is a method present in classifier{ SVC, Logistic Regression } class of sklearn machine learning framework. This method basically returns a Numpy array, In which each element represents whether a predicted sample for x_test by the classifier lies to the right or left side of the Hyperplane and also how far from the HyperPlane.\n",
    "\n",
    "It also tells us that how confidently each value predicted for x_test by the classifier is Positive ( large-magnitude Positive value ) or Negative ( large-magnitude Negative value).\n",
    "\n",
    "Math behind the Decision Function method:\n",
    "Let’s consider the SVM for linearly-separable binary class classification problem:\n",
    "\n",
    "**Cost Function:**\n",
    "\n",
    "![Imgur](https://imgur.com/awpwVRv.png)\n",
    "\n",
    "### What Actually happens when we pass a data instance to Decision Function method ?\n",
    "\n",
    "This data sample is substituted in this hypothesis whose model parameters have been found by minimizing the cost function and returns the value outputted by this hypothesis which would be >1 if actual output is 1 or <-1 if the actual output is 0. This returned value indeed represents on which side of the hyperplane and also how far from it the given data sample lie.\n",
    "\n",
    "Overall, Decision Function output represents whether a predicted sample for x_test by the classifier lies to the right side or left side of hyperplane and also how far from it. It also tells us how confidently each value predicted for x_test by the classifier is Positive ( large-magnitude Positive value ) or Negative ( large-magnitude Negative value)\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1wES-wWN4ZxX"
   },
   "source": [
    "After we train a kernel SVM model, we will be getting support vectors and their corresponsing coefficients $\\alpha_{i}$\n",
    "\n",
    "Check below scikit-learn documentation for better understanding of these attributes: \n",
    "\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html\n",
    "\n",
    "<img src='https://i.imgur.com/K11msU4.png' width=500>\n",
    "\n",
    "Here we will be implementing the *`decision_function()`* of kernel SVM, and the *`decision_function()`* here means that, based on the value return by *`decision_function()`* the model will classify the data point either as positive or negative.\n",
    "\n",
    "For example, In logistic regression After traning the models with the optimal weights $w$ we get, we will find the value \n",
    "\n",
    "### $$\\frac{1}{1+\\exp(-(wx+b))}$$, \n",
    "\n",
    "If this value comes out to be < 0.5 we will mark it as negative class, else its positive class\n",
    "\n",
    "Another Example In Linear SVM After traning the models with the optimal weights $w$ we get, we will find the value of $sign(wx+b)$, if this value comes out to be -ve we will mark it as negative class, else its positive class.\n",
    "\n",
    "Similarly in Kernel SVM After traning the models with the coefficients $\\alpha_{i}$ we get, we will find the value of \n",
    "\n",
    "### $sign(\\sum_{i=1}^{n}(y_{i}\\alpha_{i}K(x_{i},x_{q})) + intercept)$, here $K(x_{i},x_{q})$\n",
    "\n",
    "is the RBF kernel. If this value comes out to be -ve we will mark $x_{q}$ as negative class, else its positive class.\n",
    "\n",
    "RBF kernel is defined as: \n",
    "\n",
    "# $K(x_{i},x_{q})$ = $exp(-\\gamma ||x_{i} - x_{q}||^2)$\n",
    "\n",
    "For better understanding [refer here](https://scikit-learn.org/stable/modules/svm.html#svm-mathematical-formulation)\n",
    "</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MuBxHiCQ4Zxc"
   },
   "source": [
    "#### Now here's our workflow\n",
    "\n",
    "> 1. Split the data into $X_{train}$(60), $X_{cv}$(20), $X_{test}$(20)\n",
    "\n",
    "> 2. Train $SVC(gamma=0.001, C=100.)$ on the ($X_{train}$, $y_{train}$)\n",
    "\n",
    "> 3. Get the decision boundry values $f_{cv}$ on the $X_{cv}$ data  i.e. ` `$f_{cv}$ ```= decision_function(```$X_{cv}$```)```  <font color='red'>you need to implement this decision_function()</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "fCgMNEvI4Zxf"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.datasets import make_classification\n",
    "import numpy as np\n",
    "from sklearn.svm import SVC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "ANUNIqCe4Zxn"
   },
   "outputs": [],
   "source": [
    "X, y = make_classification(n_samples=5000, n_features=5, n_redundant=2,\n",
    "                           n_classes=2, weights=[0.7], class_sep=0.7, random_state=15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tHie1zqH4Zxt"
   },
   "source": [
    "### Pseudo code\n",
    "\n",
    "clf = SVC(gamma=0.001, C=100.)<br>\n",
    "clf.fit(Xtrain, ytrain)\n",
    "\n",
    "<font color='green'>def</font> <font color='blue'>decision_function</font>(Xcv, ...): #use appropriate parameters <br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<font color='green'>for</font> a data point $x_q$ <font color='green'>in</font> Xcv: <br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<font color='grey'># code to implement $(\\sum_{i=1}^{\\text{all the support vectors}}(y_{i}\\alpha_{i}K(x_{i},x_{q})) + intercept)$, here the values $y_i$, $\\alpha_{i}$, and $intercept$ can be obtained from the trained model</font><br>\n",
    "   <font color='green'>return</font> <font color='grey'><i># the decision_function output for all the data points in the Xcv</i></font>\n",
    "    \n",
    "fcv = decision_function(Xcv, ...)  <i># based on your requirement you can pass any other parameters </i>\n",
    "\n",
    "<b>Note</b>: Make sure the values you get as fcv, should be equal to outputs of clf.decision_function(Xcv)\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "### A note on what is dual_coef in SVC\n",
    "\n",
    "In your `clf`, \n",
    "\n",
    "`coef_` are the weights assigned to the features; (Note it only works for linear SVM)\n",
    "\n",
    " `support_vectors_` and `support_` are the support vectors and the corresponding index;\n",
    "\n",
    "`dual_coef_` is the coefficients of the support vector in the decision function; and \n",
    "\n",
    "`intercept_` is the bias in decision function.\n",
    "\n",
    "In linear SVM, $w^Tx+b=0$ is the decision boundary, and $w$ is the coefficients of the support vectors, $b$ is the bias, all defined above.\n",
    "\n",
    "#### Some more on dual_coef if SVC from [here](https://stackoverflow.com/questions/22816646/the-dimension-of-dual-coef-in-sklearn-svc)\n",
    "\n",
    "\n",
    "The dual coefficients of a sklearn.svm.SVC in the multiclass setting are tricky to interpret. There is an explanation in the [scikit-learn documentation](http://scikit-learn.org/stable/modules/svm.html#multi-class-classification). The sklearn.svm.SVC uses [libsvm](http://www.csie.ntu.edu.tw/~cjlin/libsvm/) for the calculations and adopts the same data structure for the dual coefficients. Another explanation of the organization of these coefficients is in the [FAQ](http://www.csie.ntu.edu.tw/~cjlin/libsvm/faq.html#f402). In the case of the coefficients you find in the fitted SVC classifier, interpretation goes as follows:\n",
    "\n",
    "The support vectors identified by the SVC each belong to a certain class. In the dual coefficients, they are ordered according to the class they belong to.\n",
    "Given a fitted SVC estimator, e.g.\n",
    "\n",
    "    from sklearn.svm import SVC\n",
    "    svc = SVC()\n",
    "    svc.fit(X, y)\n",
    "\n",
    "you will find\n",
    "\n",
    "    svc.classes_   # represents the unique classes\n",
    "    svc.n_support_ # represents the number of support vectors per class\n",
    "\n",
    "\n",
    "#### From official guide - dual_coef\n",
    "\n",
    "dual_coef_ => ndarray of shape (n_classes -1, n_SV)\n",
    "\n",
    "Dual coefficients of the support vector in the decision function (see Mathematical formulation), multiplied by their targets. For multiclass, coefficient for all 1-vs-1 classifiers. \n",
    "\n",
    "### RBF Kernel Notes\n",
    "\n",
    "\n",
    "||x - x'||² is the squared Euclidean distance between two feature vectors (2 points).\n",
    "Gamma is a scalar that defines how much influence a single training example (point) has.\n",
    "\n",
    "\n",
    "### C Parameter\n",
    "\n",
    "It controls the trade off between smooth decision boundary and classifying training points correctly. A large value of c means you will get more training points correctly. Large value of c means you will get more intricate decision curves trying to fit in all the points.\n",
    "\n",
    "\n",
    "### Gamma Parameter\n",
    "\n",
    "It defines how far the influence of a single training example reaches. If it has a low value it means that every point has a far reach and conversely high value of gamma means that every point has close reach.\n",
    "\n",
    "If gamma has a very high value, then the decision boundary is just going to be dependent upon the points that are very close to the line which effectively results in ignoring some of the points that are very far from the decision boundary. This is because the closer points get more weight and it results in a wiggly curve as shown in previous graph.On the other hand, if the gamma value is low even the far away points get considerable weight and we get a more linear curve.\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "### Now the code for this\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dual_coeff.shape  (546,)\n",
      "support_vectors  (546, 5)\n",
      "intercept  [2.33183324]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.datasets import make_classification\n",
    "import numpy as np\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn.model_selection import train_test_split\n",
    "import math\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "X, y = make_classification(n_samples=5000, n_features=5, n_redundant=2,\n",
    "                           n_classes=2, weights=[0.7], class_sep=0.7, random_state=15)\n",
    "\n",
    "x, x_test, y, y_test = train_test_split(X, y, test_size=0.2, train_size=0.8)\n",
    "\n",
    "X_train, x_cv, y_train, y_cv = train_test_split(x, y, test_size = 0.25, train_size = 0.75)\n",
    "\n",
    "clf = SVC(gamma=0.001, C=100.)\n",
    "\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "dual_coeff = clf.dual_coef_[0]\n",
    "\n",
    "support_vectors = clf.support_vectors_\n",
    "\n",
    "print('dual_coeff.shape ', dual_coeff.shape)\n",
    "print('support_vectors ', support_vectors.shape)\n",
    "\n",
    "intercept = clf.intercept_\n",
    "\n",
    "print('intercept ', intercept)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decision_function(Xcv):\n",
    "  sample_output_rbf = 0\n",
    "  y_predicted = []\n",
    "  gamma = clf._gamma\n",
    "  \n",
    "  for x_q in Xcv:\n",
    "    kernel_sum = 0\n",
    "    for i in range(len(support_vectors)):\n",
    "      squared_distance = (np.linalg.norm(support_vectors[i] - x_q)**2)\n",
    "      rbf_k = np.exp(-gamma * (squared_distance))\n",
    "      kernel_sum += dual_coeff[i]*rbf_k    \n",
    "    \n",
    "    sample_output_rbf = kernel_sum + intercept\n",
    "    y_predicted.append(sample_output_rbf[0])\n",
    "    \n",
    "  return np.array(y_predicted)    \n",
    "      "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "inbuilt_decision_function: first 5 elements  [-1.85244347 -4.23656834 -3.36141632 -3.15123116 -4.12626119]\n",
      "custom_decision_function_result: first 5 elements  [-1.85244347 -4.23656834 -3.36141632 -3.15123116 -4.12626119]\n"
     ]
    }
   ],
   "source": [
    "svm_decision_inbuilt_result_x_cv = clf.decision_function(x_cv)\n",
    "# svm_decision_inbuilt_result_x_cv = clf.decision_function(x_test)\n",
    "# svm_decision_inbuilt_result_x_test = clf.decision_function(x_test)\n",
    "# print('x_test.shape x', svm_decision_inbuilt_result_x_test.shape)\n",
    "\n",
    "print('inbuilt_decision_function: first 5 elements ', svm_decision_inbuilt_result_x_cv[:5] )\n",
    "\n",
    "custom_decision_function_result_x_cv = decision_function(x_cv)\n",
    "print('custom_decision_function_result: first 5 elements ', custom_decision_function_result_x_cv[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementing Platt Scaling to find P(Y==1|X)\n",
    "\n",
    "Here we will be doing the following\n",
    "\n",
    "\n",
    "- Applying SGD algorithm with ($f_{cv}$, $y_{cv}$) and find the weight $W$ intercept $b$ ```Note: here our data is of one dimensional so we will have a one dimensional weight vector i.e W.shape (1,)``` \n",
    "\n",
    "- Note1: We have to change the values of $y_{cv}$ as mentioned in the above image. you will calculate y+, y- based on data points in train data\n",
    "\n",
    "- Note2: the Sklearn's SGD algorithm doesn't support the real valued outputs. We have to use below logrithmic transformation\n",
    "\n",
    "<img src='https://i.imgur.com/zKYE9Oc.png'>\n",
    "\n",
    "if Y[i] is 1, it will be replaced with y+ value else it will replaced with y- value\n",
    "\n",
    "- Finally, for a given data point from $X_{test}$, $P(Y=1|X) = \\frac{1}{1+exp(-(W*f_{test}+ b))}$ where ` `$f_{test}$ ```= decision_function(```$X_{test}$```)```, W and b will be learned as metioned in the above step\n",
    "\n",
    "Check this paper for reference on [Platt Scaling and Isotonic Regression.](https://drive.google.com/open?id=133odBinMOIVb_rh_GQxxsyMRyW-Zts7a)\n",
    "\n",
    "\n",
    "<img src='https://i.imgur.com/CAMnVnh.png'>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Randomly initializing weights (w) and intercept values (b)\n",
    "# dim — size of the w vector we want (or number of features or parameters in this case)\n",
    "def initialize_weights(dim):\n",
    "    ''' In this function, we will initialize our weights and bias\n",
    "    w — weights, a numpy array of size\n",
    "    b — bias, a scalar\n",
    "    '''\n",
    "    #initialize the weights to zeros array of (1,dim) dimensions\n",
    "    #you use zeros_like function to initialize zero, check this link https://docs.scipy.org/doc/numpy/reference/generated/numpy.zeros_like.html\n",
    "    #initialize bias to zero\n",
    "    w = np.zeros_like(dim)\n",
    "    # Above will initialize all w with 0.\n",
    "    b = 0\n",
    "    return w,b\n",
    "  \n",
    "def sigmoid(z):\n",
    "    return 1.0/(1 + np.exp(-z)) \n",
    "\n",
    "\n",
    "\n",
    "def logloss(y_true, y_pred):\n",
    "    \n",
    "    len_y_true = len(y_true)\n",
    "    \n",
    "    number_of_plus = np.count_nonzero(y_true == 1)\n",
    "    number_of_minus = np.count_nonzero(y_true == 0)\n",
    "    \n",
    "    y_plus = (number_of_plus+1)/(number_of_minus+2)\n",
    "    y_minus = 1/(number_of_minus+2)\n",
    "    \n",
    "    sum_of_loss = 0\n",
    "    \n",
    "    for i in range(0, len_y_true):\n",
    "      if (y_true[i] == 1):\n",
    "        sum_of_loss += ((y_plus * np.log10(y_pred[i])) + ((1- y_plus) * np.log10(1-y_pred[i])))\n",
    "      else:\n",
    "        sum_of_loss += ((y_minus * np.log10(y_pred[i])) + ((1 - y_minus) * np.log10(1-y_pred[i])))\n",
    "        \n",
    "    loss = (-1/len_y_true) * sum_of_loss  \n",
    "    return loss\n",
    "    \n",
    "    \n",
    "  \n",
    "def gradient_dw(x,y,w,b,alpha,N):\n",
    "    '''In this function, we will compute the gardient w.r.to w '''\n",
    "    z = np.dot(w, x) + b\n",
    "    dw = x*(y - sigmoid(z)) - ((alpha)*(1/N) * w)\n",
    "    return dw\n",
    "  \n",
    "  \n",
    "def gradient_db(x,y,w,b):\n",
    "    z = np.dot(w, x) + b\n",
    "    db = y - sigmoid(z)\n",
    "\n",
    "    return db"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(X_train, y_train, X_test, y_test, epochs, alpha, eta0, tol=1e-3):\n",
    "    \"\"\" In this function, we will implement logistic regression\"\"\"\n",
    "    # Here eta0 is learning rate\n",
    "    # implement the code as follows\n",
    "    # initialize the weights (call the initialize_weights(X_train[0]) function)\n",
    "    w, b = initialize_weights(X_train[0])\n",
    "    # for every epoch\n",
    "    train_loss = []\n",
    "    test_loss = []\n",
    "    N = len(X_train)\n",
    "\n",
    "    loss_threshold = 0.0001\n",
    "\n",
    "    for epoch in range(epochs):\n",
    "        # for every data point(X_train,y_train)\n",
    "        for row in range(N - 1):\n",
    "            # compute gradient w.r.to w (call the gradient_dw() function)\n",
    "            delta_weights = gradient_dw(\n",
    "                X_train[row], y_train[row], w, b, alpha, len(X_train)\n",
    "            )\n",
    "\n",
    "            # compute gradient w.r.to b (call the gradient_db() function)\n",
    "            delta_bias = gradient_db(X_train[row], y_train[row], w, b)\n",
    "\n",
    "            # update w, b\n",
    "            w = w + eta0 * delta_weights\n",
    "            b = b + eta0 * delta_bias\n",
    "\n",
    "        # predict the output of x_train[for all data points in X_train] using w,b\n",
    "        # y_prediction_train is a numpy array (vector) containing all predictions (0/1) for the examples in X\n",
    "        y_prediction_train = [  \n",
    "        sigmoid(np.dot(w, x_row) + b) for x_row in X_train\n",
    "        ]\n",
    "\n",
    "        # compute the loss between predicted and actual values (call the loss function)\n",
    "        # store all the train loss values in a list\n",
    "        train_loss.append(logloss(y_train, y_prediction_train))\n",
    "\n",
    "        # predict the output of x_test[for all data points in X_test] using w,b\n",
    "        y_prediction_test = [\n",
    "            sigmoid(np.dot(w, x_row) + b) for x_row in X_test\n",
    "        ]\n",
    "\n",
    "        print(\n",
    "            f\"For EPOCH No : {epoch} Train Loss is : {logloss(y_train, y_prediction_train)} and Test Loss is : {logloss(y_test, y_prediction_test)}\"\n",
    "        )\n",
    "\n",
    "        # compute the loss between predicted and actual values (call the loss function)\n",
    "        test_loss.append(logloss(y_test, y_prediction_test))\n",
    "\n",
    "      \n",
    "\n",
    "    return w, b, train_loss, test_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "For EPOCH No : 0 Train Loss is : 0.2710610406358637 and Test Loss is : [0.29884728 0.2992636  0.30152214 0.30237133 0.28943891]\n",
      "For EPOCH No : 1 Train Loss is : 0.24910931184003132 and Test Loss is : [0.29777406 0.29871392 0.30215838 0.30380725 0.28013576]\n",
      "For EPOCH No : 2 Train Loss is : 0.23288264544617943 and Test Loss is : [0.29747904 0.29899958 0.3028957  0.30529273 0.27259011]\n",
      "For EPOCH No : 3 Train Loss is : 0.22073217358551833 and Test Loss is : [0.29772793 0.29985471 0.30370263 0.30679862 0.26639499]\n",
      "For EPOCH No : 4 Train Loss is : 0.21150997309733618 and Test Loss is : [0.29835816 0.30109753 0.30455661 0.30830643 0.2612464 ]\n",
      "For EPOCH No : 5 Train Loss is : 0.20442223640473065 and Test Loss is : [0.2992571  0.30260404 0.30544158 0.30980458 0.25691872]\n",
      "For EPOCH No : 6 Train Loss is : 0.19891629786651815 and Test Loss is : [0.30034614 0.304289   0.30634611 0.31128592 0.25324363]\n",
      "For EPOCH No : 7 Train Loss is : 0.19460224664607043 and Test Loss is : [0.30156976 0.30609327 0.30726197 0.31274611 0.25009433]\n",
      "For EPOCH No : 8 Train Loss is : 0.19120058409697535 and Test Loss is : [0.30288824 0.30797528 0.30818327 0.31418264 0.24737409]\n",
      "For EPOCH No : 9 Train Loss is : 0.18850760978293196 and Test Loss is : [0.30427281 0.3099055  0.30910569 0.31559419 0.2450081 ]\n",
      "For EPOCH No : 10 Train Loss is : 0.1863724385822939 and Test Loss is : [0.30570238 0.31186276 0.31002611 0.31698018 0.24293778]\n",
      "For EPOCH No : 11 Train Loss is : 0.18468157567826063 and Test Loss is : [0.30716133 0.31383174 0.31094224 0.31834056 0.24111665]\n",
      "For EPOCH No : 12 Train Loss is : 0.1833484195317538 and Test Loss is : [0.30863797 0.31580131 0.31185241 0.31967558 0.23950743]\n",
      "For EPOCH No : 13 Train Loss is : 0.18230600991063034 and Test Loss is : [0.31012351 0.31776333 0.31275539 0.3209857  0.23807985]\n",
      "For EPOCH No : 14 Train Loss is : 0.18150194066926556 and Test Loss is : [0.31161129 0.31971188 0.31365032 0.32227152 0.23680914]\n",
      "For EPOCH No : 15 Train Loss is : 0.1808947374079546 and Test Loss is : [0.31309624 0.32164264 0.31453656 0.32353369 0.23567477]\n",
      "For EPOCH No : 16 Train Loss is : 0.18045124103835908 and Test Loss is : [0.31457451 0.32355249 0.31541369 0.32477293 0.23465966]\n",
      "For EPOCH No : 17 Train Loss is : 0.18014469214763792 and Test Loss is : [0.31604317 0.3254392  0.31628142 0.32598997 0.23374941]\n",
      "For EPOCH No : 18 Train Loss is : 0.17995331047320015 and Test Loss is : [0.31749996 0.32730124 0.31713958 0.32718552 0.23293185]\n",
      "For EPOCH No : 19 Train Loss is : 0.1798592288742398 and Test Loss is : [0.31894322 0.32913758 0.31798808 0.32836031 0.23219659]\n",
      "For EPOCH No : 20 Train Loss is : 0.17984768435632134 and Test Loss is : [0.32037168 0.33094756 0.3188269  0.32951502 0.23153472]\n",
      "For EPOCH No : 21 Train Loss is : 0.17990639773229003 and Test Loss is : [0.32178441 0.33273084 0.31965606 0.33065034 0.23093855]\n",
      "For EPOCH No : 22 Train Loss is : 0.1800250932794355 and Test Loss is : [0.32318075 0.33448731 0.32047563 0.33176691 0.23040142]\n",
      "For EPOCH No : 23 Train Loss is : 0.1801951234001767 and Test Loss is : [0.32456024 0.336217   0.32128571 0.33286536 0.2299175 ]\n",
      "For EPOCH No : 24 Train Loss is : 0.18040917282735233 and Test Loss is : [0.32592257 0.33792011 0.32208639 0.33394628 0.22948172]\n",
      "For EPOCH No : 25 Train Loss is : 0.18066102365371914 and Test Loss is : [0.32726759 0.3395969  0.32287782 0.33501026 0.22908958]\n",
      "For EPOCH No : 26 Train Loss is : 0.18094536728148147 and Test Loss is : [0.32859523 0.34124774 0.32366014 0.33605783 0.22873714]\n",
      "For EPOCH No : 27 Train Loss is : 0.18125765286647882 and Test Loss is : [0.32990551 0.34287303 0.3244335  0.33708951 0.22842088]\n",
      "For EPOCH No : 28 Train Loss is : 0.18159396436986913 and Test Loss is : [0.3311985  0.34447322 0.32519805 0.33810582 0.22813768]\n",
      "For EPOCH No : 29 Train Loss is : 0.1819509201997316 and Test Loss is : [0.33247434 0.34604878 0.32595394 0.33910721 0.22788477]\n",
      "For EPOCH No : 30 Train Loss is : 0.18232559081458308 and Test Loss is : [0.33373319 0.3476002  0.32670136 0.34009415 0.22765964]\n",
      "For EPOCH No : 31 Train Loss is : 0.18271543070254417 and Test Loss is : [0.33497527 0.34912798 0.32744044 0.34106706 0.22746006]\n",
      "For EPOCH No : 32 Train Loss is : 0.18311822193712535 and Test Loss is : [0.33620079 0.35063265 0.32817136 0.34202636 0.22728401]\n",
      "For EPOCH No : 33 Train Loss is : 0.1835320271102146 and Test Loss is : [0.33741001 0.3521147  0.32889427 0.34297244 0.22712966]\n",
      "For EPOCH No : 34 Train Loss is : 0.1839551499028417 and Test Loss is : [0.33860316 0.35357464 0.32960933 0.34390568 0.22699537]\n",
      "For EPOCH No : 35 Train Loss is : 0.18438610190965835 and Test Loss is : [0.33978053 0.35501297 0.33031669 0.34482643 0.22687964]\n",
      "For EPOCH No : 36 Train Loss is : 0.18482357460945392 and Test Loss is : [0.34094239 0.35643019 0.33101651 0.34573504 0.22678111]\n",
      "For EPOCH No : 37 Train Loss is : 0.1852664155902703 and Test Loss is : [0.34208901 0.3578268  0.33170893 0.34663183 0.22669853]\n",
      "For EPOCH No : 38 Train Loss is : 0.1857136083079954 and Test Loss is : [0.34322066 0.35920325 0.3323941  0.34751712 0.22663078]\n",
      "For EPOCH No : 39 Train Loss is : 0.1861642547920974 and Test Loss is : [0.34433764 0.36056002 0.33307216 0.34839122 0.22657682]\n",
      "For EPOCH No : 40 Train Loss is : 0.18661756081952574 and Test Loss is : [0.34544021 0.36189757 0.33374326 0.34925439 0.22653569]\n",
      "For EPOCH No : 41 Train Loss is : 0.18707282316370105 and Test Loss is : [0.34652865 0.36321634 0.33440752 0.35010693 0.22650653]\n",
      "For EPOCH No : 42 Train Loss is : 0.1875294185945975 and Test Loss is : [0.34760323 0.36451675 0.33506507 0.3509491  0.22648853]\n",
      "For EPOCH No : 43 Train Loss is : 0.18798679436177052 and Test Loss is : [0.34866423 0.36579924 0.33571605 0.35178115 0.22648095]\n",
      "For EPOCH No : 44 Train Loss is : 0.18844445993749695 and Test Loss is : [0.34971191 0.3670642  0.33636058 0.35260331 0.22648311]\n",
      "For EPOCH No : 45 Train Loss is : 0.1889019798341687 and Test Loss is : [0.35074652 0.36831204 0.33699878 0.35341584 0.22649437]\n",
      "For EPOCH No : 46 Train Loss is : 0.18935896734032645 and Test Loss is : [0.35176832 0.36954314 0.33763077 0.35421895 0.22651416]\n",
      "For EPOCH No : 47 Train Loss is : 0.18981507904461406 and Test Loss is : [0.35277757 0.37075788 0.33825666 0.35501285 0.22654193]\n",
      "For EPOCH No : 48 Train Loss is : 0.1902700100374586 and Test Loss is : [0.3537745  0.37195661 0.33887656 0.35579776 0.22657718]\n",
      "For EPOCH No : 49 Train Loss is : 0.19072348969728806 and Test Loss is : [0.35475936 0.37313969 0.33949059 0.35657388 0.22661944]\n",
      "w_coef  1.1693760531988473\n",
      "intercept b  -0.05990993706025272\n"
     ]
    }
   ],
   "source": [
    "alpha=0.0001\n",
    "eta0=0.0001\n",
    "# N=len(X_train)\n",
    "epochs=50\n",
    "\n",
    "w, b, cv_log_loss, test_loss = train(custom_decision_function_result_x_cv, y_cv, x_test, y_test, epochs, alpha, eta0)\n",
    "print('w_coef ', w)\n",
    "print('intercept b ', b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAreUlEQVR4nO3deXxV1bn/8c8DCcQCMghGIAgEEAQSQBC1wiVYqlitonUAwar1XrVeq15tf2rbn1Zq63SdetVfta21DhWRW2frWOLUqjggKJMIqEEcUBEiM3l+f6x9zCGehJDkZOec832/Xvt1zh7PszCe56y19l7L3B0REZGaWsUdgIiItExKECIikpIShIiIpKQEISIiKSlBiIhISkoQIiKSkhKESI4ws5PN7IW445DMoQQhGcvMVpjZhLjjaAgzKzOzKjOrrLEcEHdsIgl5cQcgksM+dPeiuIMQqY1qEJJ1zKytmV1vZh9Gy/Vm1jba19XMHjGzNWb2uZk9b2aton0XmNlKM1tnZovN7Dsprr2fmX1kZq2Tth1lZvOi96PN7FUzW2tmH5vZtQ0sQ7mZXW5mr0TXetDMuiTtP8LM3o7KUW5meyft62VmfzOzT83sMzO7sca1/9vMvjCz5WZ2aEPik9ygBCHZ6BfA/sBwYBgwGvhltO98oALoBhQCPwfczAYCZwH7unsH4BBgRc0Lu/vLwFfAQUmbTwD+Gr2/AbjB3XcF+gEzG1GOHwI/AroDW4HfAZjZXsA9wLlROR4DHjazNlHiegR4D+gD9ARmJF1zP2Ax0BW4CviTmVkjYpQspgQh2WgqMN3dP3H3T4FLgROjfVsIX7i93X2Luz/vYUCybUBbYLCZ5bv7Cnd/t5br3wNMATCzDsD3om2J6/c3s67uXunuL9URZ4+oBpC8tEvaf6e7v+XuXwH/FzguSgDHA4+6+1PuvgX4b2AX4NuEZNgD+Jm7f+XuG909uWP6PXf/g7tvA/4S/VsU1vmvKTlLCUKyUQ/CL+iE96JtAFcDS4EnzWyZmV0I4O5LCb/IfwV8YmYzzKwHqf0VODpqtjoaeN3dE593KrAXsMjM5pjZ4XXE+aG7d6qxfJW0/4MaZcgn/PLfrnzuXhUd2xPoRUgCW2v5zI+SzlsfvW1fR4ySw5QgJBt9CPROWt8z2oa7r3P38929GDgCOC/R1+Duf3X3MdG5DlyZ6uLuvoDwBX0o2zcv4e7vuPsUYPfo/Fk1agU7o1eNMmwBVtcsX9RE1AtYSUgUe5qZbkCRRlOCkEyXb2YFSUseobnnl2bWzcy6AhcDdwGY2eFm1j/6Uv2S0LRUZWYDzeygqFawEdgAVNXxuX8FzgH+DbgvsdHMpplZt+hX/Zpoc13Xqcs0MxtsZt8CpgOzoqahmcBhZvYdM8sn9KtsAv4JvAKsAq4ws3bRv8mBDfx8yXFKEJLpHiN8mSeWXwGXAa8C84D5wOvRNoABwNNAJfAv4GZ3n03of7iC8Av9I0IN4KI6PvceYBzwD3dfnbR9IvC2mVUSOqwnu/uGWq7RI8VzED9I2n8ncHsUTwFwNoC7LwamAf8Txft94PvuvjlKIN8H+gPvEzrkj6+jHCK1Mk0YJNLymFk5cJe7/zHuWCR3qQYhIiIpKUGIiEhKamISEZGUVIMQEZGUsuZe6a5du3qfPn0afP5XX31Fu3YNvV09c6ncuUXlzi31Kfdrr7222t27pdqXNQmiT58+vPrqqw0+v7y8nLKysqYLKEOo3LlF5c4t9Sm3mb1X2z41MYmISEpKECIikpIShIiIpJQ1fRAikrm2bNlCRUUFGzduTMv1O3bsyMKFC9Ny7ZYsudwFBQUUFRWRn59f7/OVIEQkdhUVFXTo0IE+ffqQjvmL1q1bR4cOHZr8ui1dotzuzmeffUZFRQV9+/at9/lqYhKR2G3cuJHddtstLclBwMzYbbfddrqGpgQhIi2CkkN6NeTfN+cTxJo1MH06LFqUe9VPEZG65HyCMINLLoG5czvFHYqIxOijjz5i8uTJ9OvXj5EjR/K9732PJUuWUFxczOLFi7c79txzz+XKK7efcHDFihUMHTq0SWIpKytr1IO/TSXnE0THjtClC6xaVRB3KCISE3fnqKOOoqysjHfffZfXXnuNyy+/nI8//pjJkyczY8aMr4+tqqpi1qxZTJ48OcaIm0fOJwiAvn1h1apd4g5DRGIye/Zs8vPzOeOMM77eNmzYMMaOHcuUKVO49957v97+3HPP0bt3b3r37p3qUkDodD/llFMoKSlhxIgRzJ49G4D169dz3HHHMXjwYI466ij222+/HdYU7rnnHkpKShg6dCgXXHABANu2bePkk09m6NChlJSUcN111wHwu9/9jsGDB1NaWtokCUy3uQLFxfCvf6kGIdISnHsuzJ3btNccPLgtN99c+/633nqLkSNHptxXUlJCq1atePPNNxk2bBgzZsxgypQpdX7eTTfdhJkxf/58Fi1axMEHH8ySJUu4+eab6dy5MwsWLOCtt95i+PDhdV7nww8/5IILLuC1116jc+fOHHzwwTzwwAP06tWLlStX8tZbbwGwZs0aAK644gqWL19O27Ztv97WGKpBEGoQH39cwLZtcUciIi3RlClTmDFjBlu3buWBBx7g2GOPrfP4F154gWnTpgEwaNAgevfuzZIlS3jhhRe+/mU/dOhQSktL67zOnDlzKCsro1u3buTl5TF16lSee+45iouLWbZsGT/5yU94/PHH2XXXXQEoLS1l6tSp3HXXXeTlNf73v2oQhASxZUsrPvwQevWKOxqR3Hb99U1/zXXrNgFtat0/ZMgQZs2aVev+yZMnc/DBBzNu3DhKS0spLCxs+iB3QufOnXnzzTd54okn+P3vf8/MmTO57bbbePTRR3nuued4+OGH+c1vfsM///nPRn2OahCEJiaA5cvjjUNE4nHQQQexadMmbr311q+3zZs3j+effx6Afv360bVrVy688MIdNi8BjB07lrvvvhuAJUuW8P777zNw4EAOPPBAZs6cCcCCBQuYP39+ndcZPXo0zz77LKtXr2bbtm3cc889jBs3jtWrV1NVVcUPfvADLrvsMl5//XWqqqr44IMPGD9+PFdeeSVffvkllZWVDf0nAZQggFCDACUIkVxlZtx///08/fTT9OvXjyFDhnDRRRexxx57fH3MlClTWLRoEUcfffQOr3fmmWdSVVVFSUkJxx9/PLfffjtt27blzDPP5NNPP2Xw4MH88pe/ZMiQIXTs2LHW63Tv3p0rrriC8ePHM2zYMEaOHMmRRx7JypUrKSsrY/jw4UybNo3LL7+cbdu2MW3atK87xs8++2w6derUuH8Yd8+KZeTIkd5Qmza5m1X5xRc3+BIZa/bs2XGHEAuVu2VZsGBBWq+/du3atF6/vrZu3eobNmxwd/elS5d6nz59fNOmTWn7vJrlTvXvDLzqtXyvqg8CaNMGunXbxPLlupNJRNJn/fr1jB8/ni1btuDu3HzzzbRpU3vfSNyUICLdu29k2TIlCBFJnw4dOrSIJ6TrS30Qke7dN6gPQiRGobVD0qUh/75KEJE99tjIhx9CmuYrEZE6FBQU8NlnnylJpIlH80EUFOxcK4mamCI9eoTMsGIFDBoUbywiuaaoqIiKigo+/fTTtFx/48aNO/3lmA2Sy52YUW5nKEFE9thjAxBudVWCEGle+fn5OzXT2c4qLy9nxIgRabt+S9XYcquJKZKoQSxbFnMgIiIthBJEpEuXzRQU6GE5EZEEJYiIGfTpoxqEiEiCEkSS4mLVIEREEpQgkvTtG2oQutNOREQJYjvFxbB2LXzxRdyRiIjEL60JwswmmtliM1tqZhem2H+emS0ws3lm9oyZ9U7at6eZPWlmC6Nj+qQzVtCoriIiydKWIMysNXATcCgwGJhiZoNrHPYGMMrdS4FZwFVJ++4Arnb3vYHRwCfpijUhkSDUUS0ikt4axGhgqbsvc/fNwAzgyOQD3H22u6+PVl8CigCiRJLn7k9Fx1UmHZc2qkGIiFRL55PUPYEPktYrgP3qOP5U4O/R+72ANWb2N6Av8DRwobtvN2u0mZ0GnAZQWFhIeXl5g4OtrKzkjTfK2XXXA3nxxU8oL3+nwdfKJJWVlY36d8tUKnduUbkbpkUMtWFm04BRwLhoUx4wFhgBvA/cC5wM/Cn5PHe/FbgVYNSoUV5WVtbgGMrLyykrK2OvvWDjxp6UlfVs8LUySaLcuUblzi0qd8Oks4lpJdArab0o2rYdM5sA/AI4wt03RZsrgLlR89RW4AFgnzTG+rW+fdXEJCIC6U0Qc4ABZtbXzNoAk4GHkg8wsxHALYTk8EmNczuZWbdo/SBgQRpj/VpxcRjRddu2HR4qIpLV0pYgol/+ZwFPAAuBme7+tplNN7MjosOuBtoD95nZXDN7KDp3G/BT4Bkzmw8Y8Id0xZqsb1/YsgU+/LA5Pk1EpOVKax+Euz8GPFZj28VJ7yfUce5TQGn6okst+VbXXr3qPlZEJJvpSeoaiovDq/ohRCTXKUHUsOeeYWRXJQgRyXVKEDW0aROalvQ0tYjkOiWIFHSrq4iIEkRKiWG/RURymRJECsXFsGoVbNgQdyQiIvFRgkghcavre+/FG4eISJyUIFJI3OqqZiYRyWVKEClo2G8RESWIlPbYAwoKVIMQkdymBJGCmW51FRFRgqiFbnUVkVynBFGL4uJQg3CPOxIRkXgoQdSib19Yuxa++CLuSERE4qEEUQvd6ioiuU4Joha61VVEcp0SRC2SJw4SEclFShC12HVX2G03JQgRyV1KEHUYNAgWLIg7ChGReChB1KGkBObP162uIpKblCDqUFICX34JFRVxRyIi0vyUIOpQWhpe582LNw4RkTgoQdRh6NDwOn9+vHGIiMRBCaIOnTpBr15KECKSm5QgdiDRUS0ikmuUIHagpAQWLYLNm+OORESkeSlB7EBpKWzZAosXxx2JiEjzUoLYgZKS8KpmJhHJNUoQOzBwIOTlKUGISO5RgtiBNm3CkBtKECKSa5Qg6qG0VAlCRHKPEkQ9lJTA++/DmjVxRyIi0nyUIOoh0VH91lvxxiEi0pyUIOpBdzKJSC5Ka4Iws4lmttjMlprZhSn2n2dmC8xsnpk9Y2a9a+zf1cwqzOzGdMa5I716QceOShAiklvSliDMrDVwE3AoMBiYYmaDaxz2BjDK3UuBWcBVNfb/GnguXTHWl5mG3BCR3JPOGsRoYKm7L3P3zcAM4MjkA9x9truvj1ZfAooS+8xsJFAIPJnGGOtNkweJSK5JZ4LoCXyQtF4RbavNqcDfAcysFXAN8NO0RbeTEpMHffDBjo8VEckGeXEHAGBm04BRwLho05nAY+5eYWZ1nXcacBpAYWEh5eXlDY6hsrKyzvO3bu0IjODuu+dxwAGfN/hzWpodlTtbqdy5ReVuIHdPywIcADyRtH4RcFGK4yYAC4Hdk7bdDbwPrABWA2uBK+r6vJEjR3pjzJ49u879a9a4g/vllzfqY1qcHZU7W6ncuUXlrh3wqtfyvZrOGsQcYICZ9QVWApOBE5IPMLMRwC3ARHf/JLHd3acmHXMyoSP7G3dBNaeOHWHPPdVRLSK5I219EO6+FTgLeIJQQ5jp7m+b2XQzOyI67GqgPXCfmc01s4fSFU9TKCnR/NQikjvS2gfh7o8Bj9XYdnHS+wn1uMbtwO1NHVtDlJTAE0+EyYPatIk7GhGR9NKT1DuhpAS2btXkQSKSG5QgdkJpaXhVP4SI5AIliJ0wcCDk5ytBiEhuUILYCfn5YfIgdVSLSC5QgthJGpNJRHKFEsROKi0Nw21o8iARyXZKEDtJkweJSK5QgthJmjxIRHLFTiUIM+tsZqXpCiYTFBWFYTfUUS0i2W6HCcLMyqOZ3boArwN/MLNr0x9ay5SYPEgJQkSyXX1qEB3dfS1wNHCHu+9HGIE1Z40aBW+8EYbcEBHJVvVJEHlm1h04DngkzfFkhDFjYMOGkCRERLJVfRLEdMKIrEvdfY6ZFQPvpDeslm3MmPD6/PPxxiEikk47TBDufp+7l7r7mdH6Mnf/QfpDa7kKC2HAAHjhhbgjERFJn/p0Ul8VdVLnm9kzZvZpNEVoThszJiSIqqq4IxERSY/6NDEdHHVSH06YArQ/8LN0BpUJxoyBzz7T0N8ikr3q1UkdvR4G3OfuX6Yxnowxdmx4VT+EiGSr+iSIR8xsETASeMbMugEb0xtWy9e/P+y+u/ohRCR71aeT+kLg28Aod98CfAUcme7AWjqzUItQghCRbFWfTup8YBpwr5nNAk4FPkt3YJlgzBhYvhxWrow7EhGRplefJqb/R2heujla9om25bzE8xCqRYhINsrb8SHs6+7Dktb/YWZvpiugTDJ8OLRrFzqqjz8+7mhERJpWfWoQ28ysX2IlepJ6W/pCyhx5eXDAAapBiEh2qk+C+BkwOxrV9VngH8D56Q0rc4wdG0Z2/VI3/4pIlqnPXUzPAAOAs4GfAAOBLmmOK2OMGQPu8M9/xh2JiEjTqteEQe6+yd3nRcsm4Lo0x5Ux9tsvNDWpmUlEsk1Dpxy1Jo0ig7VrB/vsoyeqRST7NDRBeJNGkeHGjIFXXoFNm+KORESk6dSaIMxsvpnNS7HMBwqbMcYWb8yYkBxefTXuSEREmk5dz0Ec3mxRZLjkB+YOPDDeWEREmkqtNQh3f6+upTmDbOm6dYOBA9VRLSLZpaF9EFLD2LHw4ouaQEhEsocSRBMZMwa++AIWLIg7EhGRpqEE0UQS/RC63VVEskV9hvtOdTfT82Z2nZnt1hxBZoLiYujeXf0QIpI96lOD+DvwKDA1Wh4GXgU+Am6v60Qzm2hmi81sqZldmGL/eWa2IEo6z5hZ72j7cDP7l5m9He1r8WOlmoVahBKEiGSL+gz3PcHd90lan29mr7v7PmY2rbaTzKw1cBPwXaACmGNmD7l7civ9G4SZ6tab2Y+Bq4DjgfXAD939HTPrAbxmZk+4+5qdK17zGjcO7rsPliyBvfaKOxoRkcapTw2itZmNTqyY2b5A62h1ax3njQaWuvsyd98MzKDGVKXuPtvd10erLwFF0fYl7v5O9P5D4BOgWz1ijdXh0ZMjDz4YbxwiIk2hPjWIfwduM7P2hDGY1gKnmlk74PI6zusJfJC0XgHsV8fxpxKas7YTJac2wLsp9p0GnAZQWFhIeXl5nQWpS2VlZaPOTxgwYCR33FHFvvu+0ehrNYemKnemUblzi8rdQO5erwXoCHTcieOPAf6YtH4icGMtx04j1CDa1tjeHVgM7L+jzxs5cqQ3xuzZsxt1fsKvfuVu5v7RR01yubRrqnJnGpU7t6jctQNe9Vq+V+tzF1NHM7sWeAZ4xsyuMbOO9cg9K4FeSetF0baa158A/AI4wsNQ4ontuxI6x3/h7i/V4/NahEmTwvwQjzwSdyQiIo1Tnz6I24B1wHHRshb4cz3OmwMMMLO+ZtYGmAw8lHyAmY0AbiEkh0+StrcB7gfucPdZ9SlIS1FaCr17wwMPxB2JiEjj1CdB9HP3Szx0Ni9z90uB4h2d5O5bgbOAJ4CFwEx3f9vMppvZEdFhVwPtgfvMbK6ZJRLIccC/ASdH2+ea2fCdLFsszEIt4qmnoLIy7mhERBquPp3UG8xsjLu/AGBmBwIb6nNxd38MeKzGtouT3k+o5by7gLvq8xkt0aRJcMMN8OSTcPTRcUcjItIw9alBnAHcZGYrzGwFcCNwelqjynBjxkCXLmpmEpHMtsMahLu/CQyLOo1x97Vmdi4wL82xZay8vPBMxMMPw9atYV1EJNPUe7A+d1/r7muj1fPSFE/WOPLIMLqrBu8TkUzV0NFcrUmjyEKHHAIFBWpmEpHM1dAE4U0aRRZq1w4mTAjDbrj+tUQkA9WaIMxsnZmtTbGsA3o0Y4wZa9IkeO89ePPNuCMREdl5tXafunuH5gwkG33/++G5iAcfhOHD445GRGTnaEa5NNp9d/j2t9UPISKZSQkizSZNgrlzQ1OTiEgmUYJIsyOjGTA0R4SIZBoliDQbMAAGD1Yzk4hkHiWIZjBpEjz3HHz+edyRiIjUnxJEMzj6aNi2DWbOjDsSEZH6U4JoBvvsE25z/f3v9dCciGQOJYhmYAannx4emHvllbijERGpHyWIZjJ1KrRvD7fcEnckIiL1owTRTDp0gBNOgBkzYM2auKMREdkxJYhmdMYZsGED3Hln3JGIiOyYEkQzGjEC9t1XndUikhmUIJrZGWfAggXw4otxRyIiUjcliGZ2/PGw666hFiEi0pIpQTSzdu3ghz+EWbNg9eq4oxERqZ0SRAxOPx02bYK//CXuSEREaqcEEYOhQ+HAA+HWW9VZLSItlxJETE4/HZYsgfLyuCMREUlNCSImxxwDXbqos1pEWi4liJjssgucdBLcfz98/HHc0YiIfJMSRIxOPx22bFEtQkRaJiWIGA0cGKYkvfZaTSYkIi2PEkTMfv1rWLcOrr467khERLanBBGzkhKYMgVuuAE++ijuaEREqilBtACXXgqbN8Nvfxt3JCIi1ZQgWoD+/eGUU8JkQu+9F3c0IiKBEkQLcfHF4XX69HjjEBFJSGuCMLOJZrbYzJaa2YUp9p9nZgvMbJ6ZPWNmvZP2nWRm70TLSemMsyXo1Qt+/OMwPtOSJXFHIyKSxgRhZq2Bm4BDgcHAFDMbXOOwN4BR7l4KzAKuis7tAlwC7AeMBi4xs87pirWluOgiaNsWLrkk7khERNJbgxgNLHX3Ze6+GZgBHJl8gLvPdvf10epLQFH0/hDgKXf/3N2/AJ4CJqYx1hahsBDOPTfMW/3mm3FHIyK5Li+N1+4JfJC0XkGoEdTmVODvdZzbs+YJZnYacBpAYWEh5Y0Y+a6ysrJR5zeV/ffPo127/TnzzDX85jdvpf3zWkq5m5vKnVtU7oZJZ4KoNzObBowCxu3Mee5+K3ArwKhRo7ysrKzBMZSXl9OY85vSRRfBL3/ZlYKCMvbfP72f1ZLK3ZxU7tyicjdMOpuYVgK9ktaLom3bMbMJwC+AI9x9086cm63OOQe6dYPzzoNt2+KORkRyVToTxBxggJn1NbM2wGTgoeQDzGwEcAshOXyStOsJ4GAz6xx1Th8cbcsJ7dvDNdfAv/4Fv/td3NGISK5KW4Jw963AWYQv9oXATHd/28ymm9kR0WFXA+2B+8xsrpk9FJ37OfBrQpKZA0yPtuWMadPg8MPh5z+Hd96JOxoRyUVp7YNw98eAx2psuzjp/YQ6zr0NuC190bVsZuHJ6iFDwlPWzz4LrVvHHZWI5BI9Sd2C9egB118PL74IN94YdzQikmuUIFq4H/4QDjss3Nm0dGnc0YhILlGCaOESTU1t2sCPfgRVVXFHJCK5QgkiA/TsGZqann8ebrop7mhEJFcoQWSIk06CQw+FCy+Ed9+NOxoRyQVKEBnCDG69FfLy4OSTYcuWuCMSkWynBJFBiorg5pvhhRfgrLPAPe6IRCSbtYixmKT+pk6Ft9+Gyy+HQYPgv/4r7ohEJFspQWSgyy6DxYvh/PNhwIDwxLWISFNTE1MGatUK7rgDRoyAKVNg3ry4IxKRbKQEkaHatYOHHoJdd4Xvfx8++ijuiEQk2yhBZLCePeHhh2H1apg0CTZsiDsiEckmShAZbp994K674OWXw6B+etJaRJqKOqmzwFFHwRVXhIfo8vLgz3+G/Py4oxKRdNq0KUwFsHBh6Jf8wQ+a/jOUILLEBReE2sPPfw7r1sG990JBQdxRiUhjrVkTksDChbBoUfXrsmXVLQbDhilByA5cdBF07Aj/+Z9hBNgHHwyz04lIy+YOFRXVCSA5GXz8cfVxbdvCXnuFOxhPOCE8C7X33mFbOihBZJkzzwx3Np18MkyYAI89Bl26xB2ViEAYIufdd6uTQCIRLFoElZXVx3XqFL74DzusOgkMGgR9+zbvxGFKEFlo2rRQczj+eCgrgyefhD32iDsqkdzx1Vfb1wYSieCdd2Dr1urjiorCl/+PflSdCPbeG3bfPYy/FjcliCw1aVKoPRx5JIwdC48+mr5qqEiu+vTT7ZuDEsv771cf07o19O8fvvgnTapOAgMHQocOsYVeL0oQWew734Gnnw5DcYwcGSYeOuGEuKMSySxVVeELPzkBJJLCZ59VH/etb4VawNix29cG+vcPE35lIiWILLf//jB3bhiSY+pUmD0bbrgh/DGLSLXNm6tvG01OAosXw/r11cfttlv44j/66OoksPfe0KtXuN00myhB5ICiopAYLr44jAL70kswc2b4oxbJNWvXVncMJyeDd9+Fbduqj+vdO/w/Mm5cdRIYNAi6dYsv9uamBJEj8vLgt78Nf+wnngijRoW5JXr3jjsykabnDqtWVSeBp58ewGWXhfcfflh9XH5+GBG5pASOPXb7/oF27eKLv6VQgsgxhxwSmpymTg23wo4dO4Q771SikMy0ZUt4YCz5AbLE69q11cd961uFDB0K3/3u9reNFhdr1IG6KEHkoB49Quf1VVfBpZd2Ye+9wzAdP/sZ7LJL3NGJfNOXX4a+gETTUCIRLF26/W2jPXqEL/8TT6xOAnvvDYsXv8D48WUxRZ+5lCByVOvW4cnrfv1e4X//9wAuuSSM4XTddeHW2JZwD7bklsTdQomO4eTXVauqj8vLC81Ce+8dxiFLJIGBA8NDoqksWdI8Zcg2ShA5bvfdN3HvvXDGGXD22eF/uIMPhmuvhSFD4o5OstHateGLv+ayZAls3Fh9XOJp4okTQxJILH37qlmouShBCADjx8Mbb4SO64svhqFDQ03iggvggAPijk4yzdatsHx5+NJPfPknEkFybaBVq/CFP2hQ6B8YODC8Hzgw3C2kmmy8lCDka3l5oRZxwgnwP/8DN94YBvwbOzYkiu99T//DSrXEnUJLloTnB5YsqU4E7767fd9A587hS/+QQ8JrYunXLwxAJy2TEoR8Q9eucOmlodP6T3+Ca64JT2MPHQrnnw/HHKNRYnOFexhO4p13Ui9ffVV9bNu24anhIUNCU+XAgWF4l4EDw8NlknmUIKRW7dvDOeeEEWJnzAh3PZ1yCpx1VvgCOPHEMJxHc44uKU3PPcxpvnRp6iX5dtHWrUOTUP/+4ZmaAQNCEthrr+x8kjjXKUHIDuXnh2QwbRq8+CLceWd4Evuuu6B799AkNW1amLRETVAt0+bN8N578MornVmwIDQBJZZly7YfSiI5CRxwQEgCiaVPH3UQ5xIlCKk3MxgzJiw33ACPPBKSxQ03hGaooqLQxjxxYpiLolOnuCPOHdu2hSeEV6wIy/Ll4Yt/+fKwVFSEmgIMA8LzLsXFoQ9gwoTwOmBASAp77qkkIIEShDRIQUHoizjmGFi9Gh54AB5/HGbNCv0WrVuHgQIPOSQklFGjWv7Qxi3Zxo3wwQfhOYHE8t57YVmxIqwndwoD9OwZagLjx4fX4mJYs+YNjjlmBN27q7YnO5bWBGFmE4EbgNbAH939ihr7/w24HigFJrv7rKR9VwGHAa2Ap4Bz3MNvIGlZunaFf//3sGzdCi+/HJLF44/DJZeEX65mMHgwjB5dvQwZojtY3MMc4qtWhRpARcX2y8qVITF88sn255mFSaD69oX99guTQ/XpE5bevcOSak7y8vIv6dGjOUom2SBtCcLMWgM3Ad8FKoA5ZvaQuy9IOux94GTgpzXO/TZwICFxALwAjAPK0xWvNI28PDjwwLD8+tfw+efwyithefllePjh8MQ2hA7N4uLt730fNCg0dxQWZnbn94YN4Uv944+/+frRRyEZrFoVluQ7gRK6dAlNdkVFsM8+4Qt/zz2rl549M3eOAckc6axBjAaWuvsyADObARwJfJ0g3H1FtK+qxrkOFABtAAPygY+RjNOlS+iTmDgxrLuHJpFXXoG3364eTuGZZ7Z/irZ169AB3rNnWIqKwnrnztVLly7htVOnML9F27ZN02yyeXP40l6/vvq1sjKMB7RmTXhNXj777JvLhg2pr92hQ0h+PXqESZy6dw/vE6+9eoXyar4OaQnSmSB6Ah8krVcA+9XnRHf/l5nNBlYREsSN7r6w6UOU5mYWmkX69t1+e/I4PMuXh6aVRBPLwoXw1FOhKWZH2rYNHbAFBWFp3Tp8ZvLSqhVUVo4mLy8kgy1bwrJ5M2za9M22/Nrk54fktNtuYendO/zaT6zvvntYCgvDa7du+uKXzNIiO6nNrD+wN1AUbXrKzMa6+/M1jjsNOA2gsLCQ8vLyBn9mZWVlo87PVC2t3AUF1WPy17RxYyvWrcujsjKftWvzvn5fWZnHpk2t2Lw5LMnvq6qMRM+Ve3jvDl27bqGgYC15eR4tVeTlOfn5Veyyyzbatq2ioGAbBQWJ1220b7+Vdu220r79Ntq120qbNlX1qrF89VX13URxa2n/vZuLyt0w6UwQK4FeSetF0bb6OAp4yd0rAczs78ABwHYJwt1vBW4FGDVqlJeVlTU42PLychpzfqZSuXOLyp1bGlvudD73OAcYYGZ9zawNMBl4qJ7nvg+MM7M8M8sndFCriUlEpBmlLUG4+1bgLOAJwpf7THd/28ymm9kRAGa2r5lVAMcCt5jZ29Hps4B3gfnAm8Cb7v5wumIVEZFvSmsfhLs/BjxWY9vFSe/nUN3PkHzMNuD0dMYmIiJ109BaIiKSkhKEiIikpAQhIiIpKUGIiEhKShAiIpKSZcsAqWb2KfBeIy7RFVjdROFkEpU7t6jcuaU+5e7t7t1S7ciaBNFYZvaqu4+KO47mpnLnFpU7tzS23GpiEhGRlJQgREQkJSWIarfGHUBMVO7conLnlkaVW30QIiKSkmoQIiKSkhKEiIiklPMJwswmmtliM1tqZhfGHU86mdltZvaJmb2VtK2LmT1lZu9Er53jjLGpmVkvM5ttZgvM7G0zOyfanu3lLjCzV8zszajcl0bb+5rZy9Hf+73RXC1Zx8xam9kbZvZItJ4r5V5hZvPNbK6ZvRpta/Dfek4nCDNrDdwEHAoMBqaY2eB4o0qr24GJNbZdCDzj7gOAZ6L1bLIVON/dBwP7A/8Z/TfO9nJvAg5y92HAcGCime0PXAlc5+79gS+AU+MLMa3OYftJxnKl3ADj3X140vMPDf5bz+kEAYwGlrr7MnffDMwAjow5prRx9+eAz2tsPhL4S/T+L8Ck5owp3dx9lbu/Hr1fR/jS6En2l9sTU/YC+dHiwEGECbkgC8sNYGZFwGHAH6N1IwfKXYcG/63neoLoCXyQtF4Rbcslhe6+Knr/EVAYZzDpZGZ9gBHAy+RAuaNmlrnAJ8BThFka10SzPUL2/r1fD/wfoCpa343cKDeEHwFPmtlrZnZatK3Bf+tpnVFOMou7u5ll5X3PZtYe+F/gXHdfG35UBtla7mhmxuFm1gm4HxgUb0TpZ2aHA5+4+2tmVhZzOHEY4+4rzWx34CkzW5S8c2f/1nO9BrES6JW0XhRtyyUfm1l3gOj1k5jjaXJmlk9IDne7+9+izVlf7gR3XwPMBg4AOplZ4odhNv69HwgcYWYrCE3GBwE3kP3lBsDdV0avnxB+FIymEX/ruZ4g5gADojsc2gCTgYdijqm5PQScFL0/CXgwxliaXNT+/Cdgobtfm7Qr28vdLao5YGa7AN8l9L/MBo6JDsu6crv7Re5e5O59CP8//8Pdp5Ll5QYws3Zm1iHxHjgYeItG/K3n/JPUZvY9Qptla+A2d/9NvBGlj5ndA5QRhgD+GLgEeACYCexJGC79OHev2ZGdscxsDPA8MJ/qNumfE/ohsrncpYQOydaEH4Iz3X26mRUTfll3Ad4Aprn7pvgiTZ+oiemn7n54LpQ7KuP90Woe8Fd3/42Z7UYD/9ZzPkGIiEhqud7EJCIitVCCEBGRlJQgREQkJSUIERFJSQlCRERSUoKQjGZmbmbXJK3/1Mx+1UTXvt3MjtnxkY3+nGPNbKGZza6xvY+ZbYhG5kwsP2zCzy1LjHYqkoqG2pBMtwk42swud/fVcQeTYGZ5SWP/7MipwH+4+wsp9r3r7sObLjKR+lMNQjLdVsK8u/9Vc0fNGoCZVUavZWb2rJk9aGbLzOwKM5sazZ8w38z6JV1mgpm9amZLonF+EoPgXW1mc8xsnpmdnnTd583sIWBBinimRNd/y8yujLZdDIwB/mRmV9e30GZWaWbXWZjr4Rkz6xZtH25mL0Vx3Z8Y+9/M+pvZ0xbmh3g9qYztzWyWmS0ys7steZAqyXlKEJINbgKmmlnHnThnGHAGsDdwIrCXu48mDBH9k6Tj+hDGszkM+L2ZFRB+8X/p7vsC+wL/YWZ9o+P3Ac5x972SP8zMehDmJDiIMD/DvmY2yd2nA68CU939Zyni7FejiWlstL0d8Kq7DwGeJTwVD3AHcIG7lxKeHk9svxu4KZof4ttAYnTPEcC5hPlQigljGYkAamKSLBCNznoHcDawoZ6nzUkMgWxm7wJPRtvnA+OTjpvp7lXAO2a2jDAi6sFAaVLtpCMwANgMvOLuy1N83r5Aubt/Gn3m3cC/EYY6qUttTUxVwL3R+7uAv0UJspO7Pxtt/wtwXzQ+T093vx/A3TdGMRDFWxGtzyUkxFRNXZKDlCAkW1wPvA78OWnbVqJaspm1ApKnmUweh6cqab2K7f+/qDkWjQMG/MTdn0jeEY3981VDgm8CDR0zJ/nfYRv6TpAkamKSrBANPjaT7aeSXAGMjN4fQZhVbWcda2atojb7YmAx8ATw42gYccxsr2j0zLq8Aowzs64WprqdQmgaaqhWVI9OegLwgrt/CXyR1Ax1IvBsNJNehZlNiuJta2bfasRnS47QrwXJJtcAZyWt/wF40MzeBB6nYb/u3yd8ue8KnOHuG83sj4SmmNejTt1P2cE0ju6+yswuJAw7bcCj7l6fYZf7RU0/Cbe5++8IZRltZr8kjO9/fLT/JEJfybeAZcAp0fYTgVvMbDqwBTi2Hp8tOU6juYpkIDOrdPf2ccch2U1NTCIikpJqECIikpJqECIikpIShIiIpKQEISIiKSlBiIhISkoQIiKS0v8HuzuwxkAdXsEAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.array([i for i in range(0, 50)])\n",
    "\n",
    "cv_log_loss_arr = np.array(cv_log_loss)\n",
    "\n",
    "plt.plot(x, cv_log_loss_arr, \"-b\", label = 'CV log loss')\n",
    "\n",
    "plt.legend(loc=\"upper right\")\n",
    "plt.grid()\n",
    "\n",
    "plt.xlabel('Number of Epoch')\n",
    "plt.ylabel('Log Loss ')\n",
    "\n",
    "plt.title('Loss vs Epoch ')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementing the custom decision funtion on x_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of custom_decision_function_result_x_test  (1000,)\n",
      "First 5 elements prob_arr_x_test_custom_decision_func  [0.06530148068064014, 0.03790157324095724, 0.28405713154541296, 0.06839044552468132, 0.051415767200302775]\n"
     ]
    }
   ],
   "source": [
    "custom_decision_function_result_x_test = decision_function(x_test)\n",
    "print('Shape of custom_decision_function_result_x_test ', custom_decision_function_result_x_test.shape)\n",
    "\n",
    "prob_arr_x_test_custom_decision_func = []\n",
    "\n",
    "for i in custom_decision_function_result_x_test:\n",
    "  i_proba = 1/(1 + np.exp((-w * i) - b) )\n",
    "  prob_arr_x_test_custom_decision_func.append(i_proba)\n",
    "  \n",
    "  \n",
    "print('First 5 elements prob_arr_x_test_custom_decision_func ', prob_arr_x_test_custom_decision_func[:5])\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "8E&F_LR_SVM.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": ""
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}