Santender_v3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "### Competition Objective:\n",
    "\n",
    "[**Link to Kaggle Competition**](https://www.kaggle.com/c/santander-value-prediction-challenge)\n",
    "\n",
    "Santander Group is asking Data scientists to help them identify the value of transactions for each potential customer. This is a first step that Santander needs to nail in order to personalize their services at scale.\n",
    "\n",
    "Let's get the Business Context a little bit here. One of the paramount problems in Banking is predicting the value of transactions for each potential customer.\n",
    "\n",
    "Because if as a Banker I can predict the number or value the amount of Transactions that a Customer will bring to the Bank, then this value can become the basis on which other personalized services can be provided.\n",
    "\n",
    "Now coming back to the specific challenge here, This Value of transactions is a continuous variable and its prediction is a regression problem.\n",
    "\n",
    "The Training dataset contains a target variable (which is the value of transaction per customer).\n",
    "\n",
    "And a specialty of this problem here is that we are dealing with a “sparse tabular dataset” and which is completely anonymized. This makes it very difficult to exploit domain knowledge.\n",
    "\n",
    "This means, here, the feature variables come with no description whatsoever and with anonymized names like “48df886f9” that provide little meaning as to what they represent.\n",
    "\n",
    "---\n",
    "\n",
    "One can guess that they are possibly aggregated transactions (by day, by week, by month…) or an individual transaction that could be further grouped by some financial instrument (debit, credit, loan, …).\n",
    "\n",
    "So as a Data Scientist no one would like this kind of anonymized data. We would like to know the feature name and their clear definitions etc.\n",
    "\n",
    "But even the Competition host the Santander Group cant help much here. As for most companies, the data they possess is a valuable asset that cannot be compromised.\n",
    "\n",
    "This is probably the best that Santander can do or the best that their legal/compliance team would approve of. It would either be obfuscated data or no data at all that can be made publicly available. So we have to work with this constraint here.\n",
    "\n",
    "Hence the point is, this competition doesn’t seem to offer a lot of room for either sophisticated models or smart feature engineering employing domain knowledge because all the features are anonymized. If no data-leak or other similar kind of “property” is found in the dataset, the winning solution has all chances to be just a very deep stack of lgbm/xgboost/nn models.\n",
    "\n",
    "### Problem Statement\n",
    "\n",
    "This project aims to solve a high-dimensional sparse regression problem. The variable that we attempt to predict is the value of transactions for each potential customer from 4991 feature variables.\n",
    "\n",
    "### RMSE vs RMSLE\n",
    "\n",
    "1.  In the case of RMSE, the presence of outliers can explode the error term to a very high value. But, in the case of RMLSE the outliers are drastically scaled-down, therefore, nullifying their effect.\n",
    "\n",
    "Evaluation metric RMSLE is usually used when you don’t want to penalize huge differences in the predicted and the actual values and when both predicted and the actual values are large numbers.\n",
    "\n",
    "RMSE explodes in magnitude as soon as it encounters an outlier. In contrast, even on the introduction of the outlier, the RMSLE error is not affected much\n",
    "\n",
    "2.  RMSLE metric only considers the relative error between the Predicted and the actual value and the scale of the error is not significant. On the other hand, RMSE value Increases in magnitude if the scale of error increases.\n",
    "\n",
    "So RMSLE is especially useful for business cases where the under-estimation of the target variable is not acceptable but overestimation can be tolerated.\n",
    "\n",
    "---\n",
    "\n",
    "#### What Boosting Algorithm Fundamentally is in Machine Learning\n",
    "\n",
    "To understand the absolute basics of the need for Boosting algorithm, lets ask a basic question — If a data point is incorrectly predicted by our first model, and then the next (probably all models), will combining the predictions provide better results? Such questions are handled by boosting algorithm.\n",
    "\n",
    "So, Boosting is a sequential technique which works on the principle of an ensemble, where each subsequent model attempts to correct the errors of the previous model. The succeeding models are dependent on the previous model.\n",
    "\n",
    "The basic principle behind the working of the boosting algorithm is to generate multiple weak learners and combine their predictions to form one strong rule. These weak rules are generated by applying base Machine Learning algorithms on different distributions of the data set. These algorithms generate weak rules for each iteration. After multiple iterations, the weak learners are combined to form a strong learner that will predict a more accurate outcome. Note that a weak learner is one which is slightly better than random guessing. For example, a decision tree whose predictions are slightly better than 50%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\n",
    "from sklearn.decomposition import TruncatedSVD\n",
    "from sklearn import preprocessing, model_selection, metrics\n",
    "from sklearn.model_selection import train_test_split\n",
    "import lightgbm as lgb\n",
    "import xgboost as xgb\n",
    "from catboost import CatBoostRegressor\n",
    "\n",
    "from IPython.display import display # Allows the use of display() for DataFrames\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# While Reading data from Local Machine\n",
    "train_df = pd.read_csv('../../input/santander-value-prediction-challenge/train.csv')\n",
    "test_df = pd.read_csv('../../input/santander-value-prediction-challenge/test.csv')\n",
    "\n",
    "# But in Kaggle Kernel, and before final submission\n",
    "# comment-out the above line and un-comment below line to read the full train.csv\n",
    "# test_df = pd.read_csv('../input/santander-value-prediction-challenge/test.csv')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of train_df  (4459, 4993)\n",
      "Shape of Test_df  (49342, 4992)\n"
     ]
    }
   ],
   "source": [
    "\n",
    "print('Shape of train_df ', train_df.shape)\n",
    "print('Shape of Test_df ', test_df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>target</th>\n",
       "      <th>48df886f9</th>\n",
       "      <th>0deb4b6a8</th>\n",
       "      <th>34b15f335</th>\n",
       "      <th>a8cb14b00</th>\n",
       "      <th>2f0771a37</th>\n",
       "      <th>30347e683</th>\n",
       "      <th>d08d1fbe3</th>\n",
       "      <th>6ee66e115</th>\n",
       "      <th>...</th>\n",
       "      <th>3ecc09859</th>\n",
       "      <th>9281abeea</th>\n",
       "      <th>8675bec0b</th>\n",
       "      <th>3a13ed79a</th>\n",
       "      <th>f677d4d13</th>\n",
       "      <th>71b203550</th>\n",
       "      <th>137efaa80</th>\n",
       "      <th>fb36b89d9</th>\n",
       "      <th>7e293fbaf</th>\n",
       "      <th>9fc776466</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>000d6aaf2</td>\n",
       "      <td>38000000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>000fbd867</td>\n",
       "      <td>600000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0027d6b71</td>\n",
       "      <td>10000000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0028cbf45</td>\n",
       "      <td>2000000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>002a68644</td>\n",
       "      <td>14400000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 4993 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          ID      target  48df886f9  0deb4b6a8  34b15f335  a8cb14b00  \\\n",
       "0  000d6aaf2  38000000.0        0.0          0        0.0          0   \n",
       "1  000fbd867    600000.0        0.0          0        0.0          0   \n",
       "2  0027d6b71  10000000.0        0.0          0        0.0          0   \n",
       "3  0028cbf45   2000000.0        0.0          0        0.0          0   \n",
       "4  002a68644  14400000.0        0.0          0        0.0          0   \n",
       "\n",
       "   2f0771a37  30347e683  d08d1fbe3  6ee66e115  ...  3ecc09859  9281abeea  \\\n",
       "0          0          0          0          0  ...        0.0        0.0   \n",
       "1          0          0          0          0  ...        0.0        0.0   \n",
       "2          0          0          0          0  ...        0.0        0.0   \n",
       "3          0          0          0          0  ...        0.0        0.0   \n",
       "4          0          0          0          0  ...        0.0        0.0   \n",
       "\n",
       "   8675bec0b  3a13ed79a  f677d4d13  71b203550  137efaa80  fb36b89d9  \\\n",
       "0        0.0          0          0          0          0          0   \n",
       "1        0.0          0          0          0          0          0   \n",
       "2        0.0          0          0          0          0          0   \n",
       "3        0.0          0          0          0          0          0   \n",
       "4        0.0          0          0          0          0          0   \n",
       "\n",
       "   7e293fbaf  9fc776466  \n",
       "0          0          0  \n",
       "1          0          0  \n",
       "2          0          0  \n",
       "3          0          0  \n",
       "4          0          0  \n",
       "\n",
       "[5 rows x 4993 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>48df886f9</th>\n",
       "      <th>0deb4b6a8</th>\n",
       "      <th>34b15f335</th>\n",
       "      <th>a8cb14b00</th>\n",
       "      <th>2f0771a37</th>\n",
       "      <th>30347e683</th>\n",
       "      <th>d08d1fbe3</th>\n",
       "      <th>6ee66e115</th>\n",
       "      <th>20aa07010</th>\n",
       "      <th>...</th>\n",
       "      <th>3ecc09859</th>\n",
       "      <th>9281abeea</th>\n",
       "      <th>8675bec0b</th>\n",
       "      <th>3a13ed79a</th>\n",
       "      <th>f677d4d13</th>\n",
       "      <th>71b203550</th>\n",
       "      <th>137efaa80</th>\n",
       "      <th>fb36b89d9</th>\n",
       "      <th>7e293fbaf</th>\n",
       "      <th>9fc776466</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>000137c73</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>00021489f</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0004d7953</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>00056a333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>00056d8eb</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 4992 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          ID  48df886f9  0deb4b6a8  34b15f335  a8cb14b00  2f0771a37  \\\n",
       "0  000137c73        0.0        0.0        0.0        0.0        0.0   \n",
       "1  00021489f        0.0        0.0        0.0        0.0        0.0   \n",
       "2  0004d7953        0.0        0.0        0.0        0.0        0.0   \n",
       "3  00056a333        0.0        0.0        0.0        0.0        0.0   \n",
       "4  00056d8eb        0.0        0.0        0.0        0.0        0.0   \n",
       "\n",
       "   30347e683  d08d1fbe3  6ee66e115  20aa07010  ...  3ecc09859  9281abeea  \\\n",
       "0        0.0        0.0        0.0        0.0  ...        0.0        0.0   \n",
       "1        0.0        0.0        0.0        0.0  ...        0.0        0.0   \n",
       "2        0.0        0.0        0.0        0.0  ...        0.0        0.0   \n",
       "3        0.0        0.0        0.0        0.0  ...        0.0        0.0   \n",
       "4        0.0        0.0        0.0        0.0  ...        0.0        0.0   \n",
       "\n",
       "   8675bec0b  3a13ed79a  f677d4d13  71b203550  137efaa80  fb36b89d9  \\\n",
       "0        0.0        0.0        0.0        0.0        0.0        0.0   \n",
       "1        0.0        0.0        0.0        0.0        0.0        0.0   \n",
       "2        0.0        0.0        0.0        0.0        0.0        0.0   \n",
       "3        0.0        0.0        0.0        0.0        0.0        0.0   \n",
       "4        0.0        0.0        0.0        0.0        0.0        0.0   \n",
       "\n",
       "   7e293fbaf  9fc776466  \n",
       "0        0.0        0.0  \n",
       "1        0.0        0.0  \n",
       "2        0.0        0.0  \n",
       "3        0.0        0.0  \n",
       "4        0.0        0.0  \n",
       "\n",
       "[5 rows x 4992 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we have 4459 rows in train set and 49342 rows in test set. We also have 4993 columns in total including the target and id column.\n",
    "\n",
    "* Test set is almost 10 times as that of train set.\n",
    "* In the Train Dataset, the Number of columns is more than the number of train rows. That’s right: Our data set is wider than it is tall. So need to be careful with feature selection / engineering.\n",
    "* We have an anonymised ID feature together with a numerical target variable. The typical values of this target seem to be relatively high.\n",
    "* The remaining 4991 features are completely anonymous and appear to be primarily in integer or double format. Also here we find a large numerical range of values; in one instance from 7,000 to 17,020,000. Many columns have large numbers of zeros."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 4459 entries, 0 to 4458\n",
      "Columns: 4993 entries, ID to 9fc776466\n",
      "dtypes: float64(1845), int64(3147), object(1)\n",
      "memory usage: 169.9+ MB\n"
     ]
    }
   ],
   "source": [
    "train_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 49342 entries, 0 to 49341\n",
      "Columns: 4992 entries, ID to 9fc776466\n",
      "dtypes: float64(4991), object(1)\n",
      "memory usage: 1.8+ GB\n"
     ]
    }
   ],
   "source": [
    "test_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initial Observations looking at the above data\n",
    "\n",
    "- Column name does not mean anything now, as they are all anonymized\n",
    "- The dataframe is full of zero values.\n",
    "- The dataset is a sparse tabular one refer [this](https://www.kaggle.com/c/santander-value-prediction-challenge/discussion/59128)\n",
    "\n",
    "Target Variable:\n",
    "\n",
    "First doing some scatter plot of the target variable to check for visible outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfQAAAGHCAYAAABGYKDlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAApRUlEQVR4nO3de5QcdZ338ffXJASIICABMRCCkCOryIJmuTw8x40XFMGFHC8HFFxxXeJ6WS/srg8Ii+Ci6OLiZfFRAVlBEFDUbASUJy6w3gANJICASLgIRDBZQrgESEj4Pn9UDTbD9EzPTHd1d837dU6fVFf9uvvbNTCfqV/9qn6RmUiSpP72vG4XIEmSxs9AlySpBgx0SZJqwECXJKkGDHRJkmrAQJckqQYMdGkcIuKqiDi9A+87KyIyIuaUz+eWz7du92eV79+R7zEWETE/Iu6JiKcj4sRu1yP1CwNdGiQivlmGZ0bEUxGxIiKujIgPRsSUQc3fAhzb4vueGBG/abGMe4HtgKWtV95SDUdGxGNDbGr5e3RSRGwJfAU4FZgBfH7Q9rkNP5tmjyO7UPpz/giTqja52wVIPeonwLuAScB04LXAScC7IuJ1mbkGIDNXtfuDI2KjzFwHPNDu926mE99jjHak+L10SWbeP8T2X1L8oTPgM8CuFH+QDHh4NB8YEVMy86nRFir1Go/QpaGtzcwHMnN5Zi7NzNOAucArgY8PNBrcVR0Rb4mIGyPiiYhYFRH/HRHblkeNnwRePvhIslz+YER8PyLWAJ8Z5mhvn4hYGhFPRsR1EfGqhs9+ztF3Y1d9RMwF/gOY1lDDiU2+x5YRcU5EPFR+l59ExMsHf1ZEvC4ifhMRa8pejJ2G26kRMTMifhARj5aP70fE9gPvCSwpm95Z1jer8fWZua78uTyQmQ8AjwPrGp7vAfy/su5VEXF5RPxZw+cP7Nd3RMQVEfEE8L6ImBwRXyhf91C5/NWIuKrhtRERH4+IO8p9clNEHNFQ3l3lv78uP+MqpAoZ6FKLMvM3wI+Btw61PSJeBFwInAP8GfBq4Fvl5ouAfwNuozjC3K5cN+CTwGXAKyi6nJv5PPB/gDnAncAlEbFpi1/hl8BHKUJwoIbPN2n7TWBv4BBgr/I1P46ITRraTKXopv8bYF9gC+BrzT48Ip4H/CewLfCa8vFiYEFEBMX+OKBsvldZ370tfrcB04Avlq+fS3G0/sOI2GhQu1OA/wu8DFgA/CNwJPC3wD4UvxvfOeg1JwPvBT5Yvu4U4OsRcVBDzZTfYTue3WsgdZxd7tLo3AK8vsm2FwNTgIsz8/flumfOmZdHz+vLI8nBLsrMsxrazmryGf+SmZeXbd4D3EcRPGc1af+MzFwXEQ8Xi0PWMPDZs4GDgb/MzJ+W694F3AMc3vBZk4EPZuZtZZvPA2dHROTQk0S8Dtgd2Dkz7y5f805gGfC6zPxJRDxYtl05XI3DfMfvDfou7wEeoQjbnzds+vfMvLih3UeAzw28PiI+yp/+uCAipgFHA2/IzJ+Vq++KiL0oAv5SYGW5/sGx1C6NV98foUfE2VEMWhpxsFHZjba0fPwuIlZXUKLqJYBmMxrdQHHu/TcR8b2IeH9ETG/xfRe32O7qgYXMfAy4ieJosZ3+DHh60Gc9PMRnrR0I89IfgI2ALYd53z8MhHn5vneWr2vLd4iInSPi22W3+CPAHyl+z80c1HRxw2teALwI+FVDXdn4vKxvY4peiscGHsD7gZ3bUbs0XnU4Qv8mcDpw7kgNM/NjA8sR8ffAnp0rSzX1Moqu7ufIzA0R8QaKLts3UHTPnhIRf5mZN4zwvmvaUNvTFH9wNBo8Kn+8Gv+YWd9k21gOFNo17eMlFL0W7wOWU9R4C8UfGo1Gu78HvtNfUfRUNHJAnXpC3x+hl12CzxqhW/6V/uNy0NDPImLXIV76DuCCSopULUTEbhTdsBc3a5OFqzPzJOAvKI4+Dy03r6MYNT8e+zTUMw3YDbi1XLUS2DQiNm9ov8eg17dSw60Uvxv2bfiszSnO798ypqr/9L4vbjydEBEvoThVMZ73HXivF1KMeP9MZv4kM28FNmOEA5ey9+EBip/XwHtF4/OyvrXAjpm5bNBj4PTKuvLf8f6MpTGpwxH6UM4A/i4zb4+IvSkGv7x2YGNE7AjsBFzRpfrU+6aWg9yeR3HZ2uuATwDX0WQgWUTsQ3F+/XKKrt49gR34U1jdDewYEa+kOMp7NDPXjrKu4yNiJcUfCidQhMi3y23XUhx5nhIRXwD+HPjAoNffDWwcEftTjCh/PDMfb2xQ/n/znxQDvuYDq4FPU5yL/jZj9xPgRuD88pw1wL8D19Oe/xcfAv4HOCoi7qW4jv1UntuTMJQvAR+PiN9R/LzeRzGw7X6AzHy0HCPw+TLsfwo8n+IPrKcz8wxgBfAE8MaIuBt4svxjQapE3x+hDxYRzwf+F/DdiFgKfJ1nX7cKcBjFwKUNFZen/vF6il/m9wD/RTFI7ETg1QPXoA/hYWA/im7f2ylGtf9LZp5Xbv8exUj2/6I4mn7HGOo6pnzf64HZwJsHXRN/OLA/xfnu+cA/N744M39JMRL9grKGjzO091CcQ15Y/rspcEBmPjGGmgc+OylGza8EriwfDwDzmgyiG+37P03RG7I7xWDEr1B8/1b+aPo8xRUJ/wFcU677AfBkQ5t/pvhv4B+Bm4FFFFc83FV+/nrgwxQj5f9AMaJfqky04f+jriu78C7JzN3KrsHbMnNwiDe2X0IxOveXVdUoqb+Uvyd+npl/3+1apFbU7gg9Mx+huJzk7fDMzSD+fGB7eT59SxpG8Eqa2CJixyjuIf/SiNgtIr5EcaR/Trdrk1rV94EeERdQhPNLI+K+iHgvRbfjeyPiBoqusUMaXnIYcGE7uvgk1cbTwF9TnF64muLc+Jsys9XLCaWuq0WXuyRJE13fH6FLkiQDXZKkWujr69C33nrrnDVrVrfLkCSpMtddd93/ZOZzbivd14E+a9YsFi92zIokaeKIiN8Ptd4ud0mSasBAlySpBgx0SZJqwECXJKkGDHRJkmrAQJckqQYMdEmSasBAlySpBgx0SZJqoNI7xUXEJGAxsDwz3zxo21TgXOBVwIPAoZl5d5X1SZI0VjsdcynN5i8N4K7PHtTRz6/6CP0jwK1Ntr0XeCgzdwG+AHyusqokSRqH4cIcIMs2nVRZoEfE9sBBwFlNmhwCnFMuXwy8LiKiitokSRqP4cJ8NG3Go8oj9C8CHweebrJ9BnAvQGauBx4GXji4UUTMj4jFEbF45cqVHSpVkqT+UkmgR8SbgRWZed143yszz8jMOZk5Z/r058weJ0nShFTVEfp+wMERcTdwIfDaiDhvUJvlwA4AETEZeAHF4DhJknpaK+eHO30OuZJAz8xjM3P7zJwFHAZckZlHDGq2EHh3ufy2sk2nTzlIkjRuXzh0j2G3VzHKvdLL1gaLiE8BizNzIfAN4FsRsQxYRRH8kiT1vJN+ePOQ67fcdApLTnhDJTVUHuiZeRVwVbl8QsP6J4G3V12PJEnj9dDjT41qfSd4pzhJksZhwZLl3S4BMNAlSRqXUy+/rem2LTaZUlkdBrokSeOwfPUTTbedePDLK6vDQJckaRyGuxxt3p4zKqvDQJckaRx65fpqA12SpBow0CVJqgEDXZKkGjDQJUmqAQNdkqQaMNAlSaoBA12SpBow0CVJqgEDXZKkDpgUw91Drv0MdEmSxujwM69uuu0de+9QYSUGuiRJY/aLO1Y13XbyvFdUWImBLklSLRjokiTVgIEuSVINGOiSJI3RfjtvNar1nWSgS5I0RucftS+zt5n2rHWzt5nG+UftW3ktBrokSWO0y7GXcvuKNc9at2zFGhYsWV55LQa6JEljsOtxl7E+n7s+gX/87g2V12OgS5I0Bk9uGCLNS+ufbr6tUwx0SZJqwECXJKkGDHRJksZg8jBzrwwe+V4FA12SpDGYOmVS022Ljp5bXSGlSgI9IjaOiF9FxA0RcXNEnDREmyMjYmVELC0ff1tFbZIkjcWadRu6XcKzTK7oc9YCr83MxyJiCvDziPhRZl4zqN1FmfmhimqSJGlMunGd+UgqCfTMTOCx8umU8lH9mH5Jktrg6O8s7XYJz1HZOfSImBQRS4EVwKLMvHaIZm+NiBsj4uKIqHZmeEmSWjTcZeabTOnO8LTKPjUzN2TmHsD2wF4RsdugJj8EZmXm7sAi4Jyh3ici5kfE4ohYvHLlyo7WLEnSaJ3ylt278rmV/xmRmauBK4EDBq1/MDPXlk/PAl7V5PVnZOaczJwzffr0jtYqSdJozdtzRlc+t6pR7tMjYotyeRNgf+C3g9ps1/D0YODWKmqTJKkOqhrlvh1wTkRMovgj4juZeUlEfApYnJkLgQ9HxMHAemAVcGRFtUmSNCpbbjqFhx5/asj13VLVKPcbgT2HWH9Cw/KxwLFV1CNJ0nisfWroa9Czi9dveac4SZJGYcGS5Tz+1NNDblv9xHOP2qtioEuSNAqnXn5bt0sYkoEuSdIoLF/9RLdLGJKBLknSKEyKYaZZ6yIDXZKkUdgwzMi3/XbeqsJKns1AlyRpFKZtNPS0qQGcf9S+1RbTwECXJGkUHu+xaVMHGOiSJI1Csw73bk8haqBLklQDBrokSTVgoEuSVAMGuiRJLTp+wU3dLqEpA12SpBZdcO293S6hKQNdkqQWDXdTmW4z0CVJalGv3vYVDHRJklo2ZZjU7OZtX8FAlySpZU9uaN7l3s3bvoKBLklSLRjokiTVgIEuSVKLNp869ExrzdZXyUCXJKlFkyYNHZvN1lep+xVIktQnHnr8qVGtr5KBLklSCxYsWd7tEoZloEuS1IJTL7+t2yUMy0CXJKkFy1c/0e0ShmWgS5LUguFu+toLt4Q10CVJasFw07K8Y+8dKqujGQNdkqRxOnneK7pdQjWBHhEbR8SvIuKGiLg5Ik4aos3UiLgoIpZFxLURMauK2iRJqoPJFX3OWuC1mflYREwBfh4RP8rMaxravBd4KDN3iYjDgM8Bh1ZUnyRJz3H8gps475p7ul1GSyo5Qs/CY+XTKeVj8OmIQ4BzyuWLgddF9MAoA0nShNRPYQ4VnkOPiEkRsRRYASzKzGsHNZkB3AuQmeuBh4EXVlWfJEmNvn1t/4Q5VBjombkhM/cAtgf2iojdxvI+ETE/IhZHxOKVK1e2tUZJkgY8Pdyw9h5U+Sj3zFwNXAkcMGjTcmAHgIiYDLwAeHCI15+RmXMyc8706dM7XK0kSf2hqlHu0yNii3J5E2B/4LeDmi0E3l0uvw24IjP77O8jSVJdTGkxIWdvM62zhbSoqlHu2wHnRMQkij8ivpOZl0TEp4DFmbkQ+AbwrYhYBqwCDquoNkmSnmPa1CmsfmL4WdRmbzONRUfPraagEVQS6Jl5I7DnEOtPaFh+Enh7FfVIkjSSh5uEeQB3ffagaotpgXeKkyRpCFtsOmVU67vNQJckaQjNRnH16uguA12SpCE0O38+0nn1bjHQJUmqAQNdkqQaMNAlSaoBA12SpBow0CVJGmT3T/642yWMmoEuSdIgj6zd0O0SRs1AlyRpFHrl3u2DGeiSJI1Cr9y7fTADXZKkBguWLO92CWNioEuS1ODUy2/rdgljYqBLktRg+eonul3CmBjokiTVgIEuSVKL9tt5q26X0JSBLklSi84/at9ul9CUgS5JUg0Y6JIk1YCBLklS6fgFN3W7hDEz0CVJKp13zT3dLmHMDHRJkmrAQJckiZFv+brJlN6OzN6uTpKkiox0y9dT3rJ7RZWMjYEuSRIj3/J13p4zKqpkbAx0SZKASRHdLmFcDHRJkoANmU23zd5mWoWVjI2BLkkSMGOLTYZc/zxg0dFzK61lLCoJ9IjYISKujIhbIuLmiPjIEG3mRsTDEbG0fJxQRW2SJAG8ZtfpQ65/5z4zK65kbKo6Ql8P/ENmvgzYB/hgRLxsiHY/y8w9ysenKqpNkqSmN5X5wfXDX87WKyoJ9My8PzOvL5cfBW4Fenu4oCRpwtj1uMuabluzbkOFlYxd5efQI2IWsCdw7RCb942IGyLiRxHx8morkyRNVE9uaD4grl9MrvLDIuL5wPeAj2bmI4M2Xw/smJmPRcSBwAJg9hDvMR+YDzBzZn+c15AkqdMqO0KPiCkUYX5+Zn5/8PbMfCQzHyuXLwOmRMTWQ7Q7IzPnZOac6dOHHsAgSdJEU9Uo9wC+Adyamac1afOish0RsVdZ24NV1CdJUjObT53U7RJaUlWX+37Au4CbImJpue4TwEyAzPwa8Dbg/RGxHngCOCxzmKv8JUmqwI0nHdDtElpSSaBn5s+BYe+pl5mnA6dXUY8kSQMOP/PqbpfQFt4pTpI0YR1+5tX84o5V3S6jLQx0SdKENVKYN7sdbC8y0CVJauKf3vjSbpfQMgNdkqQmen0O9EYGuiRJNWCgS5I0hLs/e1C3SxgVA12SNCEtWNIfs6i1ykCXJE1Ip15+W7dLaCsDXZI0If1h9RPdLqGtDHRJ0oS00eTmETh7m2kVVtIeBrokaUJau/7pptsWHT23ukLaxECXJKkGDHRJkmrAQJckqQZaCvSIeNFo1kuSpGq1eoT+uybrb2lXIZIkVWWXYy/tdglt12qgx3NWRGwONB8iKElSj1qf3a6g/SYPtzEi7gUS2CQi7hm0+YXABZ0qTJKkTqjbLV8HDBvowBEUR+eXAe9qWJ/AHzOzXvfNkyTV3kk/vHnY7fvtvFVFlbTXsIGemf8NEBFbZ+bj1ZQkSVLnPPT4U8NuP/+ofSuqpL1aPYe+ISI+HRF3RsTDABHxhoj4UAdrkyRJLWo10L8I7AYcTtHdDnAz8P4O1CRJUlf0a3c7jHwOfcA8YJfMXBMRTwNk5vKImNGxyiRJarP9T7tq2O392t0OrR+hr2NQ+EfEdODBtlckSVKH3L5iTbdL6JhWA/27wDkRsRNARGwHnA5c2KnCJElS61oN9E8AdwE3AVsAtwN/AE7qTFmSJFVrkyn9Pb1JS+fQM3Md8DHgY2VX+/9kZg3vsyNJmqhOecvu3S5hXFoK9Ih4yaBVm0UEwFrg/sz0FrCSpJ52+JlXD7t93p79Pc671f6FZRTd7MsaHrcD9wBrI+J7EbFtsxdHxA4RcWVE3BIRN0fER4ZoExHx5YhYFhE3RsQrR/91JEka2i/uWNXtEjqq1cvWjgLmAicC9wIzgeOBq4H/Bj4HfAV4W5PXrwf+ITOvj4jNgOsiYlFmNs7W9iZgdvnYG/hq+a8kSc8y65j6zZY2Xq0G+kkU16E/WT5fFhEfAH6XmV+PiCMpjtiHlJn3A/eXy49GxK3ADJ49/eohwLnluflrImKLiNiufK0kSYBh3kyrXe7PA2YNWjcTmFQur6H18/GzgD2BawdtmkFx9D/gvnKdJEkddcQ+M7tdwri1eoT+ReCKiPgPitDdHnhPuR7gQIru92FFxPOB7wEfzcxHRlts+R7zgfkAM2f2/w9AktS6Tk19evK8V3TkfavU0hF6Zv4r8DfAiyi6xl8MvDczP1duX5CZbxruPSJiCkWYn5+Z3x+iyXJgh4bn25frBtdyRmbOycw506dPb6V8SVJNHPeDm7pdQs8a8Qg9IiYBvwNelpk/HsuHRHGN2zeAWzPztCbNFgIfiogLKQbDPez5c0lSozXrNrT9Pft5QpZGIwZ6Zm6IiA3AxhTXnY/FfsC7gJsiYmm57hMU5+HJzK8Bl1F03S8DHqfo0pckqWP223mrvp6QpdFozqF/JyI+QzFY7Zm7xGXmnSO9ODN/DsQIbRL4YIv1SJL0LHd/9qBul9BVrQb66eW/+w9an/xppLskSeqSVu/l3t93rJckqeYMakmSaqDVm8FMBj4A/CWwNQ3nwzPz1Z0pTZIktarVI/QvAO8Dfgq8iuJ68m2AKzpUlyRJz3L8gubXoPf7XObt0OoeeAvwpsz8ErC+/Hce8JpOFSZJUqPzrrmn6bZ+n8u8HVoN9E35033Wn4iITTPztxT3ZJckqav6fS7zdhg20CPiHeXircBflMuLgRMj4niGuDWrJEnttsuxzrA2kpGO0L9e/vsRijnNAY4GXgn8FeUkKZIkdcqux13G+hy53UQ30ij3AMjMXw+syMzbgdd3sihJkgY8ucE0b8VIgT4pIl7DMLdtzUxHukuS1GUjBfpUilnSmgV6Ai9pa0WSJI3CEfvM7HYJPWGkQF+TmQa2JKkrFiwZeez1yfNeUUElvc8r8SVJPeu4HzS/mQw4w1qjkQJ92ClPJUnqpDXrNnS7hL4xbKBn5mZVFSJJUqPhbvWq57LLXZLUk84f5lav4GC4wQx0SVJPGunqcwfDPZuBLklSDRjokqSeM9LlahtPcsz2YAa6JKnnjHS52m8/fWBFlfQPA12S1HO8XG30DHRJUk85/Myru11CXzLQJUk94/Azr+YXd6wato2Xqw3NQJck9YyRwhy8XK0ZA12SpBow0CVJqgEDXZLUN7bdbKNul9CzKgn0iDg7IlZExG+abJ8bEQ9HxNLycUIVdUmSesdIk7Fsu9lGXHvc/hVV038mV/Q53wROB84dps3PMvPN1ZQjSeolrYxuN8yHV8kRemb+FBh56KIkacJpJcw1sl46h75vRNwQET+KiJd3uxhJUjVaCfP9dt6qgkr6W1Vd7iO5HtgxMx+LiAOBBcDsoRpGxHxgPsDMmd5cQJImgvOP2rfbJfS8njhCz8xHMvOxcvkyYEpEbN2k7RmZOScz50yfPr3SOiVJ6lU9EegR8aKIiHJ5L4q6HuxuVZKkTtv/tKtGbDPZmVJbUkmXe0RcAMwFto6I+4BPAlMAMvNrwNuA90fEeuAJ4LDMzCpqkyR1z+0r1gy7fXLAslMOqqia/lZJoGfmO0bYfjrFZW2SpAmg1ZHthnnreqLLXZI0cbQa5l88dI/OF1MjBrokqVKtXnM+b88ZHa6kXgx0SZJqwECXJFWmlVHt4CQsY2GgS5IqM9Ko9gHet330DHRJUiUWLFneUru7P+vI9rEw0CVJlfjYRUuH3b751EmG+TgY6JKkSox0t7AbTzqgkjrqykCXJHXc3p9e1O0Sas9AlyR13B8fXTfs9tnbTKuokvoy0CVJXbfo6LndLqHvGeiSpI7a9bjLht0+xSRqC3ejJKmjntww/HC4U9++RzWF1JyBLknqmOMX3DRiG+/Z3h6VTJ8qSZp4djn2UtaPcK3afjtvVU0xE4BH6JKktmslzAHOP2rfzhczQRjokqS2ayXM1V4GuiSpK47YZ2a3S6gVA12S1FY7HXNpS+1OnveKDlcysRjokqS22f2TPx7xnu3gYLhOMNAlSW1x/IKbeGTthhHb7bfzVg6G6wADXZLUFuddc8+IbY7YZ6Zh3iEGuiRp3HY51vPm3eaNZSRJ4zKrxUFwm0+d1OFKJjaP0CVJY9bKrV0H3HjSAR2sRAa6JGnMWjlvDnD3Zw/qcCUy0CVJYzKao3N1noEuSRqTVo/OvSNcNQx0SdKoLViyvKV2R+wz05HtFalklHtEnA28GViRmbsNsT2ALwEHAo8DR2bm9VXUJkkTzU7HXNrS3dzGy/Pm1arqCP2bwHDDG98EzC4f84GvVlCTJE04syoK89nbTKvgU9SokkDPzJ8Cq4ZpcghwbhauAbaIiO2qqE2SJooqB7EtOnpuZZ+lQq+cQ58B3Nvw/L5y3XNExPyIWBwRi1euXFlJcZJUB+e3OIhtvJx4pTt6JdBblplnZOaczJwzffr0bpcjSX2jiq52wHu1d0mvBPpyYIeG59uX6yRJfWRydLuCiatXAn0h8NdR2Ad4ODPv73ZRklQX+592Vcc/Y3LAslMc2d4tVV22dgEwF9g6Iu4DPglMAcjMrwGXUVyytozisrX3VFGXJE0ErUye4iVm/a+SQM/Md4ywPYEPVlGLJE0krc6Epv7XK13ukqQ2O/zMq1tq5zXj9WCgS1JN/eKO4W7/8SdeM14PBrok1VAVg+DUWwx0Saqh21esaamdM6HVh4EuSTXT6kA4Z0Krl0pGuUuSqrHrcZe11M7L1OrHQJekPjSeKVDtZq8nA12S+sx4ri0PsJu9pjyHLkl9ZLxToN5lV3ttGeiS1EfOG8cUqNtutlEbK1GvMdAlqU+M99rya4/bvz2FqCcZ6JLUBw4/8+qWry0fiqPa689Al6Q+0OptXAcLDPOJwlHuktSDFixZzkcvWjqq1xjcE5tH6JLUY8YS5l5bLgNdknrMaMMcvLZcBrok9ZSxjGS3q11goEtSTxntSPb9dt6qQ5Wo3xjoktSnNp4UnH/Uvt0uQz3CQJekHjGae7TP3mYav/30gR2sRv3Gy9YkqQe0EubbbraRd3tTUwa6JHXY/qddNa67vA0wzDUcA12SOmg8U502mr3NtLa8j+rLc+iS1CHtCnOARUfPbdt7qZ4MdElqswVLlrc1zL00Ta2wy12S2qhd58sbeWmaWmGgS1Kb7HTMpWSb39O7wKlVBrokjdPun/wxj6zd0Nb33G/nrTwy16hUFugRcQDwJWAScFZmfnbQ9iOBU4Hl5arTM/OsquqTpKG081x4I4+81W6VBHpETAK+AuwP3Af8OiIWZuYtg5pelJkfqqImSRqJYa5+UtUo972AZZl5Z2auAy4EDqnosyVpVNo9Sn3A5DDM1TlVdbnPAO5teH4fsPcQ7d4aEa8Gfgd8LDPvHdwgIuYD8wFmzpzZgVIlTWQelatf9dKguB8CF2Tm2oh4H3AO8NrBjTLzDOAMgDlz5rR7QKmkCWqXYy9lfYd+oxjmqkJVXe7LgR0anm/Pnwa/AZCZD2bm2vLpWcCrKqpN0gQ365jOhPm2m21kmKsyVR2h/xqYHRE7UQT5YcA7GxtExHaZeX/59GDg1opqkzRBHX7m1fzijlVtf19nRVM3VBLombk+Ij4EXE5x2drZmXlzRHwKWJyZC4EPR8TBwHpgFXBkFbVJmpjGeq589jbTvK+6elJk9u9p6Dlz5uTixYu7XYakLliwZDkfvWhppZ9pmKsXRMR1mTln8PpeGhQnSSM6fsFNnHfNPZV/rufC1esMdEl9Ye9PL+KPj66r/HO/eOgezNtzRuWfK42WgS6pp+163GU8uaH6U4PeS139xkCX1HO6dTQ+wO519SMDXVJP6MSMZaMVwF2GufqUgS6pq/Y/7SpuX7Gm22V4rlx9z0CX1HFVXWJmV7kmMgNdUkdUFeJ2k0sFA11S23VyopMBm0+dxI0nHdDZD5H6iIEuqW12OuZSOn2BmXdrk4ZmoEsakyqvD/doXBqZgS6pZVWOSHfGMml0DHRJw+rUFKPNbDwp+O2nD6zs86S6MNClGhnrlKDd4CVmUnsZ6FIf68YUouNhiEudY6BLfaTq7u92cWS61HkGutSD+u3Iu5kj9pnJyfNe0e0ypAnBQJe6rIqbsFTBo3Cpuwx0qSJ1Ce5GjkiXeoeBLrXJ8Qtu4rxr7ul2GR03OWDZKQ5uk3qNgS6NQb8MTnNKUGniMNA1YdWxCxy8NEyaqAx01UoVk4P0Eru/JQ0w0NV3Jsq56sE88pY0HANdlarL9dWdtN/OW3H+Uft2uwxJfcZAV0vqer65mwxuSe1koE8g/TIyuy4CuMtuckkVMdAr5FFufTk4TVK3VRboEXEA8CVgEnBWZn520PapwLnAq4AHgUMz8+4qavO8rsAucEn9rZJAj4hJwFeA/YH7gF9HxMLMvKWh2XuBhzJzl4g4DPgccGinazPM683JQSRNFFUdoe8FLMvMOwEi4kLgEKAx0A8BTiyXLwZOj4jIzI52Up96+W2dfHtVwHPVklRdoM8A7m14fh+wd7M2mbk+Ih4GXgj8T2OjiJgPzAeYOXPmuAv7w+onxv0eGh+vr5ak8eu7QXGZeQZwBsCcOXPGffT+4i02YbmhPmqeb5ak3lJVoC8Hdmh4vn25bqg290XEZOAFFIPjOuqf3vjSCXsO3ZHZklQfVQX6r4HZEbETRXAfBrxzUJuFwLuBq4G3AVd0+vw58MxMVN0IdY9yJUntUkmgl+fEPwRcTnHZ2tmZeXNEfApYnJkLgW8A34qIZcAqitCvxLw9ZzjFpCSpr1V2Dj0zLwMuG7TuhIblJ4G3V1WPJEl18rxuFyBJksbPQJckqQYMdEmSasBAlySpBgx0SZJqwECXJKkGDHRJkmrAQJckqQYMdEmSaiAquF16x0TESuD3bXzLrRk0Xas6yv1dHfd1ddzX1Zmo+3rHzJw+eGVfB3q7RcTizJzT7TomCvd3ddzX1XFfV8d9/Wx2uUuSVAMGuiRJNWCgP9sZ3S5ggnF/V8d9XR33dXXc1w08hy5JUg14hC5JUg0Y6KWIOCAibouIZRFxTLfr6UcRcXZErIiI3zSs2yoiFkXE7eW/W5brIyK+XO7vGyPilQ2veXfZ/vaIeHc3vkuvi4gdIuLKiLglIm6OiI+U693fbRYRG0fEryLihnJfn1Su3ykiri336UURsVG5fmr5fFm5fVbDex1brr8tIt7Ypa/U8yJiUkQsiYhLyufu61Zk5oR/AJOAO4CXABsBNwAv63Zd/fYAXg28EvhNw7p/BY4pl48BPlcuHwj8CAhgH+Dacv1WwJ3lv1uWy1t2+7v12gPYDnhlubwZ8DvgZe7vjuzrAJ5fLk8Bri334XeAw8r1XwPeXy5/APhauXwYcFG5/LLyd8tUYKfyd86kbn+/XnwARwPfBi4pn7uvW3h4hF7YC1iWmXdm5jrgQuCQLtfUdzLzp8CqQasPAc4pl88B5jWsPzcL1wBbRMR2wBuBRZm5KjMfAhYBB3S8+D6Tmfdn5vXl8qPArcAM3N9tV+6zx8qnU8pHAq8FLi7XD97XAz+Di4HXRUSU6y/MzLWZeRewjOJ3jxpExPbAQcBZ5fPAfd0SA70wA7i34fl95TqN37aZeX+5/ACwbbncbJ/7sxilsptxT4ojR/d3B5RdwEuBFRR/9NwBrM7M9WWTxv32zD4ttz8MvBD3dau+CHwceLp8/kLc1y0x0FWZLPrCvKyijSLi+cD3gI9m5iON29zf7ZOZGzJzD2B7iiO9XbtbUT1FxJuBFZl5Xbdr6UcGemE5sEPD8+3LdRq/P5Zdu5T/rijXN9vn/ixaFBFTKML8/Mz8frna/d1BmbkauBLYl+K0xeRyU+N+e2aflttfADyI+7oV+wEHR8TdFKc+Xwt8Cfd1Swz0wq+B2eVIyo0oBlcs7HJNdbEQGBg5/W7gPxvW/3U5+nof4OGyq/hy4A0RsWU5QvsN5To1KM8TfgO4NTNPa9jk/m6ziJgeEVuUy5sA+1OMWbgSeFvZbPC+HvgZvA24ouwtWQgcVo7M3gmYDfyqki/RJzLz2MzcPjNnUfweviIzD8d93Zpuj8rrlQfFKODfUZwbO67b9fTjA7gAuB94iuKc1Xspzmf9F3A78BNgq7JtAF8p9/dNwJyG9/kbikEsy4D3dPt79eID+N8U3ek3AkvLx4Hu747s692BJeW+/g1wQrn+JRQhsQz4LjC1XL9x+XxZuf0lDe91XPkzuA14U7e/Wy8/gLn8aZS7+7qFh3eKkySpBuxylySpBgx0SZJqwECXJKkGDHRJkmrAQJckqQYMdGmCKmcOmzuG130zIk5uf0WSxmPyyE0k1VFmvrzbNUhqH4/QJUmqAQNdmqAi4u6IeH1EnBgR34mIcyPi0bIrfk5Duz0j4vpy20UUd+dqfJ83R8TSiFgdEb+MiN3L9YdGxF0RsXn5/E0R8UBETK/0i0oThIEuCeBgiskwtqC4D/bpAOXcBguAbwFbUdxm860DL4qIPYGzgfdR3Hb268DCiJiamRcBvwS+HBEvpLj3/N9m5spqvpI0sRjokgB+npmXZeYGivD+83L9PsAU4IuZ+VRmXkwxmdGA+cDXM/PaLKYYPQdYW74O4IMUM2ZdBfwwMy+p4LtIE5KBLgnggYblx4GNy+koXwwsz2dP+vD7huUdgX8ou9tXR8RqimkrXwzPTDf6XWA34N86V74kA13ScO4HZpTTtQ6Y2bB8L/DpzNyi4bFpZl4AEBF7UMzmdgHw5aqKliYiA13ScK4G1gMfjogpEfEWYK+G7WcCfxcRe5dzrU+LiIMiYrOI2Bg4D/gE8B6KPww+UPk3kCYIA11SU5m5DngLcCSwCjgU+H7D9sXAURSD6B6imJf6yHLzKcC9mfnVzFwLHAGcHBGzq6pfmkicD12SpBrwCF2SpBow0CVJqgEDXZKkGjDQJUmqAQNdkqQaMNAlSaoBA12SpBow0CVJqgEDXZKkGvj/tmLrzrw0TRgAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Keeping below lines commented out during development\n",
    "\n",
    "plt.figure(figsize=(8,6))\n",
    "plt.scatter(range(train_df.shape[0]), np.sort(train_df['target'].values))\n",
    "plt.xlabel('index', fontsize=12)\n",
    "plt.ylabel('Target', fontsize=12)\n",
    "plt.title('Distribution of Target', fontsize=14)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So there's not too much of outliers (visibly) but the distribution range is high. Now want to do a histogram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtoAAAH2CAYAAABOaCvjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAApPUlEQVR4nO3de5hlV10n/O8vaQICari0wHQ6JkoeUPHGtAGCIgOOhptBXyBhGAkaJjjiBVEx6PsK4uAr6iPC6CARMIkiBiIMgQnGyCXoCwQS7ndabulATENIuCkY+L1/nN2kqFR3TnVq1amq/nye5zx1ztprn/OrVaf7fGvV2ntXdwcAAFhbhy26AAAA2IoEbQAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhA0Aa2nKp6TFV1Vd15hW3bpm1PXaH/Mat8jZ9Zm4q3rqq6V1VdUlVfmMb4+1bo03PcPrr+1e9fVT2hqn5y0XUAG9u2RRcAsAH8nyT3SvLJVezzmMz+D33BiIK2kOcn+dckD0nyxSQfXKHPvZY9flmSdyR56pK2L40o7iZ4QpJ/SvLSBdcBbGCCNnDI6+69SfYuuo55VdXNu3ujBc8bqKrDktwlydO7+zX769fdb1q235eSfGp5+0HWsCnGCtiaLB0BDnkrLR2pqv9SVW+rqs9X1Wer6l1V9bhp2+uS/HCSey9Z2vC6JfseX1X/MO37hap6dVUdv8LrPqGqPlpV/1ZVb66qE6bHZ61Q232q6iVVdU2SS6ZtP1BV51XVnqr616r6QFX9blV9w7LXeV1V/VNVnVhVb5/6vq2q7jEtpfndqvpkVV1dVWdV1a3mGLNvqqo/qapPVNWXptf+5aqqfXUn+UpmnzP/z01Z/lFVt6iqZ1bVu6cxvbKqXlFVd13W70Bjdcuqek5VfXp6jpdN491TrUuf54enn9nnpp/fhVV1tyXbP5rkW5M8asnP/6yD+d6Arc2MNrCVHV5Vy/+fO/zGdqqqH0zyV0meneTXMguLd01y5NTl56bthyd53NT22Wnf70lycZL3Zra8pJOckeTiqrpnd79j6vfYJM/MbGnFS5J8e5K/XvIay70wyYuSPCzX/999dJK3JzkryeeSfFeS30rybUlOWbb/nZP8QZKnJ/l8kt9Pcv502zbV+h1Tn6uSPOkA43NYZstt7j693ruSPCjJHyXZnuQ3pu0/mNnyiucneV4OfvnHzZN8Y5L/kdnynttm9jN4Y1V9R3dfuaz/SmN1ZpKHZ7Yc5dIk95/6Lf/eHpTk5VP9/3Vq/vUk/1hV39Pdlyf5iSQX5OuXt2yav4gA60fQBray9x/kfvdMck13P2FJ29/vu9Pd762qzybZtsLyht/KLFDev7uvSZKquijJR5M8JclPTkH1KUle1d2P3bdjVV2Z5G/3U9N53f114be7v9Z3mkn+/zIL/OdU1eO7+9NLut8uyQnd/eGp/2GZBcpju/tHpj4XVtV9Mguk+w3aSR6YWYj+6e4+a2r7+2km/Feq6o+6e29VfWbatuemLAPp7muTLB2nw5NcmORfkjwys19Ylvq6saqquyT5L0nO6O7fn5ovqqpbJvmFZfs+K8nF3X3Skv1fm+TDSX4lyRO6+21rubwF2LosHQG2sp9I8gPLbvecY7+3JLlNVf1VVT24qo5cxWveJ8kr94XsJOnuz2Y2c/zDU9NR0+0ly/Z9eZLr9vO8L1veMC3feEZV/XNm4f7fk/xlkkpy3LLuH9wXsif7fgm5cFm/9yc5at8SkP24T5KvZjYDv9RfJTkiNzy48SarqkfU7Owl12Q2Rl9IcuvM1oAvt3ys7pHZmCwf7/OWvcZxmf1l4YXTkppt019EvpjkjZl93wBzM6MNbGXv7u7dSxtWWEpyA919cVU9PLPZzpdN+12c5Ind/c4b2f22WfnsJVcmuc10/07T16uWve5XqupT+3nelZ7zL5L8SGaz6G/PLHwen+RPk9xiWd/PLHv85QO0b8tsWcz+Qv9tk1zd3V9e1n7lku1rpqoekuTcJGcn+e0kn8os6F+QG36fyQ3HasXxzmxGfKlvmb4+f7ot9/E5SwZIImgDrKi7z0tyXlXdOsl9kzwjyd9V1VHd/dUD7Hp1kjuu0H7HXB9q9wXBb1naYVoScfv9lbSs7y2SnJTkqd39rCXt332A2tbK1UluW1VHLAvbd1yyfS2dkmR3dz9mX0NV3Sz7D/S97PHS8f7IkvY7LOu3b6nNk5P8wwrPu/wXC4ADsnQE4AC6+/Pd/cokz81sZvR206YvJfmGFXa5OMkDq+ob9zVM9x+S5HVT057p9vBl+z4080+A3DyzWed/X9b+mDn3vykuzuzzY3n9j8osjL5xjV/vlrnh7PpPZY4DWydvzix8L693+eMPZLaW/ru6+9IVbkv/mrG/nz/A15jRBlimqp6W2Wzna5N8IrP11L+Y5O3TObeT2VlFfq6qTk7yz0k+190fSPI7SR6c5NVV9YzMAt6vZxYWn5Yk3f3VqvrtJH9eVc/LbO3wt2V2dpJrM1sWcUDdfW1VvSmzgw8/mdlyip9JsmMNhuDGvCqzs4n8WVVtT/KezA6QfGyS/7e797f85WD9XZKHVtUzk7wyya7MlvVcM8/O3f3+qvrrJL8zHQR6WZL7ZfbLTzKNd3d3VT0+ycur6ogkL85sXO+Q5IQkH+/uP5r2eW+SH6qqB2e2ZOZT3f3Rm/qNAluLGW2AG7okyTGZnc3iosyWjVyc2Sns9nlGkldndtq6t2Q2451p1vO+mZ394+zMDk78fJIf3ndqv6nf85L8cpL/nNlBkKdldjq5zixsz+ORmYXGP83sFH9XJvmlVX2nB2FaOvOgzL6/X8/sVHgPSvLEJL854CX/PLPTEp6c5BWZhfqHZP5xSpLTM7uK55MyW3f/XUkeP2372vN09wWZHfR4q8x+thdmdirEO+brZ+qfnNkM+Isz+/k/dXXfEnAoqO7lS9kAWISq2pVZaHt0d//louvZ6qrqVzML0cd0twMdgTVn6QjAAlTVsZnNqP5jZrPf35HZhV4+kv2fS5uDNC3xuFtmZ2f5apIfSvKrSV4sZAOjCNoAi/GvmQW/R2d22r/PZHamizO6+4uLLGyL+lxmB5uekdmykCsyu/LnUxZYE7DFWToCAAADOBgSAAAGGBa0q+oFVXVVVb17hW2/UlVdVbefHldVPbuqdlfVO6vq7kv6nlpVH5pup46qFwAA1tLINdpnJfmTJOcsbayqnUl+NF9/KdsHJDluut0jyXOS3KOqbpvZ+rldmZ3y6rKqOr+7l18y+Ovc/va372OOOWZtvgsAANiPyy677FPdvX2lbcOCdne/vqqOWWHTMzM7j+nLl7SdlOScni0Yf1NVHVlVd8rsXLQXdffVSVJVFyU5McmLDvTaxxxzTC699NKb/k0AAMABVNXH9rdtXddoV9VJSa5YetGGyY4kly95vGdq2187AABsaOt2er+qumVm54j90UHPf3pmV/7K0UcfPeIlAABgbus5o/3tSY5N8o6q+miSo5K8tarumNn5THcu6XvU1La/9hvo7jO7e1d379q+fcVlMgAAsG7WLWh397u6+1u6+5juPiazZSB37+4rk5yf5NHT2UfumeTa7v5kkguT/GhV3aaqbpPZbPiF61UzAAAcrJGn93tRkjcmuUtV7amq0w7Q/YIkH06yO8mfJ/m5JJkOgvydJG+Zbk/bd2AkAABsZFvyypC7du1qZx0BAGC0qrqsu3ettM2VIQEAYABBGwAABhC0AQBgAEEbAAAGELQBAGAAQRsAAAYQtAEAYABBGwAABhC0AQBgAEEbAAAGELQBAGAAQRsAAAYQtAEAYABBe43t2Hl0qmru246dRy+6ZAAABti26AK2mk/suTwnP/cNc/c/93EnDKwGAIBFMaMNAAADCNoAADCAoA0AAAMI2gAAMICgDQAAAwjaAAAwgKANAAADCNoAADCAoA0AAAMI2gAAMICgDQAAAwjaAAAwgKANAAADCNoAADCAoA0AAAMI2gAAMICgDQAAAwjaAAAwgKANAAADCNoAADCAoA0AAAMI2gAAMICgDQAAAwjaAAAwgKANAAADCNoAADCAoA0AAAMI2gAAMICgDQAAAwjaAAAwgKANAAADCNoAADCAoA0AAAMI2gAAMICgDQAAAwjaAAAwgKANAAADCNoAADDAsKBdVS+oqquq6t1L2v6gqt5fVe+sqpdV1ZFLtj25qnZX1Qeq6seWtJ84te2uqjNG1QsAAGtp5Iz2WUlOXNZ2UZK7dff3JPlgkicnSVV9Z5JTknzXtM//qqrDq+rwJH+a5AFJvjPJI6e+AACwoQ0L2t39+iRXL2v7++6+bnr4piRHTfdPSvI33f2l7v5Ikt1Jjp9uu7v7w9395SR/M/UFAIANbZFrtH8myaum+zuSXL5k256pbX/tAACwoS0kaFfVbya5LskL1/A5T6+qS6vq0r17967V0wIAwEFZ96BdVY9J8uAkj+runpqvSLJzSbejprb9td9Ad5/Z3bu6e9f27dvXvG4AAFiNdQ3aVXVikicl+fHu/uKSTecnOaWqbl5VxyY5Lsmbk7wlyXFVdWxVHZHZAZPnr2fNAABwMLaNeuKqelGS+ya5fVXtSfKUzM4ycvMkF1VVkrypu3+2u99TVS9O8t7MlpQ8vru/Mj3Pzye5MMnhSV7Q3e8ZVTMAAKyVYUG7ux+5QvPzD9D/6UmevkL7BUkuWMPSAABgOFeGBACAAQRtAAAYQNAGAIABBG0AABhA0AYAgAEEbQAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhA0AYAgAEEbQAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhA0AYAgAEEbQAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhA0AYAgAEEbQAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhA0AYAgAEEbQAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhA0AYAgAEEbQAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhA0AYAgAEEbQAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhA0AYAgAEEbQAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhA0AYAgAGGBe2qekFVXVVV717SdtuquqiqPjR9vc3UXlX17KraXVXvrKq7L9nn1Kn/h6rq1FH1AgDAWho5o31WkhOXtZ2R5NXdfVySV0+Pk+QBSY6bbqcneU4yC+ZJnpLkHkmOT/KUfeEcAAA2smFBu7tfn+TqZc0nJTl7un92kocuaT+nZ96U5MiqulOSH0tyUXdf3d2fSXJRbhjeAQBgw1nvNdp36O5PTvevTHKH6f6OJJcv6bdnattf+w1U1elVdWlVXbp37961rRoAAFZpYQdDdncn6TV8vjO7e1d379q+fftaPS0AAByU9Q7a/zItCcn09aqp/YokO5f0O2pq2187AABsaOsdtM9Psu/MIacmefmS9kdPZx+5Z5JrpyUmFyb50aq6zXQQ5I9ObQAAsKFtG/XEVfWiJPdNcvuq2pPZ2UN+L8mLq+q0JB9L8oip+wVJHphkd5IvJvnpJOnuq6vqd5K8Zer3tO5efoAlAABsOMOCdnc/cj+b7r9C307y+P08zwuSvGANSwMAgOFcGRIAAAYQtAEAYABBGwAABhC0AQBgAEEbAAAGELQBAGAAQRsAAAYQtAEAYABBGwAABhC0AQBgAEEbAAAGELQBAGAAQRsAAAYQtAEAYABBGwAABhC0AQBgAEEbAAAGELQBAGAAQRsAAAYQtAEAYABBGwAABhC0AQBgAEEbAAAGELQBAGAAQRsAAAYQtAEAYABBGwAABhC0AQBgAEEbAAAGELQBAGAAQRsAAAYQtAEAYABBGwAABhC0AQBgAEEbAAAGELQBAGAAQRsAAAYQtAEAYABBGwAABhC0AQBgAEEbAAAGELQBAGAAQRsAAAYQtAEAYABBGwAABhC0AQBgAEEbAAAGELQBAGAAQRsAAAYQtAEAYABBGwAABhC0AQBggIUE7ar65ap6T1W9u6peVFW3qKpjq+qSqtpdVedW1RFT35tPj3dP249ZRM0AALAa6x60q2pHkl9Msqu775bk8CSnJHlGkmd2952TfCbJadMupyX5zNT+zKkfAABsaItaOrItyTdU1bYkt0zyyST3S3LetP3sJA+d7p80Pc60/f5VVetXKgAArN66B+3uviLJHyb5eGYB+9oklyW5pruvm7rtSbJjur8jyeXTvtdN/W+3njUDAMBqLWLpyG0ym6U+Nsl/SHKrJCeuwfOeXlWXVtWle/fuvalPBwAAN8kilo78SJKPdPfe7v73JC9Ncu8kR05LSZLkqCRXTPevSLIzSabt35zk08uftLvP7O5d3b1r+/bto78HAAA4oEUE7Y8nuWdV3XJaa33/JO9N8tokD5v6nJrk5dP986fHmba/prt7HesFAIBVW8Qa7UsyO6jxrUneNdVwZpJfT/LEqtqd2Rrs50+7PD/J7ab2JyY5Y71rBgCA1dp2413WXnc/JclTljV/OMnxK/T9tyQPX4+6AABgrbgyJAAADCBoAwDAAII2AAAMIGgDAMAAgjYAAAwgaAMAwACCNgAADCBoAwDAAII2AAAMIGgDAMAAgjYAAAwwV9CuqnvP0wYAAMzMO6P9P+dsAwAAkmw70MaquleSE5Jsr6onLtn0TUkOH1kYAABsZgcM2kmOSHLrqd83Lmn/bJKHjSoKAAA2uwMG7e6+OMnFVXVWd39snWoCAIBN78ZmtPe5eVWdmeSYpft09/1GFAUAAJvdvEH7JUn+LMnzknxlXDkAALA1zBu0r+vu5wytBAAAtpB5T+/3iqr6uaq6U1Xddt9taGUAALCJzTujfer09deWtHWSb1vbcgAAYGuYK2h397GjCwEAgK1krqBdVY9eqb27z1nbcgAAYGuYd+nIDyy5f4sk90/y1iSCNgAArGDepSO/sPRxVR2Z5G9GFAQAAFvBvGcdWe4LSazbBgCA/Zh3jfYrMjvLSJIcnuQ7krx4VFEAALDZzbtG+w+X3L8uyce6e8+AegAAYEuYa+lId1+c5P1JvjHJbZJ8eWRRAACw2c0VtKvqEUnenOThSR6R5JKqetjIwgAAYDObd+nIbyb5ge6+KkmqanuSf0hy3qjCAABgM5v3rCOH7QvZk0+vYl8AADjkzDuj/XdVdWGSF02PT05ywZiSAABg8ztg0K6qOye5Q3f/WlX9ZJIfnDa9MckLRxcHAACb1Y3NaP9xkicnSXe/NMlLk6Sqvnva9pCBtQEAwKZ1Y+us79Dd71reOLUdM6QiAADYAm4saB95gG3fsIZ1AADAlnJjQfvSqvpvyxur6rFJLhtTEgAAbH43tkb7CUleVlWPyvXBeleSI5L8xMC6AABgUztg0O7uf0lyQlX9pyR3m5r/T3e/ZnhlAACwic11Hu3ufm2S1w6uBQAAtgxXdwQAgAEEbQAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhA0AYAgAEEbQAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhgIUG7qo6sqvOq6v1V9b6quldV3baqLqqqD01fbzP1rap6dlXtrqp3VtXdF1EzAACsxqJmtJ+V5O+6+65JvjfJ+5KckeTV3X1ckldPj5PkAUmOm26nJ3nO+pcLAACrs+5Bu6q+Ocl9kjw/Sbr7y919TZKTkpw9dTs7yUOn+yclOadn3pTkyKq607oWDQAAq7SIGe1jk+xN8hdV9baqel5V3SrJHbr7k1OfK5PcYbq/I8nlS/bfM7UBAMCGtYigvS3J3ZM8p7u/P8kXcv0ykSRJd3eSXs2TVtXpVXVpVV26d+/eNSsWAAAOxiKC9p4ke7r7kunxeZkF73/ZtyRk+nrVtP2KJDuX7H/U1PZ1uvvM7t7V3bu2b98+rHgAAJjHugft7r4yyeVVdZep6f5J3pvk/CSnTm2nJnn5dP/8JI+ezj5yzyTXLlliAgAAG9K2Bb3uLyR5YVUdkeTDSX46s9D/4qo6LcnHkjxi6ntBkgcm2Z3ki1NfAADY0BYStLv77Ul2rbDp/iv07SSPH10TAACsJVeGBACAAQRtAAAYQNAGAIABBG0AABhA0AYAgAEEbQAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhA0AYAgAEEbQAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhA0AYAgAEEbQAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhA0AYAgAEE7UU7bFuqau7bjp1HL7piAADmsG3RBRzyvnpdTn7uG+bufu7jThhYDAAAa8WMNgAADCBoAwDAAII2AAAMIGgDAMAAgjYAAAwgaAMAwACCNgAADCBoAwDAAII2AAAMIGgDAMAAgjYAAAwgaAMAwACCNgAADCBoAwDAAII2AAAMIGgDAMAAgjYAAAwgaAMAwACCNgAADCBoAwDAAII2AAAMIGgDAMAAgjYAAAwgaAMAwACC9mZz2LZU1dy3HTuPXnTFAACHpG2LLoBV+up1Ofm5b5i7+7mPO2FgMQAA7I8ZbQAAGEDQBgCAARYWtKvq8Kp6W1W9cnp8bFVdUlW7q+rcqjpiar/59Hj3tP2YRdUMAADzWuSM9i8led+Sx89I8szuvnOSzyQ5bWo/LclnpvZnTv0AAGBDW0jQrqqjkjwoyfOmx5XkfknOm7qcneSh0/2TpseZtt9/6g8AABvWoma0/zjJk5J8dXp8uyTXdPd10+M9SXZM93ckuTxJpu3XTv2/TlWdXlWXVtWle/fuHVg6AADcuHUP2lX14CRXdfdla/m83X1md+/q7l3bt29fy6cGAIBVW8R5tO+d5Mer6oFJbpHkm5I8K8mRVbVtmrU+KskVU/8rkuxMsqeqtiX55iSfXv+yAQBgfus+o93dT+7uo7r7mCSnJHlNdz8qyWuTPGzqdmqSl0/3z58eZ9r+mu7udSwZAABWbSOdR/vXkzyxqnZntgb7+VP785Pcbmp/YpIzFlQfAADMbaGXYO/u1yV53XT/w0mOX6HPvyV5+LoWBgAAN9FGmtEGAIAtQ9AGAIABBG0AABhA0AYAgAEEbQAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhA0AYAgAEEbQAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhA0AYAgAEEbQAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhA0AYAgAEEbQAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhA0AYAgAEEbW5gx86jU1Vz33bsPHrRJQMAbDjbFl0AG88n9lyek5/7hrn7n/u4EwZWAwCwOZnRBgCAAQRtAAAYQNAGAIABBG0AABhA0AYAgAEEbQAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhA0AYAgAEEbQAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhA0AYAgAEEbQAAGEDQBgCAAQRtAAAYQNCGDWjHzqNTVXPfduw8etElAwDLbFt0AQx22LZU1aKrYJU+sefynPzcN8zd/9zHnTCwGgDgYKx70K6qnUnOSXKHJJ3kzO5+VlXdNsm5SY5J8tEkj+juz9QsJT4ryQOTfDHJY7r7retd96b11etWFdgSoQ0AYC0sYunIdUl+pbu/M8k9kzy+qr4zyRlJXt3dxyV59fQ4SR6Q5LjpdnqS56x/yQAAsDrrHrS7+5P7ZqS7+3NJ3pdkR5KTkpw9dTs7yUOn+yclOadn3pTkyKq60/pWDQAAq7PQgyGr6pgk35/kkiR36O5PTpuuzGxpSTIL4Zcv2W3P1AYAABvWwoJ2Vd06yd8meUJ3f3bptu7uzNZvr+b5Tq+qS6vq0r17965hpQAAsHoLCdpVdbPMQvYLu/ulU/O/7FsSMn29amq/IsnOJbsfNbV9ne4+s7t3dfeu7du3jyseAADmsO5BezqLyPOTvK+7/2jJpvOTnDrdPzXJy5e0P7pm7pnk2iVLTAAAYENaxHm0753kp5K8q6rePrX9RpLfS/LiqjotyceSPGLadkFmp/bbndnp/X56XasFAICDsO5Bu7v/Kcn+rqBy/xX6d5LHDy0KAADWmEuwAwDAAII2AAAMIGiz7nbsPDpVNfdtx86jF10yAMCqLeJgSA5xn9hzeU5+7hvm7n/u404YWA0AwBhmtAEAYABBGwAABhC0AQBgAEEbYIty4DHAYjkYEmCLcuAxwGKZ0QYAgAEEbQAAGEDQBgCAAQRtNr7DtjmgCwDYdBwMycb31esc0AUAbDpmtAEAYABBGwAABhC0AQBgAEEbAAAGELQBAGAAQRsAAAYQtAEAYABBGwAABhC0AQBgAEEbAAAGELQBAGAAQZtD3o6dR6eqVnXbsfPoRZcNAGxw2xZdAFvAYdtSVYuu4qB9Ys/lOfm5b1jVPuc+7oRB1QAAW4WgzU331etWFVSFVADgUGDpCAAADCBoAwDAAII2AAAMIGgDAMAAgjYAAAwgaAMAwACCNgAADCBoAwDAAC5Yw9azya9UCQBsDYI2W48rVQIAG4ClIwAAMICgDQAAAwjaAAAwgKANh6AdO49OVa3qtmPn0YsuGwA2FQdDwiHoE3suX9UBo4mDRgFgtcxow8GYTiG4YWaDV1nPRrTaWXYz7ABsdGa04WCs9hSC//0+YwPuFjil4Wpn2Tfi9wDMfmn+xJ7LV7XPfzhqZ664/OODKmIz2irvI0Eb1sMWCMIA87A0jbWwVd5Hlo4Am9NGW74DAMuY0QY2J38lAGCDM6MNHBo24Ay4A0ABtjYz2sB8pqA6r8NvdvN85d+/NLCgVdqAM+AOAAXY2gRtYD4HEVSFSNbbas9UsBHPUgBsHYI2wBo5mNNRsbb8lQDYSARtgDWy6UPeOiwPWu0+G23GeSPOmG/Emlh7fs6b06YJ2lV1YpJnJTk8yfO6+/cWXBKwla0ydG4Jg5cHHcw+G+2XkVX/MnUQF6tabUDa9L/gMRc/581pUwTtqjo8yZ8m+c9J9iR5S1Wd393vXWxlwJa1ytCZrMMH26EY/je7g3kfjb6SLAtnmdmhY1ME7STHJ9nd3R9Okqr6myQnJRG0gUPHBjxzynCH4i8Xo3/OqxzTdVmCMHjZ0kbrn2TD/SK/2vC/HmO0FWyWoL0jydKf/p4k91hQLQCslw0WOreE1Y7pesywr8NZjTZa/+EO4r19yI3ROqjuXnQNN6qqHpbkxO5+7PT4p5Lco7t/fkmf05OcPj28S5IPrHuhye2TfGoBr7uZGbPVMV6rY7xWx3itjvFaHeO1OsZr9RY1Zt/a3dtX2rBZZrSvSLJzyeOjprav6e4zk5y5nkUtV1WXdveuRdaw2Riz1TFeq2O8Vsd4rY7xWh3jtTrGa/U24phtlkuwvyXJcVV1bFUdkeSUJOcvuCYAANivTTGj3d3XVdXPJ7kws9P7vaC737PgsgAAYL82RdBOku6+IMkFi67jRix06comZcxWx3itjvFaHeO1OsZrdYzX6hiv1dtwY7YpDoYEAIDNZrOs0QYAgE1F0D4IVXViVX2gqnZX1RkrbL95VZ07bb+kqo5ZQJkbxhzj9Ziq2ltVb59uj11EnRtFVb2gqq6qqnfvZ3tV1bOn8XxnVd19vWvcSOYYr/tW1bVL3l+/td41biRVtbOqXltV762q91TVL63Qx3tsMud4eY9NquoWVfXmqnrHNF6/vUIfn5GTOcfLZ+QyVXV4Vb2tql65wrYN9f7aNGu0N4o5Lwd/WpLPdPedq+qUJM9IcvL6V7t4c45Xkpy79Lzoh7izkvxJknP2s/0BSY6bbvdI8pwc2hdwOisHHq8k+cfufvD6lLPhXZfkV7r7rVX1jUkuq6qLlv2b9B673jzjlXiP7fOlJPfr7s9X1c2S/FNVvaq737Skj8/I680zXonPyOV+Kcn7knzTCts21PvLjPbqfe1y8N395ST7Lge/1ElJzp7un5fk/nXIXXrsa+YZL5bo7tcnufoAXU5Kck7PvCnJkVV1p/WpbuOZY7xYors/2d1vne5/LrMPqx3LunmPTeYcLybTe+bz08ObTbflB4P5jJzMOV4sUVVHJXlQkuftp8uGen8J2qu30uXgl/+n+7U+3X1dkmuT3G5dqtt45hmvJPm/pj9Rn1dVO1fYzvXmHVOud6/pT7OvqqrvWnQxG8X0J9XvT3LJsk3eYys4wHgl3mNfM/1Z/+1JrkpyUXfv9/3lM3Ku8Up8Ri71x0melOSr+9m+od5fgjYbwSuSHNPd35Pkolz/myishbdmdnnc703yP5P878WWszFU1a2T/G2SJ3T3Zxddz0Z3I+PlPbZEd3+lu78vs6s4H19Vd1twSRvaHOPlM3JSVQ9OclV3X7boWuYlaK/ejV4OfmmfqtqW5JuTfHpdqtt4bnS8uvvT3f2l6eHzkvzHdapts5rnPcikuz+770+z0/n4b1ZVt19wWQs1rQX92yQv7O6XrtDFe2yJGxsv77GVdfc1SV6b5MRlm3xGrmB/4+Uz8uvcO8mPV9VHM1uKer+q+qtlfTbU+0vQXr15Lgd/fpJTp/sPS/KaPnRPWH6j47Vs7eePZ7YGkv07P8mjpzND3DPJtd39yUUXtVFV1R33rc+rquMz+3/vkP1Qn8bi+Une191/tJ9u3mOTecbLe+x6VbW9qo6c7n9DZgfCv39ZN5+Rk3nGy2fk9br7yd19VHcfk1meeE13/9dl3TbU+8tZR1Zpf5eDr6qnJbm0u8/P7D/lv6yq3ZkdpHXK4iperDnH6xer6sczO7r/6iSPWVjBG0BVvSjJfZPcvqr2JHlKZgfIpLv/LLMrpD4wye4kX0zy04updGOYY7weluS/V9V1Sf41ySmH6of65N5JfirJu6Z1oUnyG0mOTrzHVjDPeHmPXe9OSc6ezjh1WJIXd/crfUbu1zzj5TPyRmzk95crQwIAwACWjgAAwACCNgAADCBoAwDAAII2AAAMIGgDAHDIqaoXVNVVVfXuOfo+s6rePt0+WFXXzPUazjoCAMChpqruk+TzSc7p7rmvYFpVv5Dk+7v7Z26srxltgE2oqvpGbmctsLaPVtWvLur1AebR3a/P7FzbX1NV315Vf1dVl1XVP1bVXVfY9ZFJXjTPa7hgDcDmtPRqcQ9O8ufL2v51NU9WVUd095fXojCATezMJD/b3R+qqnsk+V9J7rdvY1V9a5Jjk7xmniczow2wCXX3lftuSa5Z2pbkVknOqaorq+oLVfXWqnrw0v2nWeenTmsUr0nywqn9Z6rq41X1xap6RVX9XFX1sn0fMs32/FtVfaSqnl5VR0zbXpfkW5P8wb7Z9cFDAbAmqurWSU5I8pLpSrDPzddPYCSzK02e191fmec5zWgDbD23TvKqJP93ZjPbJyd5aVV9T3e/f0m/Jyb5H0l2JamquleS5yV5cpKXJfnhJL+79Imr6scyC+W/lOT1mV2K/M+S3DzJryb5ySTvSPKCJM8Z9P0BjHBYkmu6+/sO0OeUJI9fzRMCsIV09zu6+8+6+13dvbu7n57krUketqzrxd39+1OfDyX5xSR/393P6O4PdvefZxa4l/rNJH/Q3X/R3f/c3a9N8utJfraqqruvTvKVJJ9bMsMOsOF192eTfKSqHp7MZh+q6nv3bZ/Wa98myRvnfU5BG2CLqapbVdXvV9V7q+ozVfX5zGatj17W9dJlj++a5M3L2i5Z9vg/JvnNqvr8vluSv85sucod1+hbABiuql6UWWi+S1XtqarTkjwqyWlV9Y4k70ly0pJdTknyN72KU/ZZOgKw9fxhkhMzW8rxoSRfTHJOkiOW9fvCQTz3YUl+O8lLVti29yCeD2AhuvuR+9l04n76P3W1ryFoA2w9P5jZeWH/Nkmq6hZJvj3JB29kv/cn+YFlbccve/zWJHft7t0HeJ4vJzl8/nIBtiZBG2Dr+WCSn6iqlyf59yRPSXKLOfZ7dpJ/qqpfS/K/k9wnyU8s6/O0JK+sqo8leXGS65LcLcnx3f2kqc9Hk/xQVf1Vki9196du2rcDsDlZow2w9TwxyVVJ/jGzs4+8abp/QN39xiT/LbODIt+Z5KFJnpHk35b0uTDJg5L8p8zWc785yRlJPr7kqX4ryc4k/xzLSYBDmEuwA7BfVfXMJD/S3d+96FoANhtLRwD4mmnZyEVJPp/kR5L8bJLfWGhRAJuUGW0Avqaqzk1y3yTfnOQjmV0Z7VmrOZ0VADOCNgAADOBgSAAAGEDQBgCAAQRtAAAYQNAGAIABBG0AABhA0AYAgAH+fxX6cS1YGdw6AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(12, 8))\n",
    "# sns.distplot(train_df['target'].values, bins=50, kde=False)\n",
    "sns.histplot(train_df['target'].values, bins=50, kde=False)\n",
    "plt.xlabel('Target', fontsize=14)\n",
    "plt.title('Histogram of Target', fontsize=16)\n",
    "plt.show()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a right skewed distribution with majority of the data points having low value. \n",
    "This is another big reason that this Competition host have chosen the evaluation metric as RMSLE (Root Mean Squared Logarithmic Error.).\n",
    "\n",
    "So to plot histogram in this case lets do the plot with log of target variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsYAAAH0CAYAAAA+MuuOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAgPElEQVR4nO3de7TtZ13f+/dXI5yCqFgihUAMlWABL1FTQusNSyuXcg6gLQfC0aDYOBQ6xMtR0KNyhoMWFaQXKxUrDXbIrQc8UMUqcLzUUwkGityRqAESQggHy8UoNvCcP+bcutjsnezLWnvttffrNcYca87n9/vN33c+a2Xls5/1/J7frLUCAICz3aftdwEAAHA6EIwBACDBGAAAKsEYAAAqwRgAACrBGAAAKsEYoKqZud3M/F8z86GZWTNzwX7XtJ9m5pqZ+b79rgPgVBKMgVNqZq6YmV/e7zqO4Furr6m+qrpL9Z6dG7d1r1t67EfR29qeOjNvPob9HjczHz3Kto/OzON2NP3t6meO8fyn6/cU4LgIxgAb96zettZ601rrfWutjx+2/bvaBOZDj5uqJx3Wdsxm5pyZmZOueo+stW5ca92033UcbmZus981AGcuwRg4rczM18zMlTPz5zNzw8w8a2cYmpnbz8wvbEc4b5iZp8zML8/MFbfyvt8wM2+amY/NzHtm5ocOBdOZ+c02wfdrtqO/v3n48WutD20D8/vWWu+rVvWhHa+fNDPvmJk/205D+ImZ+Z92nP+pM/Pm7ajtH1Yfq24/M/eamd/aft53zMxDDx+9nZnzZuaFM/Mn28evzMyF222Pq360uu+O0evHdZIOn0oxM98+M3+wrfMDM/Nr23D/1Oqy6h/uOP8Dtsd88cy8atsnH9yOLH/2jvc8Z/v9PfS5njUzz97Z/zPzm9u2Z8zMjdX/u23/npl548z86cxcNzP/bmY+Z8dxj9v240Nm5u0zc9PMvHxmPntm/tHMvHM7beY/zMxfO9n+As4MgjFw2piZ86pfrf5b9WXV46vHVP98x27PrL62emT196ovrb76Vt73K6r/WL20+uLqydVTqidud/mG6t9Xv9tm5PcbTqD8P20zHePe1XdWj65+6LB97lFdWv3jbd1/Uf1SdXN1/+pxbULubXfUfrvqN6o/b/O5/051ffWq7bYXtemTd/RXI9cvOoH6j2pmLq7+TfV/Vl9YPbD6z9vNz6heXL1qx/n/68zcvvq16qPV/dp8v/5u9dwdb/1928/8bW0+/6e16Z/D/W/VtPk+f/O27RNtRuzvuz3mftW/Puy421bfWz12W/PF1UvaBPlvrB5RPazN9wugc/a7AIAdvrN6b/Wda61PVG+bmSdXPzszP9wmOH1r9c1rrVdWzczjq2tv5X2/p/qttdaPbl//wXbE9Qeqf73W+uDM3FT9xXb097ittX5sx8trZuaftQl+P7yj/TbVN621btjW/qA2QfPr11rXbdu+u+2o6Naj24TCb1lrre0+3169v3rYWuvF23nDNx9j7bc/yjzj29/CMee3Cf4vX2t9pHpX9fvbbR+dmT+rPrbz/DNz2fY9v2l7TDNzefUbM3PPtdbVbUbpf3yt9ZLt9idVDz7C+f94rfW9OxvWWv9ix8trZub7q5fNzGXbn53a/D/uCWutd2zf//nVd1d3Xmt9YNv2surr2vzjAjjLCcbA6eTe1Wt2BJuq32kTKO/ZJiB+RvXaQxvXWn86t37h2b2rXzms7XeqH52Zz1prffhkC5+Zf9RmBPOe1WdWn7597HTtoVC89beq9x4KxVu/12Y09JCvaDPS/JH55CnJt6u+4ARKvam66Ajtv3+EtkNe2SYM//HM/Fr169VLDwXeo7h39cbD9vmvbT7bfbbTIv5Gn/y9XDPz2uruh73X6w5/85n5e21G/e9dfXabvr7N9j3fu93tY4dC8dYN1fsOheIdbfe5hc8BnEVMpQAOir1a9eGk33dm7l+9sM3Ugf+5zTSQ/6NNiN/pT0/g7T+tekObMLvzca/qZ0/g/dZa6+rDH91CP2zD7ZdXj6re3SaQvn1m7noC5++WznUUn9RvM/P5bf6h87Y201K+os1fEmoTjg+5+Qjn/R9HaPP/QqDyywA4vbytuv/M7Pzd9FVt5uL+4fbxP9osJVb95RzcLzqG9/3Kw9q+qs0I7i2Neh6rr6yuW2v92Frr99Za76w+/xiOe3t118MC5sV98u/m17cZhf7AEQLtB7f7/EWfOjq9q9ZaN6+1/p+11lOqL2kzTeJht3D+t1VfPDN32NH2d9t8trettT5Uva9P/l7Ozte34OI2Afi711q/u9b6g+pEQzrAXxKMgf3wWTNz0WGPC9qsm3vX6mdm5t4z8w+rp1c/vda6aa310TYXb/34zDxwZu5T/bs2v8tuaRTymdXXbleGuNfMPLbNRVk/sUuf5w+q82bmsTPzN2fmO9pcNHhrXtnmornnzcyXbkeef6rNSOehz/OLbf7c/7KZ+dqZucdsVu545qGVKaprqs+fmS+fmTvNzG3bRTPzsJn5rpn5su1o7aXVHdqE30Pn/6KZ+cLt+T9jW/dN1S9sV6f4mjYj3C/djlBX/cvq+2fmkTPzhW2+T3fp1keU39nme/6kbX88ps00FoCTIhgD++Gr26w8sfPxjO1c24e0mYrwhjYh+AXVD+449vuq/1K9vM1qDW+srmqzasMRrbVe3+ZP7t9YvblN2H569dO78WHWWv+p+snqX2zr+QfVjxzDcZ9os1rDbdvMtX1e9bQ2wfDPt/vc1ObGI3/UZmWNt2/3u2P1J9u3ekn1iurV1Y0dWyg/Hv+9zQoOr9qe//uqb1tr/Zft9p9rE5Kv2p7/K7d1P6j6rO1ne1mbVT++dcf7PqP6D21WBHnNtu2XuoXvZdVa641tLtz7nuqtbVa1cJc+4KTN9iJngANpOzr6ruon11oHfmWBmfnSNv8ouHit9SkXnZ3pZua/Vb+z1vqn+10LcPaxKgVwoMzMl7VZieC1bf6c/wPbr7u6du+pMjOPbHNx2TurC9pMpfj9NnOLz2jbaRkPqn6rzYWK/6TN/OV/sp91AWcvwRg4iL6nzfq/N7cZXf2atdatrWV8urpD9eNtlij7k+o321xUdjb8Oe8TbW7Y8ZNtpva9tXrIWuuqfa0KOGuZSgEAALn4DgAAKsEYAACq02SO8Z3udKd1wQUX7HcZAACc4V73utd9YK117pG2nRbB+IILLuiqq1xrAQDA3pqZdx1tm6kUAACQYAwAAJVgDAAAlWAMAACVYAwAAJVgDAAAlWAMAACVYAwAAJVgDAAAlWAMAACVYAwAAJVgDAAAlWAMAACVYAwAAJVgDAAAlWAMAACVYAwAAJVgDAAAlWAMAABVnbPfBQBw9nr+le8+rv0vveT8PaoEwIgxAABUgjEAAFSCMQAAVIIxAABUgjEAAFSCMQAAVIIxAABUgjEAAFSCMQAAVIIxAABUgjEAAFSCMQAAVIIxAABUgjEAAFSCMQAAVIIxAABUgjEAAFSCMQAAVIIxAABUgjEAAFSCMQAAVIIxAABUgjEAAFSCMQAAVIIxAABUgjEAAFTHEIxn5u4z8xsz89aZecvMfNe2/akzc93MvGH7eOiOY54yM1fPzDtm5kF7+QEAAGA3nHMM+9xcfe9a6/Uzc4fqdTPzyu22Z621nrFz55m5T/Xo6r7VXatXzcy91lof383CAQBgN93qiPFa6/q11uu3zz9Sva067xYOeXj1wrXWx9Zaf1xdXd1vN4oFAIC9clxzjGfmgurLqiu3TU+cmTfOzHNn5o7btvOq9+w47NqOEKRn5vKZuWpmrrrxxhuPv3IAANhFxxyMZ+Yzq5dUT1prfbh6dvUF1UXV9dUzj+fEa63nrLUuXmtdfO655x7PoQAAsOuOKRjPzGe0CcW/uNZ6adVa64a11sfXWp+ofq6/mi5xXXX3HYffbdsGAACnrWNZlWKqn6/ettb6qR3td9mx2yOrN2+fv7x69MzcdmbuUV1YvXb3SgYAgN13LKtSfGX1TdWbZuYN27YfrB4zMxdVq7qm+vaqtdZbZubF1VvbrGjxBCtSAABwurvVYLzW+p1qjrDpFbdwzNOqp51EXQAAcEody4gxAMAp8/wr331c+196yfl7VAlnG7eEBgCABGMAAKgEYwAAqARjAACoBGMAAKgEYwAAqARjAACoBGMAAKgEYwAAqARjAACo3BIaAE7Y8d66uNy+GE5nRowBACDBGAAAKsEYAAAqwRgAACrBGAAAKsEYAAAqwRgAACrBGAAAKsEYAAAqwRgAACrBGAAAKsEYAAAqwRgAACrBGAAAKsEYAAAqwRgAACrBGAAAKsEYAAAqwRgAACrBGAAAKsEYAAAqwRgAACrBGAAAKsEYAAAqwRgAACrBGAAAKsEYAAAqwRgAACrBGAAAKsEYAAAqwRgAACrBGAAAKsEYAAAqwRgAACrBGAAAKsEYAAAqwRgAACrBGAAAKsEYAAAqwRgAACrBGAAAKsEYAAAqwRgAACrBGAAAKsEYAAAqwRgAACrBGAAAKsEYAAAqwRgAACrBGAAAKsEYAACqOme/CwAAOJs9/8p3H/cxl15y/h5UghFjAABIMAYAgEowBgCA6hiC8czcfWZ+Y2beOjNvmZnv2rZ/7sy8cmbeuf16x237zMy/mpmrZ+aNM/Ple/0hAADgZB3LiPHN1feute5T3b96wszcp3py9eq11oXVq7evqx5SXbh9XF49e9erBgCAXXarwXitdf1a6/Xb5x+p3ladVz28et52t+dVj9g+f3j1C2vjNdXnzMxddrtwAADYTcc1x3hmLqi+rLqyuvNa6/rtpvdVd94+P696z47Drt22AQDAaeuYg/HMfGb1kupJa60P79y21lrVOp4Tz8zlM3PVzFx14403Hs+hAACw644pGM/MZ7QJxb+41nrptvmGQ1Mktl/fv22/rrr7jsPvtm37JGut56y1Ll5rXXzuueeeaP0AALArjmVViql+vnrbWuundmx6eXXZ9vll1ct2tH/zdnWK+1cf2jHlAgAATkvHckvor6y+qXrTzLxh2/aD1dOrF8/M46t3VY/abntF9dDq6uqm6lt2s2AAANgLtxqM11q/U81RNj/wCPuv6gknWRcAAJxS7nwHAAAJxgAAUAnGAABQCcYAAFAJxgAAUAnGAABQCcYAAFAJxgAAUAnGAABQCcYAAFAJxgAAUAnGAABQCcYAAFAJxgAAUAnGAABQCcYAAFAJxgAAUAnGAABQCcYAAFAJxgAAUAnGAABQCcYAAFAJxgAAUAnGAABQCcYAAFAJxgAAUAnGAABQCcYAAFAJxgAAUAnGAABQCcYAAFAJxgAAUAnGAABQ1Tn7XQAAG8+/8t3Htf+ll5y/R5UAnJ2MGAMAQIIxAABUgjEAAFSCMQAAVIIxAABUgjEAAFSCMQAAVNYxBoDTmvWt4dQxYgwAAAnGAABQCcYAAFAJxgAAUAnGAABQCcYAAFAJxgAAUAnGAABQCcYAAFAJxgAAULklNACcUdxCGk6cEWMAAEgwBgCASjAGAIBKMAYAgEowBgCASjAGAIDKcm3ACbIkFABnGiPGAACQYAwAAJVgDAAAlWAMAACVYAwAAJVgDAAAlWAMAACVdYwB4C8d7/rcwJnlVkeMZ+a5M/P+mXnzjranzsx1M/OG7eOhO7Y9ZWaunpl3zMyD9qpwAADYTccyleKK6sFHaH/WWuui7eMVVTNzn+rR1X23x/zMzHz6bhULAAB75VaD8Vrrt6sPHuP7Pbx64VrrY2utP66uru53EvUBAMApcTIX3z1xZt64nWpxx23bedV7duxz7bYNAABOaycajJ9dfUF1UXV99czjfYOZuXxmrpqZq2688cYTLAMAAHbHCQXjtdYNa62Pr7U+Uf1cfzVd4rrq7jt2vdu27Ujv8Zy11sVrrYvPPffcEykDAAB2zQkF45m5y46Xj6wOrVjx8urRM3PbmblHdWH12pMrEQAA9t6trmM8My+oHlDdaWaurX60esDMXFSt6prq26vWWm+ZmRdXb61urp6w1vr4nlQOAAC76FaD8VrrMUdo/vlb2P9p1dNOpigAADjV3BIaAAASjAEAoBKMAQCgEowBAKASjAEAoBKMAQCgEowBAKASjAEAoBKMAQCgEowBAKASjAEAoBKMAQCgEowBAKASjAEAoBKMAQCgEowBAKASjAEAoBKMAQCgqnP2uwA4Gz3/yncf1/6XXnL+HlUCABxixBgAABKMAQCgEowBAKASjAEAoBKMAQCgEowBAKASjAEAoBKMAQCgEowBAKASjAEAoBKMAQCgEowBAKASjAEAoBKMAQCgqnP2uwAA4OB4/pXvPq79L73k/D2qBHafEWMAAEgwBgCASjAGAIBKMAYAgEowBgCAyqoUwBnClfIAnCwjxgAAkGAMAACVYAwAAJVgDAAAlWAMAACVYAwAAJVgDAAAlWAMAACVYAwAAJVgDAAAlVtCA3AUx3ub7XKrbeBgM2IMAAAJxgAAUAnGAABQCcYAAFAJxgAAUAnGAABQCcYAAFAJxgAAUAnGAABQCcYAAFAJxgAAUAnGAABQCcYAAFDVOftdAMBB8Pwr333cx1x6yfl7UAkAe8WIMQAAJBgDAEAlGAMAQHUMc4xn5rnVw6r3r7W+aNv2udWLqguqa6pHrbX+ZGam+pfVQ6ubqsettV6/N6UDnN2Od96zOc8At+xYRoyvqB58WNuTq1evtS6sXr19XfWQ6sLt4/Lq2btTJgAA7K1bDcZrrd+uPnhY88Or522fP696xI72X1gbr6k+Z2busku1AgDAnjnROcZ3Xmtdv33+vurO2+fnVe/Zsd+127ZPMTOXz8xVM3PVjTfeeIJlAADA7jjpi+/WWqtaJ3Dcc9ZaF6+1Lj733HNPtgwAADgpJxqMbzg0RWL79f3b9uuqu+/Y727bNgAAOK2daDB+eXXZ9vll1ct2tH/zbNy/+tCOKRcAAHDaOpbl2l5QPaC608xcW/1o9fTqxTPz+Opd1aO2u7+izVJtV7dZru1b9qBmAADYdbcajNdajznKpgceYd9VPeFkiwIAgFPNne8AAKBjGDEGgNOFu/0Be8mIMQAAJBgDAEAlGAMAQCUYAwBAJRgDAEAlGAMAQCUYAwBAJRgDAEDlBh8AwAHnxi/sFiPGAACQYAwAAJVgDAAAlWAMAACVYAwAAJVVKQDOGsd75T7A2caIMQAAJBgDAEBlKgUAu8h0DeAgM2IMAAAJxgAAUJlKAQDASTreaVSXXnL+HlVycowYAwBAgjEAAFSCMQAAVIIxAABUgjEAAFRWpQCAU8pNUPbfmbKCArvPiDEAACQYAwBAZSoFcJry52Y4Nfy3Bn/FiDEAACQYAwBAJRgDAEAlGAMAQOXiOwCAW2Td47OHYAwA7BmrXhxMZ+v3zVQKAABIMAYAgMpUCqCz909mALCTEWMAAMiIMacBV/sCe8VfQ4DjYcQYAAASjAEAoBKMAQCgEowBAKASjAEAoBKMAQCgEowBAKASjAEAoBKMAQCgEowBAKASjAEAoBKMAQCgEowBAKASjAEAoBKMAQCgEowBAKCqc/a7AID98Pwr373fJQBwmjFiDAAACcYAAFCZSnHcjvfPr5decv4eVQIAwG4SjAEAdpFrGA4uUykAACDBGAAAKsEYAACqk5xjPDPXVB+pPl7dvNa6eGY+t3pRdUF1TfWotdafnFyZAACwt3ZjxPjr1loXrbUu3r5+cvXqtdaF1au3rwEA4LS2F1MpHl49b/v8edUj9uAcAACwq052ubZV/frMrOpn11rPqe681rp+u/191Z1P8hwAB5IlmwAOlpMNxl+11rpuZj6veuXMvH3nxrXW2obmTzEzl1eXV51/vptgAAAcKzcc2xsnNZVirXXd9uv7q1+q7lfdMDN3qdp+ff9Rjn3OWuvitdbF55577smUAQAAJ+2Eg/HM3H5m7nDoefX11Zurl1eXbXe7rHrZyRYJAAB77WSmUty5+qWZOfQ+z19r/eeZ+b3qxTPz+Opd1aNOvkwAANhbJxyM11p/VH3pEdr/v+qBJ1MUAACcau58BwAAnfyqFADHxNJlAJzujBgDAECCMQAAVIIxAABUgjEAAFQuvnNBEABwxpN3jo0RYwAASDAGAIDKVAoOoOP9c9Cll5x/Wp4DADi9CMYH3InMGRLiAAA+lakUAACQEWM4EEztAIC9Z8QYAAAyYnzasc4gAMD+EIzhDOQfWABw/EylAACABGMAAKhMpYBdYeoCABx8RowBACDBGAAAKsEYAAAqwRgAACoX33EWcGEcAHAsjBgDAECCMQAAVKZScAyOdyrCpZecv0eVAADsHSPGAACQYAwAAJVgDAAAlTnGZyXLlwEAfCojxgAAkGAMAACVqRTsAVM1AICDyIgxAAAkGAMAQCUYAwBAJRgDAEAlGAMAQCUYAwBAJRgDAEBlHeM9Z01fAICDwYgxAAAkGAMAQCUYAwBAJRgDAEAlGAMAQCUYAwBAJRgDAEAlGAMAQCUYAwBAJRgDAEAlGAMAQCUYAwBAJRgDAEAlGAMAQCUYAwBAJRgDAEAlGAMAQCUYAwBAJRgDAEAlGAMAQCUYAwBAJRgDAEAlGAMAQCUYAwBAJRgDAEAlGAMAQLWHwXhmHjwz75iZq2fmyXt1HgAA2A17Eoxn5tOrf1M9pLpP9ZiZuc9enAsAAHbDXo0Y36+6eq31R2utv6heWD18j84FAAAnba+C8XnVe3a8vnbbBgAAp6Vz9uvEM3N5dfn25Udn5h07Nt+p+sCpr+qMpk93nz7dXfpz9+nT3aU/d58+3X0Hok8fu7+n//yjbdirYHxddfcdr++2bftLa63nVM850sEzc9Va6+I9qu2spE93nz7dXfpz9+nT3aU/d58+3X369OTs1VSK36sunJl7zMxtqkdXL9+jcwEAwEnbkxHjtdbNM/PE6teqT6+eu9Z6y16cCwAAdsOezTFea72iesUJHn7EKRacFH26+/Tp7tKfu0+f7i79ufv06e7Tpydh1lr7XQMAAOw7t4QGAIBOg2A8M8+dmffPzJt3tH3uzLxyZt65/XrH/azxoDlKn/7jmXnLzHxiZlytehyO0p8/OTNvn5k3zswvzczn7GOJB85R+vTHtv35hpn59Zm5637WeNAcqU93bPvemVkzc6f9qO0gOsrP6FNn5rrtz+gbZuah+1njQXO0n9GZ+afb36dvmZmf2K/6Dpqj/Iy+aMfP5zUz84Z9LPFA2vdgXF1RPfiwtidXr15rXVi9evuaY3dFn9qnb66+ofrtU17NwXdFn9qfr6y+aK31JdUfVE851UUdcFf0qX36k2utL1lrXVT9cvUjp7qoA+6KPrVPm5m7V19fvftUF3TAXdER+rN61lrrou3jRK+jOVtd0WF9OjNf1+bOuF+61rpv9Yx9qOuguqLD+nOt9b8e+vmsXlK9dB/qOtD2PRivtX67+uBhzQ+vnrd9/rzqEaeypoPuSH261nrbWusdRzmEW3CU/vz1tdbN25evabNWN8foKH364R0vb1+5AOI4HOV3adWzqu9Pfx6XW+hPTtBR+vQ7qqevtT623ef9p7ywA+qWfkZnZqpHVS84pUWdAfY9GB/Fndda12+fv6+6834WA7fiW6tf3e8izgQz87SZeU+bmyIZMT5JM/Pw6rq11u/vdy1nkCdup/w81zS/XXGv6qtn5sqZ+a2Z+dv7XdAZ4qurG9Za79zvQg6a0zUY/6W1WTbDSAenpZn5oerm6hf3u5YzwVrrh9Zad2/Tn0/c73oOspm5XfWD+QfGbnp29QXVRdX11TP3tZozwznV51b3r/736sXb0U5OzmMyWnxCTtdgfMPM3KVq+9WfVjjtzMzjqodVj13WPdxtv1h9434XccB9QXWP6vdn5po2031ePzN/Y1+rOsDWWjestT6+1vpE9XPV/fa7pjPAtdVL18Zrq09ULhI9CTNzTptril6037UcRKdrMH55ddn2+WXVy/axFvgUM/PgNvM2/5e11k37Xc+ZYGYu3PHy4dXb96uWM8Fa601rrc9ba12w1rqgTQD58rXW+/a5tAPr0IDN1iPbXNTMyfm/q6+rmpl7VbepPrCfBZ0B/n719rXWtftdyEG078F4Zl5Q/W71hTNz7cw8vnp69Q9m5p1tvsFP388aD5oj9enMPHJmrq3+TvUrM/Nr+1vlwXGUn9Gfru5QvXK7LM6/3dciD5ij/Xc/M2+emTe2WUXhu/a1yAPmKH3KCTpKf/7EzLxp+zP6ddV372uRB8xR+vS51d/cLjn2wuoyf4E7Nrfw3/yjM43ihLnzHQAAdBqMGAMAwOlAMAYAgARjAACoBGMAAKgEYwAAqARjAACoBGOAU2pmPrrj8YmZ+bMdrx97imp4wHZdcwB2OGe/CwA4m6y1PvPQ8+2tmr9trfWq43mPmTlnrXXzbtcGcLYzYgxwGpiZ+83M787Mf5+Z62fmp2fmNju2r5l5wvaOoO/ctn3/dt/3zsy3bfe553bbbWfmGTPz7pm5YWb+7cz8tZm5ffWr1V13jFTfdV8+NMBpRjAGOD18vM0thu/U5tbtD6y+87B9HlFdUt1nZh5cfU/196t7Vg84bN+nV/eqLtpuP6/6kbXWn1YPqd671vrM7eO9e/B5AA4cwRjgNLDWet1a6zVrrZvXWtdUP1t97WG7/fO11gfXWn9WPar692utt6y1bqqeeminmZnq8uq7t/t/pPpn1aNPxWcBOKjMMQY4DczMvaqfqi6ubtfm9/PrDtvtPTue37W66ijbzt2+x+s2GXlziurTd7FkgDOOEWOA08Ozq7dXF661Pqv6wTZhdqe14/n11d12vL77jucfqP6suu9a63O2j8/eceHfzvcBYEswBjg93KH6cPXRmflb1Xfcyv4vrr5lZu49M7erfvjQhrXWJ6qfq541M59XNTPnzcyDtrvcUP31mfns3f4QAAeZYAxwevi+6tLqI21C7Ytuaee11q9W/6r6jerq6jXbTR/bfv2BQ+0z8+HqVdUXbo99e/WC6o+2q2BYlQKgmrX8RQ3goJuZe1dvrm5rjWOAE2PEGOCAmplHbtcrvmP149V/EooBTpxgDHBwfXv1/uoP26yDfGvzkgG4BaZSAABARowBAKASjAEAoBKMAQCgEowBAKASjAEAoBKMAQCgqv8fX6qa8hUgg64AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(12,8))\n",
    "sns.distplot( np.log1p(train_df[\"target\"].values), bins=50, kde=False)\n",
    "plt.xlabel('Target', fontsize=12)\n",
    "plt.title(\"Log of Target Histogram\", fontsize=14)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "So this above figure represents more meaningfull view of the Target values.\n",
    "\n",
    "# Checking for missing / null values in data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All Features in Train data with NaN Values = 0\n"
     ]
    }
   ],
   "source": [
    "print(\"All Features in Train data with NaN Values =\", str(train_df.columns[train_df.isnull().sum() != 0].size) )\n",
    "# print(\"All Features in Test data with NaN Values =\", str(test_df.columns[train_df.isnull().sum() != 0].size) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Remove constant columns from data with the help of Constant or Variable Standard Deviation / Variance\n",
    "\n",
    "Variance defines how the data is spread across the mean. It is calulcated by taking the square of difference of every value from the mean value for a variable. One of the statistical intution is that if the feature variance is very less, then the feature will add less contribution to the model. However, I donot follow this blindly as most of the deep learning and boosting modles are robust to such issues. But, variance can give an idea about the features which can be discarded. Atleast the features having zero variance can be discarded because they are essently constant features. (again, these features might not be significant when considered individually, but can be useful in the row wise aggregated features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_df rows and columns after removing constant columns:  (4459, 4737)\n",
      "Following `256` Constant Column\n",
      " are removed\n",
      "['d5308d8bc', 'c330f1a67', 'eeac16933', '7df8788e8', '5b91580ee', '6f29fbbc7', '46dafc868', 'ae41a98b6', 'f416800e9', '6d07828ca', '7ac332a1d', '70ee7950a', '833b35a7c', '2f9969eab', '8b1372217', '68322788b', '2288ac1a6', 'dc7f76962', '467044c26', '39ebfbfd9', '9a5ff8c23', 'f6fac27c8', '664e2800e', 'ae28689a2', 'd87dcac58', '4065efbb6', 'f944d9d43', 'c2c4491d5', 'a4346e2e2', '1af366d4f', 'cfff5b7c8', 'da215e99e', '5acd26139', '9be9c6cef', '1210d0271', '21b0a54cb', 'da35e792b', '754c502dd', '0b346adbd', '0f196b049', 'b603ed95d', '2a50e001c', '1e81432e7', '10350ea43', '3c7c7e24c', '7585fce2a', '64d036163', 'f25d9935c', 'd98484125', '95c85e227', '9a5273600', '746cdb817', '6377a6293', '7d944fb0c', '87eb21c50', '5ea313a8c', '0987a65a1', '2fb7c2443', 'f5dde409b', '1ae50d4c3', '2b21cd7d8', '0db8a9272', '804d8b55b', '76f135fa6', '7d7182143', 'f88e61ae6', '378ed28e0', 'ca4ba131e', '1352ddae5', '2b601ad67', '6e42ff7c7', '22196a84c', '0e410eb3d', '992e6d1d3', '90a742107', '08b9ec4ae', 'd95203ded', '58ad51def', '9f69ae59f', '863de8a31', 'be10df47c', 'f006d9618', 'a7e39d23d', '5ed0abe85', '6c578fe94', '7fa4fcee9', '5e0571f07', 'fd5659511', 'e06b9f40f', 'c506599c8', '99de8c2dc', 'b05f4b229', '5e0834175', 'eb1cc0d9c', 'b281a62b9', '00fcf67e4', 'e37b65992', '2308e2b29', 'c342e8709', '708471ebf', 'f614aac15', '15ecf7b68', '3bfe540f1', '7a0d98f3c', 'e642315a5', 'c16d456a7', '0c9b5bcfa', 'b778ab129', '2ace87cdd', '697a566f0', '97b1f84fc', '34eff114b', '5281333d7', 'c89f3ba7e', 'cd6d3c7e6', 'fc7c8f2e8', 'abbbf9f82', '24a233e8f', '8e26b560e', 'a28ac1049', '504502ce1', 'd9a8615f3', '4efd6d283', '34cc56e83', '93e98252a', '2b6cef19e', 'c7f70a49b', '0d29ab7eb', 'e4a0d39b7', 'a4d1a8409', 'bc694fc8f', '3a36fc3a2', '4ffba44d3', '9bfdec4bc', '66a866d2f', 'f941e9df7', 'e7af4dbf3', 'dc9a54a3e', '748168a04', 'bba8ce4bb', 'ff6f62aa4', 'b06fe66ba', 'ae87ebc42', 'f26589e57', '963bb53b1', 'a531a4bf0', '9fc79985d', '9350d55c1', 'de06e884c', 'fc10bdf18', 'e0907e883', 'c586d79a1', 'e15e1513d', 'a06067897', '643e42fcb', '217cd3838', '047ebc242', '9b6ce40cf', '3b2c972b3', '17a7bf25a', 'c9028d46b', '9e0473c91', '6b041d374', '783c50218', '19122191d', 'ce573744f', '1c4ea481e', 'fbd6e0a0b', '69831c049', 'b87e3036b', '54ba515ee', 'a09ba0b15', '90f77ec55', 'fb02ef0ea', '3b0cccd29', 'fe9ed417c', '589e8bd6f', '17b5a03fd', '80e16b49a', 'a3d5c2c2a', '1bd3a4e92', '611d81daa', '3d7780b1c', '113fd0206', '5e5894826', 'cb36204f9', 'bc4e3d600', 'c66e2deb0', 'c25851298', 'a7f6de992', '3f93a3272', 'c1b95c2ec', '6bda21fee', '4a64e56e7', '943743753', '20854f8bf', 'ac2e428a9', '5ee7de0be', '316423a21', '2e52b0c6a', '8bdf6bc7e', '8f523faf2', '4758340d5', '8411096ec', '9678b95b7', 'a185e35cc', 'fa980a778', 'c8d90f7d7', '080540c81', '32591c8b4', '5779da33c', 'bb425b41e', '01599af81', '1654ab770', 'd334a588e', 'b4353599c', '51b53eaec', '2cc0fbc52', '45ffef194', 'c15ac04ee', '5b055c8ea', 'd0466eb58', 'a80633823', 'a117a5409', '7ddac276f', '8c32df8b3', 'e5649663e', '6c16efbb8', '9118fd5ca', 'ca8d565f1', '16a5bb8d2', 'fd6347461', 'f5179fb9c', '97428b646', 'f684b0a96', 'e4b2caa9f', '2c2d9f267', '96eb14eaf', 'cb2cb460c', '86f843927', 'ecd16fc60', '801c6dc8e', 'f859a25b8', 'ae846f332', '2252c7403', 'fb9e07326', 'd196ca1fd', 'a8e562e8e', 'eb6bb7ce1', '5beff147e', '52b347cdc', '4600aadcf', '6fa0b9dab', '43d70cc4d', '408021ef8', 'e29d22b59']\n"
     ]
    }
   ],
   "source": [
    "const_columns_to_remove = []\n",
    "\n",
    "for col in train_df.columns:\n",
    "    if col != 'ID' and col != 'target':\n",
    "        if train_df[col].std() == 0:\n",
    "            const_columns_to_remove.append(col)\n",
    "\n",
    "# Now remove that array of const columns from the data\n",
    "train_df.drop(const_columns_to_remove, axis=1, inplace=True)\n",
    "\n",
    "test_df.drop(const_columns_to_remove, axis=1, inplace=True)\n",
    "\n",
    "# Print to see the reduction of columns\n",
    "print('train_df rows and columns after removing constant columns: ', train_df.shape)\n",
    "\n",
    "print('Following `{}` Constant Column\\n are removed'.format(len(const_columns_to_remove)))\n",
    "print(const_columns_to_remove)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4459, 4737)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "---\n",
    "\n",
    "# Remove Duplicate Columns\n",
    "\n",
    "**I will be using the duplicated() function of pandas - here's how it works:**\n",
    "\n",
    "Suppose the columns of the data frame are `['alpha','beta','alpha']`\n",
    "\n",
    "`df.columns.duplicated()` returns a boolean array: a `True` or `False` for each column. If it is `False` then the column name is unique up to that point, if it is `True` then the column name is duplicated earlier. For example, using the given example, the returned value would be `[False,False,True]`. \n",
    "\n",
    "`Pandas` allows one to index using boolean values whereby it selects only the `True` values. Since we want to keep the unduplicated columns, we need the above boolean array to be flipped (ie `[True, True, False] = ~[False,False,True]`)\n",
    "\n",
    "Finally, `df.loc[:,[True,True,False]]` selects only the non-duplicated columns using the aforementioned indexing capability. \n",
    "\n",
    "**Note**: the above only checks columns names, *not* column values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[False False False ... False False False]\n"
     ]
    }
   ],
   "source": [
    "print(train_df.columns.duplicated())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train rows and columns after removing duplicate columns:  (4459, 4737)\n"
     ]
    }
   ],
   "source": [
    "train_df = train_df.loc[:,~train_df.columns.duplicated()]\n",
    "print('Train rows and columns after removing duplicate columns: ', train_df.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### So from above I can see there's no change in number of columns - hence there is no duplicated columns\n",
    "\n",
    "## Handling Sparse data\n",
    "\n",
    "**What is Sparse data**\n",
    "\n",
    "As an example, let's say that we are collecting data from a device which has 12 sensors. And you have collected data for 10 days.\n",
    "\n",
    "The data you have collected is as follows:\n",
    "\n",
    "[![enter image description here][1]][1]\n",
    "\n",
    "\n",
    "The above is an example of sparse data because most of the sensor outputs are zero. Which means those sensors are functioning properly but the actual reading is zero. Although this matrix has high dimensional data (12 axises) it can be said that it contains less information.\n",
    "\n",
    "So basically, sparse data means that there are many gaps present in the data being recorded. For example, in the case of the sensor mentioned above, the sensor may send a signal only when the state changes, like when there is a movement of the door in a room. This data will be obtained intermittently because the door is not always moving. Hence, this is sparse data.\n",
    "\n",
    "  [1]: https://i.stack.imgur.com/Af5IH.png\n",
    "\n",
    "\n",
    "First lets have a look at or train_df data again, that how much of sparse data is there. And as we can see there are plenty of '0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>target</th>\n",
       "      <th>48df886f9</th>\n",
       "      <th>0deb4b6a8</th>\n",
       "      <th>34b15f335</th>\n",
       "      <th>a8cb14b00</th>\n",
       "      <th>2f0771a37</th>\n",
       "      <th>30347e683</th>\n",
       "      <th>d08d1fbe3</th>\n",
       "      <th>6ee66e115</th>\n",
       "      <th>...</th>\n",
       "      <th>3ecc09859</th>\n",
       "      <th>9281abeea</th>\n",
       "      <th>8675bec0b</th>\n",
       "      <th>3a13ed79a</th>\n",
       "      <th>f677d4d13</th>\n",
       "      <th>71b203550</th>\n",
       "      <th>137efaa80</th>\n",
       "      <th>fb36b89d9</th>\n",
       "      <th>7e293fbaf</th>\n",
       "      <th>9fc776466</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>000d6aaf2</td>\n",
       "      <td>38000000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>000fbd867</td>\n",
       "      <td>600000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0027d6b71</td>\n",
       "      <td>10000000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0028cbf45</td>\n",
       "      <td>2000000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>002a68644</td>\n",
       "      <td>14400000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 4737 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          ID      target  48df886f9  0deb4b6a8  34b15f335  a8cb14b00  \\\n",
       "0  000d6aaf2  38000000.0        0.0          0        0.0          0   \n",
       "1  000fbd867    600000.0        0.0          0        0.0          0   \n",
       "2  0027d6b71  10000000.0        0.0          0        0.0          0   \n",
       "3  0028cbf45   2000000.0        0.0          0        0.0          0   \n",
       "4  002a68644  14400000.0        0.0          0        0.0          0   \n",
       "\n",
       "   2f0771a37  30347e683  d08d1fbe3  6ee66e115  ...  3ecc09859  9281abeea  \\\n",
       "0          0          0          0          0  ...        0.0        0.0   \n",
       "1          0          0          0          0  ...        0.0        0.0   \n",
       "2          0          0          0          0  ...        0.0        0.0   \n",
       "3          0          0          0          0  ...        0.0        0.0   \n",
       "4          0          0          0          0  ...        0.0        0.0   \n",
       "\n",
       "   8675bec0b  3a13ed79a  f677d4d13  71b203550  137efaa80  fb36b89d9  \\\n",
       "0        0.0          0          0          0          0          0   \n",
       "1        0.0          0          0          0          0          0   \n",
       "2        0.0          0          0          0          0          0   \n",
       "3        0.0          0          0          0          0          0   \n",
       "4        0.0          0          0          0          0          0   \n",
       "\n",
       "   7e293fbaf  9fc776466  \n",
       "0          0          0  \n",
       "1          0          0  \n",
       "2          0          0  \n",
       "3          0          0  \n",
       "4          0          0  \n",
       "\n",
       "[5 rows x 4737 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "---\n",
    "\n",
    "# Check and handle total memory of data\n",
    "\n",
    "`get_dummies` pandas function converts categorical variables into indicator variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Memory usage is  168.978 MB\n",
      "(4459, 4737)\n"
     ]
    }
   ],
   "source": [
    "def print_memory_usage_of_df(df):\n",
    "    bytes_per_mb = 0.000001\n",
    "    memory_usage = round(df.memory_usage().sum() * bytes_per_mb, 3)\n",
    "    print('Memory usage is ', str(memory_usage) + \" MB\")\n",
    "\n",
    "print_memory_usage_of_df(train_df)\n",
    "print(train_df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4459, 9195)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummy_encoded_train_df = pd.get_dummies(train_df)\n",
    "dummy_encoded_train_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Memory usage is  188.825 MB\n"
     ]
    }
   ],
   "source": [
    "print_memory_usage_of_df(dummy_encoded_train_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "We see that the memory usage of the dummy_encoded_train_df data  frame is larger compared to the original, because now the number of columns have increased in the data frame.\n",
    "\n",
    "##### So lets apply `sparse=True` if it reduces the memory-usages to some extent.\n",
    "\n",
    "This parameter `sparse` defaults to False. If True the encoded columns are returned as **SparseArray**. By setting `sparse=True` we create a sparse data frame directly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4459, 9195)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummy_encoded_sparse_train_df = pd.get_dummies(train_df, sparse=True)\n",
    "dummy_encoded_sparse_train_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Memory usage is  168.965 MB\n"
     ]
    }
   ],
   "source": [
    "print_memory_usage_of_df(dummy_encoded_sparse_train_df)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "But looks like in this case the reduction in memory_size was not a huge amount. So lets try some other alternative\n",
    "\n",
    "## [Pandas Sparse Structures](https://pandas.pydata.org/pandas-docs/stable/user_guide/sparse.html#sparse-data-structures)\n",
    "\n",
    "Pandas provides data structures for efficient storage of sparse data. In these structures, zero values (or any other specified value) are not actually stored in the array. Rather, you can view these objects as being “compressed” where any data matching a specific value (NaN / missing value, though any value can be chosen, including 0) is omitted. The compressed values are not actually stored in the array.\n",
    "\n",
    "Storing only the non-zero values and their positions is a common technique in storing sparse data sets.\n",
    "\n",
    "This hugely reduces the memory usage of our data set and “compress” the data frame.\n",
    "\n",
    "In our example, we will convert the one-hot encoded columns into SparseArrays, which are 1-d arrays where only non-zero values are stored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def convert_df_to_sparse_array(df, exclude_columns=[]):\n",
    "    df = df.copy()\n",
    "    exclude_columns = set(exclude_columns)\n",
    "\n",
    "    for (column_name, column_data) in df.iteritems():\n",
    "        if column_name in exclude_columns:\n",
    "            continue\n",
    "        df[column_name] = pd.SparseArray(column_data.values, dtype='uint8')\n",
    "\n",
    "    return df\n",
    "\n",
    "# Now convert our earlier dummy_encoded_train_df with above function and check memory_size\n",
    "\n",
    "# train_data_post_conversion_to_sparse_array = convert_df_to_sparse_array(dummy_encoded_train_df)\n",
    "# print('Sparse Array Train_DF rows and columns: ', train_data_post_conversion_to_sparse_array.shape)\n",
    "# print_memory_usage_of_df(train_data_post_conversion_to_sparse_array)\n",
    "\n",
    "# Commenting the above out - for running the Notebook faster during my development \n",
    "# Because df.iteritems() will take a huge time to process the data - see warning below"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**We see the that the memory_usage is substantially reduced now**\n",
    "\n",
    "### A warning on using df.iteritems()\n",
    "\n",
    "The df.iteritems() iterates over columns and not rows. Generally iteration over dataframes is an anti-pattern, and something we should avoid, unless you want to get used to a lot of waiting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Sparse Data Removal (Following simpler plain-vanilla technique)\n",
    "\n",
    "## For this notebook, I will go with the easier approach to handle sparse data - which is just to drop it from the dataframe\n",
    "\n",
    "like below code, I will do this for the sake of running this notebook faster for now\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def drop_sparse_from_train_test(train, test):\n",
    "    column_list_to_drop_data_from = [x for x in train.columns if not x in ['ID','target']]\n",
    "    for f in column_list_to_drop_data_from:\n",
    "        if len(np.unique(train[f]))<2:\n",
    "            train.drop(f, axis=1, inplace=True)\n",
    "            test.drop(f, axis=1, inplace=True)\n",
    "    return train, test\n",
    "\n",
    "train_df, test_df = drop_sparse_from_train_test(train_df, test_df)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "---\n",
    "\n",
    "# Split data into Train and Test for Model Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "X_train = train_df.drop(['ID', 'target'], axis=1)\n",
    "\n",
    "y_train = np.log1p(train_df['target'].values)\n",
    "# Note for Target I am using log1p => This mathematical function calculates natural logarithmic value of x+1 where x belongs to all the input array elements.\n",
    "\n",
    "X_test_original = test_df.drop('ID', axis=1)\n",
    "\n",
    "X_train_split, X_validation, y_train_split, y_validation = train_test_split(X_train, y_train, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "---\n",
    "\n",
    "# LightGBM Model Training\n",
    "\n",
    "### Some Fundamentals of LightGBM Model\n",
    "\n",
    "It is a **gradient boosting** model that makes use of tree based learning algorithms and is considered to be a fast processing algorithm.\n",
    "\n",
    "While in other Gradient Boosting algorithms, trees grow horizontally, in LightGBM the Trees grow vertically, meaning it grows leaf-wise while the other algorithms grow level-wise. \n",
    "\n",
    "LightGBM chooses the leaf with large loss to grow and it can lower down more loss than a level wise algorithm when growing in the same leaf.\n",
    "\n",
    "![img](https://i.imgur.com/pzOP2Lb.png)\n",
    "\n",
    "[Source of Image](https://www.analyticsvidhya.com/blog/2017/06/which-algorithm-takes-the-crown-light-gbm-vs-xgboost/)\n",
    "\n",
    "Light GBM is prefixed as Light because of its high speed. Light GBM can handle the large size of data and takes lower memory to run.\n",
    "\n",
    "Another reason why Light GBM is so popular is because it focuses on accuracy of results. LGBM also supports GPU learning and thus data scientists are widely using LGBM for data science application development.\n",
    "\n",
    "**Leaf growth technique in LightGBM**\n",
    "\n",
    "LightGBM uses leaf-wise (best-first) tree growth. It chooses to grow the leaf that minimizes the loss, allowing a growth of an imbalanced tree. Because it doesn’t grow level-wise, but leaf-wise, over-fitting can happen when data is small. In these cases, it is important to control the tree depth.\n",
    "\n",
    "### When to use LightGBM ?\n",
    "\n",
    "LightGBM is not preferred for a small volume of datasets as it can easily overfit small data due to its sensitivity. Hence, it generally advised for data having more than 10,000+ rows, though there is no fixed threshold that helps in deciding the usage of LightGBM.\n",
    "\n",
    "## What are LightGBM Parameters?\n",
    "\n",
    "While, LightGBM has more than 100 parameters that are given in the [documentation of LightGBM](https://github.com/microsoft/LightGBM), let's checkout the most important ones.\n",
    "\n",
    "#### Control Parameters\n",
    "\n",
    "**Max depth**: It gives the depth of the tree and also controls the overfitting of the model. If you feel your model is getting overfitted lower down the max depth.\n",
    "\n",
    "**Min_data_in_leaf**: Leaf minimum number of records also used for controlling overfitting of the model.\n",
    "\n",
    "**Feature_fraction**: It decides the randomly chosen parameter in every iteration for building trees. If it is 0.7 then it means 70% of the parameter would be used.\n",
    "\n",
    "**Bagging_fraction**: It checks for the data fraction that will be used in every iteration. Often, used to increase the training speed and avoid overfitting.\n",
    "\n",
    "**Early_stopping_round**: If the metric of the validation data does show any improvement in last early_stopping_round rounds. It will lower the imprudent iterations.\n",
    "\n",
    "**Lambda**: It states regularization. Its values range from 0 to 1.\n",
    "\n",
    "**Min_gain_to_split**: Used to control the number of splits in the tree.\n",
    "\n",
    "### Core Parameters\n",
    "\n",
    "**Task**: It tells about the task that is to be performed on the data. It can either train on the data or prediction on the data.\n",
    "\n",
    "**Application**: This parameter specifies whether to do regression or classification. LightGBM default parameter for application is regression.\n",
    "\n",
    "**Binary**: It is used for binary classification.\n",
    "\n",
    "**Multiclass**: It is used for multiclass classification problems.\n",
    "\n",
    "**Regression**: It is used for doing regression.\n",
    "\n",
    "**Boosting**: It specifies the algorithm type.\n",
    "\n",
    "**rf** :  Used for Random Forest.\n",
    "\n",
    "**Goss**: Gradient-based One Side Sampling.\n",
    "\n",
    "**Num_boost_round**: It tells about the boosting iterations.\n",
    "\n",
    "**Learning_rate**: The role of learning rate is to power the magnitude of the changes in the approximate that gets updated from each tree’s output. It determines the contribution of each tree on the final outcome and controls how quickly the algorithm proceeds down the gradient descent (learns); Typical values between 0.001–0.3. Smaller values make the model robust to the specific characteristics of each individual tree, thus allowing it to generalize well. Smaller values also make it easier to stop prior to overfitting; however, they increase the risk of not reaching the optimum with a fixed number of trees and are more computationally demanding. This hyperparameter is also called shrinkage. Generally, the smaller this value, the more accurate the model can be but also will require more trees in the sequence.\n",
    "\n",
    "**Num_leaves**: It gives the total number of leaves that would be present in a full tree, default value: 31\n",
    "\n",
    "### Metric Parameter\n",
    "\n",
    "It takes care of the loss while building the model. Some of them are stated below for classification as well as regression.\n",
    "\n",
    "**Mae**: Mean absolute error.\n",
    "\n",
    "**Mse**: Mean squared error.\n",
    "\n",
    "**Binary_logloss**: Binary Classification loss.\n",
    "\n",
    "**Multi_logloss**: Multi Classification loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def light_gbm_model_run(train_x, train_y, validation_x, validation_y, test_x):\n",
    "    params = {\n",
    "        \"objective\" : \"regression\",\n",
    "        \"metric\" : \"rmse\",\n",
    "        \"num_leaves\" : 100,\n",
    "        \"learning_rate\" : 0.001,\n",
    "        \"bagging_fraction\" : 0.6,\n",
    "        \"feature_fraction\" : 0.6,\n",
    "        \"bagging_frequency\" : 6,\n",
    "        \"bagging_seed\" : 42,\n",
    "        \"verbosity\" : -1,\n",
    "        \"seed\": 42\n",
    "    }\n",
    "    \n",
    "    # Given its a regression case, I am using the RMSE as the metric.\n",
    "\n",
    "    lgbm_train = lgb.Dataset(train_x, label=train_y)\n",
    "    \n",
    "    lg_validation = lgb.Dataset(validation_x, label=validation_y)\n",
    "    \n",
    "    evals_result_lgbm = {}\n",
    "\n",
    "    model_light_gbm = lgb.train(params, lgbm_train, 5000,\n",
    "                      valid_sets=[lgbm_train, lg_validation],\n",
    "                      early_stopping_rounds=100,\n",
    "                      verbose_eval=150,\n",
    "                      evals_result=evals_result_lgbm )\n",
    "\n",
    "    pred_test_light_gbm = np.expm1(model_light_gbm.predict(test_x, num_iteration=model_light_gbm.best_iteration ))\n",
    "\n",
    "    return pred_test_light_gbm, model_light_gbm, evals_result_lgbm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: bagging_frequency\n",
      "Training until validation scores don't improve for 100 rounds\n",
      "[150]\ttraining's rmse: 1.66447\tvalid_1's rmse: 1.63998\n",
      "[300]\ttraining's rmse: 1.57647\tvalid_1's rmse: 1.59273\n",
      "[450]\ttraining's rmse: 1.49838\tvalid_1's rmse: 1.55444\n",
      "[600]\ttraining's rmse: 1.42899\tvalid_1's rmse: 1.5229\n",
      "[750]\ttraining's rmse: 1.36605\tvalid_1's rmse: 1.49779\n",
      "[900]\ttraining's rmse: 1.30903\tvalid_1's rmse: 1.47722\n",
      "[1050]\ttraining's rmse: 1.25715\tvalid_1's rmse: 1.46109\n",
      "[1200]\ttraining's rmse: 1.20994\tvalid_1's rmse: 1.44797\n",
      "[1350]\ttraining's rmse: 1.16675\tvalid_1's rmse: 1.43745\n",
      "[1500]\ttraining's rmse: 1.12708\tvalid_1's rmse: 1.42923\n",
      "[1650]\ttraining's rmse: 1.09046\tvalid_1's rmse: 1.42251\n",
      "[1800]\ttraining's rmse: 1.05679\tvalid_1's rmse: 1.41778\n",
      "[1950]\ttraining's rmse: 1.02548\tvalid_1's rmse: 1.41419\n",
      "[2100]\ttraining's rmse: 0.995928\tvalid_1's rmse: 1.4113\n",
      "[2250]\ttraining's rmse: 0.96824\tvalid_1's rmse: 1.40915\n",
      "[2400]\ttraining's rmse: 0.941909\tvalid_1's rmse: 1.40749\n",
      "[2550]\ttraining's rmse: 0.917232\tvalid_1's rmse: 1.40622\n",
      "[2700]\ttraining's rmse: 0.893893\tvalid_1's rmse: 1.40514\n",
      "[2850]\ttraining's rmse: 0.87174\tvalid_1's rmse: 1.40444\n",
      "[3000]\ttraining's rmse: 0.850888\tvalid_1's rmse: 1.40394\n",
      "[3150]\ttraining's rmse: 0.831132\tvalid_1's rmse: 1.40382\n",
      "[3300]\ttraining's rmse: 0.812395\tvalid_1's rmse: 1.40365\n",
      "[3450]\ttraining's rmse: 0.794587\tvalid_1's rmse: 1.40377\n",
      "Early stopping, best iteration is:\n",
      "[3351]\ttraining's rmse: 0.806206\tvalid_1's rmse: 1.4036\n"
     ]
    }
   ],
   "source": [
    "# Training and output of LightGBM Model\n",
    "predicted_y_test_org_light_gbm, model_lgbm, evals_result = light_gbm_model_run(X_train_split, y_train_split, X_validation, y_validation, X_test_original)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "\n",
    "## Hyper-Parameter Tuning in LightGBM\n",
    "\n",
    "Parameter Tuning is an important part that is usually done by data scientists to achieve a good accuracy, fast result and to deal with overfitting. Let us see quickly some of the parameter tuning you can do for better results.\n",
    "\n",
    "**num_leaves**: This parameter is responsible for the complexity of the model. I normally start by trying values in the range [10,100]. But if you have a solid heuristic to choose tree depth you can always use it and set num_leaves to 2^tree_depth - 1\n",
    "\n",
    "[LightGBM Documentation](https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html) says in respect -\n",
    "This is the main parameter to control the complexity of the tree model. Theoretically, we can set num_leaves = 2^(max_depth) to obtain the same number of leaves as depth-wise tree. However, this simple conversion is not good in practice. The reason is that a leaf-wise tree is typically much deeper than a depth-wise tree for a fixed number of leaves. Unconstrained depth can induce over-fitting. Thus, when trying to tune the num_leaves, we should let it be smaller than 2^(max_depth). For example, when the max_depth=7 the depth-wise tree can get good accuracy, but setting num_leaves to 127 may cause over-fitting, and setting it to 70 or 80 may get better accuracy than depth-wise.\n",
    "\n",
    "**Min_data_in_leaf**: Assigning bigger value to this parameter can result in underfitting of the model. Giving it a value of 100 or 1000 is sufficient for a large dataset.\n",
    "\n",
    "**Max_depth**: Controls the depth of the individual trees. Typical values range from a depth of 3–8 but it is not uncommon to see a tree depth of 1. Smaller depth trees are computationally efficient (but require more trees); however, higher depth trees allow the algorithm to capture unique interactions but also increase the risk of over-fitting. Larger training data sets are more tolerable to deeper trees.\n",
    "\n",
    "**num_iterations**: Num_iterations specifies the number of boosting iterations (trees to build). The more trees you build the more accurate your model can be at the cost of:\n",
    "    - Longer training time\n",
    "    - Higher chance of over-fitting\n",
    "So typically start with a lower number of trees to build a baseline and increase it later when you want to squeeze the last % out of your model.\n",
    "\n",
    "It is recommended to use smaller `learning_rate` with larger `num_iterations`. Also, we should use `early_stopping_rounds` if we go for higher `num_iterations` to stop your training when it is not learning anything useful.\n",
    "\n",
    "**early_stopping_rounds** - \"early stopping\" refers to stopping the training process if the model's performance on a given validation set does not improve for several consecutive iterations. This parameter will stop training if the validation metric is not improving after the last early stopping round. It should be defined in pair with a number of iterations. If we set it too large we increase the chance of over-fitting. **The rule of thumb is to have it at 10% of your `num_iterations`**.\n",
    "\n",
    "\n",
    "#### So for my above implementation of LightGBM, initially for two of the LightGBM parameters as below got me a score of 1.47953 (in Kaggle Public Board)\n",
    "\n",
    "```\n",
    "\"num_leaves\" : 40,\n",
    "\"learning_rate\" : 0.004,\n",
    "```\n",
    "\n",
    "And now if I only tune these parameters as below\n",
    "\n",
    "```\n",
    "\"num_leaves\" : 100,\n",
    "\"learning_rate\" : 0.001,\n",
    "```\n",
    "#### I got my score very very slightly updated to 1.4714 (in Kaggle Public Board)\n",
    "\n",
    "I also tried the below one (keeping 'num_leaves' at 70 to avoid over-fitting)\n",
    "\n",
    "```\n",
    "\"num_leaves\" : 70,\n",
    "\"learning_rate\" : 0.001,\n",
    "```\n",
    "With this - I got a score of 1.47234 (in Kaggle Public Board)\n",
    "\n",
    "## Features Importance in LightGBM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        feature  split      gain\n",
      "4135  f190486d6   6790  6.931083\n",
      "2378  58e2e02e6   6115  4.920505\n",
      "3470  eeb9cd3aa   5559  4.045366\n",
      "2617  9fd594eec   4122  3.176494\n",
      "4025  15ace8c9f   5251  3.011257\n",
      "8     20aa07010   3674  1.897522\n",
      "3576  58232a6fb   3506  1.457728\n",
      "834   6eef030c1   3736  1.278441\n",
      "1459  b43a7cfd5   4265  1.250970\n",
      "2690  fb0f5dbfe   4565  1.124296\n",
      "3666  491b9ee45   2672  1.045985\n",
      "1484  024c577b9   2917  1.013810\n",
      "4348  1702b5bf0   3069  0.937800\n",
      "4190  f74e8f13d   3565  0.930632\n",
      "566   66ace2992   3089  0.887587\n",
      "4513  c47340d97   3228  0.879622\n",
      "3727  d6bb78916   3369  0.867906\n",
      "2082  58e056e12   3565  0.833137\n",
      "863   fc99f9426   2466  0.772764\n",
      "3816  adb64ff71   2670  0.747453\n",
      "4033  5c6487af1   2540  0.728116\n",
      "4458  190db8488   3137  0.724895\n",
      "3224  ced6a7e91   1676  0.658301\n",
      "3796  ed8ff54b5    757  0.643408\n",
      "2137  241f0f867   2461  0.641603\n",
      "3872  2288333b4   1234  0.626267\n",
      "537   26fc93eb7   2553  0.604212\n",
      "3891  50e4f96cf   1138  0.589821\n",
      "2619  fb387ea33    992  0.577047\n",
      "828   6786ea46d    827  0.571777\n",
      "1380  6cf7866c1    993  0.557521\n",
      "4346  e176a204a   2328  0.533167\n",
      "34    87ffda550   1349  0.528923\n",
      "2214  1931ccfdd   2005  0.503001\n",
      "853   bc70cbc26   1000  0.501886\n",
      "4321  c5a231d81   2368  0.500888\n",
      "3474  324921c7b   2478  0.485531\n",
      "3784  70feb1494   1912  0.485520\n",
      "213   186b87c05    347  0.483651\n",
      "3010  703885424   1951  0.476819\n",
      "2934  91f701ba2   2038  0.436217\n",
      "4022  62e59a501   2045  0.434886\n",
      "3988  45f6d00da   1365  0.419313\n",
      "624   0c9462c08   1222  0.405490\n",
      "545   0572565c2   1728  0.400984\n",
      "1750  5f341a818    830  0.385933\n",
      "645   6619d81fc   1633  0.374199\n",
      "1850  5a1589f1a   1562  0.373156\n",
      "1712  2ec5b290f   2076  0.370076\n",
      "3854  73687e512   1291  0.364064\n"
     ]
    }
   ],
   "source": [
    "gain_light_gbm = model_lgbm.feature_importance('gain')\n",
    "\n",
    "feature_imp_light_gbm = pd.DataFrame({'feature': model_lgbm.feature_name(),\n",
    "                                      'split': model_lgbm.feature_importance('split'),\n",
    "                                      'gain': 100 * gain_light_gbm / gain_light_gbm.sum()\n",
    "}).sort_values('gain', ascending=False)\n",
    "\n",
    "print(feature_imp_light_gbm[:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## XGBoost Model Training\n",
    "\n",
    "### Note on XGBoost\n",
    "\n",
    "Below we will be using **XGBoost** which is an advanced version of Gradient boosting method, it literally means eXtreme Gradient Boosting. XGBoost dominates structured or tabular datasets on classification and regression predictive modeling problems. The XGBoost library implements the [gradient boosting decision tree algorithm](https://en.wikipedia.org/wiki/Gradient_boosting).\n",
    "\n",
    "Different from the traditional gradient descent technique, gradient enhancement helps to predict the optimal gradient of the additional model. This technique can reduce the output error at each iteration.\n",
    "\n",
    "In practice what we do in order to build the learner is to:\n",
    "\n",
    "- Start with single root (contains all the training examples)\n",
    "\n",
    "- Iterate over all features and values per feature, and evaluate each possible split loss reduction:\n",
    "\n",
    "- gain = loss(father instances) - (loss(left branch)+loss(right branch))\n",
    "\n",
    "- The gain for the best split must be positive (and > min_split_gain parameter), otherwise we must stop growing the branch.\n",
    "\n",
    "**Leaf growth**\n",
    "\n",
    "XGboost splits up to the specified max_depth hyperparameter and then starts pruning the tree backwards and removes splits beyond which there is no positive gain. It uses this approach since sometimes a split of no loss reduction may be followed by a split with loss reduction. XGBoost can also perform leaf-wise tree growth (as LightGBM).\n",
    "\n",
    "Normally it is impossible to enumerate all the possible tree structures q. A greedy algorithm that starts from a single leaf and iteratively adds branches to the tree is used instead. Assume that I_L and I_R are the instance sets of left and right nodes after the split. Then the loss reduction after the split is given by,\n",
    "\n",
    "![](https://i.imgur.com/jzyLh81.png)\n",
    "\n",
    "## Differences in LightGBM & XGBoost\n",
    "\n",
    "LightGBM uses a novel technique of Gradient-based One-Side Sampling (GOSS) to filter out the data instances for finding a split value while XGBoost uses pre-sorted algorithm & Histogram-based algorithm for computing the best split. Here instances mean observations/samples.\n",
    "\n",
    "Let's see how pre-sorting splitting works-\n",
    "\n",
    "- For each node, enumerate over all features\n",
    "\n",
    "- For each feature, sort the instances by feature value\n",
    "\n",
    "- Use a linear scan to decide the best split along that feature basis information gain\n",
    "\n",
    "- Take the best split solution along all the features\n",
    "\n",
    "In simple terms, Histogram-based algorithm splits all the data points for a feature into discrete bins and uses these bins to find the split value of histogram. While, it is efficient than pre-sorted algorithm in training speed which enumerates all possible split points on the pre-sorted feature values, it is still behind GOSS in terms of speed.\n",
    "\n",
    "## XGBoost Model Parameters\n",
    "\n",
    "I am explaining only those Parameters that I will be implementing below in my function. For an exhaustive explanation of all of them [see here](https://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/)\n",
    "\n",
    "**objective [default=reg:linear]**\n",
    "\n",
    "This defines the loss function to be minimized. Mostly used values are:\n",
    "\n",
    "- binary:logistic –logistic regression for binary classification, returns predicted probability (not class)\n",
    "\n",
    "- multi:softmax –multiclass classification using the softmax objective, returns predicted class (not probabilities)\n",
    "you also need to set an additional num_class (number of classes) parameter defining the number of unique classes\n",
    "\n",
    "- multi:softprob –same as softmax, but returns predicted probability of each data point belonging to each class.\n",
    "\n",
    "**eval_metric [ default according to objective ]**\n",
    "\n",
    "The metric to be used for validation data. The default values are rmse for regression and error for classification.\n",
    "Typical values are:\n",
    "\n",
    "- rmse – root mean square error\n",
    "\n",
    "- mae – mean absolute error\n",
    "\n",
    "- logloss – negative log-likelihood\n",
    "\n",
    "- error – Binary classification error rate (0.5 threshold)\n",
    "\n",
    "- merror – Multiclass classification error rate\n",
    "\n",
    "- mlogloss – Multiclass logloss\n",
    "\n",
    "- auc: Area under the curve\n",
    "\n",
    "\n",
    "**eta [default=0.3]**\n",
    "\n",
    "- Analogous to learning rate in GBM.\n",
    "- Makes the model more robust by shrinking the weights on each step.\n",
    "- Typical final values to be used: 0.01-0.2\n",
    "\n",
    "**colsample_bytree**:  We can create a random sample of the features (or columns) to use prior to creating each decision tree in the boosted model. That is, tuning Column Sub-sampling in XGBoost By Tree. This is controlled by the colsample_bytree parameter. The default value is 1.0 meaning that all columns are used in each decision tree. A fraction (e.g. 0.6) means a fraction of columns to be subsampled. We can evaluate values for colsample_bytree between 0.1 and 1.0 incrementing by 0.1.\n",
    "\n",
    "#### A Note on regularization in XGBoost\n",
    "\n",
    "XGBoost adds built-in regularization to achieve accuracy gains beyond gradient boosting. Regularization is the process of adding information to reduce variance and prevent overfitting.\n",
    "\n",
    "Although data may be regularized through hyperparameter fine-tuning, regularized algorithms may also be attempted. For example, Ridge and Lasso are regularized machine learning alternatives to LinearRegression.\n",
    "\n",
    "XGBoost includes regularization as part of the learning objective, as contrasted with gradient boosting and random forests. The regularized parameters penalize complexity and smooth out the final weights to prevent overfitting. XGBoost is a regularized version of gradient boosting.\n",
    "\n",
    "Mathematically, XGBoost's learning objective may be defined as follows:\n",
    "\n",
    "### obj(θ) = l(θ) + Ω (θ)\n",
    "\n",
    "Here, **l(θ)**  is the loss function, which is the Mean Squared Error (MSE) for regression, or the log loss for classification, and **Ω (θ)** is the regularization function, a penalty term to prevent over-fitting. Including a regularization term as part of the objective function distinguishes XGBoost from most tree ensembles.\n",
    "\n",
    "The learning objective for the th boosted tree can now be rewritten as follows:\n",
    "\n",
    "![img](https://i.imgur.com/IRNCrvM.png)\n",
    "\n",
    "**reg_alpha and reg_lambda** : First note the loss function is defined as\n",
    "\n",
    "![img](https://i.imgur.com/aw1Hod9.png)\n",
    "\n",
    "##### So the above is how the regularized objective function looks like if you want to allow for the inclusion of a L1 and a L2 parameter in the same model\n",
    "\n",
    "`reg_alpha` and `reg_lambda` control the L1 and L2 regularization terms, which in this case limit how extreme the weights at the leaves can become. Higher values of alpha mean more L1 regularization. See the documentation [here](http://xgboost.readthedocs.io/en/latest///parameter.html#parameters-for-tree-booster).\n",
    "\n",
    "Since L1 regularization in GBDTs is applied to leaf scores rather than directly to features as in logistic regression, it actually serves to reduce the depth of trees. This in turn will tend to reduce the impact of less-predictive features. We might think of L1 regularization as more aggressive against less-predictive features than L2 regularization.\n",
    "\n",
    "These two regularization terms have different effects on the weights; L2 regularization (controlled by the lambda term) encourages the weights to be small, whereas L1 regularization (controlled by the alpha term) encourages sparsity — so it encourages weights to go to 0. This is helpful in models such as logistic regression, where you want some feature selection, but in decision trees we’ve already selected our features, so zeroing their weights isn’t super helpful. For this reason, I found setting a high lambda value and a low (or 0) alpha value to be the most effective when regularizing.\n",
    "\n",
    "##### [From this Paper](https://arxiv.org/pdf/1603.02754.pdf)\n",
    "\n",
    "You find the mathematical underpinnings for XGBoost model by Tianqi Chen et al. A couple of mathematical deviations of this model form the classic Friedman's GBM are:\n",
    "\n",
    " - Regularized (penalized) parameters (and remember that parameters in the boosting are the function, trees, or linear models): L1 and L2 are available.\n",
    "\n",
    "   [1]: https://i.stack.imgur.com/oJlNo.png\n",
    "\n",
    "\n",
    "\n",
    "## Using XGBoost DMatrix\n",
    "\n",
    "XGBoost provides a way to convert our training and testing data into DMatrix. DMatrix is an internal data structure that is used by XGBoost, which is optimized for both memory efficiency and training speed. You can construct DMatrix from multiple different sources of data.\n",
    "The best part is that converting a dataset into DMatrix is really easy.\n",
    "\n",
    "The following code converts the training and testing data into DMatrix format.\n",
    "\n",
    "```py\n",
    "# convert to DMatrix\n",
    "d_train = xgb.DMatrix(x_train, y_train)\n",
    "\n",
    "d_test = xgb.DMatrix(x_val, y_val)\n",
    "```\n",
    "\n",
    "To convert data to DMatrix, we need to give the features of the dataset and then the labels as well.\n",
    "\n",
    "To read more about DMatrices check the [documentation](https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.DMatrix).\n",
    "\n",
    "\n",
    "## Watchlist\n",
    "\n",
    "named list of xgb.DMatrix datasets to use for evaluating model performance. Metrics specified in either eval_metric or feval will be computed for each of these datasets during each boosting iteration, and stored in the end as a field named evaluation_log in the resulting object. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def xgb_model_run(train_x, train_y, validation_x, validation_y, test_x):\n",
    "    params = {\n",
    "        'objective': 'reg:squarederror', \n",
    "          'eval_metric': 'rmse',\n",
    "          'eta': 0.001,\n",
    "          'max_depth': 10, \n",
    "          'subsample': 0.6, \n",
    "          'colsample_bytree': 0.6,\n",
    "          'alpha':0.001,\n",
    "          'random_state': 42\n",
    "          \n",
    "    }\n",
    "\n",
    "    training_data = xgb.DMatrix(train_x, train_y)\n",
    "    validation_data = xgb.DMatrix(validation_x, validation_y)\n",
    "\n",
    "    watchlist = [(training_data, 'train'), (validation_data, 'valid')]\n",
    "\n",
    "    model_xgb = xgb.train(params, training_data, 50, watchlist, maximize=False, early_stopping_rounds=100, verbose_eval=100 )\n",
    "\n",
    "    data_test = xgb.DMatrix(test_x)\n",
    "    predict_test_xgb = np.expm1(model_xgb.predict(data_test, ntree_limit=model_xgb.best_ntree_limit ) )\n",
    "\n",
    "    return predict_test_xgb, model_xgb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Training XGB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\ttrain-rmse:14.08766\tvalid-rmse:14.07682\n",
      "[49]\ttrain-rmse:13.42471\tvalid-rmse:13.41326\n"
     ]
    }
   ],
   "source": [
    "predicted_y_test_org_xgbm, model_xgb = xgb_model_run(X_train_split, y_train_split, X_validation, y_validation, X_test_original)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Hyper-Parameter Tuning in XGBoost\n",
    "\n",
    "As an example, on the above mode, for our XGBoost function we could fine-tune five hyperparameters. The ranges of possible values that we could consider could be as below:\n",
    "\n",
    "```\n",
    "{\"learning_rate\"    : [0.05, 0.10, 0.15, 0.20, 0.25, 0.30 ] ,\n",
    " \"max_depth\"        : [ 3, 4, 5, 6, 8, 10, 12, 15],\n",
    " \"min_child_weight\" : [ 1, 3, 5, 7 ],\n",
    " \"gamma\"            : [ 0.0, 0.1, 0.2 , 0.3, 0.4 ],\n",
    " \"colsample_bytree\" : [ 0.3, 0.4, 0.5 , 0.7 ] }\n",
    "```\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## CatBoost Model Training\n",
    "\n",
    "CatBoost is another competitor to XGBoost, LightGBM and H2O. “CatBoost” name comes from two words “Category” and “Boosting”.\n",
    "\n",
    "The library works well with multiple Categories of data, such as audio, text, image including historical data.\n",
    "\n",
    "The CatBoost library can be used to solve both classification and regression challenge. For classification, you can use **“CatBoostClassifier”** and for regression, **“CatBoostRegressor“**.\n",
    "\n",
    "**[Yandex](https://yandex.com/)** is relying heavily on Catboost for ranking, forecasting and recommendations. This model is serving more than 70 million users each month.\n",
    "\n",
    "\"CatBoost is an algorithm for gradient boosting on decision trees. Developed by Yandex researchers and engineers, it is the successor of the MatrixNet algorithm that is widely used within the company for ranking tasks, forecasting and making recommendations. It is universal and can be applied across a wide range of areas and to a variety of problems.\"\n",
    "\n",
    "Overall some of the algorithmic enhancements that **Catboost** brought:\n",
    "\n",
    "- 1. For data with **categorical** features the accuracy of CatBoost would be better compared to other algorithms.\n",
    "\n",
    "2. **Better over-fitting handling**: - CatBoost uses the implementation of ordered boosting, an alternative to the classic boosting algorithm, which will be specially significant on small datasets\n",
    "\n",
    "3. **GPU-training**: - The versions of CatBoost available from pip install (pip install catboost) and conda install (conda install catboost) have GPU support out-of-the-box. You just need to specify that you want to train your model on GPU in the corresponding HP (will be shown below).\n",
    "\n",
    "For [GPU system requirements of CatBoost](https://catboost.ai/docs/concepts/python-installation.html#gpu-system-requirements)\n",
    "\n",
    "The versions of CatBoost available from pip install and conda install have GPU support out-of-the-box.\n",
    "Devices with compute capability 3.0 and higher are supported in compiled packages.\n",
    "Training on GPU requires NVIDIA Driver of version 418.xx or higher.\n",
    "The Python version of CatBoost for CUDA of compute capability 2.0 can be built from source.\n",
    "\n",
    "To check Compute Capability of your CUDA-GPU check this [NVIDIA Official Link](https://developer.nvidia.com/cuda-gpus#compute)\n",
    "\n",
    "Further for [Training on GPU](https://catboost.ai/docs/features/training-on-gpu.html)\n",
    "\n",
    "The parameters that enable and customize training on GPU are set in the constructors of the classes - CatBoost (fit), CatBoostClassifier (fit), CatBoostRegressor (fit).\n",
    "`task_type` - The processing unit type to use for training. Possible values are - \"CPU\" or \"GPU\" . An example below\n",
    "\n",
    "```python\n",
    "model = CatBoostClassifier(iterations=1000,\n",
    "                           task_type=\"GPU\",\n",
    "                           devices='0:1')\n",
    "model.fit(train_data,\n",
    "          train_labels,\n",
    "          verbose=False)\n",
    "```\n",
    "\n",
    "**Categorical features handling in CatBoost Algorithm**\n",
    "\n",
    "[The below is taken from this paper](http://learningsys.org/nips17/assets/papers/paper_11.pdf)\n",
    "\n",
    "Categorical features have a discrete set of values called categories which are not necessary comparable with each other; thus, such features cannot be used in binary decision trees directly. A common practice for dealing with categorical features is converting them to numbers at the preprocessing time, i.e., each category for each example is substituted with one or several numerical values. The most widely used technique which is usually applied to low-cardinality categorical features is one-hot encoding: the original feature is removed and a new binary variable is added for each category [14]. One-hot encoding can be done during the preprocessing phase or during training, the latter can be implemented more efficiently in terms of training time and is implemented in CatBoost.\n",
    "\n",
    "For further details on this red [CatBoost's documentation](https://catboost.ai/docs/concepts/algorithm-main-stages_cat-to-numberic.html)\n",
    "\n",
    "**Leaf growth algorithm in CatBoost**\n",
    "\n",
    "Catboost grows a balanced tree. In each level of such a tree, the feature-split pair that brings to the lowest loss (according to a penalty function) is selected and is used for all the level’s nodes. It is possible to change its policy using the grow-policy parameter.\n",
    "\n",
    "## CatBoost Training Parameters\n",
    "\n",
    "Let’s look at the common parameters in CatBoost:\n",
    "\n",
    "**loss_function** alias as **objective** — Metric used for training. These are regression metrics such as root mean squared error for regression and logloss for classification.\n",
    "\n",
    "**eval_metric** — Metric used for detecting over-fitting.\n",
    "\n",
    "**iterations** — The maximum number of trees to be built, defaults to 1000. It aliases are `num_boost_round`, `n_estimators`, and `num_trees`.\n",
    "Some notes on **Total num of Trees** - In **bagging** and **random forests**  the averaging of independently grown trees makes it very difficult to overfit with too many trees. However, in GBMs this function differently as each tree is grown in sequence to fix up the past tree’s mistakes. For example, in regression, GBMs will chase residuals as long as we allow them to. Also, depending on the values of the other hyperparameters, GBMs often require many trees (sometimes many thousands of trees). But also more trees, can easily overfit we must find the optimal number of trees that minimize the loss function of interest with cross validation.\n",
    "\n",
    "**learning_rate** alias **eta** — The learning rate that determines how fast or slow the model will learn. The default is usually varies between 0.01 to 0.03.\n",
    "\n",
    "**random_seed** alias **random_state** — The random seed used for training.\n",
    "\n",
    "**l2_leaf_reg** alias **reg_lambda** — Coefficient at the L2 regularization term of the cost function. The default is 3.0.\n",
    "\n",
    "**bootstrap_type** — Determines the sampling method for the weights of the objects, e.g Bayesian, Bernoulli, MVS, and Poisson.\n",
    "depth —The depth of the tree.\n",
    "\n",
    "**grow_policy** — Determines how the greedy search algorithm will be applied. It can be either SymmetricTree, Depthwise, or Lossguide.\n",
    "\n",
    "**SymmetricTree** is the default. In SymmetricTree, the tree is built level-by-level until the depth is attained. In every step, leaves from the previous tree are split with the same condition. When **Depthwise** is chosen, a tree is built step-by-step until the specified depth is achieved. On each step, all non-terminal leaves from the last tree level are split. The leaves are split using the condition that leads to the best loss improvement. In **Lossguide**, the tree is built leaf-by-leaf until the specified number of leaves is attained. On each step, the non-terminal leaf with the best loss improvement is split\n",
    "\n",
    "**min_data_in_leaf** alias **min_child_samples** — This is the minimum number of training samples in a leaf. This parameter is only used with the Lossguide and Depthwise growing policies.\n",
    "\n",
    "**max_leaves** alias num_leaves — This parameter is used only with the Lossguide policy and determines the number of leaves in the tree.\n",
    "\n",
    "**ignored_features** — Indicates the features that should be ignored in the training process.\n",
    "\n",
    "**nan_mode** — The method for dealing with missing values. The options are Forbidden, Min, and Max. The default is Min. When Forbidden is used, the presence of missing values leads to errors. With Min, the missing values are taken as the minimum values for that feature. In Max, the missing values are treated as the maximum value for the feature.\n",
    "\n",
    "**leaf_estimation_method** — The method used to calculate values in leaves. In classification, 10 Newton iterations are used. Regression problems using quantile or MAE loss use one Exact iteration. Multi classification uses one Netwon iteration.\n",
    "\n",
    "**leaf_estimation_backtracking** — The type of backtracking to be used during gradient descent. The default is `AnyImprovement`. `AnyImprovement` decreases the descent step, up to where the loss function value is smaller than it was in the last iteration. Armijo reduces the descent step until the Armijo condition is met.\n",
    "\n",
    "**boosting_type** — The boosting scheme. It can be plain for the classic gradient boosting scheme, or ordered, which offers better quality on smaller datasets.\n",
    "\n",
    "**score_function** — The score type used to select the next split during tree construction. `Cosine` is the default option. The other available options are `L2`, `NewtonL2`, and `NewtonCosine`.\n",
    "\n",
    "**early_stopping_rounds** — When True, sets the over-fitting detector type to `Iter` and stops the training when the optimal metric is achieved.\n",
    "\n",
    "classes_count — The number of classes for multi-classification problems.\n",
    "task_type — Whether you are using a CPU or GPU. CPU is the default.\n",
    "devices — The IDs of the GPU devices to be used for training.\n",
    "cat_features — The array with the categorical columns.\n",
    "text_features —Used to declare text columns in classification problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning: Overfitting detector is active, thus evaluation metric is calculated on every iteration. 'metric_period' is ignored for evaluation metric.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 1.7614936\ttest: 1.6943084\tbest: 1.6943084 (0)\ttotal: 1.28s\tremaining: 10m 41s\n",
      "50:\tlearn: 1.6747126\ttest: 1.6330977\tbest: 1.6330977 (50)\ttotal: 1m 1s\tremaining: 8m 58s\n",
      "100:\tlearn: 1.6096948\ttest: 1.5902474\tbest: 1.5902474 (100)\ttotal: 2m 1s\tremaining: 8m\n",
      "150:\tlearn: 1.5572985\ttest: 1.5589053\tbest: 1.5589053 (150)\ttotal: 3m 1s\tremaining: 7m\n",
      "200:\tlearn: 1.5167332\ttest: 1.5382647\tbest: 1.5382647 (200)\ttotal: 4m 2s\tremaining: 6m 1s\n",
      "250:\tlearn: 1.4827385\ttest: 1.5217760\tbest: 1.5217760 (250)\ttotal: 5m 3s\tremaining: 5m\n",
      "300:\tlearn: 1.4545436\ttest: 1.5093004\tbest: 1.5093004 (300)\ttotal: 6m 5s\tremaining: 4m 1s\n",
      "350:\tlearn: 1.4296056\ttest: 1.4994080\tbest: 1.4994080 (350)\ttotal: 7m 6s\tremaining: 3m 1s\n",
      "400:\tlearn: 1.4078157\ttest: 1.4916401\tbest: 1.4916401 (400)\ttotal: 8m 8s\tremaining: 2m\n",
      "450:\tlearn: 1.3911783\ttest: 1.4861881\tbest: 1.4861881 (450)\ttotal: 9m 10s\tremaining: 59.8s\n",
      "499:\tlearn: 1.3751665\ttest: 1.4807567\tbest: 1.4807567 (499)\ttotal: 10m 12s\tremaining: 0us\n",
      "\n",
      "bestTest = 1.48075668\n",
      "bestIteration = 499\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Now Catboost model training\n",
    "\n",
    "model_catboost = CatBoostRegressor(iterations=500,\n",
    "                                   learning_rate=0.01,\n",
    "                                   depth=10,\n",
    "                                   eval_metric='RMSE',\n",
    "                                   random_seed = 42,\n",
    "                                   bagging_temperature=0.2,\n",
    "                                   od_type='Iter',\n",
    "                                   metric_period=50,\n",
    "                                   od_wait=20\n",
    "                                   )\n",
    "\n",
    "model_catboost.fit(X_train_split, y_train_split,\n",
    "                   eval_set=(X_validation, y_validation),\n",
    "                   use_best_model=True,\n",
    "                   verbose=50\n",
    "                   )\n",
    "\n",
    "predicted_y_test_org_catboost = np.expm1(model_catboost.predict(X_test_original))\n",
    "\n",
    "# np.expm1 => Calculates exponential of all the elements subtracting 1 from all the input array elements.\n",
    "# So The output array consists of elementwise exponential value -1 corresponding to each input element.\n",
    "# This function provides greater precision than exp(x) - 1 for small values of x.\n",
    "# the expm1 \"function avoids the loss of precision involved in the direct evaluation of exp(x)-1 for small x.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Creating Output file for Submission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "submission_final = pd.read_csv('../input/santander-value-prediction-challenge/sample_submission.csv')\n",
    "\n",
    "submission_lgb = pd.DataFrame()\n",
    "submission_lgb['target'] = predicted_y_test_org_light_gbm\n",
    "\n",
    "submission_xgb = pd.DataFrame()\n",
    "submission_xgb['target'] = predicted_y_test_org_xgbm\n",
    "\n",
    "submission_catboost = pd.DataFrame()\n",
    "submission_catboost['target'] = predicted_y_test_org_catboost\n",
    "\n",
    "submission_final['target'] = (submission_lgb['target'] * 0.5 + submission_xgb['target'] * 0.3 + submission_catboost['target'] * 0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "submission_final.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "submission_final.to_csv('submission_combined_lgb_xgb_catboost.csv', index=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Further Reading and References\n",
    "\n",
    "- T. Chen, C. Guestrin, [XGBoost: A Scalable Tree Boosting System, 2016](https://arxiv.org/abs/1603.02754)\n",
    "\n",
    "- J. Friedman, [Greedy Function Approximation: A Gradient Boosting Machine 1999](http://docs.salford-systems.com/GreedyFuncApproxSS.pdf)\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
  },
  "kernelspec": {
   "display_name": "Python 3.8.10 64-bit",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}