Bootstrap_Sampling_In_Random_Forests.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"colab_type":"text","id":"RAHap1Z3FZC-"},"source":["\n","# First a note on Ensemble Learning\n","\n","The general principle of ensemble methods is to construct a linear combination of some model ﬁtting method, instead of using a single ﬁt of the method. The main principle behind the ensemble model is that a group of weak learners come together to form a strong learner, thus increasing the accuracy of the model.When we try to predict the target variable using any machine learning technique, the main causes of difference in actual and predicted values are noise, variance, and bias. Ensemble helps to reduce these factors (except noise, which is irreducible error). The noise-related error is mainly due to noise in the training data and can't be removed. However, the errors due to bias and variance can be reduced.\n","The total error can be expressed as follows: \n","\n","### Total Error = Bias + Variance + Irreducible Error \n","\n","![Imgur](https://imgur.com/q4Egoes.png)\n","\n","# Why we do Bootstrap re-sampling\n","\n","Bootstrapping resamples the original dataset with replacement many thousands of times to create simulated datasets. This process involves drawing random samples from the original dataset. Through bootstrapping you are simply taking samples over and over again from the same group of data (your sample data) to estimate how accurate your estimates about the entire population (what really is out there in the real world) is.\n","\n","If you were to take one sample and make estimates on the real population, you might not be able to estimate how accurate your estimates are - we only have one estimate and have not identified how this estimate varies with different samples that we might have encountered.\n","\n","### Bootstrap Aggregation (or Bagging for short), is a simple and very powerful ensemble method.\n","\n","An ensemble method is a technique that combines the predictions from multiple machine learning algorithms together to make more accurate predictions than any individual model.\n","\n","Bootstrap Aggregation is a general procedure that can be used to reduce the variance for those algorithm that have high variance. An algorithm that has high variance are decision trees, like classification and regression trees (CART).\n","\n","Decision trees are sensitive to the specific data on which they are trained. If the training data is changed (e.g. a tree is trained on a subset of the training data) the resulting decision tree can be quite different and in turn the predictions can be quite different.\n","\n","Bagging is the application of the Bootstrap procedure to a high-variance machine learning algorithm, typically decision trees.\n","\n","Another major drawback associated with the tree classifiers is that they are unstable. That is, a small\n","change in the training data set can result in a very different tree. The reason for this lies in the\n","hierarchical nature of the tree classifiers. An error that occurs in a node at a high level of the tree\n","propagates all the way down to the leaves below it.\n","\n","\n","And so Bagging (bootstrap aggregating) can reduce the variance and improve the\n","generalization error performance. The basic idea is to create B variants, X1, X2 , . . . , XB , of the\n","training set X, using bootstrap techniques, by uniformly sampling from X with replacement. For\n","each of the training set variants Xi , a tree Ti is constructed. The final decision for the classification\n","of a given point is in favor of the class predicted by the majority of the subclassifiers Ti , i =\n","1, 2, . . . , B.\n","\n","# Finally Principle of Random Forest with Bootstrapping\n","\n","The below text is taken from [this](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8950481) source\n","\n","Firstly, using Bootstrap resampling technique, multiple samples are randomly selected from the original training sample set x to generate a new training sample set. Then, multiple decision trees are constructed to form random forest. Finally, the random forest averages the output of each decision tree to determine the final filling result y. Due to the Bootstrap used in the random decision tree generation process, all samples are not used in a decision tree and the unused samples are called out of band (OOB). Through out of band, the accuracy of the tree can be evaluated. The other trees are evaluated according to the principle and finally average the results\n","\n","![Imgur](https://imgur.com/qj2c3ur.png)\n","\n","---\n","\n","### Now our Code Implementations\n","\n","### There will be some functions that start with the word \"grader\" ex: grader_sampples(), grader_30().. etc, we should not change those function definition. Every Grader function has to return True."]},{"cell_type":"code","execution_count":1,"metadata":{"colab":{},"colab_type":"code","id":"m6ag91ijrQOs"},"outputs":[{"name":"stdout","output_type":"stream","text":["x.shape  (506, 13)\n","y.shape  (506,)\n"]}],"source":["import numpy as np # importing numpy for numerical computation\n","from sklearn.datasets import load_boston # here we are using sklearn's boston dataset\n","from sklearn.metrics import mean_squared_error # importing mean_squared_error metric\n","\n","import random\n","from sklearn.tree import DecisionTreeRegressor\n","\n","boston = load_boston()\n","x=boston.data #independent variables\n","y=boston.target #target variable\n","\n","print(\"x.shape \", x.shape)\n","print(\"y.shape \", y.shape)"]},{"cell_type":"code","execution_count":2,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":269},"colab_type":"code","id":"kQle3T_wuOa3","outputId":"521c7bdd-5316-48d5-c534-b61d170d2c28"},"outputs":[{"data":{"text/plain":["array([[6.3200e-03, 1.8000e+01, 2.3100e+00, 0.0000e+00, 5.3800e-01,\n","        6.5750e+00, 6.5200e+01, 4.0900e+00, 1.0000e+00, 2.9600e+02,\n","        1.5300e+01, 3.9690e+02, 4.9800e+00],\n","       [2.7310e-02, 0.0000e+00, 7.0700e+00, 0.0000e+00, 4.6900e-01,\n","        6.4210e+00, 7.8900e+01, 4.9671e+00, 2.0000e+00, 2.4200e+02,\n","        1.7800e+01, 3.9690e+02, 9.1400e+00],\n","       [2.7290e-02, 0.0000e+00, 7.0700e+00, 0.0000e+00, 4.6900e-01,\n","        7.1850e+00, 6.1100e+01, 4.9671e+00, 2.0000e+00, 2.4200e+02,\n","        1.7800e+01, 3.9283e+02, 4.0300e+00],\n","       [3.2370e-02, 0.0000e+00, 2.1800e+00, 0.0000e+00, 4.5800e-01,\n","        6.9980e+00, 4.5800e+01, 6.0622e+00, 3.0000e+00, 2.2200e+02,\n","        1.8700e+01, 3.9463e+02, 2.9400e+00],\n","       [6.9050e-02, 0.0000e+00, 2.1800e+00, 0.0000e+00, 4.5800e-01,\n","        7.1470e+00, 5.4200e+01, 6.0622e+00, 3.0000e+00, 2.2200e+02,\n","        1.8700e+01, 3.9690e+02, 5.3300e+00]])"]},"execution_count":2,"metadata":{},"output_type":"execute_result"}],"source":["x[:5]"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"V2fHTdS_zpgG"},"source":["# <font color='red'><b>Task 1\n","</b></font>\n","\n","*  <font color='blue'><b>Creating samples</b></font><br>\n","    <b> Randomly create 30 samples from the whole boston data points</b>\n","    *  Creating each sample: Consider any random 303(60% of 506) data points from whole data set and then replicate any 203 points from the sampled points\n","    \n","     For better understanding of this procedure lets check this examples, assume we have 10 data points [1,2,3,4,5,6,7,8,9,10], first we take 6 data points randomly , consider we have selected [4, 5, 7, 8, 9, 3] now we will replicate 4 points from [4, 5, 7, 8, 9, 3], consder they are [5, 8, 3,7] so our final sample will be [4, 5, 7, 8, 9, 3, 5, 8, 3,7]\n","* <font color='blue'><b> Create 30 samples </b></font>\n","    *  Note that as a part of the Bagging when we are taking the random samples <b>make sure each of the sample will have different set of columns</b><br>\n","Ex: Assume we have 10 columns[1 ,2 ,3 ,4 ,5 ,6 ,7 ,8 ,9 ,10] for the first sample we will select [3, 4, 5, 9, 1, 2] and for the second sample  [7, 9, 1, 4, 5, 6, 2] and so on...\n","Make sure each sample will have atleast 3 feautres/columns/attributes\n","\n","### Step - 1 - Creating samples"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"CSVaWG1F4uCZ"},"source":["<font color='Orange'><b>Algorithm</b></font>\n","\n","![alt text](https://imgur.com/MA1rUVB.jpg)"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"f_oWoN97BhDY"},"source":["### Generating samples"]},{"cell_type":"code","execution_count":4,"metadata":{},"outputs":[],"source":["def generating_samples(input_data, target_data): \n","  \n","  # random.choice to generate random indices without replacement\n","  # selecting 303 random row indices from the input_data, without replacement\n","  rows_selected = np.random.choice(len(input_data), 303, replace=False)\n","  \n","  # now we will replicate 203 points from above selected rows\n","  # Replacing Rows => Extracting 206 reandom row indices from the abvoe rows_selected\n","  rows_203_extracted_from_rows_selected = np.random.choice(rows_selected, 203, replace=False)\n","  \n","  # Now get 3 to 13 random column indices from input_data\n","  number_of_columns_to_select = random.randint(3, 13)\n","  columns_selected = np.array(random.sample(range(0, 13), number_of_columns_to_select ))\n","  \n","  sample_data = input_data[rows_selected[:, None], columns_selected]\n","  \n","  target_of_sample_data = target_data[rows_selected]\n","  \n","  # Now Replication of Data for 203 data points out of 303 selected points\n","  \n","  replicated_203_sample_data_points = input_data[rows_203_extracted_from_rows_selected[:, None], columns_selected ]\n","  \n","  target_203_replicated_sample_data = target_data[rows_203_extracted_from_rows_selected]\n","  \n","  # Concatenating data\n","  \n","  final_sample_data = np.vstack((sample_data, replicated_203_sample_data_points ))\n","  \n","  final_target_data = np.vstack((target_of_sample_data.reshape(-1, 1), target_203_replicated_sample_data.reshape(-1, 1) ))\n","  \n","  return final_sample_data, final_target_data, rows_selected, columns_selected\n","  "]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"MivEQFlm7iOg"},"source":["<font color='cyan'> <b> Grader function - 1 </b> </fongt>"]},{"cell_type":"code","execution_count":5,"metadata":{"colab":{},"colab_type":"code","id":"AVvuhNzm7uld"},"outputs":[{"name":"stdout","output_type":"stream","text":["(506, 6)\n","(506, 1)\n","(303,)\n","(6,)\n"]},{"data":{"text/plain":["True"]},"execution_count":5,"metadata":{},"output_type":"execute_result"}],"source":["def grader_samples(a,b,c,d):\n","    length = (len(a)==506  and len(b)==506)\n","    sampled = (len(a)-len(set([str(i) for i in a]))==203)\n","    rows_length = (len(c)==303)\n","    column_length= (len(d)>=3)\n","    assert(length and sampled and rows_length and column_length)\n","    return True\n","  \n","a,b,c,d = generating_samples(x, y)\n","print(a.shape)\n","print(b.shape)\n","print(c.shape)\n","print(d.shape)\n","grader_samples(a,b,c,d)"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"b4LSsmn4Jn2_"},"source":["### Creating 30 samples\n","\n","![alt text](https://i.imgur.com/p8eZaWL.jpg)"]},{"cell_type":"code","execution_count":6,"metadata":{"colab":{},"colab_type":"code","id":"XXlKWjCcBvTk"},"outputs":[],"source":["# Use generating_samples function to create 30 samples \n","# store these created samples in a list\n","list_input_data =[]\n","list_output_data =[]\n","list_selected_row= []\n","list_selected_columns=[]\n","\n","for i in range (0, 30):\n","  a, b, c, d = generating_samples(x, y)\n","  list_input_data.append(a)\n","  list_output_data.append(b)\n","  list_selected_row.append(c)\n","  list_selected_columns.append(d)"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"MXUz9VFiMQkh"},"source":["<font color='cyan'> <b>Grader function - 2 </b></font>"]},{"cell_type":"code","execution_count":7,"metadata":{"colab":{},"colab_type":"code","id":"hCvIq8NuMWOC"},"outputs":[{"data":{"text/plain":["True"]},"execution_count":7,"metadata":{},"output_type":"execute_result"}],"source":["def grader_30(a):\n","    assert(len(a)==30 and len(a[0])==506)\n","    return True\n","grader_30(list_input_data)"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"whaHCPB0O8qF"},"source":["# Step - 2 of Task-1\n","\n","\n","## Building High Variance Models on each of the sample and finding train MSE value\n","\n","\n","*  Build a regression trees on each of 30 samples.\n","*  Computed the predicted values of each data point(506 data points) in our corpus.\n","*  Predicted house price of $i^{th}$ data point $y^{i}_{pred} =  \\frac{1}{30}\\sum_{k=1}^{30}(\\text{predicted value of } x^{i} \\text{ with } k^{th} \\text{ model})$\n","*  Now calculate the $MSE =  \\frac{1}{506}\\sum_{i=1}^{506}(y^{i} - y^{i}_{pred})^{2}$\n","\n","## Flowchart for Building regression trees\n","\n","![alt text](https://imgur.com/lt2TQQB.png)\n","<!-- ![alt text](https://imgur.com/vUDY4kv.png) -->\n","<!-- ![alt text](https://i.imgur.com/pcXfSmp.png) -->"]},{"cell_type":"code","execution_count":8,"metadata":{"colab":{},"colab_type":"code","id":"YWQp6tRwMthq"},"outputs":[],"source":["list_of_all_models_decision_tree = []\n","for i in range(0, 30):\n","  model_i = DecisionTreeRegressor(max_depth=None)\n","  model_i.fit(list_input_data[i], list_output_data[i])\n","  list_of_all_models_decision_tree.append(model_i)"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"21j8BKfAQ1U8"},"source":["## Flowchart for calculating MSE\n","\n","![alt text](https://imgur.com/CoTSGj0.png)\n","<!-- ![alt text](https://i.imgur.com/sPEE618.png) -->"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"6e-UamlHRjPy"},"source":["After getting predicted_y for each data point, we can use sklearns mean_squared_error to calculate the MSE between predicted_y and actual_y.\n","\n","# Calculating MSE"]},{"cell_type":"code","execution_count":9,"metadata":{"colab":{},"colab_type":"code","id":"qWhcvMRWRA9b"},"outputs":[{"name":"stdout","output_type":"stream","text":["MSE :  0.09994218142619152\n"]}],"source":["from sklearn.metrics import mean_squared_error\n","from statistics import median\n","\n","array_of_Y = []\n","\n","for i in range(0, 30):\n","  data_point_i = x[:, list_selected_columns[i]]\n","  target_y_i = list_of_all_models_decision_tree[i].predict(data_point_i)\n","  array_of_Y.append(target_y_i)\n","  \n","  \n","predicted_array_of_target_y = np.array(array_of_Y)\n","predicted_array_of_target_y = predicted_array_of_target_y.transpose()\n","\n","# print(predicted_array_of_target_y.shape)\n","\n","# Now to calculate MSE, first calculate the Median of Predicted Y\n","# passing axis=1 will make sure the medians are computed along axis=1\n","median_predicted_y = np.median(predicted_array_of_target_y, axis=1)\n","median_predicted_y.shape\n","\n","print(\"MSE : \", mean_squared_error(y, median_predicted_y ))"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"RuclPDMnSz8F"},"source":["# Step - 3 of Task-1\n","\n","### Calculating the OOB score\n","\n","*  Predicted house price of $i^{th}$ data point $y^{i}_{pred} =  \\frac{1}{k}\\sum_{\\text{k= model which was buit on samples not included } x^{i}}(\\text{predicted value of } x^{i} \\text{ with } k^{th} \\text{ model})$.\n","*  Now calculate the $OOB Score =  \\frac{1}{506}\\sum_{i=1}^{506}(y^{i} - y^{i}_{pred})^{2}$.\n","\n","\n","### Given a single query point predict the price of house\n","\n","Consider xq= [0.18,20.0,5.00,0.0,0.421,5.60,72.2,7.95,7.0,30.0,19.1,372.13,18.60] \n","Predict the house price for this point as mentioned in the step 2 of Task 1.\n","\n","## Flowchart for calculating OOB score\n","\n","![alt text](https://imgur.com/0wUQtcC.png)\n","<!-- ![alt text](https://i.imgur.com/95S5Mtm.png) -->\n","\n","## Now calculate the $OOB Score =  \\frac{1}{506}\\sum_{i=1}^{506}(y^{i} - y^{i}_{pred})^{2}$."]},{"cell_type":"code","execution_count":10,"metadata":{"colab":{},"colab_type":"code","id":"Fog_6DNdS-h_"},"outputs":[{"name":"stdout","output_type":"stream","text":["final_oob_score is  13.081730298143956\n"]}],"source":["# First noting that our earlier definded variable list_selected_row and list_selected_columns \n","# which has the list of selected rows and columns\n","# e.g. list_selected_row is a 2D array of the form [[], [], []...] each inner-array represnting selected row numbers\n","# for a specific sample. and len(list_selected_row) is 30 reprsenting the 30 samples we have selected for bootstrapping\n","# print(\"list_selected_row[10] \", list_selected_row[10])\n","# print(list_selected_columns)\n","\n","y_predicted_oob_median_list = []\n","\n","for i in range(0, 506):\n","  indices_for_oob_models = []\n","  \n","  # For each of i-th row I shall build a list, of sample size 30\n","  # ONLY condition being that this i-th row should not be part of the list_selected_row[i-th]\n","  # e.g. say for i = 469 and index_oob in below loop is 10 then \n","  # list_selected_row[10] (which is an array of row-numbers) should not contain the 469-th row\n","  for index_oob in range(0, 30):\n","    if i not in list_selected_row[index_oob]:\n","      indices_for_oob_models.append(index_oob)\n","      \n","  y_predicted_oob_list = []\n","  \n","  for oob_model_index in indices_for_oob_models:\n","    model_oob = list_of_all_models_decision_tree[oob_model_index]\n","    \n","    row_oob = x[i]\n","    # print('oob_model_index ', oob_model_index)\n","    \n","    # Now extract ONLY those specific columns/featues that were selected during the bootstrapping\n","    x_oob_data_point = [row_oob[columns] for columns in list_selected_columns[oob_model_index] ]\n","    # print('np.array(x_oob_data_point) ', np.array(x_oob_data_point))\n","    x_oob_data_point = np.array(x_oob_data_point).reshape(1, -1)\n","    \n","    y_predicted_oob_data_point = model_oob.predict(x_oob_data_point)\n","    y_predicted_oob_list.append(y_predicted_oob_data_point)\n","    # \n","  y_predicted_oob_list = np.array(y_predicted_oob_list)\n","  \n","  y_predicted_median = np.median(y_predicted_oob_list)\n","  y_predicted_oob_median_list.append(y_predicted_median)\n","  \n","\n","def calculate_oob_score(num_rows):\n","  oob_score = 0\n","  for i in range(0, num_rows):\n","    oob_score += ((y[i] - y_predicted_oob_median_list[i] ) ** 2)\n","  final_oob_score = oob_score/506\n","  return final_oob_score\n","\n","print(\"final_oob_score is \", calculate_oob_score(506))   \n","  "]},{"cell_type":"markdown","metadata":{},"source":["### Further notes on above OOB calculation\n","\n","The key point is that the OOB sample rows were passed through every Decition Treee that did not contain those specific OOB sample rows during the bootstrapping of training data.\n","\n","OOB error is simply the error on samples that were not seen during training. \n","\n","OOB Scoring is very useful when I dont have a large dataset and thereby if I split that dataset into training and validation set - will result in loss of useful data that otherwise could have been used for training the models. Hence in this case, we decide to extract some of the training data as the validation set by using only those data-points that were not used for training a particular sample-set.\n","\n","---\n","\n","# Task 2\n","\n","* Computing CI of OOB Score and Train MSE\n","* Repeat Task 1 for 35 times, and for each iteration store the Train MSE and OOB score\n","* After this we will have 35 Train MSE values and 35 OOB scores\n","* using these 35 values (assume like a sample) find the confidence intravels of MSE and OOB Score\n","* we need to report CI of MSE and CI of OOB Score\n","* Note: Refer the Central_Limit_theorem.ipynb to check how to find the confidence intravel\n"]},{"cell_type":"code","execution_count":11,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["(0.05013239090102449, 14.059993831916806)\n"]}],"source":["# Function to build the entire bootstrapping steps that we did above and\n","# Reurning from the function the MSE and oob score\n","def bootstrapping_and_oob(x, y):\n","\n","  # Use generating_samples function to create 30 samples \n","  # store these created samples in a list\n","  list_input_data =[]\n","  list_output_data =[]\n","  list_selected_row= []\n","  list_selected_columns=[]\n","  \n","  for i in range (0, 30):\n","    a, b, c, d = generating_samples(x, y)\n","    list_input_data.append(a)\n","    list_output_data.append(b)\n","    list_selected_row.append(c)\n","    list_selected_columns.append(d)\n","  \n","  # building regression trees\n","  list_of_all_models_decision_tree = []\n","  for i in range(0, 30):\n","    model_i = DecisionTreeRegressor(max_depth=None)\n","    model_i.fit(list_input_data[i], list_output_data[i])\n","    list_of_all_models_decision_tree.append(model_i)\n","  \n","  # calculating MSE\n","  array_of_Y = []\n","\n","  for i in range(0, 30):\n","    data_point_i = x[:, list_selected_columns[i]]\n","    target_y_i = list_of_all_models_decision_tree[i].predict(data_point_i)\n","    array_of_Y.append(target_y_i)\n","    \n","    \n","  predicted_array_of_target_y = np.array(array_of_Y)\n","  predicted_array_of_target_y = predicted_array_of_target_y.transpose()\n","\n","  # print(predicted_array_of_target_y.shape)\n","\n","  # Now to calculate MSE, first calculate the Median of Predicted Y\n","  # passing axis=1 will make sure the medians are computed along axis=1\n","  median_predicted_y = np.median(predicted_array_of_target_y, axis=1)\n","  \n","  # And now the final MSE\n","  MSE = mean_squared_error(y, median_predicted_y )\n","  \n","  # Calculating OOB Score\n","  y_predicted_oob_median_list = []\n","\n","  for i in range(0, 506):\n","    indices_for_oob_models = []\n","    \n","    # For each of i-th row I shall build a list of sample size 30\n","    # ONLY condition being that this ith row should not be part of\n","    # the list_selected_row\n","    for index_oob in range(0, 30):\n","      if i not in list_selected_row[index_oob]:\n","        indices_for_oob_models.append(index_oob)\n","        \n","    y_predicted_oob_list = []\n","    \n","    for oob_model_index in indices_for_oob_models:\n","      model_oob = list_of_all_models_decision_tree[oob_model_index]\n","      \n","      row_oob = x[i]\n","      # print('oob_model_index ', oob_model_index)\n","      \n","      x_oob_data_point = [row_oob[col] for col in list_selected_columns[oob_model_index] ]\n","      # print('np.array(x_oob_data_point) ', np.array(x_oob_data_point))\n","      x_oob_data_point = np.array(x_oob_data_point).reshape(1, -1)\n","      \n","      y_predicted_oob_data_point = model_oob.predict(x_oob_data_point)\n","      y_predicted_oob_list.append(y_predicted_oob_data_point)\n","      # \n","    y_predicted_oob_list = np.array(y_predicted_oob_list)\n","    \n","    y_predicted_median = np.median(y_predicted_oob_list)\n","    y_predicted_oob_median_list.append(y_predicted_median)\n","    \n","\n","  oob_score = 0\n","\n","  for i in range(0, 506):\n","    # oob_score = (oob_score + (y[i] - y_predicted_oob_median_list[i] ) ** 2)\n","    # 13.828377285079045\n","    oob_score += (y[i] - y_predicted_oob_median_list[i] ) ** 2\n","\n","  final_oob_score = oob_score/506\n","  \n","  return MSE, final_oob_score\n","\n","print(bootstrapping_and_oob(x, y))\n","  "]},{"cell_type":"code","execution_count":12,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["confidence_interval_mse_35  (0.09294459920700518, 0.18099695278943642)\n","confidence_interval_oob_score_35  (13.768617241526412, 14.885877160705009)\n"]}],"source":["import scipy\n","\n","x=boston.data #independent variables\n","y=boston.target #target variable\n","\n","mse_boston_35_times_arr = []\n","oob_score_boston_35_times_arr = []\n","\n","# Repeat Task 1 for 35 times, and for each iteration store the Train MSE and OOB score\n","for i in range(0, 35):\n","  mse, oob_score = bootstrapping_and_oob(x, y)\n","  mse_boston_35_times_arr.append(mse)\n","  oob_score_boston_35_times_arr.append(oob_score)\n","\n","\n","mse_boston_35_times_arr = np.array(mse_boston_35_times_arr)\n","oob_score_boston_35_times_arr = np.array(oob_score_boston_35_times_arr)\n","\n","confidence_level = 0.95\n","degrees_of_freedom = 34 # sample.size - 1\n","\n","mean_of_sample_mse_35 = np.mean(mse_boston_35_times_arr)\n","standard_error_of_sample_mse_35 = scipy.stats.sem(mse_boston_35_times_arr)\n","\n","\n","# Per document - https://www.kite.com/python/answers/how-to-compute-the-confidence-interval-of-a-sample-statistic-in-python\n","# confidence_interval = scipy.stats.t.interval(confidence_level, degrees_freedom, sample_mean, sample_standard_error)\n","confidence_interval_mse_35 = scipy.stats.t.interval(confidence_level, degrees_of_freedom, mean_of_sample_mse_35, standard_error_of_sample_mse_35 )\n","print(\"confidence_interval_mse_35 \", confidence_interval_mse_35)\n","\n","# Now calculate confidence inter for oob score\n","mean_of_sample_oob_score_35 = np.mean(oob_score_boston_35_times_arr)\n","standard_error_of_sample_oob_score_35 = scipy.stats.sem(oob_score_boston_35_times_arr)\n","\n","confidence_interval_oob_score_35 = scipy.stats.t.interval(confidence_level, degrees_of_freedom, mean_of_sample_oob_score_35, standard_error_of_sample_oob_score_35 )\n","print(\"confidence_interval_oob_score_35 \", confidence_interval_oob_score_35)\n"]},{"cell_type":"markdown","metadata":{},"source":["## Observation / Interpretation of above Confidence Interval\n","\n","By definition we know the interpretation of a 95% confidence interval for the population mean as  -  If repeated random samples were taken and the 95% confidence interval was computed for each sample, 95% of the intervals would contain the population mean.\n","\n","So in this case\n","\n","- MSE - There is a 95% chance that the confidence interval of (0.05732086175441538, 0.11667646077442519) contains the true population mean of MSE.\n","- OOB Score - There is a 95% chance that the confidence interval of (13.274222499705303, 14.427942855729313) contains the true population mean of OOB Score.\n","\n","# Task 3 (send query point \"xq\" to 30 models)\n","\n","#### We created 30 models by using 30 samples in TASK-1. Here, we need send query point \"xq\" to 30 models and perform the regression on the output generated by 30 models\n","\n","#### Flowchart for Task 3\n","\n","![alt text](https://imgur.com/ATQji1x.png)\n","<!-- ![alt text](https://i.imgur.com/Y5cNhQk.png) -->"]},{"cell_type":"code","execution_count":13,"metadata":{"colab":{},"colab_type":"code","id":"i_pUlSD-VYD1"},"outputs":[{"data":{"text/plain":["18.7"]},"execution_count":13,"metadata":{},"output_type":"execute_result"}],"source":["def predict_y_given_x_bootstrap(x_query):\n","  y_predicted_array_30_sample = []\n","  \n","  for i in range(0, 30):\n","    model_i = list_of_all_models_decision_tree[i]\n","    # Extract x for ith data point with specific number of featues from list_selected_columns\n","    x_data_point_i = [x_query[column] for column in list_selected_columns[i]]\n","    x_data_point_i = np.array(x_data_point_i).reshape(1, -1)\n","    y_predicted_i = model_i.predict(x_data_point_i)\n","    y_predicted_array_30_sample.append(y_predicted_i)\n","    \n","  y_predicted_array_30_sample = np.array(y_predicted_array_30_sample)\n","  y_predicted_median = np.median(y_predicted_array_30_sample)\n","  return y_predicted_median\n","\n","\n","xq= [0.18,20.0,5.00,0.0,0.421,5.60,72.2,7.95,7.0,30.0,19.1,372.13,18.60] \n","y_predicted_for_xq = predict_y_given_x_bootstrap(xq)\n","y_predicted_for_xq    "]}],"metadata":{"colab":{"collapsed_sections":[],"name":"Bootstrap_assignment.ipynb","provenance":[]},"interpreter":{"hash":"31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"},"kernelspec":{"display_name":"Python 3.8.10 64-bit","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.8.10"}},"nbformat":4,"nbformat_minor":2}