Pred modeling updates (#1446)

* Predictive Modeling Updates

* fix complete

---------

Co-authored-by: u51369 <[email protected]>

Author: bjodom

AI-and-Analytics/Jupyter/Predictive_Modeling_Training/AI_Kit_XGBoost_Predictive_Modeling.complete/00_Local_Setup/Local_Setup.ipynb

@@ -33,16 +33,6 @@
     "[Images Source](https://commons.wikimedia.org/wiki/File:3D_view_of_an_event_recorded_with_the_CMS_detector_in_2012_at_a_proton-proton_centre_of_mass_energy_of_8_TeV.png)"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "2f66db55-d1f8-407e-b7bb-b353ce78fa56",
-   "metadata": {},
-   "source": [
-    "# Example environment creation:\n",
-    "\n",
-    "This environment is the latest [Intel® oneAPI AI Analytics Toolkit](https://software.intel.com/content/www/us/en/develop/tools/oneapi/ai-analytics-toolkit.html) base environment, which includes data analytics and machine learning workflows and Intel optimizations for XGboost. See [here](https://software.intel.com/content/www/us/en/develop/articles/installing-ai-kit-with-conda.html) for more installation information."
-   ]
-  },
   {
    "cell_type": "markdown",
    "id": "57efb70b-7073-4baa-b544-253551c7bb58",
@@ -61,18 +51,21 @@
    "outputs": [],
    "source": [
     "import sklearn\n",
-    "#from sklearnex import patch_sklearn\n",
-    "#patch_sklearn()\n",
+    "from sklearnex import patch_sklearn\n",
+    "patch_sklearn()\n",
     "#unpatch_sklearn()\n",
     "from sklearn.model_selection import train_test_split\n",
     "from sklearn.metrics import mean_squared_error\n",
+    "import warnings\n",
+    "warnings.simplefilter(action='ignore', category=FutureWarning)\n",
     "import pandas as pd\n",
+    "from pandas import MultiIndex, Int16Dtype # if you don't import in this order you will get a pandas.Int64Index fix for FutureWarning error.\n",
     "import xgboost as xgb\n",
     "import numpy as np\n",
-    "import warnings\n",
     "from time import perf_counter\n",
     "print(\"XGB Version          : \", xgb.__version__)\n",
-    "print(\"Scikit-Learn Version : \", sklearn.__version__)"
+    "print(\"Scikit-Learn Version : \", sklearn.__version__)\n",
+    "print(\"Pandas Version       : \", pd.__version__)"
    ]
   },
   {
@@ -169,11 +162,21 @@
    "source": [
     "filename = 'HIGGS.csv'\n",
     "names =  ['class_label', 'lepton pT', 'lepton eta', 'lepton phi', 'missing energy magnitude', 'missing energy phi', 'jet 1 pt', 'jet 1 eta', 'jet 1 phi', 'jet 1 b-tag', 'jet 2 pt', 'jet 2 eta', 'jet 2 phi', 'jet 2 b-tag', 'jet 3 pt', 'jet 3 eta', 'jet 3 phi', 'jet 3 b-tag', 'jet 4 pt', 'jet 4 eta', 'jet 4 phi', 'jet 4 b-tag', 'm_jj', 'm_jjj', 'm_lv', 'm_jlv', 'm_bb', 'm_wbb', 'm_wwbb']\n",
-    "data = pd.read_csv(filename, names=names, delimiter=\",\", nrows=100000)\n",
-    "#data = pd.read_csv(filename, names=names, delimiter=\",\", nrows=1100000)\n",
+    "#data = pd.read_csv(filename, names=names, delimiter=\",\", nrows=100000)\n",
+    "data = pd.read_csv(filename, names=names, delimiter=\",\", nrows=1100000)\n",
     "print(data.shape)"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "505ce472-a525-42c3-b995-ddc2c3aa2b43",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%time p_df = pd.read_csv(\"HIGGS.csv\")"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "0d7249bc-4e6b-4a28-8894-00b14c61d4f2",
@@ -233,7 +236,7 @@
    "id": "d339991a-3485-49ef-8151-c5c8020fc586",
    "metadata": {},
    "source": [
-    "* In this scenario loading 100000 rows the balance isn't too skewed.  "
+    "* In this scenario loading 100000 rows the balance isn't too skewed, the next cell is optional."
    ]
   },
   {
@@ -293,7 +296,7 @@
    "outputs": [],
    "source": [
     "# This is the y target vector -- the ones we want to predict.\n",
-    "# print(data.iloc[:,0])"
+    "print(data.iloc[:,0])"
    ]
   },
   {
@@ -383,7 +386,7 @@
     "}\n",
     "\n",
     "# Train the model\n",
-    "warnings.filterwarnings(\"ignore\", category=UserWarning)\n",
+    "warnings.simplefilter(action='ignore', category=UserWarning)\n",
     "t1_start = perf_counter()  # Time fit function\n",
     "model_xgb= xgb.XGBClassifier(**xgb_params)\n",
     "model_xgb.fit(X_train,y_train)\n",
@@ -515,17 +518,17 @@
     "    'disable_default_eval_metric':  'true',\n",
     "    'tree_method':                  'hist', \n",
     "}\n",
+    "# Necessary for now to supress multi-threaded Future errors with respect to pandas and XGBoost\n",
+    "import os\n",
+    "os.environ['PYTHONWARNINGS']='ignore::FutureWarning'\n",
     "\n",
     "# Train the model\n",
-    "warnings.filterwarnings(\"ignore\", category=UserWarning)\n",
     "model_xgb= xgb.XGBClassifier(**xgb_params2, use_label_encoder=False)\n",
     "\n",
-    "\n",
     "# Setup grid search n_jobs=-1 uses all cores, reducing cv from 5 to 3 for speed, scoring is done using area under curve.\n",
     "grid_cv = GridSearchCV(model_xgb, param_grid, n_jobs=-1, cv=3, scoring=\"roc_auc\")\n",
     "\n",
     "# This fit function takes a while--hours, make sure you are ready.\n",
-    "\n",
     "_ = grid_cv.fit(X_train, y_train)"
    ]
   },
@@ -556,7 +559,7 @@
    "source": [
     "### Results\n",
     "\n",
-    "    grid_cv.best_score_ = 0.8002252116945674 grid cv.best_params\n",
+    "    grid_cv.best_score_ = 0.80 grid cv.best_params\n",
     "\n",
     "    {'colsample_bytree': 1, 'gamma': 0, 'learning_rate': 0.1, 'max_depth': 8, 'reg_lambda': 10, 'scale_pos_weight': 1, 'subsample': 1}\n",
     "\n",
@@ -609,7 +612,7 @@
     "\n",
     "     n_estimators:, default=100\n",
     "\n",
-    "The number of trees in the forest.  A good way to see how many trees might be useful is to plot the learning curve.  since this is a classification problem we will use log loss as our measurement where lower values are better.  \n",
+    "n_estimaters represents the number of trees in the forest.  A good way to see how many trees might be useful is to plot the learning curve.  Since this is a classification problem we will use log loss as our measurement where lower values are better.  \n",
     "\n",
     "Our orignal fit function needs to be modified to include eval_metric with the type set to logloss.  In addition we need to define the evaluation data set so that the results are evaluated after each round in order to plot them.\n"
    ]
@@ -632,7 +635,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# fit the model\n",
+    "# Fit the model\n",
     "model_xgb.fit(X_train, y_train, eval_metric='logloss', eval_set=evalset)"
    ]
   },
@@ -716,16 +719,15 @@
     "    'reg_lambda':                    10,\n",
     "    'scale_pos_weight':              1,\n",
     "    'tree_method':                  'hist', \n",
-    "    'n_estimators':                  250\n",
+    "    'n_estimators':                  1000,\n",
     "}\n",
     "\n",
     "# Train the model\n",
-    "warnings.filterwarnings(\"ignore\", category=UserWarning)\n",
-    "t = time.process_time()  # Time fit function\n",
+    "t1_start = perf_counter()  # Time fit function\n",
     "model_xgb= xgb.XGBClassifier(**xgb_params)\n",
     "model_xgb.fit(X_train,y_train, eval_metric='logloss', eval_set=evalset, verbose=True)\n",
-    "elapsed_time = time.process_time() - t\n",
-    "print (\"It took\",elapsed_time,\" to fit.\")"
+    "t1_stop = perf_counter()\n",
+    "print (\"It took\", t1_stop-t1_start,\"seconds to fit.\")"
    ]
   },
   {
@@ -780,7 +782,7 @@
    "source": [
     "## So how many trees do we need really?\n",
     "\n",
-    "* It takes awhile to watch 250 trees get evaluated, a great performance improvement is to use the XGBoost early stopping capbility.\n",
+    "* It takes awhile to watch 1000 trees get evaluated, a great performance improvement is to use the XGBoost early stopping capability.\n",
     "\n",
     "* Modify the fit function to stop the training after 10 to 15 rounds of no improvement.  \n",
     "        \n",
@@ -823,9 +825,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
+   "display_name": "Python 3 (Intel® oneAPI 2023.0)",
    "language": "python",
-   "name": "python3"
+   "name": "c009-intel_distribution_of_python_3_oneapi-beta05-python"
   },
   "language_info": {
    "codemirror_mode": {
@@ -837,7 +839,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.7"
+   "version": "3.9.15"
   },
   "nbTranslate": {
    "displayLangs": [

AI-and-Analytics/Jupyter/Predictive_Modeling_Training/AI_Kit_XGBoost_Predictive_Modeling.complete/03_XGBoost/XGBoost.ipynb

@@ -0,0 +1,7 @@
+Copyright Intel Corporation
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.