notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# High-speed-rail document analysis\n",
    "\n",
    "By [Ben Welsh](https://palewire/who-is-ben-welsh/)\n",
    "\n",
    "This analysis was conducted for April 27, 2019 story [\"How California’s faltering high-speed rail project was ‘captured’ by costly consultants.\"](https://www.latimes.com/local/california/la-me-california-high-speed-rail-consultants-20190426-story.html)\n",
    "\n",
    "It found that outside consultants have provided more than 3,000 environmental statements, business documents and other reports to the California High-Speed Rail Authority. Altogether they contain more than 152,000 pages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How we did it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import Python tools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import PyPDF2\n",
    "import pathlib\n",
    "import requests\n",
    "import pandas as pd\n",
    "from bs4 import BeautifulSoup\n",
    "from urllib.request import urlretrieve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set output directory where documents will be saved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_dir = pathlib.Path(os.getenv(\"OUTPUT_DIR\") or \"./output\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read in a list of all the URLs on the rail authority's site that contain consultant reports."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "page_list = open(\"./input/urls.csv\").read().split(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Request each page and parse out all of its PDF links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_page(url):\n",
    "    \"\"\"\n",
    "    Parse all the PDF urls from the provided URL's HTML.\n",
    "    \"\"\"\n",
    "    r = requests.get(url)\n",
    "    soup = BeautifulSoup(r.text)\n",
    "    a_list = soup.find_all(\"a\")\n",
    "    pdf_list = [a['href'] for a in a_list if a['href'].endswith(\".pdf\")]\n",
    "    return [f\"http://hsr.ca.gov{href}\" for href in pdf_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "pdf_list = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "for page in page_list:\n",
    "    pdfs = parse_page(page)\n",
    "    pdf_list.extend(pdfs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many total PDF urls were found?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3,410'"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f\"{len(pdf_list):,}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove all of the duplicates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "pdf_set = set(pdf_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many URLs remain?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3,168'"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f\"{len(pdf_set):,}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download them all."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def download_pdf(url):\n",
    "    \"\"\"\n",
    "    Download a PDF url to the output folder.\n",
    "    \"\"\"\n",
    "    filename = url.split(\"/\")[-1]\n",
    "    path = output_dir.joinpath(filename)\n",
    "    if path.exists():\n",
    "        return\n",
    "    try:\n",
    "        print(f\"Downloading {filename}\")\n",
    "        urlretrieve(url, path)\n",
    "    except Exception:\n",
    "        print(f\"Failed to download {url}\")            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "for url in pdf_set:\n",
    "    download_pdf(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get their page counts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_page_count(path):\n",
    "    \"\"\"\n",
    "    Get the page count of the provided PDF path.\n",
    "    \"\"\"\n",
    "    with open(path, 'rb') as f:\n",
    "        try:\n",
    "            pdfReader = PyPDF2.PdfFileReader(f)\n",
    "            return pdfReader.numPages\n",
    "        except:\n",
    "            return pd.np.NaN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_list = list(output_dir.glob('*.pdf'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Count the total number of documents again to check out many we actually downloaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3,129'"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f\"{len(pdf_path_list):,}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loop through all the documents and tally pages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "page_counts = dict((p, get_page_count(p)) for p in path_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(pdf_page_counts.items(), columns=[\"pdf\", \"page_count\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pdf</th>\n",
       "      <th>page_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1413</th>\n",
       "      <td>/media/palewire/Passport/hsr-pdfs/final_EIR_Me...</td>\n",
       "      <td>8923.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142</th>\n",
       "      <td>/media/palewire/Passport/hsr-pdfs/final_ERIS_F...</td>\n",
       "      <td>4297.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2108</th>\n",
       "      <td>/media/palewire/Passport/hsr-pdfs/BayCValley20...</td>\n",
       "      <td>2126.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2508</th>\n",
       "      <td>/media/palewire/Passport/hsr-pdfs/brdmtg_04211...</td>\n",
       "      <td>1523.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2421</th>\n",
       "      <td>/media/palewire/Passport/hsr-pdfs/Los_Angeles_...</td>\n",
       "      <td>1369.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    pdf  page_count\n",
       "1413  /media/palewire/Passport/hsr-pdfs/final_EIR_Me...      8923.0\n",
       "142   /media/palewire/Passport/hsr-pdfs/final_ERIS_F...      4297.0\n",
       "2108  /media/palewire/Passport/hsr-pdfs/BayCValley20...      2126.0\n",
       "2508  /media/palewire/Passport/hsr-pdfs/brdmtg_04211...      1523.0\n",
       "2421  /media/palewire/Passport/hsr-pdfs/Los_Angeles_...      1369.0"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(\"page_count\", ascending=False).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3129"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'151,703.0'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f\"{df.page_count.sum():,}\""
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}