ENH: Add calamite engine to read_excel

ci/deps/actions-310.yaml

@@ -55,4 +55,5 @@ dependencies:
   - zstandard>=0.15.2
 
   - pip:
+    - python-calamine
     - tzdata>=2022.1

ci/deps/actions-311.yaml

@@ -55,4 +55,5 @@ dependencies:
   - zstandard>=0.15.2
 
   - pip:
+    - python-calamine>=0.1.0
     - tzdata>=2022.1

ci/deps/actions-38-downstream_compat.yaml

@@ -70,4 +70,5 @@ dependencies:
   - py
 
   - pip:
+    - python-calamine
     - tzdata>=2022.1

ci/deps/actions-38-minimum_versions.yaml

@@ -59,4 +59,5 @@ dependencies:
 
   - pip:
     - pyqt5==5.15.1
+    - python-calamine==0.1.0
     - tzdata==2022.1

ci/deps/actions-38.yaml

@@ -55,4 +55,5 @@ dependencies:
   - zstandard>=0.15.2
 
   - pip:
+    - python-calamine
     - tzdata>=2022.1

ci/deps/actions-39.yaml

@@ -55,4 +55,5 @@ dependencies:
   - zstandard>=0.15.2
 
   - pip:
+    - python-calamine
     - tzdata>=2022.1

ci/deps/circle-38-arm64.yaml

@@ -54,3 +54,6 @@ dependencies:
   - xlrd>=2.0.1
   - xlsxwriter>=1.4.3
   - zstandard>=0.15.2
+
+  - pip:
+    - python-calamine

doc/source/getting_started/install.rst

@@ -345,6 +345,7 @@ xlrd                      2.0.1              excel           Reading Excel
 xlsxwriter                1.4.3              excel           Writing Excel
 openpyxl                  3.0.7              excel           Reading / writing for xlsx files
 pyxlsb                    1.0.8              excel           Reading for xlsb files
+python-calamine           0.1.0              excel           Reading for xls/xlsx/xlsb/ods files
 ========================= ================== =============== =============================================================
 
 HTML

doc/source/user_guide/io.rst

@@ -3420,7 +3420,8 @@ Excel files
 The :func:`~pandas.read_excel` method can read Excel 2007+ (``.xlsx``) files
 using the ``openpyxl`` Python module. Excel 2003 (``.xls``) files
 can be read using ``xlrd``. Binary Excel (``.xlsb``)
-files can be read using ``pyxlsb``.
+files can be read using ``pyxlsb``. Also, all this formats can be read using ``python-calamine``,
+but this library has sime limitation, for example, can't detect date in most formats.
 The :meth:`~DataFrame.to_excel` instance method is used for
 saving a ``DataFrame`` to Excel.  Generally the semantics are
 similar to working with :ref:`csv<io.read_csv_table>` data.

doc/source/whatsnew/v2.0.0.rst

@@ -279,6 +279,7 @@ Other enhancements
 - :meth:`Series.dropna` and :meth:`DataFrame.dropna` has gained ``ignore_index`` keyword to reset index (:issue:`31725`)
 - Improved error message in :func:`to_datetime` for non-ISO8601 formats, informing users about the position of the first error (:issue:`50361`)
 - Improved error message when trying to align :class:`DataFrame` objects (for example, in :func:`DataFrame.compare`) to clarify that "identically labelled" refers to both index and columns (:issue:`50083`)
+- Performance improvement in :func:`to_datetime` when format is given or can be inferred (:issue:`50465`)
 - Added support for :meth:`Index.min` and :meth:`Index.max` for pyarrow string dtypes (:issue:`51397`)
 - Added :meth:`DatetimeIndex.as_unit` and :meth:`TimedeltaIndex.as_unit` to convert to different resolutions; supported resolutions are "s", "ms", "us", and "ns" (:issue:`50616`)
 - Added :meth:`Series.dt.unit` and :meth:`Series.dt.as_unit` to convert to different resolutions; supported resolutions are "s", "ms", "us", and "ns" (:issue:`51223`)

doc/source/whatsnew/v2.1.0.rst

@@ -42,9 +42,9 @@ Other enhancements
 - Improved error message when creating a DataFrame with empty data (0 rows), no index and an incorrect number of columns. (:issue:`52084`)
 - :meth:`DataFrame.applymap` now uses the :meth:`~api.extensions.ExtensionArray.map` method of underlying :class:`api.extensions.ExtensionArray` instances (:issue:`52219`)
 - :meth:`arrays.SparseArray.map` now supports ``na_action`` (:issue:`52096`).
+- Added ``calamine`` as an engine to ``read_excel`` (:issue:`50395`)
 - :meth:`Categorical.map` and :meth:`CategoricalIndex.map` now have a ``na_action`` parameter (:issue:`44279`)
 - Add dtype of categories to ``repr`` information of :class:`CategoricalDtype` (:issue:`52179`)
--
 
 .. ---------------------------------------------------------------------------
 .. _whatsnew_210.notable_bug_fixes:

environment.yml

@@ -117,5 +117,6 @@ dependencies:
 
   - pip:
       - sphinx-toggleprompt
+      - python-calamine
       - typing_extensions; python_version<"3.11"
       - tzdata>=2022.1

pandas/compat/_optional.py

@@ -37,6 +37,7 @@
     "pyarrow": "7.0.0",
     "pyreadstat": "1.1.2",
     "pytest": "7.0.0",
+    "python-calamine": "0.1.0",
     "pyxlsb": "1.0.8",
     "s3fs": "2021.08.0",
     "scipy": "1.7.1",
@@ -64,6 +65,7 @@
     "lxml.etree": "lxml",
     "odf": "odfpy",
     "pandas_gbq": "pandas-gbq",
+    "python_calamine": "python-calamine",
     "snappy": "python-snappy",
     "sqlalchemy": "SQLAlchemy",
     "tables": "pytables",

pandas/core/config_init.py

@@ -503,11 +503,11 @@ def use_inf_as_na_cb(key) -> None:
     auto, {others}.
 """
 
-_xls_options = ["xlrd"]
-_xlsm_options = ["xlrd", "openpyxl"]
-_xlsx_options = ["xlrd", "openpyxl"]
-_ods_options = ["odf"]
-_xlsb_options = ["pyxlsb"]
+_xls_options = ["xlrd", "calamine"]
+_xlsm_options = ["xlrd", "openpyxl", "calamine"]
+_xlsx_options = ["xlrd", "openpyxl", "calamine"]
+_ods_options = ["odf", "calamine"]
+_xlsb_options = ["pyxlsb", "calamine"]
 
 
 with cf.config_prefix("io.excel.xls"):

pandas/io/excel/_base.py

@@ -149,13 +149,15 @@
     of dtype conversion.
 engine : str, default None
     If io is not a buffer or path, this must be set to identify io.
-    Supported engines: "xlrd", "openpyxl", "odf", "pyxlsb".
+    Supported engines: "xlrd", "openpyxl", "odf", "pyxlsb", "calamine".
     Engine compatibility :
 
     - "xlrd" supports old-style Excel files (.xls).
     - "openpyxl" supports newer Excel file formats.
     - "odf" supports OpenDocument file formats (.odf, .ods, .odt).
     - "pyxlsb" supports Binary Excel files.
+    - "calamine" supports Excel (.xls, .xlsx, .xlsm, .xlsb)
+      and OpenDocument (.ods) file formats.
 
     .. versionchanged:: 1.2.0
         The engine `xlrd <https://xlrd.readthedocs.io/en/latest/>`_
@@ -375,7 +377,7 @@ def read_excel(
     | Callable[[str], bool]
     | None = ...,
     dtype: DtypeArg | None = ...,
-    engine: Literal["xlrd", "openpyxl", "odf", "pyxlsb"] | None = ...,
+    engine: Literal["xlrd", "openpyxl", "odf", "pyxlsb", "calamine"] | None = ...,
     converters: dict[str, Callable] | dict[int, Callable] | None = ...,
     true_values: Iterable[Hashable] | None = ...,
     false_values: Iterable[Hashable] | None = ...,
@@ -414,7 +416,7 @@ def read_excel(
     | Callable[[str], bool]
     | None = ...,
     dtype: DtypeArg | None = ...,
-    engine: Literal["xlrd", "openpyxl", "odf", "pyxlsb"] | None = ...,
+    engine: Literal["xlrd", "openpyxl", "odf", "pyxlsb", "calamine"] | None = ...,
     converters: dict[str, Callable] | dict[int, Callable] | None = ...,
     true_values: Iterable[Hashable] | None = ...,
     false_values: Iterable[Hashable] | None = ...,
@@ -453,7 +455,7 @@ def read_excel(
     | Callable[[str], bool]
     | None = None,
     dtype: DtypeArg | None = None,
-    engine: Literal["xlrd", "openpyxl", "odf", "pyxlsb"] | None = None,
+    engine: Literal["xlrd", "openpyxl", "odf", "pyxlsb", "calamine"] | None = None,
     converters: dict[str, Callable] | dict[int, Callable] | None = None,
     true_values: Iterable[Hashable] | None = None,
     false_values: Iterable[Hashable] | None = None,
@@ -1418,13 +1420,15 @@ class ExcelFile:
         .xls, .xlsx, .xlsb, .xlsm, .odf, .ods, or .odt file.
     engine : str, default None
         If io is not a buffer or path, this must be set to identify io.
-        Supported engines: ``xlrd``, ``openpyxl``, ``odf``, ``pyxlsb``
+        Supported engines: ``xlrd``, ``openpyxl``, ``odf``, ``pyxlsb``, ``calamine``
         Engine compatibility :
 
         - ``xlrd`` supports old-style Excel files (.xls).
         - ``openpyxl`` supports newer Excel file formats.
         - ``odf`` supports OpenDocument file formats (.odf, .ods, .odt).
         - ``pyxlsb`` supports Binary Excel files.
+        - ``calamine`` supports Excel (.xls, .xlsx, .xlsm, .xlsb)
+          and OpenDocument (.ods) file formats.
 
         .. versionchanged:: 1.2.0
 
@@ -1452,6 +1456,7 @@ class ExcelFile:
             This is not supported, switch to using ``openpyxl`` instead.
     """
 
+    from pandas.io.excel._calaminereader import CalamineExcelReader
     from pandas.io.excel._odfreader import ODFReader
     from pandas.io.excel._openpyxl import OpenpyxlReader
     from pandas.io.excel._pyxlsb import PyxlsbReader
@@ -1462,6 +1467,7 @@ class ExcelFile:
         "openpyxl": OpenpyxlReader,
         "odf": ODFReader,
         "pyxlsb": PyxlsbReader,
+        "calamine": CalamineExcelReader,
     }
 
     def __init__(

pandas/io/excel/_calaminereader.py

@@ -0,0 +1,99 @@
+from __future__ import annotations
+
+from datetime import (
+    date,
+    datetime,
+    time,
+)
+from typing import (
+    TYPE_CHECKING,
+    Union,
+)
+
+from pandas.compat._optional import import_optional_dependency
+from pandas.util._decorators import doc
+
+import pandas as pd
+from pandas.core.shared_docs import _shared_docs
+
+from pandas.io.excel._base import BaseExcelReader
+
+if TYPE_CHECKING:
+    from pandas._typing import (
+        FilePath,
+        ReadBuffer,
+        Scalar,
+        StorageOptions,
+    )
+
+_CellValueT = Union[int, float, str, bool, time, date, datetime]
+
+
+class CalamineExcelReader(BaseExcelReader):
+    @doc(storage_options=_shared_docs["storage_options"])
+    def __init__(
+        self,
+        filepath_or_buffer: FilePath | ReadBuffer[bytes],
+        storage_options: StorageOptions = None,
+    ) -> None:
+        """
+        Reader using calamine engine (xlsx/xls/xlsb/ods).
+
+        Parameters
+        ----------
+        filepath_or_buffer : str, path to be parsed or
+            an open readable stream.
+        {storage_options}
+        """
+        import_optional_dependency("python_calamine")
+        super().__init__(filepath_or_buffer, storage_options=storage_options)
+
+    @property
+    def _workbook_class(self):
+        from python_calamine import CalamineWorkbook
+
+        return CalamineWorkbook
+
+    def load_workbook(self, filepath_or_buffer: FilePath | ReadBuffer[bytes]):
+        from python_calamine import load_workbook
+
+        return load_workbook(filepath_or_buffer)  # type: ignore[arg-type]
+
+    @property
+    def sheet_names(self) -> list[str]:
+        return self.book.sheet_names  # pyright: ignore
+
+    def get_sheet_by_name(self, name: str):
+        self.raise_if_bad_sheet_by_name(name)
+        return self.book.get_sheet_by_name(name)  # pyright: ignore
+
+    def get_sheet_by_index(self, index: int):
+        self.raise_if_bad_sheet_by_index(index)
+        return self.book.get_sheet_by_index(index)  # pyright: ignore
+
+    def get_sheet_data(
+        self, sheet, file_rows_needed: int | None = None
+    ) -> list[list[Scalar]]:
+        def _convert_cell(value: _CellValueT) -> Scalar:
+            if isinstance(value, float):
+                val = int(value)
+                if val == value:
+                    return val
+                else:
+                    return value
+            elif isinstance(value, date):
+                return pd.Timestamp(value)
+            elif isinstance(value, time):
+                return value.isoformat()
+
+            return value
+
+        rows: list[list[_CellValueT]] = sheet.to_python(skip_empty_area=False)
+        data: list[list[Scalar]] = []
+
+        for row in rows:
+            data.append([_convert_cell(cell) for cell in row])
+            if file_rows_needed is not None and len(data) >= file_rows_needed:
+                break
+
+        return data