Skip to content

ENH: add arrow engine to read_csv #31817

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 51 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
f22ff46
add arrow engine to read_csv
lithomas1 Feb 9, 2020
8ae43e4
fix failing test
lithomas1 Feb 9, 2020
09074df
formatting and revert unnecessary change
lithomas1 Feb 9, 2020
6be276d
remove bloat and more formatting changes
lithomas1 Feb 9, 2020
df4fa7e
Whatsnew
lithomas1 Feb 9, 2020
9cd9a6f
Merge remote-tracking branch 'upstream/master' into add-arrow-engine
lithomas1 Feb 9, 2020
ecaf3fd
Get tests up and running
lithomas1 Feb 10, 2020
b3c3287
Some fixes
lithomas1 Feb 10, 2020
474baf4
Add asvs and xfail some tests
lithomas1 Feb 11, 2020
2cd9937
address comments
lithomas1 Feb 20, 2020
48ff255
Merge branch 'master' into add-arrow-engine
lithomas1 Feb 20, 2020
3d15a56
fix typo
lithomas1 Feb 20, 2020
c969373
Merge branch 'add-arrow-engine' of github-other.com:lithomas1/pandas …
lithomas1 Feb 20, 2020
98aa134
some fixes
lithomas1 Feb 29, 2020
b9c6d2c
Fix bug
lithomas1 Apr 5, 2020
67c5db6
Fix merge conflicts
lithomas1 Apr 5, 2020
7f891a6
New benchmark and fix more tests
lithomas1 Apr 10, 2020
11fc737
Merge branch 'master' into add-arrow-engine
lithomas1 Apr 10, 2020
23425f7
More cleanups
lithomas1 Apr 10, 2020
d9b7a1f
Merge master
lithomas1 Apr 10, 2020
b8adf3c
Merge branch 'add-arrow-engine' of github-other.com:lithomas1/pandas …
lithomas1 Apr 11, 2020
01c0394
Formatting fixes and typo correction
lithomas1 Apr 11, 2020
ba5620f
skip pyarrow tests if not installed
lithomas1 Apr 12, 2020
2570c82
Address comments
lithomas1 Apr 12, 2020
b3a1f66
Get some more tests to pass
lithomas1 Apr 14, 2020
d46ceed
Fix some bugs and cleanups
lithomas1 Apr 17, 2020
d67925c
Merge branch 'master' into add-arrow-engine
lithomas1 Apr 17, 2020
6378459
Perform version checks for submodule imports too
lithomas1 May 20, 2020
9d64882
Refresh with newer pyarrow
lithomas1 May 20, 2020
852ecf9
Merge branch 'master' into add-arrow-engine
lithomas1 May 20, 2020
93382b4
Start xfailing tests
lithomas1 May 21, 2020
f1bb4e2
Get all tests to run & some fixes
lithomas1 May 27, 2020
14c13ab
Merge branch 'master' into add-arrow-engine
lithomas1 May 27, 2020
7876b4e
Lint and CI
lithomas1 May 29, 2020
4426642
Merge branch 'master' into add-arrow-engine
lithomas1 May 29, 2020
008acab
parse_dates support and fixups of some tests
lithomas1 Jun 3, 2020
2dddae7
Date parsing fixes and address comments
lithomas1 Jun 13, 2020
261ef6a
Merge branch 'master' into add-arrow-engine
lithomas1 Jun 13, 2020
88e200a
Clean/Address comments/Update docs
lithomas1 Jun 29, 2020
bf063ab
Merge branch 'master' into add-arrow-engine
lithomas1 Jun 29, 2020
ede2799
Fix typo
lithomas1 Jun 29, 2020
e8eff08
Fix doc failures
lithomas1 Jul 8, 2020
87cfcf5
Merge remote-tracking branch 'upstream/master' into add-arrow-engine
simonjayhawkins Oct 22, 2020
55139ee
wip
simonjayhawkins Oct 22, 2020
c1aeecf
more xfails and skips
simonjayhawkins Oct 22, 2020
62fc9d6
Merge branch 'master' into add-arrow-engine
lithomas1 Oct 28, 2020
b53a620
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 28, 2020
f13113d
Fix typos
lithomas1 Oct 28, 2020
f9ce2e4
Doc fixes and more typo fixes
lithomas1 Oct 28, 2020
4158d6a
Green?
lithomas1 Nov 2, 2020
d34e75f
Merge branch 'master' into add-arrow-engine
lithomas1 Nov 17, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion pandas/tests/io/parser/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -142,4 +142,16 @@ def pyarrow_xfail(request):
if "all_parsers" in request.fixturenames:
parser = request.getfixturevalue("all_parsers")
if parser.engine == "pyarrow":
pytest.xfail("pyarrow doesn't support this.")
mark = pytest.mark.xfail(reason="pyarrow doesn't support this.")
request.node.add_marker(mark)


@pytest.fixture
def pyarrow_skip(request):
"""
Fixture that skips a test if the engine is pyarrow.
"""
if "all_parsers" in request.fixturenames:
parser = request.getfixturevalue("all_parsers")
if parser.engine == "pyarrow":
pytest.skip("pyarrow doesn't support this.")
2 changes: 2 additions & 0 deletions pandas/tests/io/parser/test_comment.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@
from pandas import DataFrame
import pandas._testing as tm

pytestmark = pytest.mark.usefixtures("pyarrow_xfail")


@pytest.mark.parametrize("na_values", [None, ["NaN"]])
def test_comment(all_parsers, na_values):
Expand Down
Loading