Skip to content

BUG: fix concat to work with more iterables (GH8645) #8668

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 29, 2014
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions doc/source/whatsnew/v0.15.1.txt
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,28 @@ API changes

gr.apply(sum)

- ``concat`` permits a wider variety of iterables of pandas objects to be
passed as the first parameter (:issue:`8645`):

.. ipython:: python

from collections import deque
df1 = pd.DataFrame([1, 2, 3])
df2 = pd.DataFrame([4, 5, 6])

previous behavior:

.. code-block:: python

In [7]: pd.concat(deque((df1, df2)))
TypeError: first argument must be a list-like of pandas objects, you passed an object of type "deque"

current behavior:

.. ipython:: python

pd.concat(deque((df1, df2)))

.. _whatsnew_0151.enhancements:

Enhancements
Expand Down
6 changes: 3 additions & 3 deletions pandas/tools/merge.py
Original file line number Diff line number Diff line change
Expand Up @@ -675,7 +675,7 @@ def concat(objs, axis=0, join='outer', join_axes=None, ignore_index=False,

Parameters
----------
objs : list or dict of Series, DataFrame, or Panel objects
objs : a sequence or mapping of Series, DataFrame, or Panel objects
If a dict is passed, the sorted keys will be used as the `keys`
argument, unless it is passed, in which case the values will be
selected (see below). Any None objects will be dropped silently unless
Expand Down Expand Up @@ -731,8 +731,8 @@ class _Concatenator(object):
def __init__(self, objs, axis=0, join='outer', join_axes=None,
keys=None, levels=None, names=None,
ignore_index=False, verify_integrity=False, copy=True):
if not isinstance(objs, (list,tuple,types.GeneratorType,dict,TextFileReader)):
raise TypeError('first argument must be a list-like of pandas '
if isinstance(objs, (NDFrame, compat.string_types)):
raise TypeError('first argument must be an iterable of pandas '
'objects, you passed an object of type '
'"{0}"'.format(type(objs).__name__))

Expand Down
27 changes: 27 additions & 0 deletions pandas/tools/tests/test_merge.py
Original file line number Diff line number Diff line change
Expand Up @@ -2203,6 +2203,33 @@ def test_concat_series_axis1_same_names_ignore_index(self):
result = concat([s1, s2], axis=1, ignore_index=True)
self.assertTrue(np.array_equal(result.columns, [0, 1]))

def test_concat_iterables(self):
from collections import deque, Iterable

# GH8645 check concat works with tuples, list, generators, and weird
# stuff like deque and custom iterables
df1 = DataFrame([1, 2, 3])
df2 = DataFrame([4, 5, 6])
expected = DataFrame([1, 2, 3, 4, 5, 6])
assert_frame_equal(pd.concat((df1, df2), ignore_index=True), expected)
assert_frame_equal(pd.concat([df1, df2], ignore_index=True), expected)
assert_frame_equal(pd.concat((df for df in (df1, df2)), ignore_index=True), expected)
assert_frame_equal(pd.concat(deque((df1, df2)), ignore_index=True), expected)
class CustomIterator1(object):
def __len__(self):
return 2
def __getitem__(self, index):
try:
return {0: df1, 1: df2}[index]
except KeyError:
raise IndexError
assert_frame_equal(pd.concat(CustomIterator1(), ignore_index=True), expected)
class CustomIterator2(Iterable):
def __iter__(self):
yield df1
yield df2
assert_frame_equal(pd.concat(CustomIterator2(), ignore_index=True), expected)

def test_concat_invalid(self):

# trying to concat a ndframe with a non-ndframe
Expand Down