BUG: first() changes datetime64 data

[chrisbyboston]

I have a dataframe that contains a datetime64 column that seems to be losing some digits after a groupby.first(). I know that under the hood these are stored at nanosecond precision, but I need them to remain equal, since I'm merging back onto the original frame later on. Is this expected behavior?

import pandas as pd
from pandas import Timestamp

data = [
 {'a': 1,
  'dateCreated': Timestamp('2011-01-20 12:50:28.593448')},
 {'a': 1,
  'dateCreated': Timestamp('2011-01-15 12:50:28.502376')},
 {'a': 1,
  'dateCreated': Timestamp('2011-01-15 12:50:28.472790')},
 {'a': 1,
  'dateCreated': Timestamp('2011-01-15 12:50:28.445286')}]

df = pd.DataFrame(data)

Output is:

In [6]: df
Out[6]:
   a                dateCreated
0  1 2011-01-20 12:50:28.593448
1  1 2011-01-15 12:50:28.502376
2  1 2011-01-15 12:50:28.472790
3  1 2011-01-15 12:50:28.445286

In [7]: df.groupby('a').first()
Out[7]:
                    dateCreated
a
1 2011-01-20 12:50:28.593447936

When I compare the datetime64 in the first row to the datetime64 after the groupby.first(), the two are not equal.

[shoyer]

This looks like a bug to me.

I can reproduce this on master.

[chrisbyboston]

I have isolated it to this line.

https://github.com/pydata/pandas/blob/master/pandas/core/internals.py#L1818

I'm going to see if I can determine if this is a numpy bug or if we just need to pass some additional parameters to numpy.ndarray.astype.

[chrisbyboston]

Interesting...

numpy casting defaults to "unsafe" for backward compatibility reasons. When I pass in "safe":

result = result.astype('M8[ns]', casting='safe')

TypeError: Cannot cast array from dtype('float64') to dtype('<M8[ns]') according to the rule 'safe'

So numpy is even aware that this can't be done safely.

[chrisbyboston]

If I leave casting="safe", then all is well, as it raises an error and fails over safely:

def _groupby_function(name, alias, npfunc, numeric_only=True,
                      _convert=False):
    def f(self):
        self._set_selection_from_grouper()
        try:
            return self._cython_agg_general(alias, numeric_only=numeric_only)
        except AssertionError as e:
            raise SpecificationError(str(e))
        except Exception:
            result = self.aggregate(lambda x: npfunc(x, axis=self.axis))
            if _convert:
                result = result.convert_objects()
            return result

    f.__doc__ = "Compute %s of group values" % name
    f.__name__ = name

    return f

Result:

   a                dateCreated
0  1 2011-01-20 12:50:28.593448
1  1 2011-01-15 12:50:28.502376
2  1 2011-01-15 12:50:28.472790
3  1 2011-01-15 12:50:28.445286


                 dateCreated
a
1 2011-01-20 12:50:28.593448

I'll work up a PR and a test... and see if I break all the things.

[shoyer]

The real question is why this is being cast to float64 in the first place. We might need to duplicate an internal cython routine so that there is an int64 version that we can safely use without needing any casting.

[chrisbyboston]

@shoyer I'll take a peek at when that gets converted to float64. I didn't catch that at all, but you're right. It should be able to safely convert an int64.

[chrisbyboston]

It runs an ensure_float on this line: https://github.com/pydata/pandas/blob/master/pandas/core/groupby.py#L1508

[chrisbyboston]

Alright, looks like the cython in generated.pyx needs to have functions for group_nth_ for int8, int16, int32, and int64.

Sound right to you @shoyer?

[shoyer]

@iwschris That file is generated from pandas/src/generate_code.py, and it looks like integers were explicitly excluded, probably because something did not think they were necessary for groupby functions: https://github.com/pydata/pandas/blob/master/pandas/src/generate_code.py#L2407

[chrisbyboston]

Cool. I have a branch started that should fix this, and I'll enable integers for now, and see how things go from there.

[jreback]

@iwschris this would be a nice fix