dtypes convert to object on merge on 1.0.0rc0

[mdoom23]

dtypes convert to object on merge

Currently on 1.0.0rc0, when doing a left merge with datetime64[ns] on the right dataframe, if any rows from the left dataframe don't have a match on the right dataframe, then the result dataframe converts datetime to be object. If all items match, then it will remain as a datetime column. This previously maintained dtype in 0.25.3 and 0.24.2.

It seems to no longer maintain the dtype and populate values with NaT.

With 1.0.0rc0, after this I am able to convert to datetime column and it'll properly recognize as a NaT value.

Example with extra value in left dataframe

df1 = pd.DataFrame({'x': {0: 'a', 1: 'b', 2:'c'}, 'y': {0: '1', 1: '2', 2:'4'}})

df2 = pd.DataFrame({'y': {0: '1', 1: '2', 2:'3'}, 'z': {0: '2018-05-01', 1: '2018-05-02', 2:'2018-05-03'}})
df2['z'] = df2['z'].astype('datetime64[ns]')

result = pd.merge(df1, df2, how='left', on='y')

Output

  # 0.24.2
result.dtypes
x            object
y            object
z    datetime64[ns]
dtype: object

  # 0.25.3
result.dtypes
x            object
y            object
z    datetime64[ns]
dtype: object

  # 1.0.0rc0
result.dtypes
x            object
y            object
z            object
dtype: object

[TomAugspurger]

Thanks for the excellent report. This is a hacky fix

diff --git a/pandas/core/internals/concat.py b/pandas/core/internals/concat.py
index c6f30ef65e..bc307d6b74 100644
--- a/pandas/core/internals/concat.py
+++ b/pandas/core/internals/concat.py
@@ -242,6 +242,8 @@ def concatenate_join_units(join_units, concat_axis, copy):
         raise AssertionError("Concatenating join units along axis0")
 
     empty_dtype, upcasted_na = _get_empty_dtype_and_na(join_units)
+    if is_datetime64_dtype(empty_dtype) and upcasted_na == tslibs.iNaT:
+        upcasted_na = tslibs.NaT
 
     to_concat = [
         ju.get_reindexed_values(empty_dtype=empty_dtype, upcasted_na=upcasted_na)

In [3]: result
Out[3]:
   x  y          z
0  a  1 2018-05-01
1  b  2 2018-05-02
2  c  4        NaT

In [4]: result.dtypes
Out[4]:
x            object
y            object
z    datetime64[ns]
dtype: object

I don't recall why we're using iNaT for upcasted NA there. The git blame isn't too informative. I'll see what all fails.