`DataFrame.diff` on `datetime64` returns result inconsistent with `Series.diff`

[batterseapower]

Code Sample, a copy-pastable example if possible

>>> pd.Series(['NaT', '2019-01-01', '2019-01-02'], dtype='datetime64[ns]').diff()
0      NaT
1      NaT
2   1 days
dtype: timedelta64[ns]

>>> pd.Series(['NaT', '2019-01-01', '2019-01-02'], dtype='datetime64[ns]').to_frame().diff()
                             0
0                          NaT
1 -88855 days +00:12:43.145224
2              1 days 00:00:00

Problem description

Doing .diff() on a DataFrame of datetime64 should yield exactly time deltas in exactly the same way as on the Series i.e. index (1,0) in the DataFrame above should be NaT, not an overflown timedelta64.

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit : None python : 3.8.1.final.0 python-bits : 64 OS : Windows OS-release : 10 machine : AMD64 processor : Intel64 Family 6 Model 63 Stepping 2, GenuineIntel byteorder : little LC_ALL : None LANG : None LOCALE : English_United Kingdom.936

pandas : 1.0.1
numpy : 1.18.1
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 45.2.0.post20200210
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None

[shang-vikas]

Hi @batterseapower, I copy-pasted the same code and was able to reproduce the bug.
I did some deep dive and found out that for dataframe(ndim=2), a function diff_2d in algos.pyx is being used which is written in cython.
For series type, there is another logic which has a comment
# need to make sure that we account for na for datelike/timedelta # we don't actually want to subtract these i8 numbers
which i believe is handling the NaT values properly by some mask and i was not able to find this mask code in diff_2d function.

Just to be sure i put a systemtrace and got the following output.

System details

pandas==1.0.1
OS: Ubuntu 18.04

TRACE CODE

## https://pymotw.com/2/sys/tracing.html 
import sys
def trace_calls_and_returns(frame, event, arg):
    co = frame.f_code
    func_name = co.co_name
    if func_name == 'write':
        # Ignore write() calls from print statements
        return
    line_no = frame.f_lineno
    filename = co.co_filename
    if (event == 'call') and ('algorithms' in filename):
        print ('Call to %s on line %s of %s' % (func_name, line_no, filename))
        return trace_calls_and_returns
    if event == 'return':
        print( '%s => %s' % (func_name, arg))
    return

sys.settrace(trace_calls_and_returns)

I/Output

Series

DataFrame

Above ss clearly shows that NaT values are not handled in algos.pyx.
I would like to take this bug up and submit a PR(would be my first one).

Please correct me if i am wrong.

[batterseapower]

@shang-vikas: looks like you found the bug, nice one.

I think the code you are referring to is

pandas/pandas/core/algorithms.py

Line 1900 in 706e642

if is_timedelta:

(Series) and

pandas/pandas/_libs/algos.pyx

Line 1193 in 9e69040

def diff_2d(diff_t[:, :] arr,

(DataFrame). I'm sure the pandas project would be very happy to have a PR fixing the DataFrame case to match the Series one.

[shang-vikas]

take

[shang-vikas]

@batterseapower Should i start fixing the issue or wait for any guidelines from one of the members?

[batterseapower]

@shang-vikas I don't see why you shouldn't just submit a pull request. The pandas devs are quite responsive to applying pull requests, but are too busy to solve every issue themselves.

[jbrockmendel]

@shang-vikas pull request would be welcome. it sounds like you've already done most of the work by identifying the source of the problem