BUG: read_csv with date_parser lock file open on failure

[rsheftel]

Problem description

When using the date_parser functionality of read_csv() if the file read fails then the file is left locked open. My use case is that I am trying to enforce a strict datetime format that must include the time zone offset. Another thread stated that the way to accomplish this is with date_parser. My issue is that I would like to move a file that fails being loaded to another directory, but I cannot because the failure in the date_parser keeps the file open until the python session is terminated.

Code Sample, a copy-pastable example if possible

import pandas as pd
import datetime

def strict_parser(dates):
    datetimes = [pd.Timestamp(datetime.datetime.strptime(date, '%Y-%m-%d %H:%M:%S%z'), tz='UTC') for date in dates]
    return pd.DatetimeIndex(datetimes)

filename = 'c:/temp/data.csv'
data = pd.read_csv(filename, parse_dates=['datetime'], index_col=['datetime'], date_parser=strict_parser)

The file data.csv is:

datetime,data
2010-05-05 09:30:00-0500,10
2010-05-05 09:35:00-0500,20
2010-05-05 09:40:00,30

Output of pd.show_versions()

pandas: 0.19.1

[jreback]

We had a very similar issue, can't seem to find this now. But on windows, this is fixed I think in 0.19.2, but maybe 0.20.0. (still in dev)

can you try?

[rsheftel]

I tried with 0.19.2 and the problem remains. When 0.20.0 becomes available on one of my systems I will give that a try.

[jreback]

I thought this was solved, but was able to repro on windows.

So it seems that we are not closing on calling a date_parser function (we raise, but the top-level needs a try/finally around this to close the handle). We do this for other errors, this must have been missed.

@rsheftel would you like to do a pull-request to fix?

[rsheftel]

I don't think I am enough of an expert in the pandas code to submit a change that fixes this. Do you want me to just create a blank pull-request? (Sorry I'm new to the GitHub / collaborative world and how exactly it works)

[jreback]

no this would be a regular pull-request that has tests and makes the change.

http://pandas.pydata.org/pandas-docs/stable/contributing.html

[rsheftel]

If I gain expertise in the code base and feel I can confidently contribute I will. Thanks.