asfreq() issue with 'method' parameter

[usagliaschi]

Hi all,

I've noticed the following behaviour in .asfreq method when dealing with frames/series containing original NaN values. When calling for instance asfreq('W-FRI',method='ffill') only NaN generated by DateOffset are forward filled, while the already present NaN values are not treated. I'm trying to figure out whether this was an intentional feature or a bug. In the former case however I would say that the documentation is a bit misleading

http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.asfreq.html

Indeed, the description of parameter 'method' is exactly equivalent to that present in fillna,

http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.fillna.html

so it comes natural to expect all invalid observations be filled.

Thanks for your help,
Umberto

[jreback]

pls provide an example and pd.show_versions()

[usagliaschi]

Hello jreback,

            In [32]: pd.show_versions()

            INSTALLED VERSIONS
                            ------------------
            commit: None
            python: 2.7.9.final.0
            python-bits: 64
            OS: Darwin
            OS-release: 14.1.0
            machine: x86_64
            processor: i386
            byteorder: little
            LC_ALL: None
            LANG: None
            pandas: 0.16.0
            nose: 1.3.4
            Cython: 0.22
            numpy: 1.9.2
            scipy: 0.15.1
            statsmodels: 0.6.1
            IPython: 3.0.0
            sphinx: None
            patsy: 0.3.0
            dateutil: 2.4.2
            pytz: 2015.2
            bottleneck: None
            tables: 3.1.1
            numexpr: 2.4
            matplotlib: 1.4.3
            openpyxl: 2.2.0
            xlrd: 0.9.3
            xlwt: 0.7.5
            xlsxwriter: 0.6.7
            lxml: 3.4.2
            bs4: None
            html5lib: None
            httplib2: None
            apiclient: None
            sqlalchemy: None
            pymysql: None
            psycopg2: None

Here's an example. The original data are contained in this DataFrame

            In [48]: df
            Out[48]: 
                            foo       baz
            2015-04-01  0.740599  0.129932
            2015-04-02  0.739513  0.129732
            2015-04-03       NaN  0.129759
            2015-04-06       NaN  0.129584
            2015-04-07  0.742161  0.130395
            2015-04-08  0.745335  0.130769
            2015-04-09  0.747672  0.131258
            2015-04-13  0.750516  0.131806
            2015-04-14  0.747754  0.131592
            2015-04-15  0.750230  0.132054
            2015-04-16  0.747818  0.131769
            2015-04-17  0.742453  0.130972
            2015-04-20  0.743158  0.131129
            2015-04-21       NaN  0.131238

Notice that "Friday, 10th April" is missing. Now,

            In [49]: df.asfreq(freq='W-FRI',method='ffill')
            Out[49]: 
                          foo         baz
            2015-04-03       NaN  0.129759
            2015-04-10  0.747672  0.131258
            2015-04-17  0.742453  0.130972

As you can see forward filling took place only for the missing Friday.

Thanks again,
Umberto

[jreback]

.asfreq just expands the days, selects based on the new freq and then fills. You prob want.

In [11]: df.ffill().asfreq(freq='W-FRI',method='ffill')        
Out[11]: 
                 foo       baz
2015-04-03  0.739513  0.129759
2015-04-10  0.747672  0.131258
2015-04-17  0.742453  0.130972

[usagliaschi]

Thanks for your reply,

As you also noticed filling occurs only for NaN generated by imposing the new frequency and not on original missing data. My personal opinion is that this behaviour should be highlighted in the documentation, otherwise one may assume that all holes/NaN are filled.

Umberto