Inconsistent display of numbers in dataframe

[markrichardson]

Using pandas 0.13.1, I see see that there is an apparent inconsistency in the format in which floating-point numbers in my dataframe are displayed. In particular, the numbers in the first column appear to be displayed in the form '%1.8e', whereas the other numbers are displayed as '%1.6f'.

                   col_0       col_1       col_2       col_3
2013-01-18  3.132256e+00         NaN         NaN         NaN
2013-01-21  3.513191e+00         NaN         NaN         NaN
2013-01-22  2.581368e+00         NaN         NaN         NaN
2013-01-23  1.751014e+00         NaN         NaN         NaN
2013-01-24  2.508899e+00         NaN         NaN         NaN
2013-01-25  4.085701e+00         NaN         NaN         NaN
2013-01-28  3.400867e+00         NaN         NaN         NaN
2013-01-29  3.030346e+00         NaN         NaN         NaN
2013-01-30  2.934522e+00         NaN         NaN         NaN
2013-01-31  6.548878e-01         NaN         NaN         NaN
2013-02-01  4.743263e-01         NaN         NaN         NaN
2013-02-04  5.372891e-02         NaN         NaN         NaN
2013-02-05  8.271173e-02         NaN         NaN         NaN
2013-02-06  9.247646e-03         NaN         NaN         NaN
2013-02-07  1.499508e-03         NaN         NaN         NaN
2013-02-08  6.927766e-04         NaN         NaN         NaN
2013-02-11  7.232857e-05         NaN         NaN         NaN
2013-02-12  3.274444e-05         NaN         NaN         NaN
2013-02-13  4.520291e-07         NaN         NaN         NaN
2013-02-14  2.092958e-16         NaN         NaN         NaN
2013-02-15  0.000000e+00    6.815180         NaN         NaN
2013-02-18           NaN    7.581717         NaN         NaN
2013-02-19           NaN   15.039587         NaN         NaN
2013-02-20           NaN   10.740931         NaN         NaN
2013-02-21           NaN    4.101203         NaN         NaN
2013-02-22           NaN    9.078457         NaN         NaN
2013-02-25           NaN   13.372686         NaN         NaN
2013-02-26           NaN    4.394059         NaN         NaN
2013-02-27           NaN    4.621631         NaN         NaN
2013-02-28           NaN    5.737833         NaN         NaN
2013-03-01           NaN    3.454591         NaN         NaN
2013-03-04           NaN    2.521326         NaN         NaN
2013-03-05           NaN    9.589268         NaN         NaN
2013-03-06           NaN    8.321083         NaN         NaN
2013-03-07           NaN    8.069279         NaN         NaN
2013-03-08           NaN   16.680768         NaN         NaN
2013-03-11           NaN    9.940089         NaN         NaN
2013-03-12           NaN    5.922898         NaN         NaN
2013-03-13           NaN    2.206807         NaN         NaN
2013-03-14           NaN    9.449342         NaN         NaN
2013-03-15           NaN    0.000000    5.331617         NaN
2013-03-18           NaN         NaN    3.320303         NaN
2013-03-19           NaN         NaN    2.480900         NaN
2013-03-20           NaN         NaN    2.787095         NaN
2013-03-21           NaN         NaN    2.157647         NaN
2013-03-22           NaN         NaN    2.795969         NaN
2013-03-25           NaN         NaN    0.884890         NaN
2013-03-26           NaN         NaN    0.425347         NaN
2013-03-27           NaN         NaN    0.376083         NaN
2013-03-28           NaN         NaN    0.332580         NaN
2013-04-02           NaN         NaN    0.689754         NaN
2013-04-03           NaN         NaN    0.256095         NaN
2013-04-04           NaN         NaN    0.157584         NaN
2013-04-05           NaN         NaN    0.084014         NaN
2013-04-08           NaN         NaN    0.023838         NaN
2013-04-09           NaN         NaN    0.007898         NaN
2013-04-10           NaN         NaN    0.064029         NaN
2013-04-11           NaN         NaN    0.047283         NaN
2013-04-12           NaN         NaN    0.010981         NaN
2013-04-15           NaN         NaN    0.000285         NaN
                     ...         ...         ...         ...

[64 rows x 4 columns]

df.values
array([[  3.13225571e+00,              nan,              nan,
                     nan],
       [  3.51319052e+00,              nan,              nan,
                     nan],
       [  2.58136815e+00,              nan,              nan,
                     nan],
       [  1.75101397e+00,              nan,              nan,
                     nan],
       [  2.50889891e+00,              nan,              nan,
                     nan],
       [  4.08570093e+00,              nan,              nan,
                     nan],
       [  3.40086744e+00,              nan,              nan,
                     nan],
       [  3.03034635e+00,              nan,              nan,
                     nan],
       [  2.93452229e+00,              nan,              nan,
                     nan],
       [  6.54887773e-01,              nan,              nan,
                     nan],
       [  4.74326267e-01,              nan,              nan,
                     nan],
       [  5.37289144e-02,              nan,              nan,
                     nan],
       [  8.27117286e-02,              nan,              nan,
                     nan],
       [  9.24764587e-03,              nan,              nan,
                     nan],
       [  1.49950795e-03,              nan,              nan,
                     nan],
       [  6.92776578e-04,              nan,              nan,
                     nan],
       [  7.23285714e-05,              nan,              nan,
                     nan],
       [  3.27444365e-05,              nan,              nan,
                     nan],
       [  4.52029103e-07,              nan,              nan,
                     nan],
       [  2.09295814e-16,              nan,              nan,
                     nan],
       [  0.00000000e+00,   6.81517955e+00,              nan,
                     nan],
       [             nan,   7.58171652e+00,              nan,
                     nan],
       [             nan,   1.50395868e+01,              nan,
                     nan],
       [             nan,   1.07409307e+01,              nan,
                     nan],
       [             nan,   4.10120346e+00,              nan,
                     nan],
       [             nan,   9.07845736e+00,              nan,
                     nan],
       [             nan,   1.33726856e+01,              nan,
                     nan],
       [             nan,   4.39405879e+00,              nan,
                     nan],
       [             nan,   4.62163060e+00,              nan,
                     nan],
       [             nan,   5.73783295e+00,              nan,
                     nan],
       [             nan,   3.45459132e+00,              nan,
                     nan],
       [             nan,   2.52132635e+00,              nan,
                     nan],
       [             nan,   9.58926847e+00,              nan,
                     nan],
       [             nan,   8.32108299e+00,              nan,
                     nan],
       [             nan,   8.06927868e+00,              nan,
                     nan],
       [             nan,   1.66807682e+01,              nan,
                     nan],
       [             nan,   9.94008926e+00,              nan,
                     nan],
       [             nan,   5.92289775e+00,              nan,
                     nan],
       [             nan,   2.20680680e+00,              nan,
                     nan],
       [             nan,   9.44934225e+00,              nan,
                     nan],
       [             nan,   0.00000000e+00,   5.33161661e+00,
                     nan],
       [             nan,              nan,   3.32030302e+00,
                     nan],
       [             nan,              nan,   2.48089964e+00,
                     nan],
       [             nan,              nan,   2.78709474e+00,
                     nan],
       [             nan,              nan,   2.15764709e+00,
                     nan],
       [             nan,              nan,   2.79596877e+00,
                     nan],
       [             nan,              nan,   8.84890216e-01,
                     nan],
       [             nan,              nan,   4.25347224e-01,
                     nan],
       [             nan,              nan,   3.76083458e-01,
                     nan],
       [             nan,              nan,   3.32579876e-01,
                     nan],
       [             nan,              nan,   6.89754124e-01,
                     nan],
       [             nan,              nan,   2.56095287e-01,
                     nan],
       [             nan,              nan,   1.57583997e-01,
                     nan],
       [             nan,              nan,   8.40144066e-02,
                     nan],
       [             nan,              nan,   2.38379017e-02,
                     nan],
       [             nan,              nan,   7.89829707e-03,
                     nan],
       [             nan,              nan,   6.40289210e-02,
                     nan],
       [             nan,              nan,   4.72825870e-02,
                     nan],
       [             nan,              nan,   1.09810537e-02,
                     nan],
       [             nan,              nan,   2.84832649e-04,
                     nan],
       [             nan,              nan,   1.58283327e-06,
                     nan],
       [             nan,              nan,   3.91397716e-12,
                     nan],
       [             nan,              nan,   6.65980697e-23,
                     nan],
       [             nan,              nan,   0.00000000e+00,
          4.83556226e+00]])

df.index.values
array(['2013-01-18T00:00:00.000000000+0000',
       '2013-01-21T00:00:00.000000000+0000',
       '2013-01-22T00:00:00.000000000+0000',
       '2013-01-23T00:00:00.000000000+0000',
       '2013-01-24T00:00:00.000000000+0000',
       '2013-01-25T00:00:00.000000000+0000',
       '2013-01-28T00:00:00.000000000+0000',
       '2013-01-29T00:00:00.000000000+0000',
       '2013-01-30T00:00:00.000000000+0000',
       '2013-01-31T00:00:00.000000000+0000',
       '2013-02-01T00:00:00.000000000+0000',
       '2013-02-04T00:00:00.000000000+0000',
       '2013-02-05T00:00:00.000000000+0000',
       '2013-02-06T00:00:00.000000000+0000',
       '2013-02-07T00:00:00.000000000+0000',
       '2013-02-08T00:00:00.000000000+0000',
       '2013-02-11T00:00:00.000000000+0000',
       '2013-02-12T00:00:00.000000000+0000',
       '2013-02-13T00:00:00.000000000+0000',
       '2013-02-14T00:00:00.000000000+0000',
       '2013-02-15T00:00:00.000000000+0000',
       '2013-02-18T00:00:00.000000000+0000',
       '2013-02-19T00:00:00.000000000+0000',
       '2013-02-20T00:00:00.000000000+0000',
       '2013-02-21T00:00:00.000000000+0000',
       '2013-02-22T00:00:00.000000000+0000',
       '2013-02-25T00:00:00.000000000+0000',
       '2013-02-26T00:00:00.000000000+0000',
       '2013-02-27T00:00:00.000000000+0000',
       '2013-02-28T00:00:00.000000000+0000',
       '2013-03-01T00:00:00.000000000+0000',
       '2013-03-04T00:00:00.000000000+0000',
       '2013-03-05T00:00:00.000000000+0000',
       '2013-03-06T00:00:00.000000000+0000',
       '2013-03-07T00:00:00.000000000+0000',
       '2013-03-08T00:00:00.000000000+0000',
       '2013-03-11T00:00:00.000000000+0000',
       '2013-03-12T00:00:00.000000000+0000',
       '2013-03-13T00:00:00.000000000+0000',
       '2013-03-14T00:00:00.000000000+0000',
       '2013-03-15T00:00:00.000000000+0000',
       '2013-03-18T00:00:00.000000000+0000',
       '2013-03-19T00:00:00.000000000+0000',
       '2013-03-20T00:00:00.000000000+0000',
       '2013-03-21T00:00:00.000000000+0000',
       '2013-03-22T00:00:00.000000000+0000',
       '2013-03-25T00:00:00.000000000+0000',
       '2013-03-26T00:00:00.000000000+0000',
       '2013-03-27T00:00:00.000000000+0000',
       '2013-03-28T00:00:00.000000000+0000',
       '2013-04-02T01:00:00.000000000+0100',
       '2013-04-03T01:00:00.000000000+0100',
       '2013-04-04T01:00:00.000000000+0100',
       '2013-04-05T01:00:00.000000000+0100',
       '2013-04-08T01:00:00.000000000+0100',
       '2013-04-09T01:00:00.000000000+0100',
       '2013-04-10T01:00:00.000000000+0100',
       '2013-04-11T01:00:00.000000000+0100',
       '2013-04-12T01:00:00.000000000+0100',
       '2013-04-15T01:00:00.000000000+0100',
       '2013-04-16T01:00:00.000000000+0100',
       '2013-04-17T01:00:00.000000000+0100',
       '2013-04-18T01:00:00.000000000+0100',
       '2013-04-19T01:00:00.000000000+0100'], dtype='datetime64[ns]')

[jreback]

its the tiny value (1e-16). you can try display.chop_threshold I think

[jreback]

@gdirichlet how would you see this displayed? IIRC this is on purpose

[markrichardson]

The trade-off here seems to be retention of visual information versus loss of aesthetic appeal. Personally I value the second enough to be prepared to extract the column in question in order to examine further if I see "0.000000".

[jreback]

Here's a couple of options

not sure if this is a bug, or just what,
see also #6839

In [1]: df = DataFrame({ 'A' : [1.0,1e-16], 'B' : [np.nan,5.0] })

In [2]: df
Out[2]: 
              A   B
0  1.000000e+00 NaN
1  1.000000e-16   5

[2 rows x 2 columns]

In [4]: pd.describe_option('chop_threshold')
display.chop_threshold: [default: None] [currently: None]
: float or None
        if set to a float value, all float values smaller then the given threshold
        will be displayed as exactly 0 by repr and friends.

In [5]: pd.set_option('chop_threshold',1e-6)

In [6]: df
Out[6]: 
              A   B
0  1.000000e+00 NaN
1             0   5

[2 rows x 2 columns]

In [8]: pd.describe_option('float_format')
display.float_format: [default: None] [currently: None]
: callable
        The callable should accept a floating point number and return
        a string with the desired format of the number. This is used
        in some places like SeriesFormatter.
        See core.format.EngFormatter for an example.

In [11]: pd.set_option('float_format',lambda x: "%.6f" % x)

In [12]: df
Out[12]: 
         A        B
0 1.000000      nan
1 0.000000 5.000000

[2 rows x 2 columns]

[jreback]

closing as xref to #6842