BUG: Small numbers in scientific notation do not show the right significant figures in DataFrame.

[Vital-Fernandez]

Research

• I have searched the [pandas] tag on StackOverflow for similar questions.

• I have asked my usage related question on StackOverflow.

Link to question on StackOverflow

https://stackoverflow.com/questions/71359050/pandas-series-not-showing-small-numbers-in-scientific-notation-depending-on-firs/71359216#71359216

Question about pandas

I am using pandas dataframes to store astronomical data (this is my library) and the default unit system has many variables in scientific notation. I would like to use the to_string() function to save the data as a text file. This is an example of the workflow:

import numpy as np
import pandas as pd

myDF = pd.DataFrame(columns=['c0', 'c1', 'c2', 'c3'])

myDF.loc['r0', 'c0'] = 'abc'
myDF.loc['r0', 'c1'] = np.nan
myDF.loc['r0', 'c2'] = 1234.0
myDF.loc['r0', 'c3'] = 1.234e-18

# Save the table as a dataframe.
with open(f'output_DF.txt', 'wb') as output_file:
    string_DF = myDF.to_string()
    output_file.write(string_DF.encode('UTF-8'))

print('Output DF\n', myDF)

Calling the cell data from the dataframe returns the right value (in this case 1.234e-18) but the dataframe is stored in a text file as:

      c0   c1     c2   c3
r0  abc  NaN  1234.0  0.0

This is also how it is displayed on the terminal. I think that the right behaviour should preserve the significant figures. I am not sure if this is an issue with the dtypes in the dataframe creation or while writting the output file.

I am using pandas 1.4.1 in windows 10. I wonder if anyone could please offer me some advice on the right way to do this operation.

Thank you for your work.

[attack68]

as a workaround you can use Styler.to_string in combination with Styler.format:

>>> print(myDF.style.format(subset="c3", formatter="{:.3e}").to_string())
 c0 c1 c2 c3
r0 abc nan 1234.000000 1.234e-18

[Vital-Fernandez]

Thank you, @attack68, very much for your reply. This is an issue I don't have in some scripts so I wonder if it is due to a different OS or pandas version or the data input order.

So far I have not been able to replicate the "good behaviour" but using the pandas.Series you can see that the variables type has an impact. For example:

iimport numpy as np
import pandas as pd

print(f'Pandas {pd.__version__}')

columns = ['c0', 'c1', 'c2', 'c3']
mySeries = pd.Series(index=columns, dtype='float')

mySeries['c0'] = 'abc'
mySeries['c1'] = np.nan
mySeries['c2'] = 1234.0
mySeries['c3'] = 1.234e-18

print(mySeries)

columns = ['c0', 'c1', 'c2', 'c3']
mySeries = pd.Series(index=columns, dtype='float')

mySeries['c0'] = 123
mySeries['c1'] = np.nan
mySeries['c2'] = 1234.0
mySeries['c3'] = 1.234e-18

print(mySeries)

In the first case, with one string entry, we loose the significant figures for the small numbers:

c0       abc
c1       NaN
c2    1234.0
c3       0.0
dtype: object

In the second case, with only floats, the signficant figures are showed but we loose the smart formatting of pandas (the floats are always displayed with 6 significant figures in scientific notation)

c0    1.230000e+02
c1             NaN
c2    1.234000e+03
c3    1.234000e-18
dtype: float64

Is it possible to have several variable types in a series and keep the input user format? Is there a dtype which provides this behaviour or is it working as intended?

Thanks again.

[ArLough]

take

[ArLough]

Hi @mroeschke , I'm new to open source and looking to contribute. Is there anything else I need to do besides 'take' in order to get started on tackling the issue? I see that there is a needs discussion tag.

[mroeschke]

@ArLough I would recommend not tackling an issue with any of the Needs ... tags as further discussion might be needed to determine the action item. Instead I would suggest looking for issues with good first issue

[ArLough]

Ok, thank you! That's helpful