Skip to content

BUG: sum vs groupby.sum errors #38778

Closed
@cipriantrofin

Description

@cipriantrofin

data_source.zip

The attached data source is a subset of a larger set. I included just enough rows to show the error (when the datasource is larger, the difference of sums grows larger too).

Code Sample, a copy-pastable example

import pandas as pd
import numpy as np
pd.options.display.float_format = '{:.5f}'.format

df = pd.read_csv("data_source.csv", index_col = 0)

df.reset_index(inplace = True)

print ("Regular sum: %s\n" % df["rul_c"].sum())

print ("Regular sum on filtered column: %s\n" % df[df["cont"] == 20]["rul_c"].sum())

print ("GroupBy sum:\n%s" % df.groupby("cont")["rul_c"].sum())

Problem description

I know about floating point math and small associated "errors" but the "cont" column has an unique, not-null value, and this means all sums above should be the same (about 30880496049.43). However the groupby.sum is quite different from the expected result .45165.

The larger the dataset, the larger the difference.

Regular sum: 30880496049.429993

Regular sum on filtered column: 30880496049.429993

GroupBy sum:
cont
20   30880496049.45165
Name: rul_c, dtype: float64

Expected Output

All sums should be 30880496049.429993

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 3e89b4c
python : 3.8.5.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.18362
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : Romanian_Romania.1252

pandas : 1.2.0
numpy : 1.19.4
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.3
setuptools : 47.1.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 1.3.4
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.18.1
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.2
numexpr : 2.7.1
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.5.2
sqlalchemy : None
tables : 3.6.1
tabulate : None
xarray : None
xlrd : 1.2.0
xlwt : None
numba : None

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions