Skip to content

difference in overflows handling while doing pow on large integers #15046

Closed
@yarikoptic

Description

@yarikoptic

Output from running tests and troubleshooting

probably related to having numpy beta release or freshish numexpr...

$> nosetests -s -v --pdb pandas/tests/test_expressions.py
...
test_integer_arithmetic (pandas.tests.test_expressions.TestExpressions) ... > /home/yoh/deb/gits/pkg-exppsy/pandas/pandas/util/testing.py(1018)raise_assert_detail()
-> raise AssertionError(msg)
*(Pdb) print msg
DataFrame.iloc[:, 0] are different

DataFrame.iloc[:, 0] values are different (85.16148 %)
[left]:  [2565992168703393792, 2110287360296998863, 0, 0, 2644539488379390509, -5278486589563110205, -4689350456247753643, 8450172506621111015, -2101438300051996672, 8028209804145959321, 302875106592253, -5278486589563110205, -4053175462519618527, -4071272324936764365, 0, 0, 0, 256, 8450172506621111015, 0, -145357957422043229, 3030756035345000135, 256, 0, -4071272324936764365, -5278486589563110205, -2863221430593058543, 0, 0, 0, 4951460147845608313, 0, -1595931050845505211, -5821812116355782473, -6284992600390200471, 2435553365352800019, -1595931050845505211, -4053175462519618527, 0, 3792227444683241861, -6776596920136667815, -5821812116355782473, 0, -8795647696940200745, 4593512421086651489, 256, 0, 0, 2644539488379390509, -5821812116355782473, -7924192819420782515, 1826852818786091687, 0, 2110287360296998863, -2559158286230232783, -4689350456247753643, -6646187150092009472, 8028209804145959321, 0, 437893890380859375, 0, -5278486589563110205, 0, 2467972595799031808, 2604998672350111773, -4689350456247753643, 7437354637604294721, 0, 285311670611, 5529354540715494413, 2435553365352800019, -1595931050845505211, 256, 823543, -2559158286230232783, 0, -3660930864499998817, 0, -4642015662142636065, -4071272324936764365, 4611686018427387904, 285311670611, 1, 0, 2565992168703393792, 8028209804145959321, -145357957422043229, 8450172506621111015, -5821812116355782473, 0, -4675932718989443137, 8037740809528595931, -4071272324936764365, 0, 6353754964178307979, 0, -1595931050845505211, -1595931050845505211, -3264474685660004015, 2823291043684304491, ...]
[right]: [-9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, 302875106592253, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, 256, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, 256, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, 256, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, 437893890380859375, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, 285311670611, -9223372036854775808, -9223372036854775808, -9223372036854775808, 256, 823543, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, 285311670611, 1, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, ...]
(Pdb) up
> /home/yoh/deb/gits/pkg-exppsy/pandas/pandas/src/testing.pyx(173)pandas._testing.assert_almost_equal (pandas/src/testing.c:3274)()
-> raise_assert_detail(obj, msg, lobj, robj)
(Pdb) 
> /home/yoh/deb/gits/pkg-exppsy/pandas/pandas/src/testing.pyx(59)pandas._testing.assert_almost_equal (pandas/src/testing.c:4156)()
-> cpdef assert_almost_equal(a, b,
(Pdb) 
> /home/yoh/deb/gits/pkg-exppsy/pandas/pandas/util/testing.py(1181)assert_series_equal()
-> obj='{0}'.format(obj))
(Pdb) 
> /home/yoh/deb/gits/pkg-exppsy/pandas/pandas/util/testing.py(1313)assert_frame_equal()
-> obj='DataFrame.iloc[:, {0}]'.format(i))
(Pdb) 
> /home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py(100)run_arithmetic_test()
-> assert_func(expected, result)
(Pdb) p df
        A   B   C   D
0      30  89  97  21
1      47  22  16  80
2      78  23  84  33
3      88  25  61  59
4      61   1  41  96
5      27  67  76  98
6      37  13  13  82
7      23  60  93   1
8      20   6  18  90
9      89  97  27  73
10     13  95  37  38
11     27  53  70  77
12     33  51  44  39
13     43  89  95   5
14     36  84  51  97
15     44   8  82  31
16     84  18  50  35
17      4  52  74  37
18     23  52  73  42
19     76  99  11  67
20     59  95  87  68
21     55  18  89  73
22      4  26  64  19
23     94  59  47  36
24     43  79  37  26
25     27  42  41  84
26     17  30   4   3
27     24  37  63  88
28     66  14  92  86
29     52  38  27  97
...    ..  ..  ..  ..
9971   25  13  53  68
9972    9  46  89  93
9973   70  53  47  33
9974   10  30  47   3
9975   98  15  98  43
9976   34   5  53  76
9977   21  29  65  95
9978   30  83  67  82
9979   52   1  58  17
9980    4  84  38  79
9981   83  96  57  12
9982   13  38  93  72
9983   58  64  24  13
9984   18  46   5  43
9985   70  84  88  45
9986   79  94  57  29
9987   18  31  63  57
9988   58  90  99   5
9989    8  88  77  55
9990   46  12  61  61
9991   69  70  96  45
9992   71  67  48  25
9993   77  68  46  16
9994   88  55  80  37
9995    6  93  76  73
9996   11  68  45  52
9997   57  79   3  69
9998   63  48  79   8
9999   11   5  70  16
10000  10  32  32  88

[10001 rows x 4 columns]
(Pdb) p op
<function pow at 0x7efedaaad758>

Problem description

Not sure what to blame here -- numexpr or numpy or ... What would you expect while trying to get e.g. 30**30? max for int of original pow base type?

Output of pd.show_versions()

# Paste the output here pd.show_versions() here $> ./ci/print_versions.py

INSTALLED VERSIONS

commit: ad13f08
python: 2.7.13.candidate.1
python-bits: 64
OS: Linux
OS-release: 4.8.0-1-amd64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.19.0+git14-ga40e185
nose: 1.3.7
pip: 8.1.2
setuptools: 28.7.1
Cython: 0.25.2b0
numpy: 1.12.0b1
scipy: 0.18.1
statsmodels: 0.8.0rc1
xarray: None
IPython: 5.1.0
sphinx: 1.4.8
patsy: 0.4.1+dev
dateutil: 2.5.3
pytz: 2015.7
blosc: None
bottleneck: None
tables: 3.3.0
numexpr: 2.6.1
feather: None
matplotlib: 1.5.3
openpyxl: 2.3.0
xlrd: 1.0.0
xlwt: 0.7.5
xlsxwriter: None
lxml: 3.6.4
bs4: 4.5.1
html5lib: 0.999
httplib2: 0.9.2
apiclient: 1.5.2
sqlalchemy: 1.0.15
pymysql: None
psycopg2: 2.6.2 (dt dec mx pq3 ext lo64)
jinja2: 2.8
s3fs: None
pandas_datareader: None

Metadata

Metadata

Assignees

No one assigned

    Labels

    Compatpandas objects compatability with Numpy or Python functionsNumeric OperationsArithmetic, Comparison, and Logical operationsTestingpandas testing functions or related to the test suite

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions