Skip to content

Rounding errors when parsing floats #73

Closed
@abaco

Description

@abaco

I was running some generative tests on my code and I ran into an issue with high precision floats. For example:

> x = 0.7531531167929774
> runParser (show x) testTokenParser.float
(Right 0.7531531167929775)

Note the final 5 instead of 4. The problem seems to be in the code that parses the decimal part of a number. Indeed, the following equivalent code gives the same erroneous result:

> digits = [7,5,3,1,5,3,1,1,6,7,9,2,9,7,7,4]
> foldr (\d f -> (f + toNumber d) / 10.0) 0.0 digits
0.7531531167929775

I'm not sure about the best way to modify fraction so as to avoid rounding errors. Maybe using the native parseFloat?

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: bugSomething that should function correctly isn't.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions