Numeric value returns 16 decimal places

[Marcus]

Hi,
I’ve a simple loop over all records.

The table has several same fields (Number, Currency Style, 2 Decimal Places).

Playing with that, every field has the value 1.99

In some cases getFieldValue returns “1.9899999999999998”

Is this a bug ?

[Brendan]

It’s a byproduct of the use of the Objective-C doubleValue method. The formatting and rounding according to the decimal places and Number Format settings doesn’t happen until display time. But internally the value is as it is. You would need to perform some JavaScript rounding of the values yourself if you need to do that.

[Marcus]

Thanks, Brendan.
I tried that already by using var.toFixed(2),
but this is running in a TypeError: toFixed is not a function.
This happens only when the script is running over such an affected value, for other values toFixed is working.

[Sam Moffatt]

My guess is that the number if actually a string, use typeof to check what type it is. If it’s a string then it wouldn’t have toFixed because that’s for numbers. If it is a string then perhaps try a parseFloat as a workaround.

[Marcus]

But should this happen when the field is numeric ?
However, found a workaround to fix it.

[Sam Moffatt]

Probably shouldn’t happen but bugs happen :)

What was your workaround?

[Marcus]

Ugly and sort of tricky.
Detecting the amount of decimal places.
If > 2 split the string to x.xx and convert to number.

Still unclear for me why this happens only for some fields and some values only, even if they all have the same format and the same value.

[Dave Emme]

Sorry, but that’s life when using floating point (non-integer) numbers in a general-purpose computer. It’s not anybody’s bug. It’s down to the fact that the representation of arbitrary floating point numbers in a fixed number of bits in the hardware can result in the computer being unable to represent some values exactly. So, for example, a floating point calculation which should result in 2.000 might instead give 1.99999…

[Sam Moffatt]

Yes, this is a standard problem with floating point numbers that values aren’t represented exactly. but are an approximation. Some values approximate well and others, as you’ve seen, end up looking a little weird if you don’t round them. There is an article called What Every Computer Scientist Should Know About Floating-Point Arithmetic which is perhaps a little heavy but is detailed on how floating point numbers are merely an approximation, the general standard is IEEE 754.

Another workaround is to store everything as whole numbers and then elsewhere handle adjusting for display purposes. This allows you to work with whole numbers throughout though it requires at least two fields: one for the value and a second for number of significant digits. You could store it in a single value if and only if you have a fixed number of digits. Then it becomes more a formatting issue. The challenge with this approach is when you need to do maths with the numbers, you need to convert them to an equivalent formats and integer math is restricted in size in many CPUs which is why we have the floating point system we do. These days that is less of an issue, a 64-bit processor can store an unfathomably large number, however a 16-bit processors largest number is 65535, and half that if you want to store negative values.

This is unfortunately less a Tap Forms issue and more an issue with hardware limitations of computers.

[Author]

Posts