I am facing below issue, there is a numeric text field on my window, I am trying to set the value to 93.014757 , but anyway it shows 93.014755 which is not I expect .
Here's my window and script.
When I run program and click the button,
I am really confused where this "93.014755249" comes from.
Another test I made,
when I change the set value in script to 93.014711 , it display as 93.014709.
when I change the set value in script to 93.014705 , it display as 93.014702.
Always difference.
Further test I just made, when I enter 93.014711 at text field from screen, and when mouse leave the field, it jump to 93.014709, same when I enter 93.014705m it jump to 93.014702.
And there is no data binding.
1 Like
You are at the limit of 32-bit binary floating point's resolution. ~6.5 significant decimal digits. The tag simply cannot hold the precise value you are trying to set.
Use 64-bit floating point to get ~16 significant decimal digits.
These are fundamental limits of binary arithmetic, not anything Ignition is causing.
4 Likes
I am not using tag to pass the data, in my case, just ask user to key in the value from screen and save it to do with named query, can you suggest how to do .
I checked text numeric field property, it only int/double/long/float can choose.
"Float" is 32-bit. "Double" is 64-bit. Be sure to use 64bit types for entry field, named query parameter, and database column. If anything is 32-bit, it will re-introduce this loss of precision.
Also keep in mind that binary numbers cannot exactly represent fractional values, unless the fraction has a least common denominator that is a power of 2. Except for that rare case, you will always have rounding happening when converting entered decimals to binary, or converting binary back to decimal for display. Computer Science 101.
{ Doesn't matter whether a tag is involved--this is simply how floating point works in computer software, including Ignition. }
4 Likes
