part of a UDT holds the following memory tags set to float data type.
I am familiar with float types changing, but my experience has always shown changes around .0000000001 and truncating or rounding took care of it. Moreover, I have never experienced the change affecting the value at the resolution where we have the format string set. Using the example I have below, I expected changes to my number to show up at thousandths and further out, not at tenths and hundredths.
In the tag Browser, when I enter 5.20 for our low value (or into any of the tags), it changes to 5.199999809 which can make a huge difference for us. Am I able to restrict resolution to hundredths, at the tag level? Or does anyone have any methods to stop this from ocurring besides changing everything to strings?
That’s not a bug. Just a limitation of binary floating point. It simply cannot represent exactly any “.2”. 32-bit versus 64-bit float just changes how many “9” there are before the repeating part. Some related discussions:
TL/DR: Strings might hide your problem, but any conversion from string back to a number will yield the same. I don’t recommend hiding your problem.
I do know is a behavior of that data type, and not a bug.
Is it safe to say; As long as values of the same resolution are entered and I round the values to the same resolution when using them, I do not need to be concerned about using a value that was not intended?
Thank you for the extra explanation and related discussions. I’ll give them a look.
Yes, no need to be concerned. Simply ensure that you round at all points of display. (Formatting to some # of decimals.)
I’ll just throw this in for future reference:
A canonical answer from a StackOverflow question
Is floating point math broken?
Which references an easy to read guide:
What Every Programmer Should Know About Floating-Point Arithmetic
But also references the head spinning, more detail than you ever realized existed:
What Every Computer Scientist Should Know About Floating-Point Arithmetic