Two REAL Values Look Equal but the PLC Says No? Check Hidden Precision
A display rounded to two decimal places can hide a difference that remains in the stored floating-point values. Exact equality may therefore stay false even when the HMI shows matching numbers. Define what counts as close enough for the process instead of relying on display formatting.
Original comparison example; tolerance selection belongs to the application requirements.
What is happening?
Binary floating-point has finite precision, and many decimal fractions are not represented exactly. Calculation history and conversion can affect the final stored bits. A useful comparison may need an absolute tolerance, a relative tolerance, or an explicit integer unit, chosen from the requirement rather than an arbitrary small number.
Check these five things
-
Inspect the full available precision and the calculation path of both operands.
-
Determine whether the requirement is exact identity, a process acceptance band, or a threshold crossing.
-
Choose a documented representation and tolerance based on measurement resolution and the operating range.
-
Test values just inside and outside the acceptance boundary, including behavior near zero.
-
Handle invalid measurements and unsupported numerical values before declaring the comparison successful.
Worked example
Suppose two calculated values are 10.004 and 10.003. Both may display as 10.00, but they are not equal. In an illustrative process with an approved absolute tolerance of 0.01 unit, their difference of 0.001 is within the band. That tolerance would be inappropriate if the application required tighter resolution.
Quick diagnostic reference
| Question | Possible approach |
|---|---|
| Exact counter match? | Integer representation |
| Physical measurement agreement? | Requirement-based tolerance |
| Wide dynamic range? | Review relative and absolute criteria |
The mistake to avoid
A tolerance that is too large can accept a real process error. Numerical convenience is not a substitute for an engineering acceptance limit.
How to verify the fix
Record the chosen criterion and demonstrate correct decisions at its boundaries without depending on the HMI's rounded text.
Field-work boundary: These are educational diagnostic guides. Use the exact equipment manuals and approved site procedures. Electrical testing and machinery changes belong to qualified, authorized personnel; control hazardous energy and validate affected functions before release.
Technical reference
Siemens: REAL data type and IEEE 754 precision. The examples and diagnostic tables above are original teaching material; they do not replace the product manual.