Intermittent Tag update delays in one provider while OPC Quick Client updates normally (8.1.54)

Ignition
1

Hi everyone,

We are experiencing intermittent Tag update delays in one Standard Tag Provider on a production Gateway. I would appreciate advice on where to investigate and what diagnostics to collect.

Environment:

  • Ignition 8.1.54
  • Java 17, Windows Server 2022
  • 16 logical processors, approximately 32 GB RAM, 18 GB maximum JVM heap
  • Redundant Gateway setup
  • Two business Standard Tag Providers on the same Gateway
  • The affected provider uses both external Siemens OPC UA connections and devices connected through Ignition's built-in OPC UA server

Symptoms:

  1. Tags in the affected provider continue updating, but their values fall several seconds to tens of seconds behind. The problem eventually clears without intervention. We observed two episodes today.

  2. During previous incidents, OPC Quick Client on the affected Gateway showed the monitored PLC value updating normally, while the corresponding Tag in the affected provider was delayed.

  3. We have also observed another provider on the same Gateway updating normally for the same PLC point while the affected provider was lagging.

  4. The delay is not limited to the Designer display. It has also been observed in Gateway scripts, SQL Bridge Transaction Groups, and another Gateway consuming tags from the affected provider.

  5. Tags from multiple PLCs appear to be affected together. Memory Tag updates also appeared delayed, although we have not yet measured this separately.

  6. During recovery, a PLC heartbeat counter appears to update faster and gradually catch up. We do not have a complete recording of every update, so we cannot confirm whether intermediate values are skipped.

These observations were collected across several incidents; not every comparison was recorded simultaneously during each incident.

The heartbeat uses a Direct Tag Group with a 1,000 ms rate and Subscribed mode. After recovery, OPC diagnostics showed:

  • Revised sampling interval: 1,000 ms
  • Revised publishing interval: 1,000 ms
  • Queue size: 1
  • Status: Good

Overall Gateway CPU usage does not appear high during the issue.

My questions:

  • Given that OPC Quick Client can remain up to date while one provider's consumers lag, which part of the Tag system should we investigate first?
  • How can we distinguish delayed Tag value updates from delayed delivery to subscribers?
  • Which Gateway diagnostics, metrics, or specific loggers would be most useful during the next occurrence?

We have collected diagnostic bundles and thread dumps and can provide relevant sanitized excerpts if needed.

Thank you.