Clock Drift Warnings on Raspberry PI

Ignition
1

Hello,

I have Ignition Edge 8.1.37 running on a Raspberry PI using Ubuntu 22.04.4 LTS. Ever so often, Clock drift warnings appear in the logs which often follow with higher CPU and Memory usage. This, in turn, can start a snowball of problems that bogs down the gateway and gives tremendous performance issues with data loss. My usual solution is to reboot Raspberry PI which clears up the Clock Drift warnings and sets the gateway running normally.

This brings up the following questions:

  1. What causes these seemingly random clock drifts, and can they be mitigated?

  2. Is there a way to send a tag or alarm for a Clock Drift Warning?

  3. Is this purely a system issue that I need to address strictly on the local machine?

I've attached some images of the logs and system performance from the gateway.

Logs (1).PNG
Logs (1).PNG1211×683 99.1 KB

Performance Stats
Performance Stats1236×569 126 KB

It looks like issues started occurring yesterday around 14:53 which resulted in blanks in the memory and CPU usage.

Thanks for any help on this!

2

Typically, it is a severe memory issue with a garbage collection stall. You have a gap in your memory plot that could correspond to a memory spike.

But, it can also be a CPU hog outside Ignition running on the same machine. Running a database on the same machine is often a culprit, especially in underpowered systems.

There is no way to send an alarm for these--though an external process that watches the wrapper log might be able to help.

The close spacing of these events in your log suggest that some other process on the machine was bogging you down.

3

Today, I created a similar event by running another process on the machine in an attempt to load a syslog file which I'm guessing was rather large. Regardless, I'm not sure what other programs if any would be captivating all the PI's CPU when it is not artificially created by a user.

Are there any known processes within the gateway that would suddenly require large amounts of CPU and/or Memory?

4

Not randomly. But

  • Your own scripts can generate extreme load, or

  • Perspective clients demanding complex or long-duration history plots, or

  • To a lesser extent, Vision clients demanding large history datasets.

5

Neither Perspective or Vision are running on this Edge Gateway. There are also few if any scripts running that I know of. It could be something on the local machine that I don't know about. But at this point, I'm not sure what is causing it to happen.

Just to reiterate, you mentioned there was no way within Ignition to associate a tag with the Clock Drift Log Warning?

6

Right. You could monitor the wrapper log externally, or perhaps use logback's configuration to filter in some other manner.

You could also run your own detector as a timer event with a fixed pace, and monitor the timestamps you get within that event.

7

Hmm. Might be worth looking into creating an external wrapper monitor and/or an artificial clock drift to mimic the one in the logs. I may also look into writing a script or somethin to systematically reboot the machine every so often to PM the issue.

But, thanks for the help Phil!