# Clock Drift from PLC Timeout? **URL:** \ **Category:** Ignition\ **Created:** [May 6, 2025, 6:34pm UTC](https://forum.inductiveautomation.com/t/clock-drift-from-plc-timeout/105933 "2025-05-06T18:34:54Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![jacob\_geers](https://avatars.discourse-cdn.com/v4/letter/j/4da419/32.png) [@jacob\_geers](https://forum.inductiveautomation.com/u/jacob_geers)\ **Post date:** [May 6, 2025, 6:34pm UTC](https://forum.inductiveautomation.com/t/clock-drift-from-plc-timeout/105933/1 "2025-05-06T18:34:54Z")
Hello, Looking for input on how to troubleshoot clock drift. We get several in a row occasionally but the CPU never spikes - is this a concern? ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/5/f/5f9f34d701c1cf5c4ef19e633a96f2b8ab933225.png) ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/f/5/f5ff69b34130fa2d51090207ff234591985c16df.png) Based on the logs I notice we also get ABMicroLogixReadRequest warnings - could this be the issue? How do I go about troubleshooting? The two database issue are embarrasing but I don't think they are related. **Example #1** ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/4/4/449bd28e6738ee2c5534e0afc1ce41c4ef8fac9e.png) **Example #2** ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/d/0/d0fc039ac7a0a49ed64191b6e04a84bdb823ce71.png) ---
**Author:** ![jacob\_geers](https://avatars.discourse-cdn.com/v4/letter/j/4da419/32.png) [@jacob\_geers](https://forum.inductiveautomation.com/u/jacob_geers)\ **Post date:** [May 6, 2025, 6:48pm UTC](https://forum.inductiveautomation.com/t/clock-drift-from-plc-timeout/105933/2 "2025-05-06T18:48:20Z")
Seems to always be these devices. ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/f/1/f194b7f6c326047644f6226a67b59bafdc2b3d5c.png) ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [May 6, 2025, 6:57pm UTC](https://forum.inductiveautomation.com/t/clock-drift-from-plc-timeout/105933/3 "2025-05-06T18:57:45Z")
It's most likely the other way around - these timeouts are caused by the clock drift. ---
**Author:** ![jacob\_geers](https://avatars.discourse-cdn.com/v4/letter/j/4da419/32.png) [@jacob\_geers](https://forum.inductiveautomation.com/u/jacob_geers)\ **Post date:** [May 6, 2025, 8:20pm UTC](https://forum.inductiveautomation.com/t/clock-drift-from-plc-timeout/105933/4 "2025-05-06T20:20:33Z")
Is clock drift without high CPU a concern? ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [May 6, 2025, 8:25pm UTC](https://forum.inductiveautomation.com/t/clock-drift-from-plc-timeout/105933/5 "2025-05-06T20:25:55Z")
Yes, it can screw up history and typically indicates either untuned Java garbage collection or, in Windows, actual clock warps. Consider adding the following to your `ignition.conf` file (with unique sequence numbers): ```python wrapper.java.additional.10=-XX:+UseG1GC wrapper.java.additional.11=-XX:MaxGCPauseMillis=25 wrapper.java.additional.12=-Xlog:gc:file=logs/gc.log:t,tags:filecount=5,filesize=4M wrapper.java.additional.13=-XX:-OmitStackTraceInFastThrow ``` Restart your gateway and then watch the contents of the `gc.log` files. If you are getting actual "pause-the-world" halts of \> ~100ms, you should look for memory usage problems. ---
**Author:** ![jacob\_geers](https://avatars.discourse-cdn.com/v4/letter/j/4da419/32.png) [@jacob\_geers](https://forum.inductiveautomation.com/u/jacob_geers)\ **Post date:** [May 7, 2025, 3:04pm UTC](https://forum.inductiveautomation.com/t/clock-drift-from-plc-timeout/105933/6 "2025-05-07T15:04:47Z")
Thanks, will try over the weekend. Here is something that may be interesting, `gateway-shared-exec-engine` taking 20% of CPU when the clock drifts occur? ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/2/4/2421cd1ac14fbe529e8caa4de286bfbeccd4b753.png) ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [May 7, 2025, 3:10pm UTC](https://forum.inductiveautomation.com/t/clock-drift-from-plc-timeout/105933/7 "2025-05-07T15:10:00Z")
Might be interesting, you would need to stack trace from that thread (click the "+" button to expand) to know any more. It will probably be tricky to get a snapshot of that thread exactly when the CPU usage is high and the clock drift is happening. ---
**Author:** ![Clancy\_Cavanaugh](https://avatars.discourse-cdn.com/v4/letter/c/45deac/32.png) [@Clancy\_Cavanaugh](https://forum.inductiveautomation.com/u/Clancy_Cavanaugh)\ **Post date:** [May 7, 2025, 3:22pm UTC](https://forum.inductiveautomation.com/t/clock-drift-from-plc-timeout/105933/8 "2025-05-07T15:22:06Z")
Are you on a 3 year old minor version by chance? I remember something similar occurring on our system a few years ago. Our issue was caused by PLCs being powered off for maintenance, and Ignition appeared to be tying up threads waiting for the timeout. I believe it was fixed in version 8.1.15 though. ---
**Author:** ![jacob\_geers](https://avatars.discourse-cdn.com/v4/letter/j/4da419/32.png) [@jacob\_geers](https://forum.inductiveautomation.com/u/jacob_geers)\ **Post date:** [May 7, 2025, 3:56pm UTC](https://forum.inductiveautomation.com/t/clock-drift-from-plc-timeout/105933/9 "2025-05-07T15:56:27Z")
Here is a thread dump with one item using 17% during the time of clock drifts, but it is different than before. [site-dev\_thread\_dump20250507-115452.json](https://forum.inductiveautomation.com/uploads/short-url/cswR0HNPTwK1EKhRRqMhc9sV2DZ.json) (531.9 KB) ---
**Author:** ![jacob\_geers](https://avatars.discourse-cdn.com/v4/letter/j/4da419/32.png) [@jacob\_geers](https://forum.inductiveautomation.com/u/jacob_geers)\ **Post date:** [May 7, 2025, 3:57pm UTC](https://forum.inductiveautomation.com/t/clock-drift-from-plc-timeout/105933/10 "2025-05-07T15:57:18Z")
We are on 8.1.32. ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/c/5/c5738ef8ffd6deee7cc77a5c400490465312c4b3.png) ---
**Author:** ![jacob\_geers](https://avatars.discourse-cdn.com/v4/letter/j/4da419/32.png) [@jacob\_geers](https://forum.inductiveautomation.com/u/jacob_geers)\ **Post date:** [May 8, 2025, 1:34pm UTC](https://forum.inductiveautomation.com/t/clock-drift-from-plc-timeout/105933/11 "2025-05-08T13:34:03Z")
Thanks, we ended up applying those gateway config changes. 12:07 5/28 - apply config change and restart service ... no clock drifts 00:24 5/29 - first clock drift ... a few clock drifts 5:51 5/29 - tons of clock drifts all hour Attached is the gc.log file if you wouldn't mind taking a look? My team and I are now playing around with Visual VM but don't have much experience with it. [gc.log](https://forum.inductiveautomation.com/uploads/short-url/vqNeo0CVqxQQPJRhelG761mlus6.log) (1.5 MB) ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [May 8, 2025, 1:48pm UTC](https://forum.inductiveautomation.com/t/clock-drift-from-plc-timeout/105933/12 "2025-05-08T13:48:32Z")
You do not have a GC problem. There's no pause greater than 200ms in the entire log. Which means you have a workload problem. You will need to study regular thread dumps and extra thread dumps when the problems start. (This can be quite time consuming--you probably want IA support involved.) In my experience, this is often caused by mishandling of long-lived asynchronous threads. ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [May 8, 2025, 1:49pm UTC](https://forum.inductiveautomation.com/t/clock-drift-from-plc-timeout/105933/13 "2025-05-08T13:49:22Z")
Is your Ignition Gateway running on real hardware or in a VM? ---
**Author:** ![jacob\_geers](https://avatars.discourse-cdn.com/v4/letter/j/4da419/32.png) [@jacob\_geers](https://forum.inductiveautomation.com/u/jacob_geers)\ **Post date:** [May 8, 2025, 2:02pm UTC](https://forum.inductiveautomation.com/t/clock-drift-from-plc-timeout/105933/14 "2025-05-08T14:02:10Z")
VMware ESXi 8.0.3. I will monitor threads on my own for awhile then make a support request. Thank you both very much! ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [May 8, 2025, 2:04pm UTC](https://forum.inductiveautomation.com/t/clock-drift-from-plc-timeout/105933/15 "2025-05-08T14:04:11Z")
> [@jacob\_geers](#): > > VMware Ewww! Have you followed all of the IA guidance on setting low latency mode and avoiding overcommitted resources? (VMware is poorly configured by default for latency-sensitive applications like Ignition.) ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [May 8, 2025, 2:05pm UTC](https://forum.inductiveautomation.com/t/clock-drift-from-plc-timeout/105933/16 "2025-05-08T14:05:20Z")
[https://support.inductiveautomation.com/hc/en-us/articles/12888090388237-Considerations-and-Best-Practices-for-VMs-Hosting-Ignition](https://support.inductiveautomation.com/hc/en-us/articles/12888090388237-Considerations-and-Best-Practices-for-VMs-Hosting-Ignition)