Red Lion Messed up Big time

Anyone know the conversion to get from the timestamps that are now coming in as Jan 1 1997 to March 22, 2024?

I'm guessing its just seconds between the two dates?

1 Like

A little more detail would be nice.

1 Like

Everyone who is using red lions is getting timestamps Jan 1 1997 for today, as of last night.

Here's the info I have:

Crimson’s data and event logs will show incorrect times and possibly exhibit other failures from 03/22/24 onwards.

Crimson tracks time internally using a two methods: The first is to count the number of seconds since 01/01/1997, and the second is the count the number of 200ms ticks since that same epoch. The latter clock is used for data logging and other similar purposes where better than one second resolution is required. Both values are stored in 32-bit numbers, which works fine for the 1 second clock, but for the 200ms clock creates a wrap-around after slightly more than 27 years. To be precise, the 200ms clock will wrap-around at 01:17:39 on 03/22/24. It appears that our Australian customers have already started to encounter this problem, and we can expect other customers to see the same tomorrow morning. The symptoms will be incorrect data in data logs, and possibly crashes or other undefined behavior when viewing historical data on the HMI.
There is no work-around, save setting the clock back a year and accepting the incorrect but at least coherent dates.

3 Likes

Holy shit :laughing:

9 Likes

Right now it looks like the delta seconds to convert back to March 22 is

858993453

Hmm, I got 858993459 seconds.

jshell> import java.time.*

jshell> var i1 = Instant.parse("1997-01-01T00:00:00Z")
i1 ==> 1997-01-01T00:00:00Z

jshell> var i2 = i1.plusMillis(0xFFFFFFFFL * 200L)
i2 ==> 2024-03-22T01:17:39Z

jshell> Duration.between(i1, i2).toSeconds()
$4 ==> 858993459

Close enough for me.
Now to figure out how to merge the data into our DataPartition tables with the correct tag IDs and adjusted Millis

Yay for Friday.

It looks like Ignition has already made a few new tag IDs for the 1997 data, I imagine that is because timestamps from the red lion are backtracking some?

Looks like they have a notice posted about this at least: Crimson® | Red Lion

Devices running Crimson® 2.0, 3.0, 3.1 and 3.2 will display inaccurate timestamps on data, alarm, and event logs on and about March 22, 2024. Additionally, the device may cease to operate or automatically restart at the initial time of failure.

Operation can be restored by power-cycling the unit, and if a GMC message is displayed on an HMI, clearing the error via the touchscreen or keyboard.

Once the device is operable, communications should be restored, but incorrect dates will continue to be displayed. Viewing the data logs via the trend viewer may result in further restarts or failures.

Red Lion is urgently working on a fix. We will notify you when the update is available. In the meantime, if you need further assistance or information, please contact Red Lion support via the methods listed below.

1 Like

Is there any way for Ignition to use its own timestamp on the MQTT instead of Red Lion?

You'll have to check with Cirrus Link, I'm not sure. I know it's not possible with OPC tags.

Really, in 2024 they're still being bitten on the ass by overflows on timestamps ?

3 Likes

Indeed.

The real shit show will happen in 2038 (2038-01-19T03:14:07Z).

12 Likes

Please, Kevin, can we maintain some decorum on the forum!

2 Likes

Something similar happened to Microsoft 2022-01-01 if you recall Microsoft confirms Exchange Year 2022 problem | Hacker News

Summary of the bug: Exchange tried to store "2201010001" (i.e. date time '22-01-01 00:00) in a 32-bit signed int, but INT_MAX is 2147483647 so the result is a negative number.

Red Lion has just issued updated firmware for their devices which will correct the error and return timestamps to current dates.

3 Likes

Hi all. We lost comms to 79 Red Lion units which shut down our SCADA system for our wastewater utility. It took 36 hours to repair the damage and get us back on line. It cost us $6k in support contractors. Is anyone looking to claim costs from Red Lion?

If anyone is, I doubt IA would appreciate it being discussed here on their forum.

1 Like

OK, where would you suggest?