Expression connect callback not working on active backup gateway after failover

Module Development
21

There isn't a 1:1 relationship between connect and the listener callback.

Paul's comment suggests that your module's active gateway code should be messaging the backup gateway whenever an expression is added/removed so the inactive gateway can do connect/startup/shutdown/disconnect at the same time.

1 Like
22

Hi Paul,

Playing with this because I’m interested. Could I ask what Parse context you guys are using? If I wanted to set up a TagReferenceBinder for any provider my base path could just be blank I assume?

23

The parse context heavily depends on what your environment is.

e.x. for Ignition Extensions I set up a simple context that either references provided keywords or evaluates to tag paths:

And ultimately creating a parse context is pretty simple:

There are a ton of different implementations in the main Ignition codebase - most likely you should be creating one specific to your module(s):

image
image1070×1670 302 KB

Most likely you want to extend from com.inductiveautomation.ignition.gateway.expressions.AbstractMultiLevelExpressionParseContext to support delegation, if you want to support custom reference styles in your module.
Alternately, one of the factories in com.inductiveautomation.ignition.gateway.expressions.TagReferenceBinder might be the way to go. I think a blank (not null) path would be fine.

2 Likes
24

I have confirmed that the backup gateway has the correct set of expressions stored in its own memory and has called connect/startup/shutdown/disconnect as they are added or removed. I have tried this in 2 different ways that both produce the same results

  1. Have both gateways listen on the redundacy state change and based on its own state (active or inactive) it will disconnect from its set of expressions or query and connect to all the expressions. I have confirmed using logs and breakpoints in the code that on failover, the backup has all the expressions and calls connect on all of them as well as the master calls disconnect on all the expressions it has and that both sets are identical
  2. Alternatively I had both gateways listening for the same submission event triggered by saving the form and both keeping simultaneous sets of connected expressions. This meant both gateways have the same set of expressions and were both connected at the same time regardless of which one is active

The result of #1 is that the master gateway has its callback invoked while it active and not when it isnt active since it disconnected. Either way the backup never has its callback invoked whether it is active or not

the result of #2 is that the master gateway has its callback invoked when it is active and when it is not active and the backup never has its callback invoked.

I don’t know what else I should be doing, this is all seeming like the connect is only invoking the callback for the gateway that has the master role. If so this seems wrong to me since it should do it for whatever the active gateway so that it keeps working on failover.

25

What exactly are you using the callback for? It's purpose is to make your infrastructure re-execute the expression, the result of which you can use as you please. For tag subscriptions, the callback fires to report a tag change, the result of which becomes available through expression execution.

Short of sharing all of your code, I don't know what else to ask. (And I am not volunteering to go through your whole module--my plate is full.)

26

fair question, I cant really share my module even if I wanted to. Regardless of what I want to do with this event, it is odd that it simply doesn’t trigger on the backup gateway when it takes over and feels like a bug or poor handling. Why does the event trigger on my non-active master gateway and not on my active backup gateway when they both have the same set of connected expressions?

Regardless of the use case, how could I subscribe to expressions in a module so that it doesnt just stop working when the backup gateway takes over?

27

You're the only one reporting this problem, and the platform is obviously working, so it must be something you are failing to do. No magic wand, sorry. :man_shrugging:

28

Do you know of any other cases where expression subscriptions using connect actually fires the callback on the backup gateway? I would love to speak to whoever has done that successfully to see what I might be missing or view any available code samples they are willing to share.

I dont even care if its being done using com.inductiveautomation.ignition.common.expressions.Expression.connect, Is there any other way that has been successfully done that works on a backup gateway and not just on the master

29

I don't know of any such case, because I don't know of any usage that even needs to take master/active/inactive/backup into account. I don't see how talking to somebody will help short of showing them your code.

If you can't show your code, I recommend adding all kinds of logging to your module to trace all relevant activity, then compare master and backup logs.

30

the use case is that someone might want to subscribe to changes in tags using expressions. I have seen this use case many times.

This isn’t a use case that needs to take into account master/backup. I would expect the role of the node (backup vs master) not to matter if redundancy is implemented correctly on IA side. Logs on both gateways give me no doubt that the both call connect on the expressions. Past that its com.inductiveautomation.ignition.common.expressions.Expression code that should be firing the provided callback. My code doesn’t differentiate between master/backup and connects for both. It looks to me like the com.inductiveautomation.ignition.common.expressions.Expression code or somewhere down the line on the subscription suppresses the callback being fired depending on the gateway role.

Can you confirm that this is not the case?

31

I can confirm nothing. I can believe it might not happen on an inactive gateway, but you've said it is missing on the backup gateway, including when active. All I know about the platform suggests you are missing something.

{ You have moved from "productively asking for help" to "whining about unsupported behavior". I think I'm done in this topic. }

32

I had spent quite a while debugging on my side and was convinced the IA code was just not firing on the backup. Turns out neither this nor my code was the problem. My testing strategy was flawed. I was modifying a tag via the master gateway opc quick client assuming it would mirror on the backup which in hindsight was a bit presumptive considering I was mimicking the master gateway being down. I retested by updating the tag using the backup gateway quick client and it is now triggering on that gateway.

Sorry if I seemed to be whining, it looked like there might be bug with the IA redundancy code for expected supported redundancy behaviour.

33

tldr;

the opc quick client on master gateway was being used to test tag expression subscriptions on the backup gateway. I needed to use the opc quick client on the backup gateway to test this out. Once I did that things worked fine.