Uncertain_InitialValue for large tag provider

Ignition
1

At seemingly random time points, a large majority of our tags get the "Uncertain_InitialValue" quality.
This can be resolved by restarting the affected tag provider. However, it is unclear to us why this issue happens.

The affected tag provider has about 400 000 tags.
The gateway has 8 cores and 32GB RAM.
The gateway cpu usage is about 3%. So I figure the resources are not the issue (?)

Is this too many tags for one provider?
If so, are there any recommendations for tag provider size?
Any idea why this is happening?
Do we need to spread our tags over multiple gateways?

2

Is this an Ignition native driver or an OPC connection?

3

Ignition is connected to the tags through Ignition's OPC client to Kepware.
Then Kepware is connected to about 70 Mitsubitshi PLCs.
Poll rate is 1000ms.

Ignition OPC Client <==> Kepware Server <==> PLC

4

Any ideas?

5

I have a client with a similar setup (but a few dozen PLCs, not 70), and they have no issues.

The only thing that comes to mind is the suggestion to make each PLC its own channel in Kepware.

6

Uncertain_InitialValue is the starting quality for a tag when it initializes but has not received a value. There isn't really a path I can see for tags to go from having a value/quality back to Uncertain_InitialValue, except maybe by restarting the tag or provider.

Are you sure you're not just discovering a tag that never received a value and remained in its initial state? Or that it's not some other Uncertain quality?

7

Try this

8

This is a work around that ignores whatever the root cause is. Subscribed mode should work, or there should be a definitive reason why it does not work with that server.

9

Now that I think about it.
I think it has only happened on our develop/test gateways.
On those servers we have no licenses and work with the trial mode.

Maybe something happens when one of the developers starts a new trial session?
Or a trial session ends?

10

I haven't tried giving each PLC its own channel.
I will try to investigate this route.

11

Do you also have 2 Kepware instances? 2 sets of PLCs?

Or do you have 2x 400k tag gateways smashing a single Kepware? Or two Kepware instances both talking to your PLCs at once?

12

At the moment, we have the following setup (scaling out to more production plants in the near future):

  • three environments (develop, test, production)
  • two kepware instances (one instance for each production plant)
  • three ignition gateways per environment (one for each production plant and a group gateway)

For one production plant (so one of the kepware servers), there are three ignition gateways (three environments) looking at it.

I'm not sure how many connections are active on each PLC but there are multiple applications looking at it: kepware, citect (maybe more?). This I would have to check with engineering.

13

Dev and test point at your production PLCs? Uhm. Oy!

14

Those can only read, no writing.
Do you see any other problems?

15

Ah. That's a bit better. But you cannot develop/test stuff that writes to the/any PLCs?

The only other concern would be Kepware's capacity. But not that large a concern--I'd expect the load to be stable.

16

That is an issue indeed. We'll have to look into that later...

17

We had another event.
However, I made a mistake: it was not "Uncertain_InitialValue" but "Uncertain_LastKnownValue".
And this time the issue resolved itself after a few minutes without restarting the affected tag provider.

The issue happened after we did a deploy from test to production (might be coincidence ?).
In this deploy, we did the following:

  • copy projects from the test gateways to the production gateways
  • send tags from develop gateways to the production gateways with EAM
  • copy theme files from test gateways to the production gateways
  • sync tags from the plant production gateways to the group production gateway
18

Uncertain_LastKnownValue, when connected to a 3rd party server, happens in 3 scenarios:

  1. you've lost connection to the server. The Ignition client will automatically reconnect, resume or establish a new session, and transfer or create new subscriptions.
  2. the server has indicated to us that some subscription is no longer valid for some reason. The Ignition client will delete and then re-create it.
  3. the subscription watchdog timer has elapsed, which means we should have received either data change notifications or a keep alive notification for some subscription, but 2x the expected arrival has elapsed, and now we assume this subscription is no longer reliable. The Ignition client will delete and then re-create it.

The logs may have some indication which of these is occurring. All of them should recover automatically, assuming the server is not broken or overloaded in some way and that the subscriptions can be successfully re-created.

2 Likes
19

We are experiencing some general networking issues at our company.

So this might be the cause.
I will investigate.
Thanks!

20

In my case I observed "Uncertain_InitialValue" happening when you move project from dev environment to production and tag group for poll does not exist in the production environment.