[IGN-2343, IGN-8727] "Unable to send updates to clients" error in logs

Here's a stack trace from v8.1.19

java.io.IOException: Channel is not open.

at com.inductiveautomation.perspective.gateway.model.PageModel.send(PageModel.java:357)

at com.inductiveautomation.perspective.gateway.model.PropertySyncManager$SendUpdates.run(PropertySyncManager.java:224)

at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.base/java.util.concurrent.FutureTask.run(Unknown Source)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at com.inductiveautomation.perspective.gateway.threading.BlockingWork$BlockingWorkRunnable.run(BlockingWork.java:58)

at java.base/java.lang.Thread.run(Unknown Source)

Looking back, my request for the stacktrace was not going to be helpful, because what we really need is the stacktrace from when the Websocket became disconnected. These "unable to send updates" errors are the Gateway trying to send data to a session which is no longer connected. The Gateway has no idea why - only that the connection is no longer open. The extra logging which was put in place was done in the handling of the actual closure event. Look for something akin to "Websocket disconnected from session" when the "Unable to send updates to clients" logging first began. The stacktrace from the Websocket disconnect is the (hopefully) useful piece.

3 Likes

The same on v8.1.31
I had one Perspective session and one Designer session open and three VPN connection errors. I also noticed it took about 10 seconds to save a change in Designer. Sorry, no other details to share.

I have been getting this error all day I just found out as well.

Posts which inform us that you have encountered the issue will not help us resolve the issue for you, other than we can note on our internal ticket that additional customers are encountering the issue. For us to move forward in diagnosing, resolving, or working around the issue we require the requested stacktraces either here on the forums or through the Support team.

1 Like

Mine is identical to nminchin's trace.
Same version too, 8.1.19.

I will see if the one designer that is open can be closed tomorrow and see if it fixes it.
Then do a reset if not, and see if that does it.
Probably that will fix it.
If not, probably I'll do a ticket.
Sorry to bug.

Cody has clearly said that Nick's trace is not the one needed.

Yah, that is why I didn't post it.

There is one quark that I think is notable.

When I hover the magnifying glass on the log page, the project name is for a different project than the session-project.
The view is from that session-project.

Neither the project nor the session-project are inheritable.


We turned off our designers. Restarted the gateway. The issue went away.

Similar situation for me as rbachman stated.
I'm running v8.1.33

rbachman: " I think that the issue is related to Perspective views open in the Designer when the connection in the Designer is lost to the server. For example, if I have am editing a Perspective session in the Designer and get disconnected from our VPN, it appears that it may be the cause of the issue."

Still getting the warnings in the log even after terminating all Perspective sessions and closing the Designer. Restarting the gateway resolved the issue.

Same in 8.1.35

One of the thing I am noticing right now is that almost all logs produced have the same session-project

We forward all our logs to Splunk
We got some random occurence, but nothing major before January

The project that appear as session-project was modified. Modifications includes, for unknown reason, "\n" in text field (really not sure this is linked to the error)
image

here also the logs exported from Splunk at the moment the error begins last time
(removed logs as no longer required here)

1 Like

Great, thank you. I've attached those logs to the active ticket and will be reviewing them with a Dev tomorrow to see if we can glean anything. Was there any filtering done to these logs you provided?

Can you tell us anything about what events occurred between 23:33:37 and 23:38:37? It's clear from the logs that at least one session encountered an idle timeout (inactivity timeout), but it would be good to know if that session was a browser, designer, or mobile session, and what the user was doing. It's slightly curious because the only views reporting as idle are in a dashboard - when we should expect the View housing the dashboard to also report the idle timeout.

Could we also get more information about the projects called out in the logs and how they're used? a direct/private message is fine if you're hesitant to share that information on the forum.

At 23:30 it is the end of a break in the shop, so the operator went back to his workstation.

The session was in a Browser, Microsoft Edge.

Don't know what the operator did exactly. Right now, he uses 2 project in Ignition, so he might have switch browser Tab, loaded the other project in the same tab, or interacted with the active interface

The interface is a HMI that toggle bit, or update recipe values

I can send you, later today, a full backup of the server in a support ticket if you need

I don't think that's necessary, as this doesn't seem to be a result of the structure or content of the project in use, and is rather far more likely to be the result of some as-of-yet-unidentified communication issue between the Gateway and the Session(s). Is there any chance a Designer session was in use at this time, or are you confident that only browser sessions were in use during this time? Do these projects require authentication?

I know that there was no active user in the Designer, but I can't confirm that there was no Designer session openned (If there is a specific log, from Ignition gateway, so I can find designer session start/end, I can extract the information from Splunk)

No authentication required at this moment

1 Like

@cmallonee any luck on replicating this in-house or have an estimate on a fix? I looked at the recent Ignition change logs but didn't see this issue mentioned.

It seems the only identified solution is to restart the service? We're experiencing this on our production gateway, which is very difficult to schedule restarts. Our gateway logs our useless right now.

1 Like

I'm not directly tied to the team associated with the ticket any longer, so I don't know if they've come to any conclusion amongst themselves regarding replication. I don't see any update on the issue other than a suspicion that orphaned Designers could potentially be the culprit.

We consider orphaned Designers to be Designers which are open but their communication with the Gateway is severed for any reason. The Gateway doesn't cull or garbage collect these disconnected sessions except during a Gateway shutdown. In theory, continued development could result in updates being sent to these Designers (clients), but they aren't active to receive the update.

Again, just a theory. I suggest reaching out to Support so that we can track that more users are impacted by this.

2 Likes

am I crazy or isn’t there a way to correlate these log entries with a specific client/user/session?

I’m unsuccessfully trying to correlate wrapper log magnifying glass hover Navigation/Sidebar@L[0] with Thread [perspective-worker-3155827] id=12266073, (BLOCKED) and output of system.perspective.getSessionInfo()

may have some useful info (though not the original stacktrace)

Perspective Workstation is configured for auto-launch for all users. ‘perc' is presumably a Dell-specific (iDRAC related)? user . Unsure if it’s actively being “viewed” somewhere or perhaps just open on a server-room KVM display as this is a client’s environment. But someone clearly had to log into the Workstation instance.

though of course I can’t be sure this is the session of interest

EDIT:

here’s a script for anyone else who’s curious:

getOrphanedPerspectiveClients (Gateway Event/Message Handler)
# def handleMessage(payload): # type: ignore (ignition copy/paste)
	"""Usage
	system.util.sendRequest(
		project = 'Project Name', # env-specific
		messageHandler ='getOrphanedPerspectiveClients',
		payload = dict(
			orphanedView = 'Templates/Tanks/Tank@C$0:0$0:42' # env-specific from offending logs (gateway webpage logs magnifying glass hover or raw wrapper.log view key)
		)
	)
	"""
	def getOrphanedPerspectiveClients(payload):
		from com.inductiveautomation.ignition.common import TypeUtilities
		payload = TypeUtilities.pyToGson(payload)

		import system
		from com.inductiveautomation.ignition.gateway import IgnitionGateway
		IGg = IgnitionGateway.get(); mM = IGg.moduleManager
		PerspectiveContext = mM.resolveClass('com.inductiveautomation.perspective.gateway.api.PerspectiveContext').get(IGg).getSessionMonitor().getClientSessionsForProject(system.project.getProjectName())
		results = []
		for session in PerspectiveContext:
			for page in session.getPages():
				for view in page.getViews():
					if view.getId() == payload.orphanedView:
						results.append({'address': session.props.address, 'username': session.props.auth.user.userName})
		return results
	return getOrphanedPerspectiveClients(payload)

looks like my intuition was correct - returned {'address': u'[0:0:0:0:0:0:0:1]', 'username': u'Perc'} for me

does this mean I can kill that session without remorse as it can’t possibly be working - or do I need to ask someone to check it to be certain?

EDIT 2:

well, after all that - I just found the “Log Activity” section under the “Details” button per-user… :sob:

guess I was assuming from this thread that there would only be one (or at least not many) offending sessions, so the admittedly non-unique view id’s might not exist in multiple sessions.

For what it’s worth, evey single session in my gateway is spamming “Unable to send updates to client”, and if I pass in a view id shown under a particular user's session log activity to my function (after modifying to return a list), it returns a different address and username.

my session (Admin) is “static’ - It’s just open, not in use, so presumably my view ids wouldn’t be changing?

example: