Gateway Crashing when passing view parameter

After speaking with Dev it sounds like you are indeed attempting to pass so much data through the websocket that the websocket is tripping an internal safety setting and closing. The max websocket message size is configured in Jetty. As such, you do have the ability to configure this property, alas, I wouldn’t be able to walk you through this as I have no experience in that area. Someone with a good amount of experience in that area (nudges @pturmel) might be able to help you configure Jetty to allow for larger messages to be sent without closing the connection.

Alternatively, you could reduce the size of the data you are sending through the websocket. This might not work for your scenario, though I’d be interested to see if you encounter the same issue while passing the dataset to a running View instead of opening a Popup. My suspicion is that the size of the dataset is the sole culprit, but if the Popup is incredibly complicated then the popup could be part of the problem.

Me? Configuring Jetty? Nada.

Thank you all for your responses.
@cmallonee you are right, the gateway is not crashing, I am just being disconnected, although to the end user it seems like it crashed(it takes a refresh to recover). The popup I would not call incredibly complicated. Basically it allows the user to set up custom filters for a data set, it includes a table a dropdown, and an embedded view(which in turn is very simple). However to set up some filters, the dataset is required, as the options are from the data itself, so the view has to access it in some way. To answer your question, it does the same even if I have the view running, in an embedded view. As soon as i increase the data set size, it disconnects the same way.

Apart from reducing the dataset size, what do you think the options are in this case? Should I increase the max message size(I suppose the limit is there for a reason?-also I dont know how to do that). Or should I just store the dataset in a session property and be done?(Either way it has to be stored somewhere).

Are you actually displaying the dataset? Or just using it as a source for other operations? If the latter, try making it private, so it won’t go back and forth to the browser.

Are you able to send me the View and dataset via a direct message (click my name and select “Message”)? I could take a look and see where some changes might be warranted.

Do you need the whole dataset? Are you just filtering based on column names? If so, you could send only the headers.

@pturmel I tried making the dataset private(I also set the view parameter as private although there is no indication so I dont know if it can actually be set private). I still get the disconnect message, although now it recovers by itself. So I get a momentary disconect, then it reconects and all is ok, whreas previously it did not recover without a refresh

@cmallonee
Thanks for the offer, I will have to get back to you on Monday because I will have to clean up some of the data.
I need not only the column names, but some of the data as well since one of the filter options is actually a dropdown-select of all the possible strings within a column, so i need the actual column data. I suppose i would be able to reduce the dataset somewhat by eliminating the non-string columns, but that would not solve the problem, because the dataset is not fixed, but the user selects Start and End date, so it could still crash on the end user, if he selects a big enough date range.

So…we’re also encountering this same error in our logs. Appeared this morning after 3 weeks of a solution running with no issues. We’re on 8.0.12.

Here’s the thing: the message size reported in our logs is, you guessed it: 2,105,344

I’m going to go out on a limb and say that @akolovos and I don’t have the same project code, or anywhere close to it.

I’m planning to contact support, because this is affecting a customer, but wanted to throw this up for those involved.

@cmallonee, any chance you can confirm that the /usr/local/bin/ignition/webserver/webdefault.xml is the jetty configuration that drives Perspective? If so, it looks like injecting a servlet parameter with a larger setting will do the trick.

Sort of.

After speaking with a Dev, it sounds like there is a property in that file which would do what you want, but the websocket servlet might not exist until Perspective is launched. The Dev does believe that's the right file, and provided a potential snippet:

Update: this content removed because it was eventually found to cause as many issues as it solved,

So you could try to modify that value and see if it works for you, but it's not supported. Long-term, this should be a system property configured either in the Gateway or Project settings and until it is we don't have a supported workaround.

Hi @cmallonee,

Is there any update on this issue ?
It seems that i can’t find the place to change the maxMessage size in this file.
I’m actually in 8.1.13.

Regards

You have to add the parameter to the file. It isn’t there by default.

1 Like

We upgraded our Production server to 8.1.18, from 8.1.10, today and now we are having a problem. Note that we have upgraded our QA and Validation server and did testing pre-upgrade of the Production server. Neither of the other servers displayed this behavior. This server has not displayed this behavior prior to the upgrade.

The errors we are seeing in the logs are as follows:

Websocket connection errored out irreversibly. Closing the session.

org.eclipse.jetty.websocket.api.MessageTooLargeException: Text message size [2118304] exceeds maximum size [2097152]

at org.eclipse.jetty.websocket.api.WebSocketPolicy.assertValidTextMessageSize(WebSocketPolicy.java:140)

at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.appendFrame(SimpleTextMessage.java:56)

at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:63)

at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.onContinuationFrame(AbstractEventDriver.java:185)

at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:157)

at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:326)

at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:148)

at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:111)

at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.forwardIncoming(CompressExtension.java:169)

at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessageDeflateExtension.java:90)

at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202)

at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:225)

at org.eclipse.jetty.websocket.common.Parser.parseSingleFrame(Parser.java:259)

at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:459)

at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:440)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)

at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)

at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)

at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)

at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)

at java.base/java.lang.Thread.run(Unknown Source)

Then,

Websocket connection closed unexpectedly. code=1009, reason=Text message size [2118304] exceeds maximum size [2097152]

What we are experiencing in Perspective seems to be isolated to 2 pages. Things seem to work fine until either of those is navigated to. Once on either of these we see a message at the top of the page saying “No Connection to Gateway”.
image
Once this occurs, we can still navigate to other pages, but no functionality works.

What would cause this to only affect one out of three servers?
Why would it start now when it had not been happening before?
Note, we did not change any project or Gateway configuration before the upgrade.

Will rolling back the version help this?

BTW, I’ve checked the two pages and their embedded views and nothing in there is pulling or pushing more than a minuscule amount of data on startup. The disconnect from the Gateway happens after simply landing on the pages. No user interaction is executed before the disconnect occurs.

Something is. Are you positive this Gateway was configured the same as the other Gateways? Were they perhaps deployed at different versions? The message you’re encountering suggests that there is indeed a large amount of data (large dataset or list, or perhaps just a very complex View) that is “too large” for the websocket to handle. This issue was fixed in 8.1.15 by allowing for a configurable max websocket message size within the config files.

Workarounds:

  1. Upgrade to 8.1.15 and modify the configuration file to account for a larger maximum websocket message size. @apena , could you provide some insight into what steps are required to modify this setting?
  2. Revert back to whatever you used before, but I expect you’ll encounter this on any version between 8.1.9 and 8.1.15; the problem was first reported in 8.1.9 and wasn’t fixed until 8.1.15.

The reason nothing seems to be working on the other pages is because the websocket completely shuts down. A browser refresh should request a new (websocket) connection, but until that occurs there is nothing actively transferring data between your browser and the Gateway.

And just for clarity, the Gateway is not being affected - or crashing - , it is only the websocket connection of the singular session.

Yes, all 3 of these Gateways are essentially clones of one another. They are running on VMs and the first was created and the others were copied for that one. The only difference that I am aware of is that the Production server has a bit more memory and CPU cores (I think) than the others.

What exactly goes into the data in this websocket message?
The two main views utilize a lot of the same embedded views. The only thing that I see for these with regards to sending/receiving data are:

One of the views gets a single boolean tag value on startup, creates a log message, and sets a view param.

The other gets a small about of data from the db, no more that 20 or so records with half a dozen columns (all numeric or string values). It also creates a couple of log entries and sets some view params to “default” values.

So I’m at a loss for where that message data is coming from.

The websocket message contains all of the literal bits required to tell the browser how to render the page, so everything from the HTML which constructs the components, to the JavaScript which defines how those components change on update, to any and all properties for each component/view. So if the problem occurs when you first open the page, it’s likely the View is “too large” or complex for the websocket to send the definition of the View in use. If you encounter this after the page loads, then it’s likely the page is not the problem, but rather data being sent to the page a s the result of an updated binding or property write. Occurrences on initial load could also be a data/binding problem as the data stored in the view could result in the definition becoming too large.

1 Like

These two views don’t have any bindings that would contain large amounts of data. They are not “larger” or more complex than other views in the project. They may even be a bit less complicated/smaller that some others.
I would think that if were a view specific issue, the two views would also be crashing the WebSocket on our QA and Val servers, since the views/projects are the same across all servers.

Sorry, but without really in-depth access I’ve reached the extent to which I can help diagnose the problem. You could contact support to help examine the environments in question and locate the difference, but I’m not in a position to make guesses or troubleshoot your deployments. My help here on the forums boils down to things I know or can research.

You were right! There was some old content that had been disabled (display set to false), but there was a query binding that was still active. Since the three servers have separate DBs, the other two were not accumulating as much data as the production server, since they are only used for testing, not nearly as much data has built up in their DBs. The biding on the production server created a dataset with 58K+ rows of data.

The reason for the disabling and not summarily deleting the content was due to the nature of the facility shifting back and forth on what they may or may not want over a period of time. So in the future, if we are directed to disable content, we need to be aware of any binding and such that could be problematic.

We did have Tech Support remote in to look (before we found the above content) and they increased the socket max value. That got us up and running. I’ll be working on a bug fix branch to remedy this and look through the other content to make sure there isn’t other monsters hiding, and waiting to attack!

Thanks for your help!

1 Like

As my colleague @cmallonee mentioned, for those who need steps to add/modify the websocket max message size for Perspective, you will need to do the following to add the JVM parameter to Ignition:

  1. Open the ignition.conf file that is located in the data folder that’s found in your Ignition installation path
  2. Open the ignition.conf with any text editor
  3. Within the ignition.conf file, scroll down to where it says # Java Additional Parameters; This is the location where JVM parameters are added into Ignition and where we will be adding our web socket max message size parameter
  4. Copy/Paste the following line in the # Java Additional Parameters section of the ignition.conf file:

wrapper.java.additional.X=-Dperspective.websocket.max-message-size=#

  1. Replace X in the line with the next sequential parameter number that is seen in the file.
  2. Replace # in the line with the websocket max message size value in KB; The default size is 2048 KB when no value is specified
  3. Save changes to the ignition.conf file; A restart of the Ignition gateway service will be required for the changes to take effect. This can be done by following the instructions that are provided in the following link: Gateway Command-line Utility - gwcmd - Ignition User Manual 8.1 - Ignition Documentation

Here is an example of how the JVM parameter entry would look like :

Doing these steps, the value that was configured will take effect.

5 Likes