Hello, all, i am having an issue with my perspective project. I have a view, which I am opening as a pop-up view, and passing some parameters to it. One of the parameters is a dataset. However when the dataset is a bit larger(~1000x300) the gateway crashes when I try to open the pop-up view(I am getting the message that I have been disconnected from the gateway). With smaller datasets(~200x300), there is no issue. Why is this happening? I can solve the issue, by not passing the dataset as a parameter and storing it in a session property, but this does not seem like a good design practice.
You’ll need to look in your wrapper log for details on the crash. If it doesn’t make sense, post the last several lines of errors here (use the pre-format button (</>) after pasting so the plain text doesn’t get mangled.
Chances are, your gateway is running close to its configured memory limit (from ignition.conf) and the large dataset conversion to json (under the hood) is pushing it over the edge.
Thanks for your response, i opened the gateway logs and the error is the following:
Websocket connection closed unexpectedly. code=1009, reason=Resulting message size [2.105.344] is too large for configured max of [2.097.152]
Huh. I don’t know how to fix that. If an IA person doesn’t offer advice in the next day or so, you probably want to open a support ticket.
Just some clarification: is the Gateway crashing, or is the session being disconnected? Your posts lead me to believe you’re only experiencing a session disconnect, which is far less worrisome than your Gateway actually going down. I’m reaching out to Dev to determine where this websocket max “message size” property is configured, or if it even can be configured.
After speaking with Dev it sounds like you are indeed attempting to pass so much data through the websocket that the websocket is tripping an internal safety setting and closing. The max websocket message size is configured in Jetty. As such, you do have the ability to configure this property, alas, I wouldn’t be able to walk you through this as I have no experience in that area. Someone with a good amount of experience in that area (nudges @pturmel) might be able to help you configure Jetty to allow for larger messages to be sent without closing the connection.
Alternatively, you could reduce the size of the data you are sending through the websocket. This might not work for your scenario, though I’d be interested to see if you encounter the same issue while passing the dataset to a running View instead of opening a Popup. My suspicion is that the size of the dataset is the sole culprit, but if the Popup is incredibly complicated then the popup could be part of the problem.
Me? Configuring Jetty? Nada.
Thank you all for your responses.
@cmallonee you are right, the gateway is not crashing, I am just being disconnected, although to the end user it seems like it crashed(it takes a refresh to recover). The popup I would not call incredibly complicated. Basically it allows the user to set up custom filters for a data set, it includes a table a dropdown, and an embedded view(which in turn is very simple). However to set up some filters, the dataset is required, as the options are from the data itself, so the view has to access it in some way. To answer your question, it does the same even if I have the view running, in an embedded view. As soon as i increase the data set size, it disconnects the same way.
Apart from reducing the dataset size, what do you think the options are in this case? Should I increase the max message size(I suppose the limit is there for a reason?-also I dont know how to do that). Or should I just store the dataset in a session property and be done?(Either way it has to be stored somewhere).
Are you actually displaying the dataset? Or just using it as a source for other operations? If the latter, try making it private, so it won’t go back and forth to the browser.
Are you able to send me the View and dataset via a direct message (click my name and select “Message”)? I could take a look and see where some changes might be warranted.
Do you need the whole dataset? Are you just filtering based on column names? If so, you could send only the headers.
@pturmel I tried making the dataset private(I also set the view parameter as private although there is no indication so I dont know if it can actually be set private). I still get the disconnect message, although now it recovers by itself. So I get a momentary disconect, then it reconects and all is ok, whreas previously it did not recover without a refresh
@cmallonee
Thanks for the offer, I will have to get back to you on Monday because I will have to clean up some of the data.
I need not only the column names, but some of the data as well since one of the filter options is actually a dropdown-select of all the possible strings within a column, so i need the actual column data. I suppose i would be able to reduce the dataset somewhat by eliminating the non-string columns, but that would not solve the problem, because the dataset is not fixed, but the user selects Start and End date, so it could still crash on the end user, if he selects a big enough date range.
So…we’re also encountering this same error in our logs. Appeared this morning after 3 weeks of a solution running with no issues. We’re on 8.0.12.
Here’s the thing: the message size reported in our logs is, you guessed it: 2,105,344
I’m going to go out on a limb and say that @akolovos and I don’t have the same project code, or anywhere close to it.
I’m planning to contact support, because this is affecting a customer, but wanted to throw this up for those involved.
@cmallonee, any chance you can confirm that the /usr/local/bin/ignition/webserver/webdefault.xml is the jetty configuration that drives Perspective? If so, it looks like injecting a servlet parameter with a larger setting will do the trick.
Sort of.
After speaking with a Dev, it sounds like there is a property in that file which would do what you want, but the websocket servlet might not exist until Perspective is launched. The Dev does believe that's the right file, and provided a potential snippet:
Update: this content removed because it was eventually found to cause as many issues as it solved,
So you could try to modify that value and see if it works for you, but it's not supported. Long-term, this should be a system property configured either in the Gateway or Project settings and until it is we don't have a supported workaround.
Hi @cmallonee,
Is there any update on this issue ?
It seems that i can't find the place to change the maxMessage size in this file.
I'm actually in 8.1.13.
Regards
You have to add the parameter to the file. It isn't there by default.
1 Like
We upgraded our Production server to 8.1.18, from 8.1.10, today and now we are having a problem. Note that we have upgraded our QA and Validation server and did testing pre-upgrade of the Production server. Neither of the other servers displayed this behavior. This server has not displayed this behavior prior to the upgrade.
The errors we are seeing in the logs are as follows:
Websocket connection errored out irreversibly. Closing the session.
org.eclipse.jetty.websocket.api.MessageTooLargeException: Text message size [2118304] exceeds maximum size [2097152]
at org.eclipse.jetty.websocket.api.WebSocketPolicy.assertValidTextMessageSize(WebSocketPolicy.java:140)
at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.appendFrame(SimpleTextMessage.java:56)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:63)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.onContinuationFrame(AbstractEventDriver.java:185)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:157)
at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:326)
at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:148)
at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:111)
at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.forwardIncoming(CompressExtension.java:169)
at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessageDeflateExtension.java:90)
at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:225)
at org.eclipse.jetty.websocket.common.Parser.parseSingleFrame(Parser.java:259)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:459)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:440)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Unknown Source)
Then,
Websocket connection closed unexpectedly. code=1009, reason=Text message size [2118304] exceeds maximum size [2097152]
What we are experiencing in Perspective seems to be isolated to 2 pages. Things seem to work fine until either of those is navigated to. Once on either of these we see a message at the top of the page saying “No Connection to Gateway”.
Once this occurs, we can still navigate to other pages, but no functionality works.
What would cause this to only affect one out of three servers?
Why would it start now when it had not been happening before?
Note, we did not change any project or Gateway configuration before the upgrade.
Will rolling back the version help this?
BTW, I’ve checked the two pages and their embedded views and nothing in there is pulling or pushing more than a minuscule amount of data on startup. The disconnect from the Gateway happens after simply landing on the pages. No user interaction is executed before the disconnect occurs.
Something is. Are you positive this Gateway was configured the same as the other Gateways? Were they perhaps deployed at different versions? The message you're encountering suggests that there is indeed a large amount of data (large dataset or list, or perhaps just a very complex View) that is "too large" for the websocket to handle. This issue was fixed in 8.1.15 by allowing for a configurable max websocket message size within the config files.
Workarounds:
- Upgrade to 8.1.15 and modify the configuration file to account for a larger maximum websocket message size. @apena , could you provide some insight into what steps are required to modify this setting?
- Revert back to whatever you used before, but I expect you'll encounter this on any version between 8.1.9 and 8.1.15; the problem was first reported in 8.1.9 and wasn't fixed until 8.1.15.
The reason nothing seems to be working on the other pages is because the websocket completely shuts down. A browser refresh should request a new (websocket) connection, but until that occurs there is nothing actively transferring data between your browser and the Gateway.
And just for clarity, the Gateway is not being affected - or crashing - , it is only the websocket connection of the singular session.
Yes, all 3 of these Gateways are essentially clones of one another. They are running on VMs and the first was created and the others were copied for that one. The only difference that I am aware of is that the Production server has a bit more memory and CPU cores (I think) than the others.
What exactly goes into the data in this websocket message?
The two main views utilize a lot of the same embedded views. The only thing that I see for these with regards to sending/receiving data are:
One of the views gets a single boolean tag value on startup, creates a log message, and sets a view param.
The other gets a small about of data from the db, no more that 20 or so records with half a dozen columns (all numeric or string values). It also creates a couple of log entries and sets some view params to "default" values.
So I'm at a loss for where that message data is coming from.