At the beginning of the month, we started seeing this UascServerHelloHandler error popping up in our logs. It has been happening almost every day, and I cannot find any events on the server that correlate with when this error appears. After these errors started appearing, we updated to V8.1.38 from 8.1.36 which was installed in January, and upped the memory to 8GB from 4GB. I don't see any issues with our OPC server. I am not sure where else to look as it seems the other posts about this error as of the time of this writing do not apply to our situation. We are not using Edge, and we have a standalone installation on a Virtual Server.
?
Apologies, forgot to paste the error. [remote=/10.139.128.89:49371] Exception caught; sent ErrorMessage{error=StatusCode{name=Bad_TcpMessageTooLarge, value=0x80800000, quality=bad}, reason=max message length exceeded (1414012975 > 4128)}
It seems to come in pairs, with the port number increasing by one after each pair.
Something is trying to send 1.4GB in a single message.
Do you have an IT department that does vulnerability scanning across your datacenter?
Yeah, this is the most likely cause. It's probably trying to decode the beginnings of an HTTP request as a UA binary message.
I spoke with our IT department about this, and they say that nothing has changed. That was my first thought as the first time this popped up was 1 Mar. It has been happening nearly every day between 5 AM and 7 AM Central time for the most part, which lead me to believe that something scheduled on the server could have been the problem. I was told that nothing has been changed on that server, and no policy updates were made. We do have exceptions to the policy for that server since it handles production, and IT is pretty cool about remaining hands off and letting me manage it for the most part.
Whatever is making the connection is coming from this IP.
Set up a Wireshark capture?
Something changed.
That is the local IP address of the server. That was something else I noticed that was different from the other posts. Theirs showed localhost rather than the actual IP address. So if I understand correctly, the server is sending itself a 1.4 GB message? I will see if I can set up something in Wireshark.
It's not actually a 1.4GB message.
Something that isn't an OPC UA client is opening a TCP connection and sending data to the server, and obviously the server will attempt to decode it as OPC UA, but it's basically non-sense, and in this case fails because it thinks the client is trying to send a giant message.
1 Like
That makes sense. Thank you. I will try to get a Wireshark capture and update this post if I find anything.
A bit of an update. It might be worth noting that my installation was 7.9 upgraded to 8.1 about a year and a half ago. I was finally able to capture the event in Wireshark, but could only capture it on the loopback adapter. It seems like whatever the culprit is, it is focused solely on Ignition. We get two "GET / HTTP/1.1" on port 4096 (hence the 7.9 comment) in a row. These throw the Uasc error. A few seconds later, there is another pair of GETs on port 8060, followed almost immediately by a pair of GETs on port 8088. I can't gleam much information from the capture, but the User-Agent in all three cases is something called SecurityManagerAgent. We cannot locate any information on this agent. We have had our security team go through everything on their end, and they do not recognize that agent either.
That smells like penetration testing. On a regular basis. You should press harder on your IT group.
1 Like
I would be inclined to agree with you if it was coming from an external IP address, but the source and destination are both from the local adapter, and can only be seen when sniffing the loopback adapter. I have tried to capture this on the two network adapters on the server, and there is no trace of this in either of them. I can replicate the fault from other machines using curl in PowerShell, but in those cases I can see exactly which machine it comes from, and these do not appear on the loopback adapter.
A couple of my clients require the installation of a security snooping agent in every server (including Linux servers), so that self-probing can be done. That is what I would suspect is going on.
If that hasn't been done on your servers, then you should raise the alarm that your server might have persistent malware in place.
Phew, issue solved. You weren't very far off @pturmel. It wasn't pen-testing, but upgrade management. We use ManageEngine to make sure software is up to date. I'm not sure if it is probing every port possible on the server, or just ports it sees activity on, but it did identify a number of ports used by Ignition. It scanned every one of them looking for Java versions. Thank you both for your assistance.
Upgrade management software that's indistinguishable from malware 
1 Like