Thanks Kevin, that is looking better now!
Hi everyone, we encountered the following issues:
- RTU: Honeywell RTU2020 UCMX01
when I make any of the following changes, the class polls from Ignition stop:
- Write to RTU tag through valuechange script
- Enable unsolicited messages
- Change class poll rate
(These three change scenarios were tested independently.)
We have verified this by looking at the wireshark traffic. Only integrity polls are sent by Ignition and at a frequency different to what we have configured.
Normal traffic(Class poll)
Not normal traffic
In addition, if we want to restore to normal state, we need to power cycle the RTU again with the device connection disabled in Ignition. I cannot re-establish a normal link by re-enbaling the device in Ignition (even if I change it back to the original settings).
Here is what happened when testing another DNP3 device:
- RTU: SCADA Pack E-Series
SCADA Pack is connected according to the Ignition gateway however Ignition is not issuing any class read requests (see screenshot below). There is no class poll in wireshark traffic, only a bunch of RTU record current time.
Please upload the capture files and not screenshots.
Thanks Kevin, please see a further breakdown of events and network captures attached.
We noticed that when we write a heartbeat to the RTU, changes to the device connection in the gateway are breaking the connection.
Enabled a script which writes from a simulation tag to an analog output.
def map_tags(in_tags, out_tags):
read_res = system.tag.readBlocking(in_tags)
values = [res.value for res in read_res]
system.tag.writeBlocking(out_tags, values)
return
in_tags = ["[default]DNP3test/IgnitionScriptUse/Sine0"]
out_tags = ["[default]DNP3test/IgnitionWrite/IGN_SinewaveC1"]
test_functions.map_tags(in_tags, out_tags)
Writes were succeeding and the analog input was updating. Sequence of events below:
1047 - In the device connection settings, changed class 1 poll rate from 5 seconds to 7 secs. The tags went into a bad state as follows:
[-0.8673891, Uncertain_LastKnownValue("Whatever was updating this value has stopped doing so."), Thu Nov 30 10:47:07 AEDT 2023 (1701301627986)]
Looking at the wireshark, only integrity polls were sent from Ignition. Class polls were no longer sent for any class.
Writes were failing with error:
[Bad("Bad_InternalError: An internal error occurred as a result of a programming or configuration error.")]
1050 - disabled write script, system did not recover
1057 - Device connection disabled in Ignition gateway, restarted RTU and enabled project. Wireshark looks like comms is going through a backfilling process. Data does not seem to be restored to the RTU.
1100 - disabled connection and power cycled device
1103 - re-enabled connection. Behaviour is the same, system is not recovered.
1105 - Disabled analog output tag in designer
1105 - Restarted device connection in gateway, no change
1107 - disabled connection and power cycled device
1111 - Re-enabled device connection in gateway
1112 - Restart Ignition gateway, gateway comes back but still only issueing integrity polls but it is now processing the event data from the RTU. Tag and trend is updating but the class poll configuraiton is not working.
Other notes/information:
- When you save the device connection in designer, the next poll does not appear to backfill/proccess event data. We can see gaps in our trending.
We would be happy to set up a meeting to demonstrate as we have been able to repeat this issue. I'm not sure if other people have seen this? We have also seen this intermittently when we don't have the analog output point in the picture.
2023-11-30-dnp3-beta-comms-fail-when-writes-enabled-part-2.pcapng (432.8 KB)
2023-11-30-dnp3-beta-comms-fail-when-writes-enabled.pcapng (162.1 KB)
Can you share the Ignition logs and the OPC Item Path of the tag you're writing to?
edit: actually nevermind, hold off on this. I'm going to upload a new beta tonight or tomorrow that includes some additional logging options for the DNP3 stack that we can turn on to help debug this. I'll give you some instructions to follow once the module is available.
Also, to be clear, in the current version of the module if you have unsolicited enabled for a given class, the polls for that class do not happen. They are mutually exclusive.
The new module that I will upload changes this behavior as well. If the class poll needs to be disabled when unsolicited is enabled then it can be done by setting the poll rate to <= 0.
1 Like
We actually tested both situations, and this problem also occurred when unsolicited message was not enabled.
Additionally, we have just tried setting up a second DNP3 device. I tried to write to an Analog Tag but we get the following error:
I know that we are looking at the correct DNP3 point because I manually wrote the "4" to it from the RTU.
Sounds good, looking forward to seeing the new beta. Thanks!
Okay, new module available here (yes, same link as previously).
If there are still issue(s), narrow your focus to just one at a time.
Turn on the application layer debug logging:
Turn all the DNP3 v2 loggers to DEBUG level:
- disable the device
- start Wireshark
- enable the device
- reproduce a single issue
- stop the capture
- export the Ignition logs
- come back and post the capture, the logs, and a clear explanation of what problem you've reproduced
edit: also some screenshots of the Ignition device settings would be useful.
Hi Kevin, thanks for your support.
Here is the attachment of the requested files for investigation:
Ignition-DNP3-beta-test
I've recorded two clips here to explain what's going on：
-
Q1 12-01 2:30PM : After starting the device,
In order to make the italicized values disappear, I need to re-download the RTU program. And the tag is not updated according to the set poll rate. (updating slowly) -
Q2 12-01 2:55PM: Changing Configuration will change the tag value text to italics (uncertain state, not updating)
The above tests were all conducted with write script disabled.
Unsolicited messages disabled.
I don't know how to download the filtered Ignition log, so the log seems a bit large.
please feel free to let me know if you require other documentations.
I am happy to setup a meeting and share my screen if that helps too.
Thank you, these are very useful.
Right now it looks like the underlying issue is simply that the library lacks support for the g43v5 object/variation, and this decoding failure puts the connection into an undesirable state. I'm contacting the vendor to ensure this object/variation is not supported and figure out why the library behaves the way it does.
Support for g43v5 is only required at conformance level 4 - can you set your outstation to L3 instead? I don't know how this plays out in the real world, but the spec says that all outstations must be able to be configured for the master's conformance level, and shall not send object/variations not required by that level when configured.
1 Like
Thank you so much for your help and this discovery is very important.
I don't think we can configure the conformance level in the RTUs we are using. We can however configure the default group and variation used when the master does not ask for a specific one. Do you know what group/variations are supported? For example with the analog output, we could set it to G43V7 (single precision floating point with time).
- Configuration Options
Can you provide a list to help me determine how I should set this up, thanks!
Also, I checked in the Honeywell RTU2020 documentation, it looks like they support Level 3 conformance. However, there is no option to switch between different levels in the RTU settings, so I am not sure how the DNP3 conformance level selection mechanism is implemented.
Group 13 and 43 aren’t supported at all, in any variation.
( binary and analog output command events)
Hi Kevin, thanks for the information. I've had a talk internally and the RTU's we've played with don't seem to be able to change the conformance level as a whole. Additionally, I don't think we can disable the output command events. I'll ask our vendor if this is possible.
Do you think the IA DNP3 driver will be able to be updated to ignore the command event objects?
Cheers,
The underlying DNP3 stack will eventually but updated so it can decode them, and the driver will ignore them because they don't sensibly map to OPC UA, but it won't be before release and I have no idea what the timeline is.
If you can't disable those events this RTU is essentially only able to communicate with L4 masters. Hard to imagine it's not configurable it some way.
I second this request.
No need, it already happened, and it's available now.
1 Like
Hi Kevin,
Can you please outline the changes made to the new version of the beta module?
Cheers,
Changes/additions since the first module was uploaded:
- advanced setting to enable application layer debug logging
- scripting function
system.dnp.synchronizeTime() - allow class-based polls at the same time as unsolicited events
- modified the read-after-operate behavior so it reads less points
Nicholas, how are you time syncing the SCADAPack with Ignition? Thanks!