It is something that we should be calling out in our documentation, so this is a good exercise. Having 2 different time sources has the potential for this type of problem (and times don't always stay in sync). Making sure our documentation makes a note of how this works is something we should add as it might remind people to double check the time settings.
I fixed the time zones and I am still having issues...
Not just time zones. Precise clock synchronization. The bigger the clock error, the bigger a gap for a sample to fall out.
Hi all, I've just started some testing on the new beta with the following Setup Details:
- RTU: SCADAPack 474
- Single analog point generating sine wave buffering events every second
I've come across the following issue and would be great if someone else could replicate in their test environment.
If there are buffered events within the RTU, and a class 0 poll only is triggered, it returns the instantaneous value without the buffer as expected. Then if a class 1/2/3 is triggered and the buffers are returned from the RTU, it only processes data after the class 0. It seems to discard buffered events prior to the class 0. Hopefully the attached screen grab of the stored history explains it a bit more clearly.
So process to trigger the bug was:
- Trigger a class 0 poll only: system.dnp.demandPoll('DNP_Test',[0])
- Let buffer continue to fill for a while
- Trigger a class 1 poll or an integrity poll (Class 0, 1, 2, 3): system.dnp.demandPoll('DNP_Test',[0,1,2,3])
I can confirm that the missing historic events are being returned to Ignition by inspecting Wireshark logs, and Ignition then confirms the response which tells the RTU to clear the buffer. Ignition however never passes this historic value back to the Historian.
This has already been resolved above, I'll leave the comment above for others who come across this issue.
An additional problem that I am coming across is the triggering of individual class polling whilst Unsolicited messages are enabled on the device. When classes 1,2,3 are enabled on the Unsolicited Event Classes at the bottom of the device configuration, Ignition no longer triggers regular class polls regardless of what is set in the polling periods above.
When I remove the class 1,2,3 unsol events, the class polling continues to work as expected.
Can anyone else replicate this is their environment?
As designed. When unsol is enabled for a class the scheduled poll for that class isn’t registered.
Why is that? In my experience it is required to have both at the same time to allow for a regular polled schedule, with the ability to trigger an unsolicited message for a key event.
With most of our water utility clients, typically they would have a scheduled class poll of about 2 hours with a number of points inside the RTU that were required to be alarmed as soon as they happened in SCADA via unsolicited events, such as:
- Mains Power Failed
- Pump Fault
- Station Intruder
This then gives SCADA the best of both worlds, of having efficient class polls as well as instant updates for key points.
2 Likes
No reason in particular, I think we could easily change it to allow both. Feedback like this is why we’re doing the beta.
I think it would be very valuable to have the ability to have both.
Thanks for the clarifications Kevin
1 Like
Another thing I've stumbled across whilst testing with standard packet delays that are consistent over traditional radio networks. My setup for this test is as follows:
- Data Link Confirm Mode: Always
- Lag for inbound and outbound packets: 500ms (1 second round trip)
- Event Buffer Size: ~2500 events
- SCADA Side Device Timeout: 1000 seconds
Watching Wireshark I can see that the data chunks coming from the RTU are being correctly interpreted by Ignition as it triggers a confirm response at the end of each chunk. This confirm indicates to the RTU that all the data has been received and it clears this section of the buffer. This process continues until the buffer has been completely emptied. Watching just the packets, the comms process is acting exactly as expected.
However once this data has been received, Ignition doesn't seem to pass these historic events in to the historian, meaning that this data has been lost.
I've run this test a few times, and it seems to be occur only occasionally, so very hard to really point a finger at what is causing it.
I have attached a screenshot and packet capture here which should hopefully show what I mean:
500msDelayedPackets_AlwaysMode_IntegrityPoll.pcapng (26.8 KB)
Interestingly, when I modify the data link confirm mode to Sometimes or Never, the historian begins receiving the data as expected. The only real difference here that I can see is the number of packets required for this transfer between the two modes, which shouldn't really affect how Ignition stores the data.
1 Like
Another recommendation here would be putting pre-defined offsets on class and integrity polls relative to a fixed time of the day.
For example if Ignition is communicating to 100 DNP3 devices in the field, and they all have a scheduled class 1/2/3 poll rate of every 2 hours, in it's current state all devices will be polled at exactly the same time, every 2 hours. So if the driver is originally started at 12:00pm, then all devices will trigger a class poll at 2:00:00pm, 4:00:00pm, 6:00:00pm etc. This creates a large amount of traffic for a very short period of time, then nothing for another 2 hours. This defeats the purpose of the efficient transfer model that DNP3 is built around.
An alternative to this would be to create offsets on each of your class polls that would be used to spread these 100 devices over the 2 hour period, which should result in an evenly spread cyclic polling schedule through the network. This would significantly reduce peak throughput, and be more of a consistent throughput.
This delay could look something like the following:
- Device 1 Poll Delay: +100 seconds (This would indicate to trigger 100 seconds after every 2 hour time interval through the day - i.e. 2:01:40pm, 4:01:40pm etc.)
- Device 2 Poll Delay: +200 seconds (This would indicate to trigger 200 seconds after every 2 hour time interval through the day - i.e. 2:03:20pm, 4:03:20pm etc.)
Then for every device you would increment this delay to maintain a consistent time delay between each devices polling schedule over the two hour period. These are all numbers that are consistent with our client base, so can be thought of as a fairly standard DNP3 telemetry rollout.
I've noticed that you can kind of achieve this by modifying the configuration of the device and saving it, which then resets it's timer creating an offset between devices. Whilst this work around is not ideal, hopefully it means that it shouldn't be too hard to build in this new functionality?
The final couple of suggestions that I can see that would be highly beneficial to the driver are:
- UDP. A vast majority of our DNP3 clients use old radio networks that traditionally have very poor reliability and bandwidth issues. Most of the clients can get away with tuning the DNP3 to their limits through the poll rates and the data link confirm modes, however some require UDP as an alternative. This is to avoid the overhead TCP packets that you get which are in turn clogging up their networks. I appreciate that in a fixed line, microwave/4G or other high bandwidth solutions, this isn't an issue; however for a number of clients these are not an option due to no cell reception or other geographical reasons.
- Dual Endpoint (Listen Only). In our market, a number of IIOT devices have become popular that utilise DNP3 via a listen only connection, meaning that the connection is established by the RTU only. This would require the provisioning of a listening port for each device when the RTU would offload the data to the Gateways IP and this defined port.
I understand this is much more work to implement these two suggestions than the other suggestions previously noted in this forum, but just wanted to get in our two cents that they would be very valuable features to have going forward. Thanks to IA's efforts in bringing this updated driver into development, it is a significant improvement over the previous generation.
2 Likes
Greetings to everyone present. I have conducted some tests with the new beta version of DNP3, and the configuration details are as follows:
- RTU: ORION I/O, BRAND: NOVATECH
- 60 analog signals, internally simulated in the RTU
- 60 binary signals, internally simulated in the RTU
The generation of signal values is produced (simulated) by internal logic in the RTU, assigning random values at defined time intervals in the logic.
Here are my comments and findings:
- Functionality test, timestamp: The RTU time was configured with a one-day delay. It is expected that the simulated signals in the RTU will be sent to Ignition, and Ignition should receive the timestamp with the delayed RTU timestamp.
Result: As expected, the signals are logged in Ignition with the RTU timestamp as anticipated.
- Functionality test, Poll/unsolicited: The signal map was configured to send signals from the RTU to Ignition. Ignition receives analog signals with the following configuration:
Class 2 unsolicited activated Class 2 Poll Interval 13 seconds Integrity Poll every 20 seconds
It is expected that the simulated analog signals from the RTU will be sent according to the poll times and spontaneous changes.
Result: As expected, the signals are logged in Ignition at the expected times and spontaneous changes.
2.1 Variation of test 2: During the tests to disable "unsolicited" messages from Class 2, it is expected that the signals sent by the RTU follow the poll times. The Interval poll and integrity poll were received as expected.
2.2 Deactivation of Class 2: During the tests to disable "unsolicited" messages from Class 2, it is expected that the last message received from the RTU has the timestamp corresponding to the RTU. During this process, the last value is received with the Ignition timestamp, and then when updating the data, it is received with the RTU timestamp.
Result: Observed, the timestamp of the signal received during the deactivation of Class 2 should maintain the RTU timestamp or wait for the next reading.
- Stress test, 360 variations/second: The RTU was configured to generate 360 variations per second. Ignition was configured to receive analog signals with the following settings:
Class 2 unsolicited activated Class 2 Poll Interval 13 seconds Integrity Poll every 20 seconds
It was expected to receive 360 variations in one second in Ignition.
Result: It is observed that out of the 360 measurements, only 2 to 3 simulated values were recorded with the same timestamp. It is recommended to review the signal reception behavior as the 360 variations were not received as expected.
Tests are still ongoing for a more detailed examination of the observations.
1 Like
Modifying poll intervals or any other device setting is a more heavyweight operation than you are probably imagining. Device lifecycle is nothing more than shutdown or startup. When shutting down it has no idea if it is going to start again. When starting up it is given the current settings, with no knowledge of any prior settings, if there were even modifications.
Consequently you can't expect anything but what you observed, which is upon editing a device it does the normal startup routine that includes a poll for class 1/2/3/0.
Can you share your Tag Group settings for this test?
+1. Please.
We generally set up our important items in Class 1, and set those to unsolicited, but would like the class polling to also read Class 1, in case something got missed/dropped with the unsolicited.
Also, if we get an event flood (and we want the events from it) the device can send the unsolicited message before the next poll and before its buffers overflow. Ours are set for a fixed number of events before the unsolicited message is sent. Under normal conditions we will not reach this value before the class poll(s), but under unusual conditions we may, and, as Nicholas stated, this allows best of both worlds - reduced bandwidth under normal conditions, and high resolution data under abnormal conditions.
2 Likes
With the new driver, notice that we can set the Trip/Close code at the driver level, would really like to see this at the tag level. For breakers, we want to be able to both trip and close, and not having that ability at the tag level means we will once again have to resort to scripting.
Also, for me, Op Type for binaries should default to Pulse, but don't know if that is a common request.
You can use the scripting functions which give you per-point settings.
Understood. Would like to get away from scripting. Dream world, the tag definitions would be updated to include a Data Type of CROB to support DNP3 rather than having to choose Boolean and script. I guess that's a feature request 
1 Like
Probably unrealistic to ever have it defined on the Ignition tag. OPC tags are just one kind of tag, and they don't know anything about the underlying server or data source behind that server. There's just multiple layers of abstraction with your Ignition tag that happens to come from an OPC server that happens to be exposing tags from a DNP3 device.
Hi everyone, we are having issues getting the module to process DNP3 event data. Our setup is as follows:
1 RTU on the bench generating a sine wave to an analog input. We have confirmed multiple events are generated by looking at the DNP3 payload (DNP3 response) sent by the RTU to Ignition via Wireshark.
Tag configuration in attachment. 2023-11-28-DNP3-Troubleshooting.pdf (583.6 KB)
It seems like this issue has been raised a few times and resolved but we haven't been able to get it to work. We have confirmed that the time in the RTU is correct and we have set the suggested "Allow Back Fill Data" setting.
Wireshark payload showing that events are timestamped and apparent in the payload:
Screenshot of sine wave in Ignition:
We are expecting a smooth sine wave. When we increase the class poll rate, we do see the expected pattern but when we increase the class poll rate (to 5 seconds for example) the sine wave is not smooth. We have confirmed that Ignition is only storing one data point per class poll by looking at the data populated in the database after each poll. Only one new row is added.
Regarding the "Allow Back Fill Data" setting, we also wanted to provide feedback that if this is required for Ignition to process DNP3 events, we would also like to see it process alarms. A use case for this would be that a local alarm is raised and cleared before the data is sent to Ignition via an RTU. We may still want it to appear in the alarm summary as unacknowledged so that a user can come and acknowledge the alarm has occurred to clear it from the alarm summary.
The OPC Data Mode of your Tag Group should be left at Subscribed; you have changed it to Polled.
1 Like