V7.7 wit logix V21 Driver

Ignition
1

Using V7.7.1 RC1

I get nothing but Warnings in the gateway console related to every device I have connected via OPC. This only happens when I enable two logix V21 devices using the new V21 driver. When I disable the V21 devices all my problems go away. This symptom renders the gateway useless.

The two devices are Allen-Bradley L73S guardlogix PLC’s. It seems to me that there is something with the new V21 driver that is causing havoc with the older Ethernet drivers.

Is this a known issue? See below for reference

Time	Logger	Message

(I) 11:41:12 AM DownTimeDetailRecorder IsShortStop did not update any rows.
(I) 11:41:12 AM DownTimeDetailRecorder OperatorCellName did not update any rows.
(I) 11:41:12 AM CacheManager [[global]] Starting production cache manager, delay execute for 59 seconds.
(I) 11:41:12 AM MESObjectManager Successfully verified MES equipment objects.
(I) 11:41:11 AM ProductionModelRunner [[global]] Production Model Runner starting.
(I) 11:41:10 AM ProductionSubscription [Scanclass ‘Default[default]’] Starting Production OPC subscription (productioncontext is running=true.)
(I) 11:41:10 AM ProductionSubscription [Scanclass ‘Default[default]’] Stopped Production OPC subscription.
(I) 11:41:10 AM ProductionSubscriptionCollection Invalidating all subscriptions.
(I) 11:41:10 AM ProductionModelManager [[global]] Loading MES production model from project using serialization version 1. (Resource name = Model)
(I) 11:41:10 AM ProductionGatewayHook [Staging version] projectUpdated called on [global]. Edit Count=176
ERROR 11:41:01 AM RuntimeDataQueryManager Error collecting scheduled vs. actual data.

com.sepasoft.production.common.exception.ProductionException: Production Module Error: RunContext operation named getRunInfo not found.
more
ERROR 11:40:30 AM RuntimeDataQueryManager Error collecting scheduled vs. actual data.

com.sepasoft.production.common.exception.ProductionException: Production Module Error: RunContext operation named getRunInfo not found.
more
(I) 11:40:30 AM ControlLogixDriver[BMS Cell 25] [BMS Cell 25] Updating tagManager with edit version 44801 with 3629 tags
(I) 11:40:30 AM ControlLogixDriver[BMS Cell 25] [BMS Cell 25] Browse complete.
(I) 11:40:30 AM ABControlLogixBrowseRequest [BMS Cell 25] Browse complete.
(I) 11:40:30 AM ABControlLogixBrowseRequest [BMS Cell 25] Creating browse nodes.
ERROR 11:40:00 AM RuntimeDataQueryManager Error collecting scheduled vs. actual data.

com.sepasoft.production.common.exception.ProductionException: Production Module Error: RunContext operation named getRunInfo not found.
more
(I) 11:39:46 AM ABControlLogixBrowseRequest [BMS Cell 25] Creating browse request using HIGH priority.
(I) 11:39:46 AM ControlLogixDriver[BMS Cell 25] Comm state changed to ACTIVE
(I) 11:39:45 AM PropertiesFactory Loading properties files from jar:file:/C:/Program%20Files%20(x86)/Inductive%20Automation/Ignition/webserver/webapps/main/WEB-INF/lib/wicket-extensions-6.1.1.jar!/org/apache/wicket/extensions/Initializer.properties with loader org.apache.wicket.resource.IsoPropertiesFilePropertiesLoader@108191a
(I) 11:39:45 AM PropertiesFactory Loading properties files from jar:file:/C:/Program%20Files%20(x86)/Inductive%20Automation/Ignition/webserver/webapps/main/WEB-INF/lib/wicket-core-6.1.1.jar!/org/apache/wicket/Application.properties with loader org.apache.wicket.resource.IsoPropertiesFilePropertiesLoader@108191a
WARN 11:39:45 AM ClockDriftDetector Clock drift, degraded performance, or pause-the-world detected. lastTime=Sat Aug 16 11:39:41 EDT 2014 currentTime=Sat Aug 16 11:39:45 EDT 2014 deviation from expected 1000ms delta=2619ms.
(I) 11:39:41 AM LocalDBManagerImpl Created auto-backup of internal database “settings” in 15 seconds
(I) 11:39:41 AM ENGINE Successfully backed up instance ‘settings’ to ‘C:\Program Files (x86)\Inductive Automation\Ignition\data\db\autobackup\db_20140816_113925.tar.gz’
(I) 11:39:40 AM LogixBrowseStateManager Browse completed in 29777ms. 607 symbols, 923673 tags, 22 programs.
(I) 11:39:40 AM ControlLogixDriver[BMS Cell 25] Processor info: Vendor=1, Product Type=14, Product Code=86, Revision=19.11, Product Name=1756-L63S/B LOGIX5563SAFETY
(I) 11:39:40 AM ControlLogixDriver[BMS Cell 25] Comm state changed to INACTIVE
WARN 11:39:40 AM ControlLogixDriver[BMS Cell 25] [BMS Cell 25] Browse unsuccessfully.
WARN 11:39:40 AM ABControlLogixBrowseRequest [BMS Cell 25] Browse request failed because fail type is other than TIMEOUT, type = DISCONNECTED.
(I) 11:39:40 AM ABControlLogixBrowseRequest [BMS Cell 25] Browse request failed because fail type is other than TIMEOUT, type = DISCONNECTED.
WARN 11:39:40 AM ABSLCReadRequest Read Request timeout. First item: N31:20, Timeout Count: 1, Transaction: 2634
WARN 11:39:40 AM TimeoutDaemon ScheduledRequest[com.inductiveautomation.xopc.drivers.allenbradley.requests.ABSLCReadRequest@ed166b] request with key “2634” failed due to timeout.
WARN 11:39:40 AM ABSLCReadRequest Read Request timeout. First item: B3:16/5, Timeout Count: 1, Transaction: 3809
WARN 11:39:40 AM TimeoutDaemon ScheduledRequest[com.inductiveautomation.xopc.drivers.allenbradley.requests.ABSLCReadRequest@110ceb3] request with key “3809” failed due to timeout.
WARN 11:39:40 AM ABSLCReadRequest Read Request timeout. First item: C5:39.ACC, Timeout Count: 1, Transaction: 649
WARN 11:39:40 AM TimeoutDaemon ScheduledRequest[com.inductiveautomation.xopc.drivers.allenbradley.requests.ABSLCReadRequest@d49245] request with key “649” failed due to timeout.
WARN 11:39:40 AM ControlLogixDriver[BMS Cell 25] [BMS Cell 25] Browse unsuccessfully.
WARN 11:39:40 AM ABControlLogixBrowseRequest [BMS Cell 25] Browse request failed because fail type is other than TIMEOUT, type = DISCONNECTED.
(I) 11:39:40 AM ABControlLogixBrowseRequest [BMS Cell 25] Browse request failed because fail type is other than TIMEOUT, type = DISCONNECTED.
WARN 11:39:40 AM ABSLCReadRequest Read Request timeout. First item: N20:6, Timeout Count: 1, Transaction: 259
WARN 11:39:40 AM TimeoutDaemon ScheduledRequest[com.inductiveautomation.xopc.drivers.allenbradley.requests.ABSLCReadRequest@c1265] request with key “259” failed due to timeout.
WARN 11:39:40 AM ABSLCReadRequest Read Request timeout. First item: N7:1, Timeout Count: 1, Transaction: 3851
WARN 11:39:40 AM TimeoutDaemon ScheduledRequest[com.inductiveautomation.xopc.drivers.allenbradley.requests.ABSLCReadRequest@19be348] request with key “3851” failed due to timeout.
WARN 11:39:40 AM ControlLogixDriver[BMS Cell 25] [BMS Cell 25] Browse unsuccessfully.
WARN 11:39:40 AM ABControlLogixBrowseRequest [BMS Cell 25] Browse aborted due to 3 consecutive timeouts.
WARN 11:39:40 AM ABControlLogixBrowseRequest [BMS Cell 25] Browse aborted due to 3 consecutive timeouts.
WARN 11:39:40 AM ABControlLogixTransportPool [BMS Cell 25] Reseting CIP connection for transaction ID of 2048
WARN 11:39:40 AM TimeoutDaemon [ABControlLogixBrowseRequest] request with key “2048” failed due to timeout.
WARN 11:39:40 AM ABSLCReadRequest Read Request timeout. First item: N10:51, Timeout Count: 1, Transaction: 746
WARN 11:39:40 AM ABSLCReadRequest Read Request timeout. First item: N7:1, Timeout Count: 1, Transaction: 4460
WARN 11:39:40 AM TimeoutDaemon ScheduledRequest[com.inductiveautomation.xopc.drivers.allenbradley.requests.ABSLCReadRequest@12f58] request with key “4460” failed due to timeout.
WARN 11:39:40 AM TimeoutDaemon ScheduledRequest[com.inductiveautomation.xopc.drivers.allenbradley.requests.ABSLCReadRequest@8a2566] request with key “746” failed due to timeout.
WARN 11:39:40 AM ClockDriftDetector Clock drift, degraded performance, or pause-the-world detected. lastTime=Sat Aug 16 11:39:34 EDT 2014 currentTime=Sat Aug 16 11:39:40 EDT 2014 deviation from expected 1000ms delta=4727ms.
WARN 11:39:34 AM ClockDriftDetector Clock drift, degraded performance, or pause-the-world detected. lastTime=Sat Aug 16 11:39:30 EDT 2014 currentTime=Sat Aug 16 11:39:34 EDT 2014 deviation from expected 1000ms delta=2834ms.
WARN 11:39:30 AM CipConnectedLayer Pending keys: Set(449)
WARN 11:39:30 AM CipConnectedLayer Received unexpected ConnectedPacket: InboundConnectedPacket(448,Swapped(SlicedByteBuf(ridx: 0, widx: 456, cap: 456/456, unwrapped: UnpooledUnsafeDirectByteBuf(ridx: 1370, widx: 1370, cap: 1370))))
WARN 11:39:30 AM CipConnectedLayer Pending keys: Set(449)
WARN 11:39:30 AM CipConnectedLayer Received unexpected ConnectedPacket: InboundConnectedPacket(447,Swapped(SlicedByteBuf(ridx: 0, widx: 327, cap: 327/327, unwrapped: UnpooledUnsafeDirectByteBuf(ridx: 1370, widx: 1370, cap: 1370))))
ERROR 11:39:30 AM CipConnectedLayer Error receiving inbound connected data.

java.lang.Exception: timed out waiting for response to sequence=447
more
ERROR 11:39:30 AM CipConnectedLay

2

If you’re pointing both versions of the driver at the same PLC(s) it’s likely that you’re either exhausting the number of CIP connections and/or unconnected messages that the PLC can handle at that moment, or simply overwhelming it.

3

I understand however that is not the case in this instance.

Thanks
JP

4

So if I disable my V21 logix device and then enable it the problems go away for a few days and all works fine until it breaks again. if i repeat the disable/enable i’m good for another few days.

Thoughts?

5

Try the latest release of it: files.inductiveautomation.com/be … river.modl

If you still have problems export your logs from the gateway console and either attach them here or send them to support. Also it might help to know a little about the machine you’re running on: Is it a virtual machine? OS? CPU cores and RAM quantity?

6

Tried the new module update with no improvement.

The gateway is on running in a VM5.0 environment

The bare metal has 64gb ram
VM is allocated 10GB ram
setup as 4 core cpu
Windows 2008 R2 OS

I have attached a log file after the upgrade.
logs (3).bin.gz (737 KB)

7

How much ram is Ignition set up to use? How much is being used (check the memory tab in the gateway) when you have these devices enabled?

8

usage init used committed max
Heap Memory Usage 256 mb 515 mb 998 mb 998 mb
Non-heap Memory Usage 160 kb 89 mb 97 mb -1 bytes

9

Sorry the first was with one device enabled

here is with both enabled

usage init used committed max
Heap Memory Usage 256 mb 912 mb 998 mb 998 mb
Non-heap Memory Usage 160 kb 87 mb 97 mb -1 bytes

10

Yeah, I think you definitely need to increase the max memory available to Ignition. It looks like you’ve got the default 1gb configured.

Find ignition.conf in the installation directory and modify the wrapper.java.maxmemory setting to be 2-4gb (assuming you’re on a 64-bit install). You’ll need to restart the gateway for this to take effect.

11

When i do that the gateway will not restart and i get a NET help msg of 3547

12

The wrapper.log file should have information about why it won’t start - if you set the memory to 2048mb or 4096mb and you’re using a 32-bit Ignition then that will no doubt be the problem. Try 1536mb as the max and see if it starts.

13

Thanks Kevin! Seems to be behaving. Guess I should look at moving over to 64Bit

14

Definitely. There’s no reason to be running the 32-bit version of Ignition on the specs you’ve posted - you can’t even take advantage of them!

15

Are there any Gotchas with a 32bit to 64 bit transition? Sorry have to ask, been bit in the ass by Rockwell too many times!!!

16

Hmm. There shouldn’t be at this point. Just make a gateway backup, uninstall Ignition, install the 64-bit version, and restore your gateway backup.

If there ARE any issues, it has traditionally been restricted to something with the ignition.conf file after upgrading, like pointing to the wrong Java (which also needs to be the 64-bit version).

17

Did you ever get this issue resolved? We were working with IA V21 driver, due to needing to use Kinetix 5500 servos. We had to move to a 60 day trial of KepwareEx because the issue was too hard to resolve in a production environment.

Once we switched to Kepware the CPU usage dropped to almost nothing. With the old drive after 8 hr the CPU and memory would be maxed out and the UA sever would become unresponsive. The client user would just see a red overlay.

IA is still looking at this. It sucked up a lot of time. I got conflicting advice from tech support. I would suggest trying the Kepware driver if this is unresolved. If you call them they will give you a 60 day trail. Changing computers and getting faster CPU didn’t improve the situation but took a lot of time and resources. Some support technicians said to use the Kepware diver and forget about using IA driver without hesitation. That was surprising especially because it was fully released at the time.

We also looked at the possibility of some of the tags creating issues. This didn’t fix the issue ether. It seems that their is something specific to our install, but using the Kepware driver totally resolved it.

I am working with IA to get this resolved. We have several clients that we would like to run the driver on. This would really push up the price to have to buy Kepware for each.

18

Yes we did.

  1. upgraded to 64Bit OS on gateway with 4GB provisioned in Ignition config file.
  2. used the latest RC3 V21 driver.

I have 2 V21 L73S guardlogix processors and about 10 pre V21 CLX processors and SLC’s working fine.

JP

19

Looks like I spoke too soon. Driver caused a gateway crash again. I have removed the drive and installed Kepware Server. Gateway memory back down to 500MB and holding solid.

20

I found and fixed a memory leak in the Logix v21 driver this week. It, along with a handful of other fixes and improvements to memory usage and performance, will be included in the upcoming 7.7.2-rc1 release. You’ll be able to download just the module and install it; you won’t need to upgrade all of Ignition.