Read/Write timing out response times

Hi all,

I have just purchased Ignition 8.1.43. I have a mix of Allen Bradley and Siemens PLCs Im reading and writing from. Ive frequently been getting long response times that exceed a 3 seconds heartbeat I created on bot h PLC. The path being Siemens PLC ->Ignition -> AB PLC and also the other way around. I've dug deep into this network wise but I dont know much of networking as I'm a simple a controls guy.
For the AB PLC I have a 1756-L83E ControlLogix 5580 Controller v30.011
For the Siemens PLC I have a 1214C DC/DC/DC v4.3 firmware

I have a heartbeat signal generated using the internal clock memory of 0.5hertz from the Siemens PLC side that's read along with 4 other bits. 5 bits in total going out and its the same for going in. Attached below is my script code writing on the gateway. The settings are Fixed delay at 200 seconds on Shared Threads.

The group settings of the tags are "Suscribed" with a rate of 250ms. The rest are default settings

*(More hardware info: currently 1 AB PLC and 6 Siemen PLC. I have 8 more to add and about 50 more siemens PLCs. Its something I conviced the plant to slowly let me create a central control system. Not all will have to communicate between each other)


#Stand Signal Exchange With Caljans. Hilmot To Caljan

#Callling Libraries  for "Try" and "Except" and logging functions
import traceback	
import sys


try:
	
	# Pulling the Device list created on the Gateway Webpage. These devices are set in a multi-dim array with 
	#Slot 0, the first device, its the Hilmot_1_3 gateway and then Slot 2 is the second device, OB_Caljan1"
	connectedlist = system.device.listDevices()
	
	#Hard coding the word "Connected" So we can compare device status later on
	sConnect= "Connected"
	
	#The "getValueat(X,X)" is now picking a value from the multi-dim array created with the "connectedList" the devices are on the first digit, 'X' as in (X,2)
	#The second digit, Y, as in "(1,Y)" contains the device status
	#So this is passing the connection status of the caljans to variable to compare later
	OBcaljan1_status = connectedlist.getValueAt(1,2)
	OBcaljan2_status = connectedlist.getValueAt(1,2)
	OBcaljan3_status = connectedlist.getValueAt(1,2)
	OBcaljan4_status = connectedlist.getValueAt(1,2)
	OBcaljan5_status = connectedlist.getValueAt(1,2)
	OBcaljan6_status = connectedlist.getValueAt(1,2)	
	# Setting paths that refrence the value to pass
	#values to be read from
	Hout_C1_Estop          = ("[default]Hilmot_13/Caljan1/write/Estop")
	Hout_C1_ExtenInPOS     = ("[default]Hilmot_13/Caljan1/write/ExtenInPOS")
	Hout_C1_Heartbeat	   = ("[default]Hilmot_13/Caljan1/write/Heartbeat")
	Hout_C1_OverMidPECstop = ("[default]Hilmot_13/Caljan1/write/OverMidPECstop")
	Hout_C1_RunP		   = ("[default]Hilmot_13/Caljan1/write/RunP")		
	Cout_1_BeltRunning 	   = ('[default]Caljan1/Write/BeltRunning')
	Cout_1_Estop           = ('[default]Caljan1/Write/Estop')
	Cout_1_Healthy         = ('[default]Caljan1/Write/Healthy')
	Cout_1_Heartbeat       = ('[default]Caljan1/Write/Heartbeat')
	Cout_1_RdyToReceive    = ('[default]Caljan1/Write/RdyToReceive')
	
	Hout_C2_Estop          = ("[default]Hilmot_13/Caljan2/write/Estop")
	Hout_C2_ExtenInPOS     = ("[default]Hilmot_13/Caljan2/write/ExtenInPOS")
	Hout_C2_Heartbeat	   = ("[default]Hilmot_13/Caljan2/write/Heartbeat")
	Hout_C2_OverMidPECstop = ("[default]Hilmot_13/Caljan2/write/OverMidPECstop")
	Hout_C2_RunP		   = ("[default]Hilmot_13/Caljan2/write/RunP")		
	Cout_2_BeltRunning 	   = ('[default]Caljan2/Write/BeltRunning')
	Cout_2_Estop           = ('[default]Caljan2/Write/Estop')
	Cout_2_Healthy         = ('[default]Caljan2/Write/Healthy')
	Cout_2_Heartbeat       = ('[default]Caljan2/Write/Heartbeat')
	Cout_2_RdyToReceive    = ('[default]Caljan2/Write/RdyToReceive')		
		
	Hout_C3_Estop          = ("[default]Hilmot_13/Caljan3/write/Estop")
	Hout_C3_ExtenInPOS     = ("[default]Hilmot_13/Caljan3/write/ExtenInPOS")
	Hout_C3_Heartbeat	   = ("[default]Hilmot_13/Caljan3/write/Heartbeat")
	Hout_C3_OverMidPECstop = ("[default]Hilmot_13/Caljan3/write/OverMidPECstop")
	Hout_C3_RunP		   = ("[default]Hilmot_13/Caljan3/write/RunP")		
	Cout_3_BeltRunning 	   = ('[default]Caljan3/Write/BeltRunning')
	Cout_3_Estop           = ('[default]Caljan3/Write/Estop')
	Cout_3_Healthy         = ('[default]Caljan3/Write/Healthy')
	Cout_3_Heartbeat       = ('[default]Caljan3/Write/Heartbeat')
	Cout_3_RdyToReceive    = ('[default]Caljan3/Write/RdyToReceive')		
	
	Hout_C4_Estop          = ("[default]Hilmot_13/Caljan4/write/Estop")
	Hout_C4_ExtenInPOS     = ("[default]Hilmot_13/Caljan4/write/ExtenInPOS")
	Hout_C4_Heartbeat	   = ("[default]Hilmot_13/Caljan4/write/Heartbeat")
	Hout_C4_OverMidPECstop = ("[default]Hilmot_13/Caljan4/write/OverMidPECstop")
	Hout_C4_RunP		   = ("[default]Hilmot_13/Caljan4/write/RunP")		
	Cout_4_BeltRunning 	   = ('[default]Caljan4/Write/BeltRunning')
	Cout_4_Estop           = ('[default]Caljan4/Write/Estop')
	Cout_4_Healthy         = ('[default]Caljan4/Write/Healthy')
	Cout_4_Heartbeat       = ('[default]Caljan4/Write/Heartbeat')
	Cout_4_RdyToReceive    = ('[default]Caljan4/Write/RdyToReceive')	
	
	Hout_C5_Estop          = ("[default]Hilmot_13/Caljan5/write/Estop")
	Hout_C5_ExtenInPOS     = ("[default]Hilmot_13/Caljan5/write/ExtenInPOS")
	Hout_C5_Heartbeat	   = ("[default]Hilmot_13/Caljan5/write/Heartbeat")
	Hout_C5_OverMidPECstop = ("[default]Hilmot_13/Caljan5/write/OverMidPECstop")
	Hout_C5_RunP		   = ("[default]Hilmot_13/Caljan5/write/RunP")		
	Cout_5_BeltRunning 	  = ('[default]Caljan5/Write/BeltRunning')
	Cout_5_Estop          = ('[default]Caljan5/Write/Estop')
	Cout_5_Healthy        = ('[default]Caljan5/Write/Healthy')
	Cout_5_Heartbeat      = ('[default]Caljan5/Write/Heartbeat')
	Cout_5_RdyToReceive   = ('[default]Caljan5/Write/RdyToReceive')	
	
	Hout_C6_Estop          = ("[default]Hilmot_13/Caljan6/write/Estop")
	Hout_C6_ExtenInPOS     = ("[default]Hilmot_13/Caljan6/write/ExtenInPOS")
	Hout_C6_Heartbeat	   = ("[default]Hilmot_13/Caljan6/write/Heartbeat")
	Hout_C6_OverMidPECstop = ("[default]Hilmot_13/Caljan6/write/OverMidPECstop")
	Hout_C6_RunP		   = ("[default]Hilmot_13/Caljan6/write/RunP")		
	Cout_6_BeltRunning 	  = ('[default]Caljan6/Write/BeltRunning')
	Cout_6_Estop          = ('[default]Caljan6/Write/Estop')
	Cout_6_Healthy        = ('[default]Caljan6/Write/Healthy')
	Cout_6_Heartbeat      = ('[default]Caljan6/Write/Heartbeat')
	Cout_6_RdyToReceive   = ('[default]Caljan6/Write/RdyToReceive')	
	#Values to be writtent o
	Cin_1_Estop 		   = ('[default]Caljan1/Read/Estop')
	Cin_1_ExtendedInPos    = ('[default]Caljan1/Read/ExtendedInPos')
	Cin_1_Heartbeat        = ('[default]Caljan1/Read/Heartbeat')
	Cin_1_MidPecOverride   = ('[default]Caljan1/Read/MidPecOverride')
	Cin_1_RunP 	           = ('[default]Caljan1/Read/RunP')
	Hin_C1_BeltRunning     = ("[default]Hilmot_13/Caljan1/read/BeltRunning")
	Hin_C1_Estop     	   = ("[default]Hilmot_13/Caljan1/read/Estop")
	Hin_C1_Healthy	       = ("[default]Hilmot_13/Caljan1/read/Healthy")
	Hin_C1_HeartBeat       = ("[default]Hilmot_13/Caljan1/read/HeartBeat")
	Hin_C1_RdyToRec		   = ("[default]Hilmot_13/Caljan1/read/RdyToRec")	

	Cin_2_Estop 		   = ('[default]Caljan2/Read/Estop')
	Cin_2_ExtendedInPos    = ('[default]Caljan2/Read/ExtendedInPos')
	Cin_2_Heartbeat        = ('[default]Caljan2/Read/Heartbeat')
	Cin_2_MidPecOverride   = ('[default]Caljan2/Read/MidPecOverride')
	Cin_2_RunP 	           = ('[default]Caljan2/Read/RunP')
	Hin_C2_BeltRunning     = ("[default]Hilmot_13/Caljan2/read/BeltRunning")
	Hin_C2_Estop     	   = ("[default]Hilmot_13/Caljan2/read/Estop")
	Hin_C2_Healthy	       = ("[default]Hilmot_13/Caljan2/read/Healthy")
	Hin_C2_HeartBeat       = ("[default]Hilmot_13/Caljan2/read/HeartBeat")
	Hin_C2_RdyToRec		   = ("[default]Hilmot_13/Caljan2/read/RdyToRec")		
	
	Cin_3_Estop 		   = ('[default]Caljan3/Read/Estop')
	Cin_3_ExtendedInPos    = ('[default]Caljan3/Read/ExtendedInPos')
	Cin_3_Heartbeat        = ('[default]Caljan3/Read/Heartbeat')
	Cin_3_MidPecOverride   = ('[default]Caljan3/Read/MidPecOverride')
	Cin_3_RunP 	           = ('[default]Caljan3/Read/RunP')
	Hin_C3_BeltRunning     = ("[default]Hilmot_13/Caljan3/read/BeltRunning")
	Hin_C3_Estop     	   = ("[default]Hilmot_13/Caljan3/read/Estop")
	Hin_C3_Healthy	       = ("[default]Hilmot_13/Caljan3/read/Healthy")
	Hin_C3_HeartBeat       = ("[default]Hilmot_13/Caljan3/read/HeartBeat")
	Hin_C3_RdyToRec		   = ("[default]Hilmot_13/Caljan3/read/RdyToRec")	
	
	Cin_4_Estop 		   = ('[default]Caljan4/Read/Estop')
	Cin_4_ExtendedInPos    = ('[default]Caljan4/Read/ExtendedInPos')
	Cin_4_Heartbeat        = ('[default]Caljan4/Read/Heartbeat')
	Cin_4_MidPecOverride   = ('[default]Caljan4/Read/MidPecOverride')
	Cin_4_RunP 	           = ('[default]Caljan4/Read/RunP')
	Hin_C4_BeltRunning     = ("[default]Hilmot_13/Caljan4/read/BeltRunning")
	Hin_C4_Estop     	   = ("[default]Hilmot_13/Caljan4/read/Estop")
	Hin_C4_Healthy	       = ("[default]Hilmot_13/Caljan4/read/Healthy")
	Hin_C4_HeartBeat       = ("[default]Hilmot_13/Caljan4/read/HeartBeat")
	Hin_C4_RdyToRec		   = ("[default]Hilmot_13/Caljan4/read/RdyToRec")	
	
	Cin_5_Estop 		   = ('[default]Caljan5/Read/Estop')
	Cin_5_ExtendedInPos    = ('[default]Caljan5/Read/ExtendedInPos')
	Cin_5_Heartbeat        = ('[default]Caljan5/Read/Heartbeat')
	Cin_5_MidPecOverride   = ('[default]Caljan5/Read/MidPecOverride')
	Cin_5_RunP 	           = ('[default]Caljan5/Read/RunP')
	Hin_C5_BeltRunning     = ("[default]Hilmot_13/Caljan5/read/BeltRunning")
	Hin_C5_Estop     	   = ("[default]Hilmot_13/Caljan5/read/Estop")
	Hin_C5_Healthy	       = ("[default]Hilmot_13/Caljan5/read/Healthy")
	Hin_C5_HeartBeat       = ("[default]Hilmot_13/Caljan5/read/HeartBeat")
	Hin_C5_RdyToRec		   = ("[default]Hilmot_13/Caljan5/read/RdyToRec")	
	
	Cin_6_Estop 		   = ('[default]Caljan6/Read/Estop')
	Cin_6_ExtendedInPos    = ('[default]Caljan6/Read/ExtendedInPos')
	Cin_6_Heartbeat        = ('[default]Caljan6/Read/Heartbeat')
	Cin_6_MidPecOverride   = ('[default]Caljan6/Read/MidPecOverride')
	Cin_6_RunP 	           = ('[default]Caljan6/Read/RunP')
	Hin_C6_BeltRunning     = ("[default]Hilmot_13/Caljan6/read/BeltRunning")
	Hin_C6_Estop     	   = ("[default]Hilmot_13/Caljan6/read/Estop")
	Hin_C6_Healthy	       = ("[default]Hilmot_13/Caljan6/read/Healthy")
	Hin_C6_HeartBeat       = ("[default]Hilmot_13/Caljan6/read/HeartBeat")
	Hin_C6_RdyToRec		   = ("[default]Hilmot_13/Caljan6/read/RdyToRec")	
			
	
	
	
	
	#Initializing Arrays to read in and copy out data
	out_StandardSignals   = []
	in_StandardSignals	  = []
	
	
	if OBcaljan1_status == sConnect:
		#
		out_StandardSignals.append(Hout_C1_Estop) 
		out_StandardSignals.append(Hout_C1_ExtenInPOS)
		out_StandardSignals.append(Hout_C1_Heartbeat)
		out_StandardSignals.append(Hout_C1_OverMidPECstop)
		out_StandardSignals.append(Hout_C1_RunP)
		out_StandardSignals.append(Cout_1_BeltRunning)
		out_StandardSignals.append(Cout_1_Estop)
		out_StandardSignals.append(Cout_1_Healthy)
		out_StandardSignals.append(Cout_1_Heartbeat)
		out_StandardSignals.append(Cout_1_RdyToReceive)		
		in_StandardSignals.append (Cin_1_Estop)
		in_StandardSignals.append (Cin_1_ExtendedInPos)
		in_StandardSignals.append (Cin_1_Heartbeat)
		in_StandardSignals.append (Cin_1_MidPecOverride)
		in_StandardSignals.append (Cin_1_RunP)			
		in_StandardSignals.append (Hin_C1_BeltRunning)
		in_StandardSignals.append (Hin_C1_Estop)
		in_StandardSignals.append (Hin_C1_Healthy)
		in_StandardSignals.append (Hin_C1_HeartBeat)
		in_StandardSignals.append (Hin_C1_RdyToRec)


	if OBcaljan2_status == sConnect:
		#
		out_StandardSignals.append(Hout_C2_Estop) 
		out_StandardSignals.append(Hout_C2_ExtenInPOS)
		out_StandardSignals.append(Hout_C2_Heartbeat)
		out_StandardSignals.append(Hout_C2_OverMidPECstop)
		out_StandardSignals.append(Hout_C2_RunP)
		out_StandardSignals.append(Cout_2_BeltRunning)
		out_StandardSignals.append(Cout_2_Estop)
		out_StandardSignals.append(Cout_2_Healthy)
		out_StandardSignals.append(Cout_2_Heartbeat)
		out_StandardSignals.append(Cout_2_RdyToReceive)		
		in_StandardSignals.append (Cin_2_Estop)
		in_StandardSignals.append (Cin_2_ExtendedInPos)
		in_StandardSignals.append (Cin_2_Heartbeat)
		in_StandardSignals.append (Cin_2_MidPecOverride)
		in_StandardSignals.append (Cin_2_RunP)			
		in_StandardSignals.append (Hin_C2_BeltRunning)
		in_StandardSignals.append (Hin_C2_Estop)
		in_StandardSignals.append (Hin_C2_Healthy)
		in_StandardSignals.append (Hin_C2_HeartBeat)
		in_StandardSignals.append (Hin_C2_RdyToRec)

	if OBcaljan3_status == sConnect:
		#
		out_StandardSignals.append(Hout_C3_Estop) 
		out_StandardSignals.append(Hout_C3_ExtenInPOS)
		out_StandardSignals.append(Hout_C3_Heartbeat)
		out_StandardSignals.append(Hout_C3_OverMidPECstop)
		out_StandardSignals.append(Hout_C3_RunP)
		out_StandardSignals.append(Cout_3_BeltRunning)
		out_StandardSignals.append(Cout_3_Estop)
		out_StandardSignals.append(Cout_3_Healthy)
		out_StandardSignals.append(Cout_3_Heartbeat)
		out_StandardSignals.append(Cout_3_RdyToReceive)		
		in_StandardSignals.append (Cin_3_Estop)
		in_StandardSignals.append (Cin_3_ExtendedInPos)
		in_StandardSignals.append (Cin_3_Heartbeat)
		in_StandardSignals.append (Cin_3_MidPecOverride)
		in_StandardSignals.append (Cin_3_RunP)			
		in_StandardSignals.append (Hin_C3_BeltRunning)
		in_StandardSignals.append (Hin_C3_Estop)
		in_StandardSignals.append (Hin_C3_Healthy)
		in_StandardSignals.append (Hin_C3_HeartBeat)
		in_StandardSignals.append (Hin_C3_RdyToRec)


	if OBcaljan4_status == sConnect:
		#
		out_StandardSignals.append(Hout_C4_Estop) 
		out_StandardSignals.append(Hout_C4_ExtenInPOS)
		out_StandardSignals.append(Hout_C4_Heartbeat)
		out_StandardSignals.append(Hout_C4_OverMidPECstop)
		out_StandardSignals.append(Hout_C4_RunP)
		out_StandardSignals.append(Cout_4_BeltRunning)
		out_StandardSignals.append(Cout_4_Estop)
		out_StandardSignals.append(Cout_4_Healthy)
		out_StandardSignals.append(Cout_4_Heartbeat)
		out_StandardSignals.append(Cout_4_RdyToReceive)		
		in_StandardSignals.append (Cin_4_Estop)
		in_StandardSignals.append (Cin_4_ExtendedInPos)
		in_StandardSignals.append (Cin_4_Heartbeat)
		in_StandardSignals.append (Cin_4_MidPecOverride)
		in_StandardSignals.append (Cin_4_RunP)			
		in_StandardSignals.append (Hin_C4_BeltRunning)
		in_StandardSignals.append (Hin_C4_Estop)
		in_StandardSignals.append (Hin_C4_Healthy)
		in_StandardSignals.append (Hin_C4_HeartBeat)
		in_StandardSignals.append (Hin_C4_RdyToRec)

	if OBcaljan5_status == sConnect:
		#
		out_StandardSignals.append(Hout_C5_Estop) 
		out_StandardSignals.append(Hout_C5_ExtenInPOS)
		out_StandardSignals.append(Hout_C5_Heartbeat)
		out_StandardSignals.append(Hout_C5_OverMidPECstop)
		out_StandardSignals.append(Hout_C5_RunP)
		out_StandardSignals.append(Cout_5_BeltRunning)
		out_StandardSignals.append(Cout_5_Estop)
		out_StandardSignals.append(Cout_5_Healthy)
		out_StandardSignals.append(Cout_5_Heartbeat)
		out_StandardSignals.append(Cout_5_RdyToReceive)
		in_StandardSignals.append (Cin_5_Estop)
		in_StandardSignals.append (Cin_5_ExtendedInPos)
		in_StandardSignals.append (Cin_5_Heartbeat)
		in_StandardSignals.append (Cin_5_MidPecOverride)
		in_StandardSignals.append (Cin_5_RunP)			
		in_StandardSignals.append (Hin_C5_BeltRunning)
		in_StandardSignals.append (Hin_C5_Estop)
		in_StandardSignals.append (Hin_C5_Healthy)
		in_StandardSignals.append (Hin_C5_HeartBeat)
		in_StandardSignals.append (Hin_C5_RdyToRec)

	if OBcaljan6_status == sConnect:
		#
		out_StandardSignals.append(Hout_C6_Estop) 
		out_StandardSignals.append(Hout_C6_ExtenInPOS)
		out_StandardSignals.append(Hout_C6_Heartbeat)
		out_StandardSignals.append(Hout_C6_OverMidPECstop)
		out_StandardSignals.append(Hout_C6_RunP)
		out_StandardSignals.append(Cout_6_BeltRunning)
		out_StandardSignals.append(Cout_6_Estop)
		out_StandardSignals.append(Cout_6_Healthy)
		out_StandardSignals.append(Cout_6_Heartbeat)
		out_StandardSignals.append(Cout_6_RdyToReceive)		
		in_StandardSignals.append (Cin_6_Estop)
		in_StandardSignals.append (Cin_6_ExtendedInPos)
		in_StandardSignals.append (Cin_6_Heartbeat)
		in_StandardSignals.append (Cin_6_MidPecOverride)
		in_StandardSignals.append (Cin_6_RunP)			
		in_StandardSignals.append (Hin_C6_BeltRunning)
		in_StandardSignals.append (Hin_C6_Estop)
		in_StandardSignals.append (Hin_C6_Healthy)
		in_StandardSignals.append (Hin_C6_HeartBeat)
		in_StandardSignals.append (Hin_C6_RdyToRec)
		
		
	#Hilmot to Caljan Signals	
	out_StandardSignals_ReadIn = system.tag.readBlocking(out_StandardSignals,350)
	in_done = system.tag.writeBlocking(in_StandardSignals,out_StandardSignals_ReadIn,350)
	

except Exception, err:
	exc_type, exc_value, exc_tb = sys.exc_info()
	errtb = traceback.format_exception(exc_type, exc_value, exc_tb)
	errStr = str('Text describing script: '+str(err)+errtb[1] + errtb[2])
	logger = system.util.getLogger("OB_Cal1-6")
	logger.info(errStr)

I tried using wireshark but it looks good to me(???)
I had a heartbeat timeout generate a fault at 23:41:00.840 and this is what the wireshark shows me(pic below)

Ignition is hosted on a VM server, ( I was promised a physical computer dedicated to this but this is what I got :[ )

When I look at the ignition logger I see I have one machine that keeps losing its connection

In the connection devices configuration I have set the timeouts to the devices to 3500ms and I generate the PLC faults at 3000ms.
For the AB PLC I changed the the Max Concurrent request to 40.( I know the PLC can handle a total of 63 connections and I need a maximum of 16 devices to talk to.) and I turned off the Automatic Rebrowse Option.

Here's a pic of the Ignition dashboard status.

The devices are all at 1% overload use

What can I do to optimize these response times?
Thank you for yall's time

Here is the server information:

Sorry I thought I had this included

Also to expand on the heartbeat fault. The fault was reported at 23:41:03.840.
So I look at 23:41:00.840 and back since its my last received messaged from the Siemens PLC to the AB PLC

To expand on the network architecture. We go from our Siemens PLC on a unmanaged switched to the Host company's building server-switches, and then routed back to our machine network, back to a manged switch and then to the AB PLC. The host company swears that the ports assigned to us are totally wide open for our automation network.