Trying to get better Graph performance

I am having an issue with loading data for some XY graphs. I have Ignition Cloud on an AWS t3.medium. It's connected to a Postgres db.t4g.micro. On the page that is having issues there are 11 XY graphs. My gateway is timing out after 10 seconds (I increased it troubleshooting this issue) when trying to get Tag History. I'm the only one using the page because it's still in development. This is a demo project. I haven't implemented the Realtime toggle and considering deleting it after the page is having trouble loading 11 graphs.

Page

As this is a demo, the data doesn't come from a PLC. I created a .NET app that runs on a laptop that connects to a local Ignition Gateway and changes tags at a rate of 500ms via a separate port on the gateway from the internet. This Gateway is connected to the AWS server and uses the AWS historian for these Demo tags. The app is designed so it only changes relevant tags to the simulation. I have changed the simulation rate to 1 sec and the store and forward rate is between 2 -4 datapoints/sec.

Store & Forward from Local Gateway

The data bindings on the graphs is using Tag History Historical with starting date based on a datetimeinput and end date on another datetimeinput. They tag data path comes from a custom param.

Custom Param

The Postgres database doesn't seem to be overwhelmed, but this is the first time I've ever investigated a database for performance issues.

Posgres Metrics

Ideally I could perform the queries on the graphs as they become visible to reduce load, but I haven't found anything on pagination or loading as you scroll. What can I do to reduce the load of these graphs?

Error on Gateway

Can you show the status of the database connections when you are running this?

Gateway Web page-status-databases-History DB-Details

What you are showing should not be timing out with any standard history query.

Here's DB Connections page on the Cloud:

Cloud DB Connections

DB Details while a query ongoing

I hope this is what you are looking for.

Edit:

Metric Dashboard on DB queries

I'd suggest minimizing the load first to see what that looks like and then work from there adding things back in. For example drop the number of points in the query on each graph to like 10 it to historical mode set to a 1 minute window. See if that loads to determine if it's just a loading issue or if its something else.

I'd also monitor your network traffic through task manager and see if you're seeing a spike. Once I ran into an issue where one chart was pulling 10-20mbps of load just because it had a lot of data points and was accidently set to real time mode.

Here is some screenshots for network traffic.

Local Gateway (2 ports)

Cloud Gateway during query

The max tx speed seems a little high. I tried it a couple times after this high spike and I couldn't get it to tx that fast again.

Follow up test

Edit 2: after some further testing I believe the high tx speed correlates to when a new session is started.

Changed all graphs to only 10 points.

Page after changing all graphs to 10 points

2nd error

Edit: could my custom param paths not be built and the graphs are sending out a Tag History request for null? Seems odd that it would do this and that a null request would cause a server error.

Could it possibly be the tags on the local gateway?

Example Local Tag

I think I found my problem.

Problem
def waitForFracWell(timeout=2.0, interval=0.05):
    start = time.time()
    while time.time() - start < timeout:
        wellNum = getFracWell(12)
        if wellNum != 0:
            return wellNum
        time.sleep(interval)
    return 0

def getFracWell(numWells):
	#print("numWells passed in: ", numWells)
	base = "[IPC Demo]Pad/Well"
	placeFrac = "/HMI/Modes/Frac"
	removeFrac = "/HMI/Modes/Remove Frac"
	balldrop = "/HMI/Modes/Balldrop Frac"
	
	paths = []
	
	for x in range(numWells):
		i = x + 1
		paths.append(base + str(i) + placeFrac)
		paths.append(base + str(i) + removeFrac)
		paths.append(base + str(i) + balldrop)
	
	results = system.tag.readBlocking(paths)
	
	for x in range(numWells):
		placeVal   = results[x*3].value
		removeVal  = results[x*3 + 1].value
		balldropVal= results[x*3 + 2].value
		
		if placeVal or removeVal or balldropVal:
			return x + 1
	
	return 0

I'm wondering if these readBlocking are suspending processing for too long.

I tried to do the following, but tag.readAsync doesn't allow passing in parameters

Doesn't work
def onTagChange(initialChange, newValue, previousValue, event, executionCount):
	if (
	not initialChange 
	and newValue is not None 
	and newValue.value != previousValue.value
	#and executionCount % 2 == 0 #filters out odd repeating onTagChange
	):
		value = newValue.value
		tagPathStr = str(event.tagPath)
		tagName = tagPathStr.split("/")[-1]
		#print("New Value: ", newValue.value)
		#print("tagPath: ", tagPathStr)
		#test = system.tag.readBlocking(["[.]Zipper Entries/Frac Pin"], 1000)
		if isinstance(value, bool) and tagName.endswith("Pin Ok"):
			general.PinChange(tagName)

# in Project Library "general"

def PinChange(tagName):
	getFracWell(12, tagName)

def getFracWell(numWells, tagName):
	base = "[IPC Demo]Pad/Well"
	placeFrac = "/HMI/Modes/Frac"
	removeFrac = "/HMI/Modes/Remove Frac"
	balldrop = "/HMI/Modes/Balldrop Frac"
	
	paths = []
	
	for x in range(numWells):
		i = x + 1
		paths.append(base + str(i) + placeFrac)
		paths.append(base + str(i) + removeFrac)
		paths.append(base + str(i) + balldrop)
	
	system.tag.readAsync(paths, _callback(tagName, numWells))

def _callback(results, tagName, numWells):
	for x in range(numWells):
		placeVal   = results[x*3].value
		removeVal  = results[x*3 + 1].value
		balldropVal= results[x*3 + 2].value
		
		if placeVal or removeVal or balldropVal:
			_pinChange(x + 1, tagName)

def _pinChange(wellNum, tagName):
	if wellNum and wellNum != 0:
		wellNameTag = "[IPC Demo]Pad/Well" + str(wellNum) + "/Last Known Name"
		wellName = system.tag.readBlocking([wellNameTag])[0].value
		action = ""
		pin = 0
		FracName = ""
		VtName = ""
		WlName = ""
		WsmName = ""
		
		prefix = tagName.replace(" Pin Ok", "")

		if (prefix == "Frac"):
			action = "Frac pin entered"
			pin = system.tag.readBlocking(["[IPC Demo]Zipper Pins/Frac Pin"])[0].value
			if pin != 0:
				FracName = findUserName(pin, "FRAC")	
			#print(pin)
		elif (prefix == "VT"):
			action = "Valvetech pin entered"
			pin = system.tag.readBlocking(["[IPC Demo]Zipper Pins/VT Pin"])[0].value
			if pin != 0:
				VtName = findUserName(pin, "VT")		
		elif (prefix == "Wline"):
			action = "Wireline pin entered"
			pin = system.tag.readBlocking(["[IPC Demo]Zipper Pins/Wline Pin"])[0].value
			if pin != 0:
				WlName = findUserName(pin, "WIRELINE")
		elif (prefix == "WSM"):
			action = "WSM pin entered"
			pin = system.tag.readBlocking(["[IPC Demo]Zipper Pins/WSM Pin"])[0].value

			if pin != 0:
				WsmName = findUserName(pin, "WSM")
		else:
			print("Zipper Pin tag change didn't find any Pin Ok")
		
		if pin != 0:
			namedQuery = "Insert User Action"
			queryParams = {"wellNum": wellNum, "wellName": wellName, "action": action, "pin": pin, "frac": FracName, "wsm": WsmName, "wl": WlName, "vt": VtName, "wellPressure": "", "fracPressure": ""}
			system.db.execUpdate(namedQuery, queryParams)

You can use lambda to make a closure that captures args from the calling scope.
As in:

system.tag.readAsync(paths, lambda results: _callback(results, tagName, numWells))

So I ended up finding my issue. Rookie MQTT mistake.

I found by using a direct tag bind to a tag that for the mqtt historian tag I had to use a different tag than the one listed in the mqtt engine. What lead me down this path was reading the mqtt docs and there were two different ways to setup the edge historian for tag data. One of them pointed out that the edge historian bypassed the tag change historian. This got me wondering if this resulted in an odd historian tag. Sure enough this was the case. I added a new Param to my view for this different path and passed it to all of my Graphs.