Transforming large datasets eating too much memory

I am working on moving some projects to perspective from vision. One of the features I used on vision table was extension functions to highlight items.

On perspective I am able to get it to work by sending the data to a tag first then binding it to the table data and transforming it. The main issue is that it I am having to change the data to an array and then add the style. Doing this takes time(loading) and a lot of resources and I haven't been able to really figure it out besides limiting my results.

Any help would be appreciated. I use a dropdown menu to change the data in the table between API and SQL queries.

That seems inefficient at best. Unless you ment property and not tag.

Can you show the script for how you're doing this transformation?

If you can, @pturmel SimAids module is highly recommended (by me anyway) for this type of thing.

I'm also interested in exactly what you mean by this. Is it because you're highlighting per cell?

Perspective moves most of the UI workload to the gateway. Are you sure you don't just need to add memory? (However much memory you were giving your Vision clients, you probably need to add that much per client to the gateway for Perspective. Or more, since JSON is very wasteful space-wise.)

2 Likes

Session custom properties are the perspective equivalent of vision’s client tags.

Do NOT use tags for anything that’s only relevant to one session, like it seems to be the case here (since the user can apparently change the tag’s data by choosing things from a drop down).
The issue is that the tag is shared by all sessions, so when a user selects something and the data is modified, it is modified for everyone.

4 Likes

Not to mention if its a Large Dataset, that value is persisted in the internal DB and that has other not great consequences.

4 Likes

Hi everyone thanks for the replies.

I am using this thread to do the transformation, I did bind it to a tag which may be my problem. I will look into binding it to a property and see if this resolves the issue. Any other tips would be appreciated as well.

Perspective has come a long way since 2019. If you provide some context on what you're trying to accomplish, (i.e what items you are needing to highlight and when). Even better would be to provide the script you are using. There are many techniques that can be used today that were just not a part of the picture 4 years ago.

I am trying to replicate the vision feature extension function getBackground/ForegroundAt in perspective. Below is the script I used in vision which took the value of the dropdown menu and applied color based on if the work order was 'past due' or, in another table, if the work order existed.

if self.parent.getComponent('EmaintDropdown').selectedValue == 1:
		if self.data.getValueAt(row, 'Past Due') == '1':
			return system.gui.color('red')
		else:
			return system.gui.color('white')
	if self.parent.getComponent('EmaintDropdown').selectedValue == 0:
		if self.data.getValueAt(row, 'workorder') > 1:
			return system.gui.color('green')
		else:
			return system.gui.color('white')

Below is from my API call that takes the date and checks to see if it is past due. All of the scripting that changes the table is in the dropdown component script, it is a mix between SQL and API calls.

			for sch_date in response.json["data"]:
					dateparse = str(sch_date["sch_date"])
					shortdate = system.date.parse(dateparse[0:10],'yyyy-MM-dd')
					if system.date.parse(dateparse[0:10],'yyyy-MM-dd') < system.date.now():
						updateWOtable = system.dataset.setValue(self.getSibling("Table").props.data, N, 'Past Due', "1")

I want to highlight entire rows based on if past due = 1 or if a work order != 0, only applying when their respective dropdown selection is applied.

From what I can tell is there is no way to transform the data in the table without first binding it to another tag/property. I am not sure how to efficiently do this.

Let's take it from the top and make it clear what it is you want to achieve.

From what I understand, you have a table that has at least a column with a date, and a column with a work order.
You want, for each row, to compare the date with 'now' to find if the task that corresponds to the row is past due, and check the value in the column "work order".
Based on that, you want to color the row red if it is past due, green if work order is not 0, and leave it white otherwise.

Am I right so far ?

Questions:

  • how is the table populated ?
  • what is the dropdown for ?
  • what's that API you're using, what do you get from it ?
  • what are you fetching from your database ?

Don't do that. If it changes the table, put in on the table. Even if it needs to be triggered by something else, you can add a component method to the table and call it from somewhere else.

Also, you need to be more specific when you say "script", there are a lot of different places where you can put scripts in perspective - transforms, component methods, property change scripts...

3 Likes

You have to change your approach. Vision lets you do all kinds of processing and decision-making and UI tweaking on the client side. Those component methods run client-side and are intimately tied to Java Swing's painting process.

Perspective needs to prepare everything on the gateway and deliver it to the browser. That preparation must include all visual adjustments you want. You don't get to decide element-by-element how to display something after it is sent to the browser.

Okay, I can appreciate this but still don't understand what the correct method is to achieve this.

Lets keep it simple and assume you have a table you fill with queried data from a database, one of the columns is the date. How would you go about changing the row color based on if we are before or after the date?

I used a method that was provided by one of the ignition devs and it seems universally disliked on this thread.

Again let's keep it simple, you can make your own assumptions based on how you would achieve this.

Add a query binding to the table's data property, make the return format json, add a transform and process each row, adding a style key to the row where needed.

I never used this but I'm pretty sure it works.

You don't fill the table with this data directly. You bring the data to a custom property. Then you reprocess the data into JSON format, adding the visual elements you desire, and that goes to props.data.

{ A transform is an option, but I don't use it, because you lose data type information that the original dataset carried. BTW, you can flag the dataset custom prop as "private" to minimize browser traffic. }

1 Like

Ok thanks for the approach I will give it a shot when I have some time and let you know how it goes.

appreciate the answers

If you look at a table's sample data, you will see that "Folsom" is highlighted. Examine the objects for that row to see the kind of nested dictionary you need to style individual cells.

Appreciate all the input I have managed to get the transform working properly now on a custom private prop.

Still unsure how to reprocess the data without transform but I am content with what I have working as is working fairly quickly.

Thank you @pturmel @pascal.fragnoud

If you need a script, you'll pretty much have to use a transform anyway. The alternatives that I'm aware of are really not any better.

If you can share a sample of your dataset, and the corresponding props.data you need, I can probably whip up an expression for you using SimAids V2. It'll run a lot more efficiently than a script.