Inquiry About Asynchronous Data Retrieval via Named Queries for Paginated Tables

Hi everyone,

I'm working on a project where I need to display data in a table in Ignition. The data is retrieved using a fairly complex SQL query, and as the dataset grows, performance has become an issue. Even with indexing, the query isn't performing well (~ 12 seconds), and I anticipate having over 50,000 records within a year.

To improve performance, I'd like to implement a paginated approach where:

  1. A separate query calculates the total number of records (to determine the number of pages).
    image
  2. Only the data for the current page (e.g., 25 records per page) is loaded when the user switches pages.

My plan is to use Named Queries to handle this asynchronously, but I'm unsure if this is the best approach or if there are limitations I need to be aware of.

Here are my questions:

  1. Is it possible to use Named Queries in Ignition for asynchronous data retrieval in a paginated table?
  2. Are there best practices for implementing such a solution efficiently?
  3. Are there alternative approaches I should consider?

Any guidance, tips, or examples would be greatly appreciated!

Thanks in advance for your help!

Not sure exactly what you mean by this, these queries are triggered, same as any other type of query in Ignition.

There is nothing stopping you having a table with exactly 25 records on it and making a dynamic pagination element that can choose what page you are on.
At the end of your query, use the following (MySQL) at the end of the query and inject the correct offset and limit based on the current page. Each time you click a different page link in the custom pagination, you trigger your limited offset query and fill the table with the results.

LIMIT number_of_rows OFFSET offset_value;

The issue here is that pagination will still cripple the database, because you are now running 1+n queries to get the data, where n is the number of pages. Limiting the output still has to run the entire query on the server.
On top of this, every single session will hit the database for this many queries, which will balloon massively. This then becomes x(1+n) where x is sessions, n is pages of data.

Better query and database design is the real answer here, database tables regularly hit 50 million records without taking over a second to query, there is no reason yours can't either.

2 Likes

Try to find a way to retrieve only the data that is frequented by your default client.

Are you able to display a subset of your large dataset (say, TOP 1000 or Last 30 Days, etc.) by default, then add input fields on the page to permit a user to query for (other subsets of) the larger dataset (via auto populated start & end date input fields, for example) as needed?

1 Like

To be clear, if you arrange to do your own queries for pagination, you have to create your own UI for the pagination. You won't be able to use the table's native paging UI.

1 Like

Sorry if my question was not clearly formulated. I need to use two conditions in my query, one of which is a date filter to narrow down the datasets. However, this date filter seems to be very time-consuming.

Currently, when the query is executed, the entire dataset is always loaded, which significantly affects performance. My hope was (and still is) that there might be a way to make the query asynchronous so that when selecting a page (e.g., page 2 with 25 records per page), only the records for that specific page are actually loaded — and not all records at once.

I know similar concepts of asynchronous queries from applications like messenger services (WhatsApp etc). There, only a certain number of messages are loaded initially. As you scroll up, additional messages (e.g., 20) are dynamically fetched. This approach is far more efficient than loading the entire conversation history for, let's say a year, all at once.

My hope is to implement a similar logic for tables in Ignition...

To better illustrate this, here is an example of a query structure I got in mind to achieve this...

Calculating the total number of pages:

First, I want to calculate the total number of records based on the filter conditions and determine the number of pages:

-- Calculate the total number of records based on the filter conditions
DECLARE @TotalRecords INT;
SELECT 
    @TotalRecords = COUNT(*)
FROM [DatabaseXY].[dbo].[Orders]
WHERE 
    (:Status IS NULL OR :Status = '' OR [Status] = :Status)   -- Filter by status
    AND (
        (:OrderStartDate IS NULL OR :OrderStartDate = '' OR :OrderEndDate IS NULL OR :OrderEndDate = '')  
        OR [OrderCreationDate] BETWEEN :OrderStartDate AND :OrderEndDate  -- Date filter
    );

Next, I calculate the total number of pages:

-- Calculate the total number of pages
DECLARE @TotalPages INT;
SET @TotalPages = CEILING(CAST(@TotalRecords AS FLOAT) / :PageSize);

My idea was to use this calculation to display the number of pages in the pager area of the table.
image

However, I am unsure if these numbers can be manipulated or passed to the table component "pager".


But in case there is a way, I'd proceed with the next step...

Fetching records for the current page:

Here is the query I would use to fetch the records for the current page:

-- Fetch relevant records based on the current page and the number of records per page
SELECT 
      [ID],
      ...
      [Status],
      ...
FROM [DatabaseXY].[dbo].[Orders]
WHERE 
    (:Status IS NULL OR :Status = '' OR [Status] = :Status)   -- Filter by status
    AND (
        (:OrderStartDate IS NULL OR :OrderStartDate = '' OR :OrderEndDate IS NULL OR :OrderEndDate = '')  
        OR [OrderCreationDate] BETWEEN :OrderStartDate AND :OrderEndDate  -- Date filter
    )
ORDER BY [ID]
OFFSET (:CurrentPage - 1) * :PageSize ROWS
FETCH NEXT :PageSize ROWS ONLY;

Passing the initialOption value


on to the :PageSize parameter works as expected!

But I haven’t found a way to pass the selected page number, like "2"
image

to the :CurrentPage parameter.

By finding a way to pass the information about which page is selected to the Parameter :CurrentPage, I believe this could be an effective approach to reducing the SQL server's load.

Once again, I apologize if I’m causing more confusion than clarity. However, if anyone understands what I’m trying to achieve and has suggestions for improvement or a best practice approach, I would be truly grateful.

That almost always means the underlying tables need additional/better indices.

Unless there is a key in the data itself that would govern which page it belongs on, then the only way to know what records go on which page is to run the whole query. This is true regardless what DB or messenger service or other data source might be involved. :man_shrugging:

Fix your DB architecture and query so the query runs quickly. Full stop.

Tips for that:

  • Make sure you have and index on OrderCreationDate. Likely also on Status.
  • Make variants of your query that omit Status and/or OrderStartData and/or OrderEndDate. Do not use OR in WHERE clauses when speed matters.
  • Do not mix data types. A string is not appropriate in a datetime parameter.
2 Likes

@pturmel/@David_Stone/@Chris_Bingham: Thank you for your patience and input!

I have already implemented indices, but unfortunately without success. To add some self-criticism: As @pturmel rightly pointed out, I skipped the OR conditions in my query. The query is definitely running more efficiently now.
However, the display in the user interface (UI) is still very slow.

It’s possible that I misinterpreted and implemented the asynchronous query in my previous approach. I mistakenly focused on optimizing the SQL server, even though the user interface (UI) should actually have been the central focus.

As @David_Stone correctly mentioned, the query is always executed in full anyway, which makes optimizing the SQL server the wrong approach for enabling an asynchronous display of data. I am now wondering if it might be possible, from the perspective of the UI, to display the first records incrementally while the query continues to run in the background.

I did some further research and would like to reference the approach used in messaging services, which employ a so-called lazy loading mechanism to always display only a limited number of messages at a time. This approach supposedly relieves the gateway's memory, as it doesn't need to process the entire dataset at once or wait for the entire query to complete.

In this way, the UI could periodically, or when switching pages in a table, display only the already retrieved data while the remaining data continues to be processed in the background. The function system.util.invokeAsynchronous() might be helpful in this context.

I would be very interested in your thoughts on this. Has anyone already implemented a lazy loading mechanism or used system.util.invokeAsynchronous() for a similar approach? Are there perhaps even best practices for this?

If my intended approach is still, in your opinion, simply wrong, I’ll bury the idea of asynchronously loading data into my table and will instead try @Chris_Bingham's suggestion.

From what I see here, you have an issue where you have bad data in the database. Maybe there is a way to clean up the records in the database so that :Status is either a value or null, also :OrderStartDate is always a value or null, and also :OrderEndDate.
Is there a trigger you can run to replace the ' ' with null or something similar so that you remove most of the OR modifiers to the query?

1 Like

In my opinion, if the average user is not going to comb through tens of thousands (millions?) of rows of data every time they visit the page, then I would not build an elegant mechanism for querying for all possible data. I would build an interface which pre-populates user input fields at page load, and query only the data that the average user is expected to comb through regularly.
I would provide the ability for a user to expand search criteria to query for ‘all data’, also with a loading indicator as results are being fetched, so that the entire dataset may be exported to CSV for further processing.

4 Likes