pre-processing
Does anyone use this? What are pros and cons?
Please explain the window sizes. What this does in the database. How queries are affected.
Basically, the pre-processing system creates additional data tables (that is, the base data is stored as always), and stores the min, max, and average values for tags according to the window specified.
When querying, the most common queries are those generated by the EasyChart, where the result size is fixed. This means that for any query, there is a window size (end-start/size). If the window is bigger than that defined for the pre-processed data, it looks to that table- otherwise, it goes to the raw data.
The main benefit here is that fewer rows will need to be loaded when querying large time frames. I can’t give you much of an idea as to how many people use this, though on one particularly large system, I’ve seen it make a pretty big difference. In that case, they have the window size set to 4 minutes. This more or less means that for queries under 20 hours, the raw data is consulted, and everything else uses the processed data. With this, they can query nearly any range of time in under 15 seconds. Oh, and they’re storing something like 4 billion rows per month… (yes, really)
Regards,
1 Like
We are nowhere near that level. Seems like something that would be more common.
That’s a bit of an extreme example, but basically where it’s helpful is when you’re often viewing data and various levels that differ by a lot. For example, normally looking at one day, but then you want another screen that gives a monthly overview. By turning on pre-processing for a window size that is appropriate for a month, you can ensure that both situations take similar amounts of time.
Regards,
2 Likes
Hello
I’m a little confused in this please hlep me out.
If I set to capture data at rate of 1 min and set pre processing to 30 min and I query for a range of month with aggregation sample interval to 5 min the system doesn’t use pre processing and if aggregation interval size set to 1 hour, it use pre processing table for result.( Because interval smaple size in bigger than pre processing)
Do I understand it right?