I need to query for raw data in tag history, with no aggregations or interpolation. I found that setting returnSize to zero is key to retrieving something that is closest to the raw data.
But I also need to limit the amount of data that is returned. My user may query for years of data and I need to handle it in chunks after the queryHistory() call (I can’t chunk it “inline” during the handling of my custom StreamingDatasetWriter.writeRow() due to how our system works).
Here’s what the code looks like:
BasicTagHistoryQueryParams queryParams = BasicTagHistoryQueryParams.newBuilder()
.paths(tagPathList)
.startDate(startDate)
.endDate(endDate)
.queryFlags(Flags.of(
TagHistoryQueryFlags.BOUNDING_VALUES_YES,
TagHistoryQueryFlags.NO_INTERPOLATION,
TagHistoryQueryFlags.NO_PREPROCESSED_DATA
))
.returnSize(0)
.returnFormat(ReturnFormat.Tall)
.build();
// Use our custom writer that converts rows directly to Samples as they arrive
SampleDataset dataset = new SampleDataset(this.tagQualityCodeFormat);
tagHistoryManager.queryHistory(queryParams, dataset);
SampleDataset is my custom StreamingDatasetWriter that handles writeRow().
So, my question is: Is there a way to limit how many rows are returned without defeating the requirement to return raw data? Maybe I can throw an exception from my writeRow() when I’ve reached the limit of what I can accept?
The exception-throwing seems to work well. Would appreciate someone from Inductive Automation telling me if it’s a bad idea. Here’s what my StreamingDataSet class looks like:
private static class SampleDataset implements StreamingDatasetWriter {
@Getter private final List<Sample> samples = new ArrayList<>();
private final int maxSamples;
public SampleDataset(TagQualityCodeFormatEnum qualityCodeFormat, int maxSamples) {
this.maxSamples = maxSamples;
}
@Override
public void initialize(String[] columnNames, Class<?>[] columnTypes, boolean fixedSize, int numRows) {
...
}
@Override
public void write(Object[] row, QualityCode[] qualityCodes) {
// Check if we've reached the limit before processing this row
if (this.samples.size() >= this.maxSamples) {
throw new SampleLimitReachedException(
String.format("Sample limit of %d reached, short-circuiting data retrieval", this.maxSamples));
}
... turn row into sample ...
this.samples.add(sample);
}
@Override
public void finish() {
// Data conversion happens in write(), nothing to do here
}
@Override
public void finishWithError(Exception e) {
if (e instanceof SampleLimitReachedException) {
// Ignore SampleLimitReachedException - this is expected when we hit the limit
return;
}
// Log other errors
LOG.error("Query finished with error", e);
}
}
The downside is that the exception gets logged, which is some unfortunate noise:
jvm 1 | E [t.h.q.ResultWriter ] [15:49:10.696]: Error executing historical tag read.
jvm 1 | com.seeq.ignition.v83.gateway.link.IgnitionGatewayConnection$SampleLimitReachedException: Sample limit of 5 reached, short-circuiting data retrieval
jvm 1 | at com.seeq.ignition.v83.gateway.link.IgnitionGatewayConnection$SampleDataset.write(IgnitionGatewayConnection.java:1460)
jvm 1 | at com.inductiveautomation.historian.gateway.query.writing.TallHistoryWriter.writeRow(TallHistoryWriter.java:118)
jvm 1 | at com.inductiveautomation.historian.gateway.query.writing.TallHistoryWriter.commitRows(TallHistoryWriter.java:71)
jvm 1 | at com.inductiveautomation.historian.gateway.query.writing.HistoryWriter.readData(HistoryWriter.java:322)
jvm 1 | at com.inductiveautomation.historian.gateway.query.writing.HistoryWriter.execute(HistoryWriter.java:227)
jvm 1 | at com.inductiveautomation.historian.gateway.HistorianManagerImpl.queryHistory(HistorianManagerImpl.java:1070)
jvm 1 | at com.inductiveautomation.ignition.gateway.sqltags.history.TagHistoryManagerBridge.queryHistory(TagHistoryManagerBridge.java:99)
jvm 1 | at com.seeq.ignition.v83.gateway.link.IgnitionGatewayConnection.getSamplesViaTagHistoryManager(IgnitionGatewayConnection.java:1537)
jvm 1 | at com.seeq.ignition.v83.gateway.link.IgnitionGatewayConnection.getSamples(IgnitionGatewayConnection.java:1605)
jvm 1 | at com.seeq.link.sdk.DatasourceConnectionV2Host.signalRequest(DatasourceConnectionV2Host.java:1777)
jvm 1 | at com.seeq.link.sdk.BaseDatasourceConnection.processMessage(BaseDatasourceConnection.java:492)
jvm 1 | at com.seeq.link.sdk.BaseDatasourceConnection.lambda$processMessage$2(BaseDatasourceConnection.java:425)
jvm 1 | at com.seeq.link.sdk.BaseDatasourceConnection.lambda$processMessage$0(BaseDatasourceConnection.java:417)
jvm 1 | at com.seeq.link.sdk.utilities.DefaultConcurrentRequestsHandler.lambda$runWhenPermitted$0(DefaultConcurrentRequestsHandler.java:71)
jvm 1 | at com.seeq.link.sdk.utilities.ThreadCollection$Worker.run(ThreadCollection.java:313)
jvm 1 | at com.seeq.link.sdk.utilities.ThreadCollection.lambda$spawn$2(ThreadCollection.java:170)
jvm 1 | at java.base/java.lang.Thread.run(Unknown Source)