How to merge 2 historic tag datasets?

hi All
I am trying to display in the table historic data for selected tags.
The data for each selected tag in the tag browse tree is retrieved on the double click event using system.tag.queryTagHistory for the last 1 minute.
Then when another tag is selected i was trying to merge the datasets using
system.dataset.appendDataset but instead of having a new column in the data i have data appended to the existing columns.
Also data format is a very long number instead of date :frowning:
Any pointers how to solve it?

2nd attempt now:
instead of

system.dataset.appendDataset

i am trying to add column using

system.dataset.addColumn(dataset, [colIndex], col, colName, colType)
my big problem is what value to pass as colType ?
the values i am added to the new column are coming from the timestamp from the dataset retrieved using

system.tag.queryTagHistory

when i printed the type using

type(dataset.getValueAt(0,0))
i got reply as

class java.sql.Timestamp

Why are you trying to add a timestamp column?

If you are trying to combine the two historical datasets, then you would want to insure that the timestamps matched for all rows. If there is interpolation, or the tags are collected at different rates, that is going to be non-trivial.

Honestly your best bet is to just rerun the query with the additional tag.

The historian stores timestamps in milliseconds from epoch. It is a timestamp it just isn't in the format you would traditionally expect. You can use system.date.fromMillis() to convert if needed.

1 Like

If you are trying to combine the two historical datasets, then you would want to insure that the timestamps matched for all rows

Yes that is a good point that timestamps will not match to the millisecond but they match close enough for comparison. When adding new dataset i am actually adding 2 columns with time and actual value which is float. I tried to use system.date.fromMillis() to convert from milliseconds to "Date".
However when using "Date" as column typei get an error still
image

can you show your script?

1 Like


i missed in my first screenshot column type which is the main troublemaker

When sharing code, don't use screenshots.
Use the code formatting button:
image

and paste your code in between the 2 lines of backticks, like so:
```
code
```

you can even specify the language to add syntax highlighting:
```python
code
```

1 Like

Okay, first things first. Please don't post screen shots of code without also posting a copy of the script using the preformatted text option. It makes it really difficult for us to help you.

Second, I assume that this is just for testing purposes at this point, as the script doesn't currently do what you originally said you were attempting to do.

I think that I, at least, need a better understanding of what exactly you're attempting to accomplish.

1 Like
def runAction(self, event):
	from java.util import Date

	endTime = system.date.now()
	startTime = system.date.addMinutes(endTime, -1)
	dataSet1 = system.tag.queryTagHistory(paths=['[default]tag1'], startDate=startTime, endDate=endTime, returnSize=5, aggregationMode="Maximum", returnFormat='Wide')
	dataSet2 = system.tag.queryTagHistory(paths=['[default]tag2'], startDate=startTime, endDate=endTime, returnSize=5, aggregationMode="Maximum", returnFormat='Wide')
	
	columnData1 = []
	columnData2 = []	
	for i in range(dataSet2.getRowCount()):
		columnData1.append(system.date.fromMillis( dataSet2.getValueAt(i,0)) )	
		columnData2.append(dataSet2.getValueAt(i,1))		
	
	self.parent.parent.parent.getChild("Label").props.text = type(dataSet2.getValueAt(0,0))				
													
	newDataSet_addedTimestamp = system.dataset.addColumn(dataSet1,2,columnData1,"sin1_time",Date)	
	newDataSet_addedSin1 = system.dataset.addColumn(newDataSet_addedTimestamp,3,columnData2,"sin1_value",float)												
																																							
	self.parent.parent.getChild("Table").props.data = newDataSet_addedSin1

i am trying to display in the table component tag history values from the selected tags in the tag tree.
the code i pasted i test code to add column with the second selected tag which is hardcoded at the moment

Any reason you're not passing both tags to queryTagHistory ?

The doc says:

List[String] paths - An array of Tag paths (strings) to query. Each Tag path specified will be a column in the result dataset.

Now, I know historian query functions returns are sometimes.... funky. But start with that.

1 Like

wow thanks for pointing this out - this will be my fallback solution but i am still pushing for the original add.column solution if it works :slight_smile:
the reason for that is that when using mobile device tag have to be selected add by one so its handy to keep merging the datasets.

Honestly, while it is completly possible to merge the datasets, you're better off just using the list of selected tags and then querying the history for all tags again.

The script as it is right now will only ever query for tag1 and tag2. There is nothing that is actually getting the selected values from the tag browse tree.

1 Like

Store the tags in a list then query all of them at once. Don't make things more complicated than they actually are. Which, in programmer language, means "use the built-ins when they exist"

2 Likes

Honestly, while it is completly possible to merge the datasets, you're better off just using the list of selected tags and then querying the history for all tags again.

thats correct selection list needs to be added to the code
yes i will use the list of tags which do works reliably as i tried just now
thanks a lot for help

@pascal.fragnoud thanks for help as well as @lrose

@lrose you mentioned that system.date.fromMillis will convert the time in the historic dataset but how do i actually do it? i would prefer to avoid looping via whole dataset and calling this function.
Can i somehow format the table itself?