Time series chart with grouped by hour

I am getting all the events in my query by grouping on format(t_stamp, 'yyyy-MM-dd HH')

However, data is not showing in my time series chart now. So I think I need a datetime conversion.
What is a good way to overcome this issue efficiently?

Or maybe it is unhappy because I only have t_stamps when there was an event instead of periodically?


Tried to fill in the zeroes.
Something is wrong with my code though.
I don't see 800 rows at the end.

today=system.date.now()		
		
#get 800 hours
eightHundredHours = [[system.date.format(system.date.addHours(system.date.addHours(today,-800),x),'yyyy-MM-dd HH:00:00')] for x in range(800)]
		
#get all the dates that are in the query
datesInQuery=[]
for row in range(value.getRowCount()):
		Hour = value.getValueAt(row, 'Hour')
		datesInQuery.append([Hour])
		 
	
#get all the hours in that 800, but not in the query
excludedHours=[x for x in eightHundredHours if x not in datesInQuery]
	
counter=0	
valueHeaders = ['Hour', 'Events']
data = []
for x in range(800):
#set events to 0 if there weren't events during the hour
	if [system.date.format(system.date.addHours(system.date.addHours(today,-800),x),'yyyy-MM-dd HH:00:00')] in excludedHours:
		data.append([str(system.date.format(system.date.addHours(system.date.addHours(today,-800),x),'yyyy-MM-dd HH:00:00')),0])	
	
#get the value if there were events during the hour				
	if [system.date.format(system.date.addHours(system.date.addHours(today,-800),x),'yyyy-MM-dd HH:00:00')] in datesInQuery:
		Hour = value.getValueAt(counter, 'Hour')
		Events = value.getValueAt(counter, 'Events')
		
		data.append([Hour, Events])
		counter=counter+1

data = system.dataset.toDataSet(valueHeaders, data)	
	
return value

Sometimes data shows, and sometimes it just says component error.
I don't know why.
I am on 8.1.19 now.

I am having two different issues.
When I just return value, there are no zeroes:
image

When I return data, I only get zeroes, though I do get the 800 rows.

From my query:

I think it is not comparing the datetimes correctly. Will edit my query.

Bad.
You're converting dates to strings and then having to do grouping on strings with all the subsequent problems when the data arrives back into Ignition. There are SQL ways of doing this but they vary by SQL implementation. Search for SQL DateTime group by date and hour and your SQL variant.

1 Like

Thanks
I think I can use this.

GROUP BY DATEPART(day, [activity_dt]), DATEPART(hour, [activity_dt]);

It is not right.
When I group, then t_stamp is not allowed in select.

The canonical form of date truncation for grouping in MS SQL Server is a paired DATEDIFF & DATEADD. Look for it on google. The result column is a timestamp marking the start of the groupable period. Do not compute with dates and times in string form. Ever. Only convert to string to display.

1 Like

Thanks
I think this is helping me a ton.

However, something is messing up when I bind the data.
image

It isn't showing the hour in 24 hour.

That's purely a display issue for your table. You might also want an ORDER BY clause matching your GROUP BY clause.

1 Like

I wish for this solution.
I really hoped there would be a setting for making the timeseries use discrete data instead of interpolating. I think then I could use the query results.
I would place it in the idea thread, but I don't know if maybe the option exists and I don't know where.

Trying to append zeroes as an alternative solution.
I think my scripting in zeroes might work if I can figure out the python date equivalent of the SQL technique: dateadd(datediff())


From my query testing, after ordering it, looks exactly how I imagine it should.
Still shows incorrectly when I look at the query results from the binding.
image



Found an alternate technique using .replace.
However, I am getting an error
AttributeError: 'java.util.Date' object has no attribute 'replace'

import datetime

today=system.date.now()	
dt=today
dt = dt.replace(hour=0, minute=0, second=0, microsecond=0)
today = dt

eightHundredHours = [[system.date.format(system.date.addHours(system.date.addHours(today,-800),x),'yyyy-MM-dd HH:00:00')] for x in range(800)]
print eightHundredHours


eightHundredHours = [[system.date.addHours(system.date.addHours(today,-800),x)] for x in range(800)]
print eightHundredHours

today=system.date.now()	
start=system.date.getDate(1970,1,1)
dt=system.date.hoursBetween( start,today)
dt=system.date.addHours(start,dt)

I kind of got the python equivalent of the SQL technique: dateadd(datediff()).
So I will try that Monday. I think the issue is comparing the datetimes.

Problem is that the data is being interpolated instead of discrete.
image

My solution is not working, which is trying to add zeroes to the hours that had zero events.

Return value is showing the query data like this:
image

Return data from my script is showing like this:
image

Script

	#get a timestamp at the start of this hour
	today=system.date.now()
	start=system.date.getDate(1970,1,1)
	dt=system.date.hoursBetween( start,today)
	dt=system.date.addHours(start,dt)		
		
	#get timesstamps for each hour between 800 hours ago and now
	eightHundredHours = [[system.date.addHours(system.date.addHours(dt,-800),x)] for x in range(800)]
		
	#get all the dates that are in the query
	datesInQuery=[]
	for row in range(value.getRowCount()):
			Hour = value.getValueAt(row, 'Hour')
			datesInQuery.append([Hour])
	
	#get all the hours in that 800, but not in the query
	excludedHours=[x for x in eightHundredHours if x not in datesInQuery]
	
	counter=0	
	valueHeaders = ['Hour', 'Events']
	data = []
	for x in range(800):
	#set events to 0 if zero events during the hour
		if [system.date.addHours(system.date.addHours(dt,-800),x)] in excludedHours:
			data.append([system.date.addHours(system.date.addHours(dt,-800),x),0])	
	
	#get the value if there were events during the hour				
		if [system.date.addHours(system.date.addHours(dt,-800),x)] in datesInQuery:
			Hour = value.getValueAt(counter, 'Hour')
			Events = value.getValueAt(counter, 'Events')
			
			data.append([Hour, Events])
			counter=counter+1

	data = system.dataset.toDataSet(valueHeaders, data)	
	
	return data

I added this to the bottom, and I think I started to find an issue in the script, but I am not sure how to fix it.


The excludedHours seem to be a list of lists with one value in them.
So I am comparing dates to objects.
edit: Had to edit the images as I had the first two columns order switched.

I am getting a different date format between the two of them?


Still doesn't work and I got them all to appear the same.

#get a timestamp at the start of this hour
	today=system.date.now()
	start=system.date.getDate(1970,1,1)
	dt=system.date.hoursBetween( start,today)
	dt=system.date.addHours(start,dt)		
		
	#get timesstamps for each hour between 800 hours ago and now
	eightHundredHours = [system.date.addHours(system.date.addHours(dt,-800),x) for x in range(800)]
		
	#get all the dates that are in the query
	datesInQuery=[]
	for row in range(value.getRowCount()):
			Hour = value.getValueAt(row, 'Hour')
			datesInQuery.append(Hour)
	
	#get all the hours in that 800, but not in the query
	excludedHours=[x for x in eightHundredHours if x not in datesInQuery]
	
	counter=0	
	valueHeaders = ['Hour', 'Events']
	data = []
	for x in range(800):
		#set events to 0 if there weren't events during the hour
		if eightHundredHours[x] not in datesInQuery:
			data.append([eightHundredHours[x],0])	
	
		#get the value if there were events during the hour				
		if eightHundredHours[x] in datesInQuery:
			Hour = value.getValueAt(counter, 'Hour')
			Events = value.getValueAt(counter, 'Events')
			data.append([Hour, Events])
			counter=counter+1
			

	data = system.dataset.toDataSet(valueHeaders, data)	
#	tempheader= ['Datesinquery','generatedDates','excluded']
#	testData=[]
#	for row in range(value.getRowCount()):		
#		Hour=datesInQuery[row]
#		genHour=eightHundredHours[row]
#		excludedHour= excludedHours[row]
#		testData.append([Hour,genHour,excludedHour ])
#	data= system.dataset.toDataSet(tempheader, testData)
	
	return data
	
	

Still the comparing is not working.
I even revised my code to get all the hours from math on the query hours.

#get all the hours that are in the query
	hoursInQuery=[]
	for row in range(value.getRowCount()):
		Hour = value.getValueAt(row, 'Hour')
		hoursInQuery.append(Hour)
	
	#get generate hours before the hour at the end of the query
	lastRow=value.getRowCount()-1
	hoursToGen=671	
	generatedHours= [system.date.addHours(system.date.addHours( value.getValueAt(lastRow, 'Hour')   ,-hoursToGen),x) for x in range(hoursToGen)]
	
	#get the generated hours not in the query
	exclusiveHours= [x for x in generatedHours if x not in hoursInQuery]
	
	counter=0	
	valueHeaders = ['Hour', 'Events']
	data = []
	for x in range(hoursToGen):
		#set events to 0 if there weren't events during the hour
		if generatedHours[x] in exclusiveHours:
			data.append([generatedHours[x],0])	
	
		#get the value if there were events during the hour				
		if generatedHours[x] not in exclusiveHours:
			Hour = value.getValueAt(counter, 'Hour')
			Events = value.getValueAt(counter, 'Events')
			data.append([Hour, Events])
			counter=counter+1			

	#data = system.dataset.toDataSet(valueHeaders, data)	
	tempheader= ['Datesinquery','generatedDates','excluded']
	testData=[]
	for row in range(value.getRowCount()):			
		Hour=hoursInQuery[row]
		genHour=generatedHours[row]
		excludedHour= exclusiveHours[row]
		testData.append([Hour,genHour,excludedHour ])
	data= system.dataset.toDataSet(tempheader, testData)
	
	return data

Got help from support.
Was getting a Timestamp from SQL.
Had to convert that to a string, then parse it to datetime.

Tried a ton of other ways, but finally got it this way.

This line helped me a ton.

return [type(hoursInQuery[1]),value.getValueAt(lastRow, 'Hour'),type(generatedHours[671]),generatedHours[801],type(exclusiveHours[1])]

The component was erroring when the dates were coming in out of order. Had to fix the parsing.
HH instead of hh for the hours.