# Create pruned dataset in Python scripting

**URL:** <https://forum.inductiveautomation.com/t/create-pruned-dataset-in-python-scripting/87409>\
**Category:** Ignition\
**Created:** [March 15, 2024, 4:10pm UTC](https://forum.inductiveautomation.com/t/create-pruned-dataset-in-python-scripting/87409 "2024-03-15T16:10:52Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bushmatic](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/bushmatic/32/87014_2.png) [@Bushmatic](https://forum.inductiveautomation.com/u/Bushmatic)\
**Post date:** [March 15, 2024, 4:10pm UTC](https://forum.inductiveautomation.com/t/create-pruned-dataset-in-python-scripting/87409/1 "2024-03-15T16:10:52Z")

</div>

In a Python script transform, I have a historian dataset that looks like this. It might have any number of columns from 1 to 10 in addition to the `t_stamp` column. I want to builld it into a new dataset using only rows that come after a given timestamp -- in this case, it is 7:00am. I want to also insert a row at the exact 7:00:00 AM mark with 0 for each value. Basically I am pruning times before a cutoff time and initializing each column.

 ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/8/d/8dac761357470eddce8f6b2176aa3d19d291b90b.png)

I started along this path:

```auto
dataset = # (see above)
cutoffDate = "2024-03-15 7:00:00 AM"
for i in range(dataset.getRowCount()):
		t_stamp = dataset.getValueAt(i,"t_stamp")
		if t_stamp >= cutoffDate:
			# (insert the entire row into a new dataset...)

```

...But kind of got stuck when I realized there's not an easy way to grab a row of data out of a dataset and throw it into a new dataset (or maybe there is?). Would a pyDataSet work better?

Alternatively, is there a way for `system.tag.queryTagHistory` to stop grabbing "early" values? My dataset comes from that function, and even when supplied with the proper `cutoffDate` of `2024-03-15 7:00:00 AM`, it is grabbing a handful of values before this timestamp (in some cases but not all). Any guidance is appreciated.

---

<div class="post-metadata">

**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\
**Post date:** [March 15, 2024, 4:24pm UTC](https://forum.inductiveautomation.com/t/create-pruned-dataset-in-python-scripting/87409/2 "2024-03-15T16:24:20Z")

</div>

I would use my `where()` expression function from my Integration Toolkit. It would look something like this:

```auto
where(
	{path.to.historian.binding.prop},
	dateIsAfter(
		it()['t_stamp'],
		{path.to.cutoff.timestamp.prop}
	)
)

```

---

<div class="post-metadata">

**Author:** ![Bushmatic](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/bushmatic/32/87014_2.png) [@Bushmatic](https://forum.inductiveautomation.com/u/Bushmatic)\
**Post date:** [March 18, 2024, 4:00pm UTC](https://forum.inductiveautomation.com/t/create-pruned-dataset-in-python-scripting/87409/3 "2024-03-18T16:00:12Z")

</div>

Thanks Phil; I ended up using built-in functions to do this but at some point I do want to grab that Integration Toolkit and apply it. I keep running into problems that it would solve easily. But what ended up working for me was this:

```auto
historyDataset = # (time series / historian dataset here)
startDate = "2024-03-18 7:00:00 AM" # this is a dynamic value based on when this transform is taking place.

deletionRowIndices = []
for i in range(historyDataset.getRowCount()):
	t_stamp = historyDataset.getValueAt(i,"t_stamp")
	if t_stamp <= startDate:
		deletionRowIndices.append(i)
newDataset0 = system.dataset.deleteRows(historyDataset, deletionRowIndices)
	
# Append a row of all 0's at the startDate to initialize the chart.
	
columnCount = historyDataset.getColumnCount()
newRow = [startDate]
for j in range(columnCount - 1):
	newRow.append(0)
	
newDataset1 = system.dataset.addRow(newDataset0,0,newRow)

return newDataset1

```