# Removing duplicate rows from dataset

**URL:** <https://forum.inductiveautomation.com/t/removing-duplicate-rows-from-dataset/41844>\
**Category:** Ignition\
**Created:** [December 29, 2020, 4:47pm UTC](https://forum.inductiveautomation.com/t/removing-duplicate-rows-from-dataset/41844 "2020-12-29T16:47:39Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![hachjessed](https://avatars.discourse-cdn.com/v4/letter/h/7ab992/32.png) [@hachjessed](https://forum.inductiveautomation.com/u/hachjessed)\
**Post date:** [December 29, 2020, 4:47pm UTC](https://forum.inductiveautomation.com/t/removing-duplicate-rows-from-dataset/41844/1 "2020-12-29T16:47:39Z")

</div>

Beginning with a basic dataset, what would be the simplest way to remove duplicate rows of data?

Input ds:

```auto
ID Value 
01 val1
02 val2
03 val3
01 val1
02 val2
02 val2

```

Desired output ds:

```auto
ID Value 
01 val1
02 val2
03 val3

```

---

<div class="post-metadata">

**Author:** ![chandler](https://avatars.discourse-cdn.com/v4/letter/c/e9a140/32.png) [@chandler](https://forum.inductiveautomation.com/u/chandler)\
**Post date:** [December 29, 2020, 5:28pm UTC](https://forum.inductiveautomation.com/t/removing-duplicate-rows-from-dataset/41844/2 "2020-12-29T17:28:39Z")

</div>

You can use a Python set to remove duplicates for you:

```auto
# get a list of your header names
headers = system.dataset.getColumnHeaders(oldDataset)

# convert to pyDataset, so the data is compatible with the set constructor
pyData = system.dataset.toPyDataSet(oldDataset)

# turn the lists into tuples so they are hashable
tuples = map(tuple, pyData)

# create a set from the list of tuples 
pySet = set(tuples)

#convert set back into a list
newData = list(pySet)

#create an Ignition dataset from the headers and data without duplicates
newDataset = system.dataset.toDataSet(headers, newData)

```