Special Characters in CSV

I have issues with special characters in CSV.

I have got a dataset with strings containing some characters like “é” or “à”, encoded like this :

string = u"This is the string with é and à" (as an example)

I export the dataset as a .csv file.

When I want to upload back this file, the encoding of the characters is not recognized and I have question marks standing for them.

Is there a way to pass the utf8 encoding through the .csv to keep the special characters ?

Thank you for your help,

Can you post your full script? Using system.dataset.toCSV/fromCSV, I get the same characters out and back in:

from com.inductiveautomation.ignition.common.util import DatasetBuilder
from java.lang import String, Integer, Double, Boolean
from java.util import Date

ds = (DatasetBuilder
	.colTypes(Integer, String, Double, Boolean, Date)
	.colNames("Int", "String", "Double", "Boolean", "Date")
	.addRow(123, u"This is the string with é and à", 123, True, system.date.now())
	.addRow(None, None, None, None, None)
	.addRow(123, "Abc", 123, True, system.date.now())

t = system.dataset.toCSV(ds, True, True, False)
p = system.file.saveFile("test.csv")
print p
system.file.writeFile(p, t)

read = system.file.readFileAsString(p)
readDs = system.dataset.fromCSV(read)
pyds = system.dataset.toPyDataSet(readDs)

for col in pyds:
	for row in col:
		print row

What might be the issue is the encoding when reading the file back in - you can force an encoding argument (I’d recommend UTF-8, for sanity) to system.file.writeFile / readFileAsString to ensure things are decoded correctly.

Thank you for your answer.

Here is my dataset in a table :

I an downloading it with this script :


And uploading the data with this :


And i get this back :

Try adding the encoding argument to system.file.writeFile. If you open the file in a text editor separately, does it have encoding errors or have the properly encoded values?

1 Like

Ok I have added the encoding in writeFile() and it is working ! Thank you !

1 Like