Problem with export the data from dataset format To CSV (UTF-8)

[Kaviya_D]

Hello Everyone,

Any one can help me to solve this issue...?

I have One Perspective View where I can enter all the details to create the work order. Once I create the Order, the data will be saved into the database. Each work order data I have, the Column Name is like Remarks which is having the description about the work order purpose. That description is sometimes entered with chinese characters.

Another view, I have the Table Component, which is used to view the list of work orders that is created by the users. On the same page, I have the Export button to export the data to CSV format. That is also working quite well.

But my problem is if I have the chinese description of my work order data, which is showing as special characters, when I open the CSV file using EXCEL.

Now I want to export the CSV file with UTF-8 format. Here I mention the Script which I used to export the data to CSV for your reference.

Here myData is my Table Data

ds = system.dataset.toDataSet( header , myData )

dataset = ds

data = system.dataset.toCSV(dataset, 1, 0, 0)

Filename = 'Exportdata.csv'

system.perspective.download(Filename, data)

If I open the downloaded file using excel, chinese characters showing as special characters.

this is how its showing in excel.
but in my data set those values are in unicode format

for example : \u8c22\u8c22 - this is how my data in my dataset.
It has to shown as 谢谢 in my excel.

but its not showing as it is. What to do in this situation. How can I export the CSV with utf-8 format.

Also I tried with below condition

system.perspective.download(Filename, data,"charset=utf-8")

But Couldn't.

Kindly help me to solve this.

Thanks in advance.

[PGriffith]

In the simplest possible case, this worked as expected for me:

def runAction(self, event):
	ds = system.dataset.toDataSet(["column"], [[u"谢谢"]])
	csv = system.dataset.toCSV(ds)
	
	system.perspective.download("test.csv", csv)	

This generated this file:
test.csv (18 Bytes)

Can you confirm the data is represented in the dataset, and in the string CSV output, in the format you expect? E.g. via system.perspective.print?

@Kaviya_D

system.perspective.download(Filename, data,"charset=utf-8")

I wouldn't expect this to make the difference, but you should use text/csv; charset=utf-8 as your content type value.

[Transistor]

Welcome to the forum. Please see Wiki - how to post code on this forum.

[Kaviya_D]

@PGriffith

I wouldn't expect this to make the difference, but you should use text/csv; charset=utf-8 as your content type value.

I will try this and let you know soon, and will share the dataset data here by using print function for your better understanding about my issue.

[Kaviya_D]

@PGriffith
FYR

header = ["SNO","MACHINE_ID","Order_No","Quantity","Remarks","Date","Created By"]

myData = [[ 1, 'MC01','KJ1121',10.0, u'\u8c22\u8c22', '09 May 2024', 'KAVIYA'],[ 2, 'MC01','KJ1122',10.0, 'TEST1', '09 May 2024', 'KAVIYA'],[ 3, 'MC01','KJ1123',10.0, 'TEST2', '09 May 2024', 'KAVIYA'],[ 4, 'MC01','KJ1124',10.0, 'TEST3', '09 May 2024', 'KAVIYA'],[ 5, 'MC01','KJ1125',10.0, 'TEST4', '09 May 2024', 'KAVIYA'],[ 6, 'MC01','KJ1126',10.0, 'TEST5', '09 May 2024', 'KAVIYA']]

The Above data needs to export as CSV. So I tried the Below mentioned Script.

@Kaviya_D

ds = system.dataset.toDataSet( header , myData )

dataset = ds

data = system.dataset.toCSV(dataset, 1, 0, 0)

Filename = 'Exportdata.csv'

system.perspective.download(Filename, data)

If you see myData , the Index 0 is

[ 1, 'MC01','KJ1121',10.0, u'\u8c22\u8c22', '09 May 2024', 'KAVIYA']

this data created with chinese description so that the data stored as unicode format

u'\u8c22\u8c22'

But remaining orders are created with english description like Test1,Test2, Test3 like that.

After Conversion the 1st row data Remarks is showing special Characters which I mentioned previously.

Also I tried what you suggested in your reply

@PGriffith

you should use text/csv; charset=utf-8 as your content type value.

But no difference.

I hope now you have a clear idea about my issue.

Can you help me to solve this.?

[Transistor]

Again, please see Wiki - how to post code on this forum. Don't post code as " or > quotations.

[PGriffith]

I think the problem here is Excel, not Ignition. With your script, I get the actual data you're expecting if I open the file in a text editor:

Consider writing out the file data with a leading 'byte order mark', to make it UTF-8 with BOM, which I suspect Excel is expecting:
system.perspective.download(Filename, u'\uFEFF' + data)

Byte order mark

[Kaviya_D]

It is working fine now.
Thank you very much.