Use of Æ, Ø and Å in system.perspective.download

Ignition
,
1

I'm using a script for saving and storing the Alarm Journal to a file. The alarm journal contains the Norwegian letters æ, ø and å. I'm using the system.perspective.download-function to save the file, but I'm not able to force it to store these characters in the file.

The system.perspective.print-function prints these characters fine in the Output Console, so my guess is that the problem is in the system.perspective.download-function. I've tried both the system.dataset.toCSV and .toExcel-conversions before downloading the file. I have also opened both types of files in both Excel and in Notepad++ without any luck.

I've also experimented with the contentType in the download function, but also here without success.

Segment from script:

dataz = system.dataset.toDataSet(header,table)
csvData = system.dataset.toCSV(dataset=dataz)

system.perspective.print(csvData)

system.perspective.download(filename="Label.xls", data=csvData, contentType="text/plain; charset=utf-8")

I'm surely not the first one experiencing this, so I'm wondering if there are anyone out there who can point me in the correct direction?

2

I have no direct experience, but I suspect that there may be some misalignment between your data and filetype causing the issues rather than Ignition's download function itself. Ultimately, .xls is a legacy Excel binary file extension which may not properly encode plaintext CSV data with such characters. Unless you explicitly need that filetype to interface with some legacy software (pre-2003 ish), I would suggest trying either

  1. Use system.dataset.toExcel to obtain data and save with file extension '.xlsx'
  2. Keep using system.dataset.toCSV and save with file extension '.csv'
    • You may also want to add a UTF-8 BOM to the beginning of your CSV data if you're ultimately planning to open this in Excel since Windows applications are known to interact strangely with non-ASCII plaintext coming from the input text stream unless given explicit instructions.
      • Example: csvData = u'\ufeff' + system.dataset.toCSV(dataz)

I'm mostly commenting because I'm interested to hear how this turns out, actually. Best of luck!

3

Do you have a reverse proxy or anything in front of Ignition?
When you provide the content to download as a string, we're explicitly writing it out with UTF-8:

if (data instanceof String) {
  OutputStreamWriter writer =
      new OutputStreamWriter(response.getOutputStream(), UTF_8);
  writer.write((String) data);

If you want to make absolutely sure the download function isn't mangling the encoding somehow, you could try converting your string into a raw byte array to pass into download; either of these two should work:

from java.lang import String
bytes = String("abc").getBytes("UTF-8")

from org.apache.commons.lang3 import StringUtils
bytes = StringUtils.getBytes("abc", "UTF-8")

A byte array provided to system.perspective.download will be emitted verbatim.

4

I tested this with a 3rd party Excel module I'm working on, and I can confirm this is an issue.

I tested with this code:

		headers = ["Language", "Sample"]

		data = [
		    ["Norwegian", "Jeg har lyst på blåbær og røkt ørret på hytta i Ålesund."],
		    ["German", "Mädchen können übermäßig süße Brötchen mögen, während schöne Vögel über die Straße fliegen."],
		    ["France", "Le pêcheur préféré a déjà rêvé d'être à côté de l'île où il était né."]
		]

		dataz = system.dataset.toDataset(headers, data)
		
		system.tag.writeBlocking(["[default]Tag0"], dataz, 1000)
		
		csvData = system.dataset.toCSV(dataset=dataz)
		
		system.perspective.download(filename="utc_test.txt", data=csvData, contentType="text/plain; charset=utf-8")
		
		from java.lang import String
		bytes = String(csvData).getBytes("UTF-8")
		system.perspective.download(filename="utc_test_bytes.txt", data=bytes)

I can confirm that the data written to the dataset tag is not correct, as is the data written to the two text files. My 3rd party module also was corrupt.

I asked Claude about the two text files, this was the final response after a few questions:

So to directly answer your question: yes, this file has the identical issue as the first one — same single UTF-8-decoded-as-Latin-1 mojibake, same fix, and I've included the BOM this time so it should open correctly in Notepad++ without any extra steps.

Then I asked Claude about my Excel file generated from my 3rd party module, this was the response:

Yes, same issue — but notable because it's happening in a different way this time. An .xlsx file's text is stored internally as proper UTF-8 XML, so there's no encoding-detection ambiguity like in a plain text file. This means the mojibake was baked directly into the cell values themselves — someone (or some tool) pasted or imported already-corrupted text into the spreadsheet, rather than the file being mis-read on open.

It appears that the issue lies within the creation of the dataset.

Interesting enough is Claude was easily able to fix the text files.

5

You have constant strings in python with special characters, but without the u'...' syntax. Your special characters in this test are breaking right here, not in the export tools.

GIGO.

Jython strings containing unicode must use the prefix.

6

And indeed, making that change everything works as expected, even in the dead simple case:

def runAction(self, event):
	headers = ["Language", "Sample"]

	data = [
	    ["Norwegian", u"Jeg har lyst på blåbær og røkt ørret på hytta i Ålesund."],
	    ["German", u"Mädchen können übermäßig süße Brötchen mögen, während schöne Vögel über die Straße fliegen."],
	    ["France", u"Le pêcheur préféré a déjà rêvé d'être à côté de l'île où il était né."]
	]

	dataz = system.dataset.toDataSet(headers, data)
	csvData = system.dataset.toCSV(dataset=dataz)
	
	system.perspective.download(filename="basic.txt", data=csvData)
	system.perspective.download(filename="utc_test.txt", data=csvData, contentType="text/plain; charset=utf-8")
	
	from java.lang import String
	bytes = String(csvData).getBytes("UTF-8")
	system.perspective.download(filename="utc_test_bytes.txt", data=bytes)