# \[IGN-4991, IGN-4996\]system.file.writeFile encoding

**URL:** <https://forum.inductiveautomation.com/t/ign-4991-ign-4996-system-file-writefile-encoding/19878>\
**Category:** Ignition\
**Created:** [September 28, 2018, 11:17am UTC](https://forum.inductiveautomation.com/t/ign-4991-ign-4996-system-file-writefile-encoding/19878 "2018-09-28T11:17:01Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![systemparadox](https://avatars.discourse-cdn.com/v4/letter/s/b5e925/32.png) [@systemparadox](https://forum.inductiveautomation.com/u/systemparadox)\
**Post date:** [September 28, 2018, 11:17am UTC](https://forum.inductiveautomation.com/t/ign-4991-ign-4996-system-file-writefile-encoding/19878/1 "2018-09-28T11:17:01Z")

</div>

Unlike `system.file.readFileAsString`, `system.file.writeFile` has no encoding option.

It appears that the behaviour is the same as `readFileAsString` without any encoding option - it uses Java’s default charset, which is platform-specific. On some systems it’s always UTF-8, but on others it can change depending on the system or user locale settings.

On my Linux machine it uses UTF-8, but on the customer’s Windows machine it uses latin1. Transferring files between these systems leads to invalid encoding errors in one direction and double-encoding in the other.

You can manually encode a unicode string in python with `.encode('utf8')`. However, the result is a `str` object, so `system.file.writeFile` still does the encoding. In Linux, this results in a double-utf8-encoded string:

```auto
system.file.writeFile(path, output.encode('utf8'))

```

The only way to get `system.file.writeFile` to not do the encoding is to give it an array of bytes rather than a string. Unfortunately, bytearray() is not available in Python 2.5. In 2.6 you could just do this:

```auto
system.file.writeFile(path, bytearray(output.encode('utf8')))

```

Otherwise you can use `array`:

```auto
from array import array
system.file.writeFile(path, array('b', output.encode('utf8')))

```

Or write the file directly with Python:

```auto
from __future__ import with_statement
import codecs
with codecs.open(path, 'w', 'utf8') as file:
    file.write(output)

```

But it would be really great if an encoding option could be added to `system.file.writeFile`.

---

<div class="post-metadata">

**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\
**Post date:** [September 28, 2018, 3:04pm UTC](https://forum.inductiveautomation.com/t/ign-4991-ign-4996-system-file-writefile-encoding/19878/2 "2018-09-28T15:04:56Z")

</div>

This is a solid idea - I’ve added an internal ticket to keep track of it.

---

<div class="post-metadata">

**Author:** ![systemparadox](https://avatars.discourse-cdn.com/v4/letter/s/b5e925/32.png) [@systemparadox](https://forum.inductiveautomation.com/u/systemparadox)\
**Post date:** [September 28, 2018, 3:05pm UTC](https://forum.inductiveautomation.com/t/ign-4991-ign-4996-system-file-writefile-encoding/19878/3 "2018-09-28T15:05:47Z")

</div>

Excellent. Thanks very much.

---

<div class="post-metadata">

**Author:** ![justin.brzozoski](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/justin.brzozoski/32/13452_2.png) [@justin.brzozoski](https://forum.inductiveautomation.com/u/justin.brzozoski)\
**Post date:** [November 1, 2021, 4:02pm UTC](https://forum.inductiveautomation.com/t/ign-4991-ign-4996-system-file-writefile-encoding/19878/4 "2021-11-01T16:02:04Z")

</div>

Sorry to resurrect a zombie topic, but I’m pondering wonky behavior right around this right and am curious what the current state of `system.file.writeFile` is.

On Ignition 8.1.7 we now have a newer Jython and bytearray (yay!) but they don’t play well with `system.file.writeFile` leading to this weird dichotomy:

```auto
xls = system.dataset.toExcel(True, [dset])
with open('/tmp/t0.xls','w') as f:
 f.write(bytearray(xls))
system.file.writeFile('/tmp/t1.xls',xls)

```

This will work and output the same file via both methods, but the `write` function fails if I pass it the raw `array.array` output type from `system.dataset.toExcel` and `system.file.writeFile` will fail if I pass it the `bytearray` output. Seems like an unnecessary complication.

(And don’t do `system.file.writeFile('t0.xls',str(bytearray(xls)))` as that won’t complain but will create a corrupt file…)

EDIT: I would be able to work around this if I had a way to get from a Python bytearray back to something that `system.file.writeFile` accepted as binary. But I’m having a bit of trouble at the moment figuring it out. It didn’t like `[int(x) for x in bytearray()]`…

---

<div class="post-metadata">

**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\
**Post date:** [November 1, 2021, 4:20pm UTC](https://forum.inductiveautomation.com/t/ign-4991-ign-4996-system-file-writefile-encoding/19878/5 "2021-11-01T16:20:59Z")

</div>

I would avoid Jython’s types entirely. `toExcel` is returning a Java `byte[]`, which Jython wraps in a type that is _not_ `bytearray`. `writeFile` can directly handle that `byte[]` (and has an encoding argument now).

---

<div class="post-metadata">

**Author:** ![justin.brzozoski](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/justin.brzozoski/32/13452_2.png) [@justin.brzozoski](https://forum.inductiveautomation.com/u/justin.brzozoski)\
**Post date:** [November 1, 2021, 4:24pm UTC](https://forum.inductiveautomation.com/t/ign-4991-ign-4996-system-file-writefile-encoding/19878/6 "2021-11-01T16:24:24Z")

</div>

I am trying to do that as much as possible, but I have another parallel code path which generates ZIP files using the Python `zipfile` library and I am trying to handle the output of both the Excel and ZIP paths in common code.

I’m trying to find the trick to convert `byte_result` in the following sample back to something that `system.file.writeFile` will accept as binary:

```auto
import io
import zipfile
with io.BytesIO() as tmp_buff:
    with zipfile.ZipFile(tmp_buff, 'w', zipfile.ZIP_DEFLATED) as zfile:
        zfile.writestr('test.txt', 'Hello world!\n')
    tmp_buff.flush()
    byte_result = tmp_buff.getvalue()

```

---

<div class="post-metadata">

**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\
**Post date:** [November 1, 2021, 4:28pm UTC](https://forum.inductiveautomation.com/t/ign-4991-ign-4996-system-file-writefile-encoding/19878/7 "2021-11-01T16:28:49Z")

</div>

Seems like we should just add support for `bytearray` to `system.file.writeFile`.

---

<div class="post-metadata">

**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\
**Post date:** [November 1, 2021, 4:43pm UTC](https://forum.inductiveautomation.com/t/ign-4991-ign-4996-system-file-writefile-encoding/19878/8 "2021-11-01T16:43:29Z")

</div>

> [@justin.brzozoski](#):
>
> I’m trying to find the trick to convert `byte_result` in the following sample back to something that `system.file.writeFile` will accept as binary:

I think your missing piece is the `jarray` module, a special Jython builtin for exactly this kind of thing:

```auto
import jarray

bytes = bytearray(10)
print bytes
print jarray.array(bytes, 'b')

```

See the Jython book for more background:  
[https://jython.readthedocs.io/en/latest/DataTypes/?highlight=jarray#jython-specific-collections](https://jython.readthedocs.io/en/latest/DataTypes/?highlight=jarray#jython-specific-collections)

---

<div class="post-metadata">

**Author:** ![justin.brzozoski](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/justin.brzozoski/32/13452_2.png) [@justin.brzozoski](https://forum.inductiveautomation.com/u/justin.brzozoski)\
**Post date:** [November 1, 2021, 5:01pm UTC](https://forum.inductiveautomation.com/t/ign-4991-ign-4996-system-file-writefile-encoding/19878/9 "2021-11-01T17:01:23Z")

</div>

Still not quite there…

```auto
>>> whee = jarray.array(byte_result, 'b')
TypeError: Type not compatible with array type
>>> whee = jarray.array(bytearray(byte_result), 'b')
OverflowError: value too large for byte
>>> whee = jarray.array(list(byte_result), 'b')
TypeError: Type not compatible with array type

```

Looks like I’m not the only person stuck on this:  
[https://sourceforge.net/p/jython/mailman/message/35003179/](https://sourceforge.net/p/jython/mailman/message/35003179/)

---

<div class="post-metadata">

**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\
**Post date:** [November 1, 2021, 5:14pm UTC](https://forum.inductiveautomation.com/t/ign-4991-ign-4996-system-file-writefile-encoding/19878/10 "2021-11-01T17:14:00Z")

</div>

It seems wrong to me that `BytesIO.getvalue()` returns a `str`, which would make me lean toward a Jython bug here:

 ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/5/4/544ed79788c86051a51f2e010d4d030a2eb03d50.png)

---

<div class="post-metadata">

**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\
**Post date:** [November 1, 2021, 5:17pm UTC](https://forum.inductiveautomation.com/t/ign-4991-ign-4996-system-file-writefile-encoding/19878/11 "2021-11-01T17:17:58Z")

</div>

> [@justin.brzozoski](#):
>
> [[Jython-users] jarray broken in Jython 2.7.0? | Jython](https://sourceforge.net/p/jython/mailman/message/35003179/)

Interesting; that implicating it as a regression is odd. I'll file a separate ticket (I added a ticket for better `bytearray` support already) to investigate that Jython behavior.

---

<div class="post-metadata">

**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\
**Post date:** [November 1, 2021, 5:19pm UTC](https://forum.inductiveautomation.com/t/ign-4991-ign-4996-system-file-writefile-encoding/19878/12 "2021-11-01T17:19:00Z")

</div>

`BytesIO.getvalue()` returns a `bytes` type, which is just an alias for `str` in 2.6/2.7 🤷‍♂️

---

<div class="post-metadata">

**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\
**Post date:** [November 1, 2021, 5:27pm UTC](https://forum.inductiveautomation.com/t/ign-4991-ign-4996-system-file-writefile-encoding/19878/13 "2021-11-01T17:27:23Z")

</div>

Bleh, I always forget what a mess Python is in this area.  
@justin.brzozoski, it _may_ be worth looking at ditching `zipfile` for ‘native’ Java interaction with ZIP files. It’s more verbose, but mostly not too complicated.

---

<div class="post-metadata">

**Author:** ![justin.brzozoski](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/justin.brzozoski/32/13452_2.png) [@justin.brzozoski](https://forum.inductiveautomation.com/u/justin.brzozoski)\
**Post date:** [November 1, 2021, 5:44pm UTC](https://forum.inductiveautomation.com/t/ign-4991-ign-4996-system-file-writefile-encoding/19878/14 "2021-11-01T17:44:08Z")

</div>

Once I decided the built in ways were broken, I just built a bigger hammer:

```auto
def pythonByteString_to_jarray(pyString):
    return jarray.array(((x if x < 128 else (x - 256)) for x in bytearray(pyString)),'b')

```