# system.net.httpClient().request() is doing some uncommanded encoding on my binary data **URL:** \ **Category:** Ignition\ **Tags:** ignition81\ **Created:** [March 28, 2021, 10:02pm UTC](https://forum.inductiveautomation.com/t/system-net-httpclient-request-is-doing-some-uncommanded-encoding-on-my-binary-data/45003 "2021-03-28T22:02:13Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![mmunin](https://avatars.discourse-cdn.com/v4/letter/m/c0e974/32.png) [@mmunin](https://forum.inductiveautomation.com/u/mmunin)\ **Post date:** [March 28, 2021, 10:02pm UTC](https://forum.inductiveautomation.com/t/system-net-httpclient-request-is-doing-some-uncommanded-encoding-on-my-binary-data/45003/1 "2021-03-28T22:02:13Z")
I would like to have my Vision client user select a file, and then use `system.net.httpClient()` to POST the contents of that file via HTTP. According to the documentation, the following should work. ```python h = system.net.httpClient() path = 'C:\pathtomyfile' bytes= system.file.readFileAsBytes(path).tostring() # returns a byte string of the file data print(len(bytes)) # sanity check to make sure that our byte string has been imported correctly into memory h.request("http://example.com","POST",data=bytes) ``` When I try this with a 2x2 pixel PNG image file, the script outputs the correct number of bytes, which is 128. However, my files are ending up at my back end system corrupted. This corruption applies to all files except ones that are valid ASCII. I have HTTP Toolkit installed, and when I configure `system.net.httpClient()` to use it as a proxy, I can see from the payload hex dump that the HTTP request sent to my back end system was 152 bytes. For ASCII files, these numbers match. Does anyone know why it might be doing this, and if there’s a work around? ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [March 28, 2021, 10:05pm UTC](https://forum.inductiveautomation.com/t/system-net-httpclient-request-is-doing-some-uncommanded-encoding-on-my-binary-data/45003/2 "2021-03-28T22:05:01Z")
> [@mmunin](#): > > bytes= system.file.readFileAsBytes(path).tostring() You're doing it at this step. You can't go from arbitrary binary to UTF-8 back to binary and expected it to survive intact. Try Base64 encoding your payload before sending it and Base64 decoding on the backend. edit: or just don't call `tostring()` first, the data parameter to the POST can be a byte array. ---
**Author:** ![mmunin](https://avatars.discourse-cdn.com/v4/letter/m/c0e974/32.png) [@mmunin](https://forum.inductiveautomation.com/u/mmunin)\ **Post date:** [March 28, 2021, 11:43pm UTC](https://forum.inductiveautomation.com/t/system-net-httpclient-request-is-doing-some-uncommanded-encoding-on-my-binary-data/45003/3 "2021-03-28T23:43:38Z")
According to the Python 2.7 [documentation](https://docs.python.org/2.7/library/array.html#array.array.tostring), `tostring()` will return a string of bytes, as though it were being written as a file. Since posting this issue, I have done an additional test. When trying to send a UTF-8 encoded file that is NOT ASCII compatible, there is still corruption of the data **after** the call to `tostring()` When the file has text `été` ecoded to UTF-8 as `C3 A9 74 C3 A9`, the call to `len(bytes)` returns 5 as expected. However, my HTTP Toolkit is intercepting a payload of `c3 83 c2 a9 74 c3 83 c2 a9`. The fact that `len(bytes)` showed 5 and not 9 leads me to believe that the `tostring()` method is not doing any undesired en/decoding. According to [this](https://www.i18nqa.com/debug/bug-utf-8-latin1.html) page, these specific results I see are consistent with the data being subject to a conversion from ISO-8859-1 to UTF-8. It seems as though passing the bytes as a string will subject them to some decoding/encoding regardless. The only byte strings I can put through there that come out unchanged are ones that are only `chr(0)` to `chr(127)` The only reason I’m even using `tostring()` is because I did not observe correct behavior _at all_ when trying to pass the byte array directly (by removing `.tostring()` from line 3 above). When I do, the HTTP request has `content-type: application/json; charset=utf-8` with json content `[-61, -87, 116, -61, -87]` When I was first trying this, I thought that I was doing something wrong by passing the byte array directly, so I started looking for other solutions. It would seem the “right” way isn’t working for me. Any ideas? ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [March 28, 2021, 11:49pm UTC](https://forum.inductiveautomation.com/t/system-net-httpclient-request-is-doing-some-uncommanded-encoding-on-my-binary-data/45003/4 "2021-03-28T23:49:31Z")
Try passing the byte array as the data parameter and explicitly setting the content-type header to `application/octet-stream`. It sounds like the content type is not being automatically detected and set as it should be. ---
**Author:** ![mmunin](https://avatars.discourse-cdn.com/v4/letter/m/c0e974/32.png) [@mmunin](https://forum.inductiveautomation.com/u/mmunin)\ **Post date:** [March 28, 2021, 11:50pm UTC](https://forum.inductiveautomation.com/t/system-net-httpclient-request-is-doing-some-uncommanded-encoding-on-my-binary-data/45003/5 "2021-03-28T23:50:40Z")
Unfortunately, trying this leaves the payload unchanged. (Except now HTTP Toolkit won’t let me view it as JSON) That is to say, the binary content of the HTTP request is still the same UTF-8 encoded JSON ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [March 29, 2021, 12:00am UTC](https://forum.inductiveautomation.com/t/system-net-httpclient-request-is-doing-some-uncommanded-encoding-on-my-binary-data/45003/6 "2021-03-29T00:00:37Z")
I’m not sure I understand what you are trying to do any more. Are you trying to post JSON to an endpoint or post binary content (like the PNG file from your first post)? ---
**Author:** ![mmunin](https://avatars.discourse-cdn.com/v4/letter/m/c0e974/32.png) [@mmunin](https://forum.inductiveautomation.com/u/mmunin)\ **Post date:** [March 29, 2021, 12:07am UTC](https://forum.inductiveautomation.com/t/system-net-httpclient-request-is-doing-some-uncommanded-encoding-on-my-binary-data/45003/7 "2021-03-29T00:07:06Z")
My apologies, I am attempting to post binary data. For debugging purposes, I have since switched to a much simpler file so I can inspect the hex data directly. In HTTP Toolkit, if a request has the JSON content-type (what I got without providing the octet-stream header), it gives you a little browser to navigate the JSON. The only difference setting the octet-stream header did was change the content-type in the request. the data was identitcal: 21 bytes for the 21 characters in `[-61,-87,116,-61,-87]`. I’m not sure if you have an HTTP debugger tool like HTTP Toolkit, but you could try the following equivalent code without any need for files to see if you can replicate it. ```auto h = system.net.httpClient(proxy="http://localhost:8000") # HTTP Toolkit is running on port 8000 on the same host as this designer/client import jarray bin = [-61, -87, 116, -61, -87] # 5 bytes of binary data that encode "été" in UTF-8 bytes = jarray.array(bin, 'b') # create a Jython Byte[] print bytes h.request("http://example.domain","POST",data=bytes,headers={"content-type":"application/octet-stream"}) ``` ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [March 29, 2021, 12:19am UTC](https://forum.inductiveautomation.com/t/system-net-httpclient-request-is-doing-some-uncommanded-encoding-on-my-binary-data/45003/8 "2021-03-29T00:19:27Z")
Okay, sorry, the problem is that `system.net.httpClient` is utterly broken when it comes to trying to use binary data. I think we’ll have to fix it. As a workaround you can do something like this: ```auto from org.python.core import PyByteArray h = system.net.httpClient(proxy="http://localhost:8000") # HTTP Toolkit is running on port 8000 on the same host as this designer/client import jarray bin = [-61, -87, 116, -61, -87] # 5 bytes of binary data that encode "été" in UTF-8 bytes = jarray.array(bin, 'b') # create a Jython Byte[] data = PyByteArray(bytes) h.request("http://example.domain","POST",data=data,headers={"content-type":"application/octet-stream"}) ``` ---
**Author:** ![mmunin](https://avatars.discourse-cdn.com/v4/letter/m/c0e974/32.png) [@mmunin](https://forum.inductiveautomation.com/u/mmunin)\ **Post date:** [March 29, 2021, 12:21am UTC](https://forum.inductiveautomation.com/t/system-net-httpclient-request-is-doing-some-uncommanded-encoding-on-my-binary-data/45003/9 "2021-03-29T00:21:49Z")
Thank you so much! I now officially count you among my top 2 favorite Kevins! But… I should let you know, your workaround preserves the data but adds a bunch of leading and trailing 0’s to it, no clue why. If you had to guess, will this be fixed in 8.1.4 or 8.1.5? ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [March 29, 2021, 12:56am UTC](https://forum.inductiveautomation.com/t/system-net-httpclient-request-is-doing-some-uncommanded-encoding-on-my-binary-data/45003/10 "2021-03-29T00:56:25Z")
Certainly not 8.1.4, possibly 8.1.5. I opened an issue for it that should get some attention this week. ---
**Author:** ![mmunin](https://avatars.discourse-cdn.com/v4/letter/m/c0e974/32.png) [@mmunin](https://forum.inductiveautomation.com/u/mmunin)\ **Post date:** [September 22, 2021, 5:41pm UTC](https://forum.inductiveautomation.com/t/system-net-httpclient-request-is-doing-some-uncommanded-encoding-on-my-binary-data/45003/11 "2021-09-22T17:41:07Z")
![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/5/6/56026aa74a50e11d176fd71acde25774c705d623.png) was this the fix? ---
**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\ **Post date:** [September 22, 2021, 6:13pm UTC](https://forum.inductiveautomation.com/t/system-net-httpclient-request-is-doing-some-uncommanded-encoding-on-my-binary-data/45003/12 "2021-09-22T18:13:03Z")
No, actually, it’s this: [https://inductiveautomation.com/downloads/releasenotes/8.1.9#17176](https://inductiveautomation.com/downloads/releasenotes/8.1.9#17176) ---
**Author:** ![Trent.Boudreaux](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/trent.boudreaux/32/4138_2.png) [@Trent.Boudreaux](https://forum.inductiveautomation.com/u/Trent.Boudreaux)\ **Post date:** [January 28, 2022, 5:09pm UTC](https://forum.inductiveautomation.com/t/system-net-httpclient-request-is-doing-some-uncommanded-encoding-on-my-binary-data/45003/13 "2022-01-28T17:09:19Z")
I’m seeing this behavior when using python’s [built-in open() function](https://docs.python.org/2.7/library/functions.html#open) in Ignition 8.1.11 ```auto fileReader = open(filePath,"rb") fileData = fileReader.read() fileReader.close() ... client = system.net.httpClient() response = client.put(url=urlCurr, data=fileData, headers=headerVals) ``` However, I was able to work around the issue by switching to system.file.readFilesAsBytes() ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [January 28, 2022, 5:17pm UTC](https://forum.inductiveautomation.com/t/system-net-httpclient-request-is-doing-some-uncommanded-encoding-on-my-binary-data/45003/14 "2022-01-28T17:17:40Z")
When there are jython functions and java functions that do the same things, use the java functions. jython’s stdlib has many flaws due to its python2 heritage. Encoding problems are one of them.