Integrating Ignition with AWS S3 via boto3

Ignition
1

I am FTPing camera inspection images into an S3 bucket. I am interested in using boto3 in scripts to SELECT images from my buckets based on metadata (datetime) and then visualize in an Ignition client. Visualization is not an issue. Just do not understand how to get the boto3 to work with Ignition.

I tried using the Python SDK (placing folders/files into the pylib) but no luck and the Java SDK (trying to create a module) is a little much for me.

Anyone has had any luck/or tried using the AWS S3 SDK known as boto3?

2

Boto3 was written from the ground up to provide native support in Python versions 2.6.5+, 2.7 and 3.3+.

Unless you're testing in Ignition 8 (Jython 2.7) there's no hope of this working in Ignition - and even then, I can't find any clear information about whether the SDK is "pure" Python or actually relies on "native" C-based code (ie, CPython 2.7 would work, but Ignition's embedded Jython 2.7 would not).

3

Thanks for the reply. I had pretty much ran into a wall and you have now confirmed what I was thinking.

To provide additional information, I am working with a Ignition 7.9 running on a Windows 2008 (SP2) instance on AWS.

The work around was to use Cloudberry which allows the manipulation of the S3 objects as files stored in a mapped network drive. I then made the necessary changes to ignition.conf drive (as detailed in IA´s documentation) and I can now run the os library directory/file functions (gateway script) on it.

Furthermore, Filezilla FTP Server allows me to point at the S3 mapped directory structure and save files as is if I was saving directly to the server instance. So it now appears to work.

I am still testing and will provide additional information within the next two weeks.

Thanks again.

1 Like
4

I wasn’t sure whether to create a new thread or bring back this one, but I am basically trying to do the same thing in Ignition 8 (S3 & Sitewise). I haven’t seen much yet in regards to Jython compatibility but will update if I find something one way or the other.

1 Like
5

Could you update us on this, please?

6

For those wondering, boto3 does work on Ignition 8. You just need to place the site-packages created by building boto3 in jython into your Ignition gateway’s site-packages folder. Don’t forget to create a credentials and config file on the gateway as well. Then use import boto3 and start using it in gateway scoped scripting.

Use this links for reference

Screenshot of all the files created by building boto3 in jython.

image
image639Ă—770 49.6 KB

2 Likes
7

Thanks, this is helpful info, I will give it a try.

8

I attempted to do this, but am getting errors on using the multiprocessing module. This was supposed to be included in python 2.5, but potentially the module uses C extensions?

  File "C:...\pylib\site-packages\s3transfer\compat.py", line 101, in <module>
    from multiprocessing.managers import BaseManager
ImportError: No module named multiprocessing

Would you be able to provide more direction on how you were able to accomplish this?

9

For reference, the script did not error on import boto3, but on creating a client.

import boto3

client = boto3.client('s3') # error here
10

Did you download Jython 2.7 to build or Python? Because your folder path says \pylib…mine is just Lib

image

I didn’t try with Python. Just know that Jython worked the first time without any issues.

11

That is the filepath on my computer when it pulls the py modules from the Ignition gateway, but yes I did use python to install. Java runtime environment is strongly discouraged at my organization, so I can’t open .jar files. Any idea if OpenJDK can work for this purpose?

As a note, the previous version of boto3 (boto) can be installed and is compatible with python 2. We’ll see if this works out as well…

12

I can’t upload a zip file of my site-packages folder…its 15MB. max size allowed to upload is 4MB. Maybe message me and I can send you the zip some other way.

13

There are a number of changes that have to be made to the source code to get it running. Jython version must be 2.7.2 for urllib3’s ssl to work. There are some instances of unsupported f’{}’ style string interpolation that have to be changed to .format interpolation. Multiprocessing must be included in pylib, and that lib has to be modified in the patch.py file with a number of changes (OR S3 has to be rewritten to use threading which will work fine with no GIL). Futures is a required module that is unlisted generally. For S3, this is the minimum level of patching. For general lib (botocore) stuff, you can get by without the multiprocessing stuff. Also, the xerces saxparser issue has to be fixed to parse S3 and DynamoDB output; this has been solved on another thread here.

The most important thing is Boto3 is designed to be as lazy as possible, and uses factories at runtime to generate the Classes you use. This means getting it to import is not the same as getting it to run, you have to explore the different clients and resources at runtime to catch all the bugs.

14

@tssmetan do you have a working version of boto3 that you’d be willing to collaborate on? I’ve started a repo to try to make it easier to import modules into ignition, and boto3 would be great to include.

15

I’m not sure what functionality is missing (since it’s deprecated), but here’s a way to get the last version of boto3 that supports Python 2.7:

git clone https://github.com/boto/boto3.git
cd boto3
git checkout 5776722
jython setup.py install

But you can at least get this far without errors:

import boto3

client = boto3.client('kinesis', region_name='us-east-1')
1 Like
16

The easiest way I have found to do this is to not use the API but use the AWS CLI instead. Ignition gives you the ability to execute commands, system.util.execute(). I put all my S3 operations in a bash script and call them as I need them from inside Ignition. This has been pretty convenient in a Linux environment; I have not tried it in Windows.