[IGN-4437] Ignition occasionally forgets how Python works?

This is a difficult bug to describe and even harder to reproduce. But it's happening to me.

I have written Python scripts, custom classes, etc. for our site. 99.9% of the time, they run fine. But, every once in a while, while calling a database and transforming the returned dataset into linked custom class objects using a Python script, I get a strange error that is making me think that Ignition is not loading its Python scripts correctly somehow . . . ? Is that even possible?

So far, the error seems to come in two flavors:

  1. It tells me that one of my custom classes, which is a child class of another custom class, cannot be created because the first argument of its init function must be an instance of the parent class. This is nonsense, because this exact same code runs fine 99.9% of the time, and also because the first argument to init is an instance of the parent class (that kind of problem is easily corrected). It's as though, every so often and for no discernable reason, it doesn't understand my class hierarchy even though it had no problem with it literally just a few seconds before. Nothing changed in the interim.

  2. Even stranger, it has started to forget how Python date and timedelta objects work. Again, the code runs just fine almost all the time. But every so often, it throws an error to complain that "+" is not a valid operator between "date" and "timedelta" objects. Now, that's just plain wrong because of course you can add a timedelta to a date, that's the whole point of what a timedelta is. But more importantly, it can't be a syntax issue inside the code because the very same unchanged code runs just fine almost every time it's called. A genuine syntax issue as glaring as using a "+" in the wrong place should happen every time, but it doesn't. It's like the script interpreter on the server has suddenly forgotten how the datetime library works, or failed to load it this time, or . . . . ? I have no idea what's going on.

Reproducing the issue on command is impossible. It does it when it feels like it, and there doesn't seem to be a pattern. It strikes most often on pages which are set up to poll automatically for fresh data, but then again, those are the pages that make the most database calls, so you would expect them to be most vulnerable to odd behavior that strikes when transforming a dataset. It can happen on any page.

My experience as the developer and that of my users (these pages are viewed hundreds of times per day inside our organization) is that closing the tab on their browser and reloading the same page on a fresh tab is usually enough to make the error go away. That implies the users are not doing anything in particular to trigger the problem. In fact, it strikes on pages that load data automatically based on a pre-written named query that takes no parameters, then updates automatically with polling, so the user couldn't possibly be doing anything different when they reload than they did when they got the error. All they did was browse to the page.

And please, no responses about "you need to import datetime" or "you need to rewrite your class init". No I don't. This is not that kind of problem. This problem sometimes happens and sometimes doesn't even though the underlying code hasn't changed. It must have something to do with how Ignition is reading that code, not how it's written.

It didn't start throwing these errors until about a month ago. No, I don't know what changed then.

Has anyone else encountered this, or anything like it? Are there logs hidden away somewhere on the server I could look at to see what Ignition was doing when the trouble hit?

1 Like

Good news: You're not crazy :slight_smile:

Yours is a somewhat different manifestation (most reporters are having issues locating project scripts) but my educated guess/hope is that it's ultimately the same bug internally. I suspect it has to do with the live-update of scripts 'attached' to each client session whenever changes are made in the designer; it might be helpful (for your sanity) to see if you can prove a correlation between saving changes in the designer and issues in live sessions.

1 Like

Your hunch seems to be right. After watching these events for the last day or so, I think I can confirm that the ones I see in the designer pop up a few seconds after saving changes--not always after I save changes, but when it happens, that's what I just did. I assume the ones my users see during the day also happen when I save, though that timing is harder to check.

For bonus points, auto-refreshing pages I left running overnight made it to morning just fine but threw errors shortly after I checked in on them, i.e., just after I started doing real work for the day. So there you go.

Thanks for the response. I'll keep my eyes open for any news about a fix.

Thanks. We're tracking this at a fairly high priority (plenty of folks are reporting it in the other thread), so it should get picked up soon.

4 Likes

This problem vanished for the last few months, but then reappeared this week. I haven't seen issues with timedelta again, but I'm back to getting the occasional error message claiming that a class' init() can't run because the first argument isn't an instance of the parent class (same class as before, incidentally).

Did something change to undo the solution that was, apparently, working for a while? Do I need to update?

The original ticket (4437, mentioned in the title of this thread) hasn't been picked up to be worked on. No one's done anything on the backend.

A fix that might help, though not specifically tagged (and thus, checked as fixing 4437) recently got merged into 8.1.32; final release should be within a couple of weeks.

1 Like

No luck. We are now at 8.1.32, but we are still seeing the issue. The folks on the other thread seem to be satisfied that 8.1.32 fixed what they were seeing, so maybe this is coming from some other root cause.

By the way, the description of the problem on the other thread differs a bit from what we are seeing in that they seem to encounter a permanent error until perspective is restarted. That's not what we're seeing. Our errors seem to be momentary; we might get a few in a row after a save, but within a few minutes we're back to normal. No need to restart anything. Or possibly another save knits everything back together.

It might be worth mentioning that the errors we're seeing are very consistent. There appear to be only the two flavors described in the first post. Even after all this time, it has never manifested in any other way. When an error arises, it's either the parent-child problem or the timedelta problem (which has returned). It's still definitely connected with saving work on the designer; now that we know to look for that the pattern is pretty obvious. And I can't prove this, but my gut says that saving changes to scripts in the project library triggers the issue, as opposed to saving changes limited to the UI and/or named queries. I will see if I can nail that part down as time goes on.

This is still happening--if anything, it's suddenly gotten worse over the last few days. I am always getting the same error--the gateway scripts have classes that form an inheritance hierarchy, and suddenly one of them will say that the init() function can't run because it was given the wrong kind of class. Again, it is definitely the right kind of class because the code was running fine a moment before.

It has started to happen even when I'm not writing and saving changes. I got on this morning to work and discovered a series of messages from users asking what was wrong with Ignition. I looked at the logs and it's the same trouble, except that nobody was working on anything before I arrived this morning. So now it looks like it can just happen, no working and saving of changes necessary. I restarted the gateway, which fixed it for about an hour, but then it came back.

I did find a temporary fix: I went back into the file for the class it was claiming it couldn't load and added a single space character. That didn't change anything about how the scripts work, but it made it just different enough to force it to re-save that file when I clicked Save. And it worked! After that save, the problem stopped.

The only other insight I can offer is that the two types of errors we experience have something in common. In one type, it complains that it can't add a Python timedelta object to a Python date. In the other, it complains that it can't run the init() of a Python class because it's been given the wrong kind of class. Those two seem pretty different; but the Python class it is trying to instantiate makes use of the datetime and timedelta in some of its functions. Maybe that's the common element here? We make relatively little use of the datetime library and the timedelta object outside of these classes and some related and supporting classes, so that might explain why errors happen here and not anywhere else. I'd really prefer not to go in and re-write all of those classes (there are dozens) to use system.date, but if this explanation sounds plausible to someone in the know, I might have to try.

We are on 8.1.36 now. I will upgrade to the latest in hopes that might fix something. Otherwise, I don't know. Getting frustrated.

You mention using python's timedelta. Don't do that. Don't use any python stdlib module that has java alternatives.

Are you importing across project library scripts? If so, don't do that. Only use import with java classes and stdlib and site-packages. Outside of a particular project library script, refer to its objects only with fully-qualified names.

Report back when you've fixed any such flaws.

2 Likes

If you're not already in contact with support officially, definitely do so.
Your best bet at this point is for us to be able to replicate whatever is going on here in-house so that we can find the exact root cause and get it fixed. The fact that you're the only one encountering it unfortunately leads to the conclusion that you're (in some way) doing something Ignition and/or Jython aren't happy with. Which suggests at least starting with Phil's advice.

2 Likes

I have systematically eliminated all references to Python's datetime library throughout my project. The phrase "import datetime" no longer exists anywhere in the project library or in any view, and neither does "timedelta".

I had two trouble-free days, but as of this afternoon, the error was back:

(To re-emphasize, despite what it looks like, there is nothing wrong with the Python code. The inheritance hierarchy between Task and QCTask is written correctly and runs without errors almost every time it's called. But every so often, and without making any changes to that code, these errors just appear, usually but not always around the same time as I save ongoing work. They last for anywhere between a few seconds and half an hour, then seem to resolve themselves. This is not a minor annoyance; the errors are being thrown by objects so fundamental that users are basically paralyzed for however long this state lasts.)

Let me make sure I am understanding pturmel's comment. I use import a lot, but I can (now) say for certain that I do not use it to bring in any outside libraries or invoke anything I haven't written myself.

I only use "import" to bring variables from one script in the Project Library to a script somewhere else. For example, if I had a database table with a column called "xyzID", I would make a "ColumnNames" file in the Project Library that says "XYZ_ID_COLUMN_NAME = 'xyzID'". Then, when working on another script that is going to need to call and parse that database table, I might begin the script by saying "from ColumnNames import *", so then I can do things like "xyzID = dataset.getValueAt(i, XYZ_ID_COLUMN_NAME)". Is that an acceptable use of "import"?

How do I get in contact with support officially? Can someone refer me and bring this comment thread over so they come up to speed?

I recently had an issue with this style of importing variables while developing a new project where I have configs at project.common.configs. Changing from from project.common.configs import SOME_VAR to SOME_VAR = project.common.configs.SOME_VAR corrected my issue.

The best part? An identical call in another file in the same project didn't have any issue importing the same variable using the from x import y syntax

1 Like

No.

Everything in the project library will be automatically "imported" for you, no need to write explicit imports.
If you want to achieve short form referencing locally, rebind using references as @ryan.white just suggested, but do not use import.

https://support.inductiveautomation.com/hc/en-us
Send them an email (support@inductiveautomation.com) and reference this thread; there'll be some initial bookkeeping to ensure you have a support contract in place.

1 Like

Edit: Responded with looking other responses. What Paul said.

No. You are importing project library scripts. This is broken. Let me repeat and emphasize:

I strongly recommend this only be done within functions and methods, not at the top level of a library script module.

2 Likes

Even for a variable/constant that is going to be used by multiple functions/methods in the module?

Even then.

1 Like

So just to be clear on the Dos and Don'ts:

Let's say I make a file in the Project Library called "Constants" to store constants, like database column names and named query paths. Let's say it has a variable called "GET_PATH" that is a simple query to the database and a variable called "XYZ_ID_COLUMN_NAME" for the name of one of the columns in the table. (This is my most common use case for this sort of thing; it saves me a lot of typing, no chance of misspelling a column name, and if the column names ever change I can fix that everywhere with one edit).

Elsewhere, in another Project Library script or in a script in a view, I need to make a database call using a named query, something along the lines of

system.db.runNamedQuery(GET_PATH, {XYZ_ID_COLUMN_NAME:variable})

How?

From the discussion above, I know I should not do this:

from Constants import *
[runs query]

But which of these are acceptable?

Option 1:

from Constants import GET_PATH
from Constants import XYZ_ID_COLUMN_NAME
[runs query]

Option 2:

GET_PATH = Constants.GET_PATH
XYZ_ID_COLUMN_NAME = Constants.XYZ_ID_COLUMN_NAME
[runs query]

Option 3:

system.db.runNamedQuery(Constants.GET_PATH, {Constants.XYZ_ID_COLUMN_NAME:variable})

Tip: you are posting all your code as quotations (using >). Format them as code using the </> button instead. It will give fixed width font and preserve indents.
See Wiki - how to post code on this forum for more.

Option 1 is just as bad as any other problem import.

Option 2 would be fine if not at the top level of a script, but I consider it a bad practice.

Option 3. Always and everywhere.

Better would be to gather the functions that use those constants into the same script, then call the function with a fully-qualified name.

4 Likes