[IGN-4437] Ignition occasionally forgets how Python works?

This is a difficult bug to describe and even harder to reproduce. But it's happening to me.

I have written Python scripts, custom classes, etc. for our site. 99.9% of the time, they run fine. But, every once in a while, while calling a database and transforming the returned dataset into linked custom class objects using a Python script, I get a strange error that is making me think that Ignition is not loading its Python scripts correctly somehow . . . ? Is that even possible?

So far, the error seems to come in two flavors:

  1. It tells me that one of my custom classes, which is a child class of another custom class, cannot be created because the first argument of its init function must be an instance of the parent class. This is nonsense, because this exact same code runs fine 99.9% of the time, and also because the first argument to init is an instance of the parent class (that kind of problem is easily corrected). It's as though, every so often and for no discernable reason, it doesn't understand my class hierarchy even though it had no problem with it literally just a few seconds before. Nothing changed in the interim.

  2. Even stranger, it has started to forget how Python date and timedelta objects work. Again, the code runs just fine almost all the time. But every so often, it throws an error to complain that "+" is not a valid operator between "date" and "timedelta" objects. Now, that's just plain wrong because of course you can add a timedelta to a date, that's the whole point of what a timedelta is. But more importantly, it can't be a syntax issue inside the code because the very same unchanged code runs just fine almost every time it's called. A genuine syntax issue as glaring as using a "+" in the wrong place should happen every time, but it doesn't. It's like the script interpreter on the server has suddenly forgotten how the datetime library works, or failed to load it this time, or . . . . ? I have no idea what's going on.

Reproducing the issue on command is impossible. It does it when it feels like it, and there doesn't seem to be a pattern. It strikes most often on pages which are set up to poll automatically for fresh data, but then again, those are the pages that make the most database calls, so you would expect them to be most vulnerable to odd behavior that strikes when transforming a dataset. It can happen on any page.

My experience as the developer and that of my users (these pages are viewed hundreds of times per day inside our organization) is that closing the tab on their browser and reloading the same page on a fresh tab is usually enough to make the error go away. That implies the users are not doing anything in particular to trigger the problem. In fact, it strikes on pages that load data automatically based on a pre-written named query that takes no parameters, then updates automatically with polling, so the user couldn't possibly be doing anything different when they reload than they did when they got the error. All they did was browse to the page.

And please, no responses about "you need to import datetime" or "you need to rewrite your class init". No I don't. This is not that kind of problem. This problem sometimes happens and sometimes doesn't even though the underlying code hasn't changed. It must have something to do with how Ignition is reading that code, not how it's written.

It didn't start throwing these errors until about a month ago. No, I don't know what changed then.

Has anyone else encountered this, or anything like it? Are there logs hidden away somewhere on the server I could look at to see what Ignition was doing when the trouble hit?

Good news: You're not crazy :slight_smile:

Yours is a somewhat different manifestation (most reporters are having issues locating project scripts) but my educated guess/hope is that it's ultimately the same bug internally. I suspect it has to do with the live-update of scripts 'attached' to each client session whenever changes are made in the designer; it might be helpful (for your sanity) to see if you can prove a correlation between saving changes in the designer and issues in live sessions.

1 Like

Your hunch seems to be right. After watching these events for the last day or so, I think I can confirm that the ones I see in the designer pop up a few seconds after saving changes--not always after I save changes, but when it happens, that's what I just did. I assume the ones my users see during the day also happen when I save, though that timing is harder to check.

For bonus points, auto-refreshing pages I left running overnight made it to morning just fine but threw errors shortly after I checked in on them, i.e., just after I started doing real work for the day. So there you go.

Thanks for the response. I'll keep my eyes open for any news about a fix.

Thanks. We're tracking this at a fairly high priority (plenty of folks are reporting it in the other thread), so it should get picked up soon.

3 Likes

This problem vanished for the last few months, but then reappeared this week. I haven't seen issues with timedelta again, but I'm back to getting the occasional error message claiming that a class' init() can't run because the first argument isn't an instance of the parent class (same class as before, incidentally).

Did something change to undo the solution that was, apparently, working for a while? Do I need to update?

The original ticket (4437, mentioned in the title of this thread) hasn't been picked up to be worked on. No one's done anything on the backend.

A fix that might help, though not specifically tagged (and thus, checked as fixing 4437) recently got merged into 8.1.32; final release should be within a couple of weeks.

1 Like

No luck. We are now at 8.1.32, but we are still seeing the issue. The folks on the other thread seem to be satisfied that 8.1.32 fixed what they were seeing, so maybe this is coming from some other root cause.

By the way, the description of the problem on the other thread differs a bit from what we are seeing in that they seem to encounter a permanent error until perspective is restarted. That's not what we're seeing. Our errors seem to be momentary; we might get a few in a row after a save, but within a few minutes we're back to normal. No need to restart anything. Or possibly another save knits everything back together.

It might be worth mentioning that the errors we're seeing are very consistent. There appear to be only the two flavors described in the first post. Even after all this time, it has never manifested in any other way. When an error arises, it's either the parent-child problem or the timedelta problem (which has returned). It's still definitely connected with saving work on the designer; now that we know to look for that the pattern is pretty obvious. And I can't prove this, but my gut says that saving changes to scripts in the project library triggers the issue, as opposed to saving changes limited to the UI and/or named queries. I will see if I can nail that part down as time goes on.