Procedure / Recipe system using SFCs

PREFACE

In what is my most ambition Ignition project yet, I am trying to build a recipe system that dynamically loads variables into SFC-based procedures (the SFCs are not generated dynamically). I started here and it rapidly grew out of the scope of any examples that I could find. I am running into lots of roadbumps of various jaggedness, but so far there are only a few I can't figure out. I've seen relatively little information about a project like this on the forums, so I'm laying out my project a bit more verbosely than might be strictly necessary for my immediate problems. Most of this writeup is for my own cognitive benefit, but if it helps anyone else that's great too!

To start: I try to develop in project dev followed by export to project base. I then discard overrides and save again (QOL REQUEST: when I export something to the project that it inherits from, please automatically "Discard Overrides" since they should at that point be identical). From that point, I am under the impression that dev should behave the same as base (but it doesn't always actually do this?)

Current test setup: I have a recursive SFC (and all the affilliated DB / Transaction group stuff, described below) that I am using to test my ability to have SFCs call other SFCs. I successfully had it functioning as expected until I exported the SFC from my dev project into its parent base project. Now, when I run it from the dev project without modifications (other than the required override to even access it in the Designer), its child SFCs are all not showing up in the Chart Control panel of the dev project. Upon investigation, it seems that they DO show up in the base Designer's Chart Control!

This direct misbehavior was the impetus for this whole post (article?). But then I started writing :man_shrugging:

PROBLEMS (and gotchas, to be expanded and resolved over time)

  1. I am confused about what projects are supposed to own SFCs. I understand that they run in the gateway entirely, and you have to jump through hoops to have them interact with a client via system commands (system.sfc.*Variables()). So I am perplexed that they seem to be owned by different projects.
  2. Enclosings don't seem to be able to be used dynamically (can't set path by reference), so I have start and monitor the SFC via scripts in Action blocks, which means that the script engine is now involved in the administration of the SFCs themselves (unless these are the same engines?). I was hoping I could get further with the built-in SFC capabilties.
  3. Debugging SFC errors is very annoying, the Chart Control shows like the first 12 characters of the error and the Gateway Logs basically only tell me which block was misbehaving with a stack that is very obtuse.
  4. Putting tables in another schema other than the default seems to break the ability of the Database Query Browser to generate queries quickly via double clicking them (doesn't include the schema), it makes it more annoying to find the recipes in the Table Browser because they arent alphebetized and can't be resorted, and also caused another problem that I have already forgotten because I gave up on this attempted relatively quickly.
  5. Enclosings default to "Cancel" mode, but so far I have almost always wanted them to be in "Blocking" mode. Not toggling this correctly has resulted in a lot of hair pulling for me.

PROJECT

I'll use my basic recursiveSoftwareTest_v1 SFC as an example. It loads some variables, waits 3 seconds to pretend to do work, calls another instance of itself, waits for that instance to return, waits another 3 seconds to pretend to do more work, then exits. The net result of calling this with a max_depth of 3 is that I have a root level SFC that runs in very close to exactly 18 seconds.

Because of problem 2), I have a couple of blocks that I need to copy paste around into all of the procedures that call other SFCs. First one loads (see load) tag data from some structured folders so that the block is agnostic to whatever SFC it is in. Next one runs a child object (see run_chidl) via system.sfc.startChart, and grabs / handles its instanceId so that it can be monitored, and the final one is an enclosing that loops until the chart identified with that instanceId is no longer in a state like "Running" "Starting" or "Initial" (see await_child).

DETAILS
image

# load
def onStart(chart, step)
  # Load the parameters for this procedure
  # they are pydatasets need to derefrence w/ [0]
  params = [x[0] for x in system.db.runNamedQuery(
      "base", "get_procedure_params", 
      {"procedure_name": chart.procedure_name})
  ]
  # Load the tag values for this procedure 
  # RECIPE_ID NEEDS TO BE SELECTED BEFOREHAND TO LOAD CORRECTLY
  # I know that this will only let me use a single loaded recipe per procedure
  # Thats a problem for later
  param_tags = [
      "[HMI]SFC/procedures/{}/{}".format(chart.procedure_name, p) 
      for p in params
  ]
  param_tag_values = system.tag.readBlocking(param_tags)
  # Connect those values to this chart
  for i, p in enumerate(params):
  	setattr(chart, p, param_tag_values[i].value)
# run_child
def onStart(chart, step)
	if chart.current_depth+1 < chart.max_depth:
		chart.child_chart_id = system.sfc.startChart(
			"base", "procedures/{}".format(chart.child_chart), {
	            "procedure_name": chart.child_chart, 
	            "current_depth": chart.current_depth+1
	        }
		)
	else:
		chart.child_chart_id = None

await_child
image

# init
def onStart(chart, step)
  chart.child_chart = chart.child
  chart.child_chart_id = chart.parent['child_chart_id']
  chart.logger = chart.parent.get('logger', None)
  chart.child_chart_done = False
  chart.n_awaits = 0

#await_child
def onStart(chart, step)
	if type(chart.child_chart_id) is type(None):
		chart.child_chart_done = True
		return
		
	charts = system.sfc.getRunningCharts("procedures/{}".format(chart.child_chart))
	coi = [{"state": charts.getValueAt(row, "chartState"), 
	        "id": charts.getValueAt(row, "instanceId")} 
		    for row in range(charts.rowCount) 
	    	if chart.child_chart_id == charts.getValueAt(row, "instanceId")
	]
		
	if any([x == coi[0]['state'].toString() for x in ['Initial', 'Starting', 'Running']]):
		chart.child_chart_done = False
	else:
		chart.child_chart_done = True

For storing my procedure and recipe specifications, I have a procedure table and
recipes_{procedure_name} tables. The procedure table contains the procedure_name as well as a JSON object representing the column names and types of the parameters in the procedure. A set of scripts generates the procedure entries and recipe tables from externally managed CSV files. The active parameters for a given SFC procedure is held in a tag folder that is linked to the database via a Transaction Group triggering on the recipe_id of a given procedure. This part works well enough so far, although programming the transaction groups has been a bit annoying as it requires generating them in XML and importing them via the legacy-ish system.groups.loadFromFile. I will eventually need a view system for controlling versions, probably.

# get_procedure_params
SELECT column_name
FROM information_schema.columns
WHERE table_name = 'recipes_' || LOWER(:procedure_name) 
	AND column_name not like '%_ndx' 
	AND column_name not in ('t_stamp', 'procedure_name')

Example procedure entry + recipe table:
procedures

recipes_softwareTestRecursive_v1

tags
image


Congratulations if you got this far! Hopefully this was delivered in a comprehensible manner, I promise it was not written linearly.

My current main question is about problem 1), with project interitance. But I also am here for comments and (constructive) critiques. Especially with regards to the SFC system in general. I feel like it is not as polished as it could be, and circumventing its shortcomings consumes a large fraction of my attention for this project.

1 Like

This confusion is mostly an artifact of time.
When SFCs were designed, Ignition's project system worked differently - there was one special "global" project that held certain types of resources, and individual projects that held other types of resources. In 8.0, we simplified that process by removing any "special" kinds of projects or resources - any project can hold any kind of resources, and projects were/are now allowed to "inherit" from one another.

SFCs are a little weird, though, because as you encountered, it doesn't really make sense to "inherit" an SFC from another project, and also any inheritable project will not run its own SFCs.
The general pattern I'd recommend is to just have a single "SFC" project that holds your SFCs, and any associated scripting libraries, and that's about it.

You're not wrong. The SFC module hasn't really received a lot of developer time since its introduction; not for any particular reason, just the nature of things.
There's a dedicated group of users who make great use of SFCs, but there's (anecdotally) a larger group of folks who either don't start using SFCs or give up on SFCs quickly in favor of basically just scripting the whole thing... which has its own developer experience tradeoffs.

It would certainly be good for us to improve SFCs. You're more than welcome to file particular suggestions on our Ideas portal (https://ideas.inductiveautomation.com) - I promise we read them, even if we don't always get to them promptly. Stuff like improved indirection support seems like "low hanging fruit" that could pay dividends, for sure.

3 Likes

Can I inherit from multiple projects without collisions? The thing is I have all my scripts in base as well, and the SFCs utilize those. Do I have sfc_base inherit base and then have everything else inherit sfc_base?

I will do that! Most of the things I stated picking up are too small for "feature requests" (more in the line of ease of use) but I can definitely dump my notes over there. As always, thanks for pointers and the historical context.

Anecdote for you: This developer (me) despises the whole concept of SFCs, from long before IA existed. I won't implement anything with them. If you ask me to work on them, the only task I'll accept is their wholesale replacement with clean state-machine code, whether ladder logic or python. Absolutely despise them.

1 Like

There's no multiple inheritance - it's strictly parent -> (potentially multiple) children. You can have as long a chain as you want, but no multiple inheritance. There are no plans to change this first party, but some new functionality arriving in 8.3 should make it possible for enterprising folks like Phil (multiple inheritance is another bugbear of his) to write a module that "emulates" multiple inheritance across standard projects, presenting a new multiply-inherited project. In theory :slight_smile:

So, yes, you could have core stuff in base, then an sfc_base, then as many SFC "leaf" project as you want - but nothing more complicated than that linear chain.

1 Like

Are the SFCs in Ignition not clean state machine code? I am admittedly relatively new to the PLC-world style of programming but SFCs are one of five sanctioned "languages" in IEC 61131-3.

SFCs offer us the value of having process engineers sketch what they want and have it more or less translate directly to something they can see en vivo.

I see, ok. I might try to do this. Something like base_script -> base_sfc -> base(ist)?

SFCs are graphics that pretend to be state machine code, and occasionally, with great attention to detail, succeed. The IEC sanction of this language is to its everlasting shame.

Is there another state machine standard that would be more appropriate to use?

I recommend ladder logic for almost all control tasks, and I count state machines as control tasks. Since Ignition doesn't have ladder logic, I write such code in jython.

{ Not to disrespect Function Blocks. Many process loops are well-represented by Function Block notation. But they aren't properly considered state machines. }

2 Likes

Since Ignition doesn't have a Batch Engine it seems SFC's are the easiest way to get some procedural / state-based control over equipment. Could you explain in broad terms (or specifically) how you'd replicate some of the functionality within jython? Would this be using a Gateway script to monitor / perform actions. If I have a long batch process where parameter and process changes are needed sometimes days after starting is there simplistic way to handle this via scripting?

Good day, Phil,
Full disclosure, I see your comments on forums very often and sincerely value your opinions and admire your experience and knowledge base.

I work with batch processes, and I see Ignition SFC as a not bad orchestra director tool that could execute different pieces of process logic whether it is the PLC code in a shape of ladder, ST OR in Ignition itself in a form of scripts and simple tag read/writes.
It is very clear that the critical logic (safety interlocks, manual overrides, FB for field devices, some critical actions that should continue working even when the SCADA is down, etc.) should still reside in the PLC logic.
It seems to me that Ignition SFC could simply "fire" certain pre-programmed actions in multiple devices and to provide a great overview of the process to a greater group of people than PLC programmers.
I guess, all of us are already mixing Ladder, ST and FBD's in PLC code due to a better fit of one or another to specific cases.

Would you be able to share why exactly you dislike SFC's that much? I assume I just haven't got to this point yet. Your experience could help me and others learn about the downside of this programming technique earlier in our careers.
Again, sincerely asking not for the argument's sake. Simply craving for the advice and maybe for concrete examples of the SFC disadvantages from a more experienced engineer.

Thank you so much.

My primary objection to SFCs is that they scatter code/logic into many pieces, that can only be viewed/modified separately. This is a maintenance nightmare. And in Ignition in particular, transitions use a totally different language (expressions) from the rest of the logic (jython), fragmenting the experience further.

Arguably, you could have all action scripts delegate via one-liner to project library scripts, but then you have all of your functions in one place where a traditional procedural state machine could use them.

As for preserving state across gateway restart or project edit, a procedural state machine can easily load state from a document tag on gateway startup, maintain that state in a library script top level variable, and write back to the document tag at any desired checkpoints. Memory tags are persistent across restarts and synchronized across redundant pairs, so checkpointed state is taken care of naturally.

My final ding on SFCs is that they handcuff a programmer into a small set of state transitions. Many applications can benefit from state transitions that cannot be represented by an SFC's graph. (Or can be represented, but look like spaghetti.)

My primary objection to SFCs is that they scatter code/logic into many pieces, that can only be viewed/modified separately . This is a maintenance nightmare.

Yep. Its really quite painful to deal with the SFC interface.

And in Ignition in particular, transitions use a totally different language (expressions) from the rest of the logic (jython), fragmenting the experience further.

Also a pain. I try to keep the transitions to chart booleans that are calculated in preceding actions, where possible.

Arguably, you could have all action scripts delegate via one-liner to project library scripts

I started doing this, minimal code in the actual SFC editor (half because of maintenance and half because the editor is awful)

1 Like

Thank you, Phil,
All of your notes are truly valid.

Just thinking out loud. Again, not to argue, but just trying to be objective:

PT:My primary objection to SFCs is that they scatter code/logic into many pieces, that can only be viewed/modified separately. This is a maintenance nightmare. And in Ignition in particular, transitions use a totally different language (expressions) from the rest of the logic (jython), fragmenting the experience further.

  • Agree, it took some to get used to a different syntax. And I was frustrated as well for having to learn and understand both. But I had no issues after getting used to it. Maintenance would deal with the written code where they would follow the example. And a simple training with a detailed SOP could help to document the knowledge.
    As for me, SFC helps to break the project scripts on simpler and smaller pieces and helps to visually detect the executable piece.

PT:Arguably, you could have all action scripts delegate via one-liner to project library scripts, but then you have all of your functions in one place where a traditional procedural state machine could use them.

  • Would not this be a maintenance nightmare on the other hand? It seems like a more complex task that requires maintenance people at least not to be afraid of the scripting languages (python).
    In SFC, there is a way to templatize most of the standard actions. Programmers would need to design the templates and then to pass the parameters to the instances. In this case, the scripting portion would be mostly hidden in order to make both the troubleshooting and programming easier.

PT: My final ding on SFCs is that they handcuff a programmer into a small set of state transitions. Many applications can benefit from state transitions that cannot be represented by an SFC's graph. (Or can be represented but look like spaghetti.)

  • At the same time, complicated transitions create a similar spaghetti in the ladder logic as well. It seems to me that the SFC kinda challenges the programmer to break a complex task onto simpler and clearer steps.

And a few more thoughts,
The PLC logic is clear only to PLC programmers. Usually, for just a couple of people on site. The ones who have a PLC software license and a specific knowledge. Reading a code often requires scrolling through hundreds of lines of code and to cross-reference multiple tags that participate in the process/calculation.
It seems to me that monitoring and troubleshooting of the system may be available to a much larger group of people with Ignition SFC. Plus, process engineers that can now easily read the process and provide the feedback for the process improvements.
And isn't the state machine a mockup of the SFC? Isn't this a proof of the SFC usefulness?

I think that there might be a way to mix SFC, ST and Ladder in a very organic way, where these programming languages (or modules) are used in very specific cases.

I think you're overestimating the ease with which you might be able to get random (read -- non-SFC/python experts) programmers to implement SFCs. I was hoping to have this as a possibility as well, and the more I use them the less I think that will work gracefully. You could build a large library of templates to have the SFCs be relatively modular, but it will require quite a large amount of effort to have them able to be assembled by someone who is not really capable of cracking them open and checking that they are all wired correctly. And when something goes wrong (and something will go wrong), intervening is quite a pain. I have had multiple SFCs that would only die with a full gateway rebuild (we run contianerized), even when told in the main gateway console to die.

What I have ended up doing is developing them with the process engineer next to me, so that I can get a rough wiring of the program and then go back and clean up the underlying operation later. Alternatively, I have them design block diagrams in something like draw.io and I translate that into the SFC myself. The engine has too many peculiarities and misbehaviors for me to ever be confident that someone with limited understanding of the engine itself can do more than incredibly basic SFC implementation.

To build "functional building blocks" for the process engineers to develop processes without my direct intervention, I have developed a smaller number of carefully constructed SFCs that operate as sequential SFC executors, so that full SFC processes can be developed and then chained together with exposed parameters for use. This project took me months and it is not nearly as stable as I would like. It required scripting inside the SFC engine, tinkering with the Transaction Group system, and a series of very large scripts for regenerating tag data and XML data that I reverse engineered from downloading files using legacy controls that you have to know about to even access (shift right click in the correct menu to even see the option).

Heh. If you say so.

FWIW, I'm more against SFCs in PLCs than in Ignition, as their architecture in PLCs has no redeeming characteristics (speaking as Rockwell-focused person).

Ladder logic is the python of PLCs--it is the lingua franca, the least common denominator, the shared experience, of all programmers in that field. The better a ladder logic programmer you are, and therefore the more valuable you are with PLCs in general, the more the SFC architectures bother you. (In my experiences with others.) Who do you want more comfortable with the code when you call for help in a crisis? The ancient field tech/geeky dinosaur who dreams in ladder logic when focused on a controls problem? Or the "modern" drag-and-drop "programmer" who needs the graphical crutch?

Similarly, jython is the literal python of Ignition--it is the lingua france....yada yada of every serious Ignition developer (especially in Perspective). Same question about who you wish to cater to for the future crisis, when it comes?

(Scrolling around hundreds or thousands of lines of jython is really not a problem when broken into reasonable subroutines, due to the sidebar index in Ignition's editor. Really.)

Anyways, I don't think you want to be convinced, so this comment is unlikely to sway you. Which is OK, as there's a great deal of subjective consideration in this question that cannot be excised.