Any progress on version control system for Ignition?

Currently we have the better part of 4 years of development in Ignition.

As it’s applications in our business grow, so does the need for a better source control system. We have no source control at all and bugs are slipping through and getting harder and harder to revert. The current information from the Ignition Gateway is not exactly clear as to what has changed with each save.

Has there been any advancements in getting window objects/script libraries to push to a traditional VCS like Git/Bitbucket?

I was thinking pushing more and more development into script libraries,and having those functions called more in the GUI components in Ignition.

Root Containers can be exported to XML, but the process would have to be manual and we lose out of any script information that runs of the Window Events.

The rub is automating these tasks, if there are any ways of scripting a pull of the XML structure for a project would be cool.

I can’t be the only dev that has struggled with this. Has there been any progress at all on this front?

10 Likes

I can't be the only dev that has struggled with this.

Oh, certainly not- a few people laughed when they saw your post, saying "It must be that time of year!", since this is a subject that comes up routinely at the ICC. So if it comes up so much, why haven't we done anything?

To cut straight to the chase, this is definitely on our minds as we work on Ignition 8. We have plans to rework the project structure (something that we've been hesitant to do in the last few releases) in such a way that better version control support should be possible. That said, the idea that you'll just have your entire Ignition project in github, with everything perfectly diff-able and mergeable probably isn't going to happen.

Our main goal is to improve how resources are managed/versioned/distributed across enterprise systems (that's to say, multiple gateways). Right now, we have a set of features comprising of Save/Publish+Project Rollback+EAM to distribute copies, all of which have various shortcomings in terms of effectively managing resources in a distributed way.

Support for external VCS systems would be nice, but would probably be layered in as a secondary component to the changes I mentioned above (as in, you could have the resources sync'd to VCS, but Ignition likely wouldn't know much about that).

As for your current situation, since you mention scripting specifically, you might be able to get pretty far, as-is. If you're not familiar with external python libraries in Ignition, check this out:
https://support.inductiveautomation.com/index.php?/Knowledgebase/Article/View/98/2/importing-and-using-3rd-party-python-libraries-in-ignition

There is a folder of python libraries that is monitored by the gateway, and even sync'd to clients. Many customers use external tools (editors+VCS) to manage their libraries, and then the code inside the project is minimized to calls to these external scripts. Not complete coverage, but for complicated libraries it can get you much further towards what you want.

Hope this helps a bit!

1 Like

Also, check out the Power Scripting Module by @nmudge for some script functions to add script modules in the designer.
https://inductiveautomation.com/moduleshowcase/module/perfect-abstractions-pa-power-scripting-module

So it should be possible in the future to have one dev on the field, working on the future production server to align everything to reality, combined with a dev in the office working on other last-minute client requests for the project on a development machine. And then the projects should be mergeable without much fuss? Even if the same resources were touched?

I don't really see how this is achievable without text diffs. Or there will have to be a very complicated GUI to show all little differences between screens (also the hidden differences in the bindings).

With this workflow, I don't expect the gateway settings to be diffable. OPC connections, DB connections, user logins, etc. all differ between a development system and a production system anyway. But the ability to diff the windows and tags is a must.

And given that both have the ability to serialize and deserialize to XML (and for windows, I mean the XML you get by copy-pasting, not the base64 encoded string encapsuled in XML). It's just a matter of making these functions easily available under an "export to folder structure" button (and an accompanying "import folder structure" button).

Don't get me wrong, I like Ignition a lot. I like how quick it is to develop features on, how easy it is to push updates to the live environment. But the lack of version control is really a bottleneck when deploying a new installation.

This isn't a new observation, as Colby acknowledges. To do this, IA would have to completely ditch their internal database and represent all resources and configuration data as text, in a repeatable way (every java-->text save yields the same text or xml, in the same order and/or same file names).
To support efficient diff'ing, the text formats would have to be "tidy", and not have arbitrary de-duplication IDs that would interfere with merging diffs from different sources.
Every single existing module would have to be rewritten with a new configuration persistence interface.

Basically, you're asking IA to rewrite a large fraction of Ignition's code.

1 Like

@pturmel, thanks for giving me some insight on the issue.

I didn’t realise those IDs were arbitrary, or that the parent-child construction of the XML was arbitrary. Hence why I didn’t see issues with using that XML for diff’ing. This indeed makes it a bit more difficult.

Otoh, since it’s an algorithm that makes up where the XML nodes arrive, and that makes up the ids, the results will probably still be pretty similar and at least allow for comparing the data via diffs.

Second thought, do we really need de-duplication IDs? Can’t the users be trusted that they will use templates for most duplicated things (I do in any case). But you’d still need IDs for circular references I guess :confused:

Perhaps GUIDs can be used instead of IDs

In any case, thanks for enlightening me. It’s more complicated than I realised.

Ideally a function to export bits of code would go pretty far.

I’m struggling with the same issue different cause. Applications we’re being asked to develop oftentimes have zero or very tangential ties to the ignition ecosystem.

They mainly want us to code in ignition because it’s something that’s easily deployed with existing structure, and easily understandable by existing devs.

It’s a bummer to hear that child-parent XML is completely arbitrary. I was kind of hoping and banking on a repeatable XML portion of code as a way to better document and store files outside the ignition IDE.

As it stands right now, most of that XML is going to be incredibly difficult to push and pull automatically. Even if it weren’t the case… the result is not exactly diffable it would at least have a trace with slightly more detail than what is currently given.

Well, it's not completely arbitrary. Reference IDs for java objects that are used in more than one place are numbered in the order they are encountered, which is the order those objects were originally created, generally. And the numbering is global, not per-class or some other absolute. So, adding and removing components will cause all of those items to be renumbered in the next export, along with just about every other refid, causing unusually large diffs that will certainly clash with other changes.

1 Like

Yeah, “arbitrary” isn’t quite the word for it. But the reference ids are probably non-deterministic in some cases, as they are generated in order of being encountered, and there might be some map/set iteration in there.

Another thing that makes this tricky is the tendency to open up screens in the designer with the data flow on, which will overwrite values. That format was not designed for human consumption. But I can certainly echo Colby’s sentiment that this is very present in our current thought process, but Like phil said, it’s going to be tricky to do.

3 Likes

I’ll be happy with an easier export/import for scripting libraries.

I’ll be looking into solutions with with the 3rd Party Modules mentioned earlier. Even if I can automate pushing, but have manual pulls would be a step forward at this point.

The XML idea was promising until I opened a custom app that had a very large table inside of it as an element. Every single column/row data entry was a row in the XML, swelling it to a huge amount of text.

Thanks to you and @Colby.Clegg for the responses.

I can’t make it to the ICC this year but I look forward to any new development/updates the team is willing to share.

I’m just trying to align our Ignition Development to the best practices within the company I work with, so apologies if I came across as overly inquisitive.

That's very easy to do in python itself.

The script below copies all files *.py from a source directory to a target directory (excluding the unit tests), and it deletes the files that aren't found (I had issues once after renaming a file and keeping the old code around too, didn't understand my own error messages). It happens async, so it's pretty fast and to ignition, the files are updated almost at the same time, so everything gets loaded at the same time (you don't want a client hitting the "update" bar while only half of the files are updated).

def SendToIgnition(targetDir, sourceDir):
    
    print("copy from " + sourceDir + " to " + targetDir)
    
    foundFiles = []
    threads = []
    
    for subDir, dir, files in os.walk(sourceDir):
        for filename in files:
            path = os.path.join(subDir, filename)
            if path.endswith(".py") and not filename.startswith("test_"):
                relFile = path[len(sourceDir) + 1:]
                foundFiles.append(relFile)
                print("copy file " + relFile)
                targetFile =  os.path.join(targetDir, relFile)
                if not os.path.exists(os.path.dirname(targetFile)):
                    os.mkdir(os.path.dirname(targetFile))
                threads.append(threading.Thread(
                        target=shutil.copy,
                        args=(os.path.join(sourceDir, relFile), targetFile)
                    ))
            
    for t in threads:
        t.start()
    for t in threads:
        t.join()
    print "Coyping done"
      
    for subDir, dir, files in os.walk(targetDir):
        for filename in files:
            path = os.path.join(subDir, filename)
            relFile = path[len(targetDir) + 1:]
            if not (relFile in foundFiles):
                try:
                    if path.endswith(".py"):
                        print("remove file " + relFile)
                    os.remove(os.path.join(targetDir, relFile))
                except:
                    print("Failed to delete file" + relFile)

To copy it to a different server, I share the pylib folder from Windows (on Linux you could make an SSH connection), and I can copy a library like this, from a file placed next to the library.

sourceDir = os.path.directory(__file__)
targetDir = "\\\\hostname\\pylib"
folder = "libraryName"
# Connect to shared drive using a local account
subprocess.call('net use %s /user:127.0.0.1\\UserName P@SSW0RD' % targetDir, shell=True)

SendToIgnition(os.path.join(targetDir, libraryName), os.path.join(sourceDir, libraryName))

If you use an IDE with an interpreter, you can just run that python script from inside your IDE, and it will get synced.

2 Likes

Hi thanks for the code snippet. From my understanding, this would be using more the 3rd party libraries we are allowed to export/import to Ignition correct?

Great Idea and I was going down a similar path, but my request was more for an automated export from existing libraries within the Ignition Libraries.

I will have no issues championing the change to more self contained libraries within that folder though.