# Porting CorsoSystems Snapshot Versions/metatools "dump" (to disk) binary deserialization scripts to 8.1 gateway context **URL:** \ **Category:** Ignition\ **Tags:** scripting\ **Created:** [January 31, 2023, 8:20pm UTC](https://forum.inductiveautomation.com/t/porting-corsosystems-snapshot-versions-metatools-dump-to-disk-binary-deserialization-scripts-to-8-1-gateway-context/69234 "2023-01-31T20:20:33Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![hunterdg](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/hunterdg/32/67774_2.png) [@hunterdg](https://forum.inductiveautomation.com/u/hunterdg)\ **Post date:** [January 31, 2023, 8:20pm UTC](https://forum.inductiveautomation.com/t/porting-corsosystems-snapshot-versions-metatools-dump-to-disk-binary-deserialization-scripts-to-8-1-gateway-context/69234/1 "2023-01-31T20:20:33Z")
Hello, I'm attempting to port CorsoSystem's Snapshot Versions/metatools binary deserialization (to XML) to 8.1 gateway context via gateway message handler (for dumping to disk/diff comparison/git history). I've found that I can successfully dump .py and .json objects by simply calling `resourceData[resDataKey] = [bytearray(resData)]`, but I'm struggling with deserializing (or re-serializing) binary objects to XML. Seems the original code relies on a designer-only context `com.inductiveautomation.ignition.designer.model.DesignerContext.createSerializer()` call which obviously doesn't work in 8.1 gateway context. I've found [`com.inductiveautomation.ignition.common.xmlserialization.deserialization.XMLDeserializer.transcodeToXML​(java.io.InputStream binary, XMLSerializer serializer)`](https://files.inductiveautomation.com/sdk/javadoc/ignition81/8.1.24/com/inductiveautomation/ignition/common/xmlserialization/deserialization/XMLDeserializer.html#transcodeToXML(java.io.InputStream,com.inductiveautomation.ignition.common.xmlserialization.serialization.XMLSerializer)) and [`com.inductiveautomation.ignition.common.xmlserialization.serialization.XMLSerializer.serializeXML()`](https://files.inductiveautomation.com/sdk/javadoc/ignition81/8.1.24/com/inductiveautomation/ignition/common/xmlserialization/serialization/XMLSerializer.html#serializeXML()) but I'm a bit out of my depth regarding proper implementation. Any help/advice is much appreciated ---
**Author:** ![hunterdg](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/hunterdg/32/67774_2.png) [@hunterdg](https://forum.inductiveautomation.com/u/hunterdg)\ **Post date:** [January 31, 2023, 8:25pm UTC](https://forum.inductiveautomation.com/t/porting-corsosystems-snapshot-versions-metatools-dump-to-disk-binary-deserialization-scripts-to-8-1-gateway-context/69234/2 "2023-01-31T20:25:09Z")
reference code is here: [metatools/dump.py at master · CorsoSource/metatools · GitHub](https://github.com/CorsoSource/metatools/blob/master/shared/tools/dump.py) ---
**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\ **Post date:** [January 31, 2023, 8:42pm UTC](https://forum.inductiveautomation.com/t/porting-corsosystems-snapshot-versions-metatools-dump-to-disk-binary-deserialization-scripts-to-8-1-gateway-context/69234/3 "2023-01-31T20:42:00Z")
Some resources, in particular Vision, simply can't be decoded in a gateway context, full stop. The required Java classes aren't there. In 8.1.25, I want to say, you can set Vision to store project resources in XML format. For other resource types, there's no fully generic way to do it. Each module will have it's own classes with their own representation. ---
**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\ **Post date:** [January 31, 2023, 9:10pm UTC](https://forum.inductiveautomation.com/t/porting-corsosystems-snapshot-versions-metatools-dump-to-disk-binary-deserialization-scripts-to-8-1-gateway-context/69234/4 "2023-01-31T21:10:38Z")
Note that the superclass, CommonContext, of both Gateway and DesignerContext has a `createDeserializer` method: [https://files.inductiveautomation.com/sdk/javadoc/ignition81/8.1.24/com/inductiveautomation/ignition/common/model/CommonContext.html#createDeserializer()](https://files.inductiveautomation.com/sdk/javadoc/ignition81/8.1.24/com/inductiveautomation/ignition/common/model/CommonContext.html#createDeserializer()) ---
**Author:** ![andrew.geiger](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/andrew.geiger/32/5219_2.png) [@andrew.geiger](https://forum.inductiveautomation.com/u/andrew.geiger)\ **Post date:** [January 31, 2023, 9:36pm UTC](https://forum.inductiveautomation.com/t/porting-corsosystems-snapshot-versions-metatools-dump-to-disk-binary-deserialization-scripts-to-8-1-gateway-context/69234/5 "2023-01-31T21:36:01Z")
Hey yeah, I should probably document that's really only effective for Vision windows. It'll dump the context to a YAML and let you extract and understand the Java objects nicely. For resources that are already a text-based format, you're far better served using what's on disk and mulching it through something that normalizes it (say, ordered YAML with block quotes in case you want to do standard diff with version control). But to reinforce what Paul said - it _must_ be done from a Designer or Vision Client context. I don't make that very clear, sorry. The Gateway doesn't know how to deserialize Vision Windows. It doesn't need to since it only serves them to clients, right? I'll try to update that library with a replacement version that handles some of the differences from Ignition 7.9 and 8.x (one of which is how much vastly easier it is to deserialize stuff now - resource management's nicer!). I've got a branch on metatools that I never updated/finshed/released that handles that since the tool mostly lives as a single self-contained window that bootstraps its own environment. But since it came up, I'll take a gander at it in a few days to see what's left to finish. ---
**Author:** ![andrew.geiger](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/andrew.geiger/32/5219_2.png) [@andrew.geiger](https://forum.inductiveautomation.com/u/andrew.geiger)\ **Post date:** [January 31, 2023, 9:54pm UTC](https://forum.inductiveautomation.com/t/porting-corsosystems-snapshot-versions-metatools-dump-to-disk-binary-deserialization-scripts-to-8-1-gateway-context/69234/6 "2023-01-31T21:54:14Z")
Just in case I do manage to forget, here's a Vision window that should tide you over or get you a leg up on it. It's a (limited) ported version of a revamped rewrite that works in Ignition 8. It's never been a part of the metatools pack because I never finished refactoring it from "an enormous pile of custom scripts bootstrapped into a venv" to a proper set of script modules. And since it _can NOT_ run on a Gateway context I moved on to other things. So here's the **Versioning v3.8.2beta** with all the disclaimers "I downloaded a thing off a forum and then ran it" gives. I won't lie: this window is a piece of absurdist art. It's meant to be loaded cold, so it's _fully_ self-contained, which leads to a lot of shenanigans. Safe stuff, worked pretty reliably, etc. But be ware it's ... a bit much. You'll probably want to see the "Bootstrap" button for the code, which hoists the needed scripts to perform the extractions. The `*Binaries` property on the `Bootstrap` button contain base64 encoded jar files for direct git integration, but that's no longer used. I never tested if it still works, and I'm fairly more invested in making the current on-disk setup work from automating the CLI git instead. Sorry. But hey! It's got the PoorSql formatter binaries right in it, so that's something. Hmm, I should probably revisit and release this properly some day... Anyway, this should save at least some trouble. Hopefully. [IGN80\_Versioning\_v3.8.2b.zip](https://forum.inductiveautomation.com/uploads/short-url/41alX82cgBlvEPlPBSIOplMPe8B.zip) (3.7 MB) PS: this version's not smart enough to skip itself when the Sledgehammer button it hit and it starts to trawl all resources. Sorry for that. It takes a while for it to dump itself 🤦‍♂️ ---
**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\ **Post date:** [January 31, 2023, 10:00pm UTC](https://forum.inductiveautomation.com/t/porting-corsosystems-snapshot-versions-metatools-dump-to-disk-binary-deserialization-scripts-to-8-1-gateway-context/69234/7 "2023-01-31T22:00:40Z")
> [@andrew.geiger](#): > > The `*Binaries` property on the `Bootstrap` button contain base64 encoded jar files for direct git integration, but that's no longer used ![](https://us1.discourse-cdn.com/inductiveautomation/original/3X/c/9/c9ed87cbb1c0a38f63c638ebddce9af4e8e5c4c6.gif) Incredible hacks! ---
**Author:** ![andrew.geiger](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/andrew.geiger/32/5219_2.png) [@andrew.geiger](https://forum.inductiveautomation.com/u/andrew.geiger)\ **Post date:** [January 31, 2023, 10:27pm UTC](https://forum.inductiveautomation.com/t/porting-corsosystems-snapshot-versions-metatools-dump-to-disk-binary-deserialization-scripts-to-8-1-gateway-context/69234/8 "2023-01-31T22:27:57Z")
Look, I check-summed the binaries, so I could at least _pretend_ it's not just hotloading arbitrary code 😐 It's done in the _same_ dataset, though, so, like, re-do the binary population if you don't trust it. _And you shouldn't_ - I'm just some rando on the internet. You can also replace the `binary` bit with a file path or URL. It'll work as long as a `sha256` signature is given. You can see how they're loaded in `Root Container.Boostrap.git_load_jars`. Repackage via `Root Container.Boostrap.package_jars` if you have your own binaries and want to calculate the signature so it's willing to run. I think this preceded the cleaner version of that tooling at both https://github.com/CorsoSource/metatools/blob/master/shared/tools/venv.py https://github.com/CorsoSource/metatools/blob/master/shared/tools/hotload.py ---
**Author:** ![hunterdg](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/hunterdg/32/67774_2.png) [@hunterdg](https://forum.inductiveautomation.com/u/hunterdg)\ **Post date:** [February 1, 2023, 3:38pm UTC](https://forum.inductiveautomation.com/t/porting-corsosystems-snapshot-versions-metatools-dump-to-disk-binary-deserialization-scripts-to-8-1-gateway-context/69234/9 "2023-02-01T15:38:37Z")
Thanks very much - this is awesome stuff. Made the (perhaps poor in hindsight) decision to check this thread late last night while trying to fall asleep Range of emotions: excited - of course you've done it even better now, sad for a moment to see the disabled gateway scripts/named/queries/etc checkboxes, happy again after getting that working, Finally when digging in a bit further - oh boy I thought I was in over my head _before_, Still haven't wrapped my head around how you're loading into bootstrap.tools.snapshot.utils from bootstrap.\_\_snapshot\_utils for example. Also impressed that you've already solved omitting all the non-relevant vision window xml bits! **At the moment I'm more interested in gateway scripts/named queries than vision windows. Appreciate any confirmation if _that_ is possible from gateway context regardless.** Also curious about the apparent progression of gateway scripts export format from individual .py files (snapshot\_versions) \> consolidated .xml (metatools.dump) \> individual .yaml files (this window). I only ask as I had to regex-tweak the exported resource naming in snapshot\_versions due to some unconventional timer script names with cron syntax that can't exist in filenames, whereas that problem didn't exist with the .xml format since the script names were inside the .xml export. Again, MUCH appreciated. I love this forum. ---
**Author:** ![andrew.geiger](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/andrew.geiger/32/5219_2.png) [@andrew.geiger](https://forum.inductiveautomation.com/u/andrew.geiger)\ **Post date:** [April 28, 2023, 7:16pm UTC](https://forum.inductiveautomation.com/t/porting-corsosystems-snapshot-versions-metatools-dump-to-disk-binary-deserialization-scripts-to-8-1-gateway-context/69234/10 "2023-04-28T19:16:44Z")
Oh dear, I don't use the forum too much directly, and didn't realize I didn't get a notification. Sorry, hope this gets to you ... eventually. The bootstrap is almost pure Python hackery - it generates the module (if needed) and hoists the bits between `venv` calls into it. The name's mostly there for convenience in organizing. I think this can work from a gateway context, so long as you have the appropriate deserialization context. Python's pretty nice, since it's just a binary blob that just so happens to be the text of the script. Convenient! I don't remember if the named queries deserialize nicely in the gateway context, though. I'll crowbar that later and get back to you on that. You sleuthed out the important bits of the evolution of the effort, but there's a few revisions before this - one of which _did_ use and analyze the raw xml bits. Originally it was a 7.9 ask for "how do I version Vision windows?!" which eventually laid on the artifact and semantics dichotomy. Ignition is _not source code_ like Node.JS or Python. It's closer to, say, Photoshop or Unity, containing multiple binary resources that can be described, but are not themselves semantic descriptions of themselves. (The exception being Python scripts, which happen to be exactly what they describe.) That's why there's a core idea of "artifacts go here" for snapshot references for fast reloading and "describe things here" for semantic details. And since you can't diff things that are nondeterministically ordered - and _especially_ because scripts are block strings - the semantic bits are converted to a structured YAML file so that they can be understood. I didn't use JSON because diffing a JSON is, frankly, insane and horrible to deal with. Naming was done almost as lazily as possible, so sorry for that. You can probably fairly easily (?) intercept where it eventually converts the resource path to a file path. There's an earlier revision that would write all resources in a window to disk, but when you get to the expression level you run out of file path length (well, Windows will let you _write_ it but not access it, another insane problem). Grabbing scripts inside of objects in a Window means knowing what component had the script; I ought to rewrite it to match how Perspective puts all the binding bits in one place since the InteractionController homes all that stuff. But I think it's really super important to keep the indentation consistent, hence the flat structure in the YAML. While nesting is simpler to dump, it's much much worse to diff since indentation isn't always ignored by diffing software. Especially with Python code, of course. So making a component path the identifier and setting it as a top level item in the YAML means scripts show up consistently. Hope that helps, if you see this 😐