What's your method for resolving resource.json git conflicts?

My colleagues and I try to follow a pretty normal git development flow: we implement things on temporary branches and do peer-reviewed pull requests to merge those into a main “develop” branch. And after some period of testing and vetting, the “develop” branch is then pulled into a “production” branch.

We still have to be cautious and avoid more than one of us working on the same binary file at the same time, but we’ve been able to work on the same view or script and git was able to merge changes reasonably well.

The resource.json files are driving me nuts, though.

They always conflict, every time we touch anything in a directory, even if the changes in the code.py or view.json or whatever were merged gracefully by git.

I know we could just jump them to a clean-but-mismatched version by using something like git checkout --ours. But then we will have a commit in our history that if checked out will trigger a gateway to regenerate the resource.json and leave uncommitted files sitting around, which makes subsequent git operations propblematic. So the proper thing to do is to let Ignition re-generate them after resolving the conflict and roll that into the commit.

But I usually have Ignition stopped so it doesn’t freak out on conflict markers in files, which means my merge/rebase process now involves starting and stopping Ignition gateway after every conflict just to re-scan files and re-generate resource.json files.

Does anyone else have a better method of dealing with this? I know the IA team has a plan to let users control when Ignition re-scans the projects directories, which would at least let me keep Ignition running, but that feature isn’t here yet.

I’m mostly just venting, but also adding a little more weight to the feature request to control directory scanning manually.

The issue is that resource.json is composed of both user-supplied metadata and calculated metadata. The former belongs in git, the latter does not. This will not be resolved until IA redesigns how this file works.

IMNSHO.

{ Maybe in v8.2 ? }

4 Likes

I wonder if a standalone CLI utility could be created to generate valid signatures…

Sigh. /:

As a workaround for 8.1. I do think we’re planning to revamp this in 8.2.

3 Likes

If a CLI tool existed I would use it. But I also agree that the files should be revamped in the long run to avoid the issue entirely.

Well, it’s not currently very useful, but in case it helps anyone as a starting point, I made a thing:

Download the latest built jar from here: Update readme · paul-griffith/modification-updater@137aaac · GitHub
Run it with any Java 17 build on the command line:
image

You could, in theory, hook this in to Git in some way to automatically update the resource.json files before overwriting them in the resource directory, and, since the signature will be valid, Ignition won’t overwrite them again later.

I’ll give it a try. If it saves me from cycling Ignition on/off repeatedly, I’m happy.

Of course, almost immediately I did this:

$ java -jar ~/bin/modification-updater.jar . > resource.json 
Exception in thread "main" kotlinx.serialization.json.internal.JsonDecodingException: Expected start of the object '{', but had 'EOF' instead at path: $

Works fine as java -jar modification-updater.jar . > r0 && mv r0 resource.json though, so no biggie. Just made me chuckle that the most obvious usage pattern blows away the input file before it can read it.

The gateway is fine with the resource.json files that tool makes, but Designer is more finicky about the date format. :face_with_head_bandage:

com.inductiveautomation.ignition.common.gson.JsonParseException: unable to parse utc timestamp: 2022-06-06T18:58:26.091320830Z
	at com.inductiveautomation.ignition.common.project.resource.LastModification$GsonAdapter.deserialize(LastModification.java:161)
	at com.inductiveautomation.ignition.common.project.resource.LastModification$GsonAdapter.deserialize(LastModification.java:129)
	at com.inductiveautomation.ignition.common.gson.internal.bind.TreeTypeAdapter.read(TreeTypeAdapter.java:69)
	at com.inductiveautomation.ignition.common.gson.Gson.fromJson(Gson.java:927)
	at com.inductiveautomation.ignition.common.gson.Gson.fromJson(Gson.java:994)
	at com.inductiveautomation.ignition.common.gson.Gson.fromJson(Gson.java:967)
	at com.inductiveautomation.ignition.common.project.resource.LastModification.fromJson(LastModification.java:61)
	at com.inductiveautomation.ignition.common.util.ResourceUtil.lambda$static$0(ResourceUtil.java:57)
	at java.base/java.util.TimSort.binarySort(Unknown Source)
	at java.base/java.util.TimSort.sort(Unknown Source)
	at java.base/java.util.Arrays.sort(Unknown Source)
	at java.base/java.util.ArrayList.sort(Unknown Source)
	at java.base/java.util.stream.SortedOps$RefSortingSink.end(Unknown Source)
	at java.base/java.util.stream.Sink$ChainedReference.end(Unknown Source)
	at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(Unknown Source)
	at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
	at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
	at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
	at com.inductiveautomation.ignition.designer.workspacewelcome.RecentlyModifiedTablePanel$RecentlyModifiedModel.lambda$new$1(RecentlyModifiedTablePanel.java:140)
	at com.inductiveautomation.ignition.designer.workspacewelcome.RecentlyModifiedTablePanel$RecentlyModifiedModel.<init>(RecentlyModifiedTablePanel.java:144)
	at com.inductiveautomation.ignition.designer.workspacewelcome.RecentlyModifiedTablePanel.<init>(RecentlyModifiedTablePanel.java:56)
	at com.inductiveautomation.ignition.designer.scripteditor.workspace.WorkspaceEditor$2.createPanels(WorkspaceEditor.java:78)
	at com.inductiveautomation.ignition.designer.workspacewelcome.WorkspaceWelcomePanel.<init>(WorkspaceWelcomePanel.java:78)
	at com.inductiveautomation.ignition.designer.scripteditor.workspace.WorkspaceEditor$2.<init>(WorkspaceEditor.java:53)
	at com.inductiveautomation.ignition.designer.scripteditor.workspace.WorkspaceEditor.<init>(WorkspaceEditor.java:50)
	at com.inductiveautomation.ignition.designer.scripteditor.workspace.ScriptWorkspace.<init>(ScriptWorkspace.java:66)
	at com.inductiveautomation.ignition.designer.IgnitionDesigner.loadProject(IgnitionDesigner.java:956)
	at com.inductiveautomation.ignition.designer.IgnitionDesigner$StartupProjectDialogHandler.lambda$new$2(IgnitionDesigner.java:2021)
	at java.base/java.lang.Thread.run(Unknown Source)

Ignition v8.1.17 (b2022051210)
Java: Azul Systems, Inc. 11.0.15

But it was definitely faster using that tool to adjust the hashes than starting/stopping the gateway! :slight_smile:

Looks like the Designer parser just doesn’t like the absurd precision the tool added on the seconds. I’m trying to figure out the syntax to wrangle the DateFormatter line in your code. First time ever using Kotlin! Let’s see if I can wing it.

Well, I’m a lot less confident in this than the Kotlin version, but I wanted a learning project, so last week I ~recreated the CLI above in Rust:

I don’t know if it’ll work across all possible resources, but it’s guaranteed to be a lot faster:

time java -jar build/libs/modification-updater.jar -s src/test/resources/script
7ea951abc0ddc97f549f41a5670b06aa513b30e189050159f40e207cfe502b02
0.12s user 
0.03s system 
118% cpu 
0.123 total
time target/release/rust_experiments test/script
7ea951abc0ddc97f549f41a5670b06aa513b30e189050159f40e207cfe502b02
0.00s user
0.00s system 
38% cpu 
0.006 total

And I’m pretty sure it has the correct date-time formatting output :slight_smile:

The Rust one has some issues:

com.inductiveautomation.ignition.common.gson.JsonSyntaxException: com.inductiveautomation.ignition.common.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 16 column 2 path $

But I’m learning Rust, so I’m more prepared to fix coding issues in this one.

Don’t prioritize fixing this tool unless you’re having fun.

Disclaimer: it can be difficult to beat Paul G. to a Pull Request :wink: .. cde50dc23177881f

1 Like

Thank you for that tool, it could be very handy to implement a workaround to handle resource files for the moment!

Especially the rust tool which is really performant. However I found a small bug there: The timestamp is not the current timestamp as stated in the readme, but is always “2022-05-27T16:47:43Z”.

Yeah, @justin.brzozoski noticed that too. If you update the repository it should be correctly using the current local time.

The rust version has been updated with some edge cases I’ve discovered met, but I also don’t have 100% of tests passing, so I’m happy for help getting that to 100%.

To come back to the original question:

What’s your method for resolving resource.json git conflicts?

With the tool @PGriffith created we now use the following workflow with git hookscripts (hopefully temporary):

  • Add */resource.json to gitignore
    • This is what we have been waiting for all the time :slight_smile:
  • pre-commit: pack all resource files in the repository into one big json file. Drop all autogenerated content (signature), add original path information.
    • This way you keep the important metadata and even the last change information (if needed) in your repository. You only have one file that changes and only if you willingly apply changes to project resources.
  • post-commit, post-checkout, post-merge: Unpack the packed resource json into the original, individual files. Run the helper tool to generate new valid signatures using the metadata which is stored in the packed json file.
    • This ensures you always have a consistent repository after checking out a new branch and even directly after cloning a new repository.
    • You can slightly modify the rust tool to take your original timestamp and user as parameters if you want to keep this information.

Optional but useful: ensure that git hook scripts are always executed on all systems you are using!

  • configure the system wide git setting to use a different hook script path (a path in your repository)
  • place the hook scripts in this path in your repository
    • Now if you clone a remote repository which contains the hook scripts, the resource files are automatically unpacked and your project is ready directly after the clone.

Quite an effort only to avoid autogenerated content in the repository, but so far this works just fine. Maybe this helps someone to create his own personalized workaround.

6 Likes

Have you tried this one?

If you can get rid of all the “last modification” information this is an easy way to do it which also works. I thought it was nice to keep as much of this information as possible, but honestly git manages that for you so if it does not cause any problems with Ignition we do not really need it.

I feel like there's some missing detail here, so some questions:

  1. What would the impact be of ONLY adding */resource.json to .gitignore be?
  2. I have only a little exp with Rust, and no exp with it on Windoze; If I'm relegated to some highly managed AWS Workspace running Windows, is there hope for making any of this work?
  3. Does the Rust helper tool program do all this packing/unpacking?

Thanks!

I managed to solve all conflicts by doing the following:

Hope that helps