Can't extract .gwbk with zipfile.ZipFile on Windows (works fine in Linux)

hunterdg · May 6, 2024, 8:07pm

This code works fine in Linux, but not in Windows:

import system
import tempfile, os
import zipfile

tempDir = tempfile.gettempdir()

def unzip(gwbk_path):
	extractDir = os.path.join(tempDir, 'extracted').decode('unicode-escape')
	# Extract the .gwbk (zip) file
	with zipfile.ZipFile(gwbk_path, 'r') as zip_ref:
		zip_ref.extractall(extractDir) ### bails here ###

I have a feeling it's breaking on an empty email-profiles folder, and perhaps this cpython 2.7.4 issue is related.

any workaround ideas?

Wrapper log

INFO   | jvm 1    | 2024/05/06 14:54:47 | W [p.a.script                    ] [14:54:47.185]: Error running action 'component.onFileReceived' on _DEV/GWBK@C/root/FileUpload: Traceback (most recent call last):
INFO   | jvm 1    | 2024/05/06 14:54:47 |   File "<function:runAction>", line 2, in runAction
INFO   | jvm 1    | 2024/05/06 14:54:47 |   File "<module:_DEV.GWBK>", line 52, in runAction
INFO   | jvm 1    | 2024/05/06 14:54:47 |   File "<module:_DEV.GWBK>", line 11, in unzip
INFO   | jvm 1    | 2024/05/06 14:54:47 |   File "C:\Program Files\Inductive Automation\Ignition\user-lib\pylib\zipfile.py", line 1038, in extractall
INFO   | jvm 1    | 2024/05/06 14:54:47 |     self.extract(zipinfo, path, pwd)
INFO   | jvm 1    | 2024/05/06 14:54:47 |   File "C:\Program Files\Inductive Automation\Ignition\user-lib\pylib\zipfile.py", line 1026, in extract
INFO   | jvm 1    | 2024/05/06 14:54:47 |     return self._extract_member(member, path, pwd)
INFO   | jvm 1    | 2024/05/06 14:54:47 |   File "C:\Program Files\Inductive Automation\Ignition\user-lib\pylib\zipfile.py", line 1059, in _extract_member
INFO   | jvm 1    | 2024/05/06 14:54:47 |     arcname = arcname.translate(table)
INFO   | jvm 1    | 2024/05/06 14:54:47 | TypeError: character mapping must return integer, None or unicode
INFO   | jvm 1    | 2024/05/06 14:54:47 |  project-name=_DEV, view=_DEV/GWBK@C, component=root/FileUpload
INFO   | jvm 1    | 2024/05/06 14:54:47 | com.inductiveautomation.ignition.common.script.JythonExecException: Traceback (most recent call last):
INFO   | jvm 1    | 2024/05/06 14:54:47 |   File "<function:runAction>", line 2, in runAction
INFO   | jvm 1    | 2024/05/06 14:54:47 |   File "<module:_DEV.GWBK>", line 52, in runAction
INFO   | jvm 1    | 2024/05/06 14:54:47 |   File "<module:_DEV.GWBK>", line 11, in unzip
INFO   | jvm 1    | 2024/05/06 14:54:47 |   File "C:\Program Files\Inductive Automation\Ignition\user-lib\pylib\zipfile.py", line 1038, in extractall
INFO   | jvm 1    | 2024/05/06 14:54:47 |     self.extract(zipinfo, path, pwd)
INFO   | jvm 1    | 2024/05/06 14:54:47 |   File "C:\Program Files\Inductive Automation\Ignition\user-lib\pylib\zipfile.py", line 1026, in extract
INFO   | jvm 1    | 2024/05/06 14:54:47 |     return self._extract_member(member, path, pwd)
INFO   | jvm 1    | 2024/05/06 14:54:47 |   File "C:\Program Files\Inductive Automation\Ignition\user-lib\pylib\zipfile.py", line 1059, in _extract_member
INFO   | jvm 1    | 2024/05/06 14:54:47 |     arcname = arcname.translate(table)
INFO   | jvm 1    | 2024/05/06 14:54:47 | TypeError: character mapping must return integer, None or unicode
INFO   | jvm 1    | 2024/05/06 14:54:47 | 
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.core.PyException.doRaise(PyException.java:211)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.core.Py.makeException(Py.java:1654)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.core.Py.makeException(Py.java:1658)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.core.Py.makeException(Py.java:1662)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.core.Py.makeException(Py.java:1666)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.pycode._pyx18.unzip$1(<module:_DEV.GWBK>:39)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.pycode._pyx18.call_function(<module:_DEV.GWBK>)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.core.PyTableCode.call(PyTableCode.java:173)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.core.PyBaseCode.call(PyBaseCode.java:150)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.core.PyFunction.__call__(PyFunction.java:426)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.pycode._pyx18.runAction$2(<module:_DEV.GWBK>:59)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.pycode._pyx18.call_function(<module:_DEV.GWBK>)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.core.PyTableCode.call(PyTableCode.java:173)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.core.PyBaseCode.call(PyBaseCode.java:150)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.core.PyFunction.__call__(PyFunction.java:426)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.pycode._pyx17.runAction$1(<function:runAction>:2)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.pycode._pyx17.call_function(<function:runAction>)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.core.PyTableCode.call(PyTableCode.java:173)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.core.PyBaseCode.call(PyBaseCode.java:306)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.core.PyFunction.function___call__(PyFunction.java:474)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.core.PyFunction.__call__(PyFunction.java:469)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at org.python.core.PyFunction.__call__(PyFunction.java:464)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at com.inductiveautomation.ignition.common.script.ScriptManager.runFunction(ScriptManager.java:847)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at com.inductiveautomation.ignition.common.script.ScriptManager.runFunction(ScriptManager.java:829)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at com.inductiveautomation.ignition.gateway.project.ProjectScriptLifecycle$TrackingProjectScriptManager.runFunction(ProjectScriptLifecycle.java:868)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at com.inductiveautomation.ignition.common.script.ScriptManager$ScriptFunctionImpl.invoke(ScriptManager.java:1010)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at com.inductiveautomation.ignition.gateway.project.ProjectScriptLifecycle$AutoRecompilingScriptFunction.invoke(ProjectScriptLifecycle.java:950)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at com.inductiveautomation.perspective.gateway.script.ScriptFunctionHelper.invoke(ScriptFunctionHelper.java:161)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at com.inductiveautomation.perspective.gateway.script.ScriptFunctionHelper.invoke(ScriptFunctionHelper.java:98)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at com.inductiveautomation.perspective.gateway.action.ScriptAction.runAction(ScriptAction.java:80)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at com.inductiveautomation.perspective.gateway.action.ActionDecorator.runAction(ActionDecorator.java:18)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at com.inductiveautomation.perspective.gateway.action.SecuredAction.runAction(SecuredAction.java:44)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at com.inductiveautomation.perspective.gateway.model.ActionCollection$ActionSequence$ExecuteActionsTask.lambda$call$0(ActionCollection.java:263)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at com.inductiveautomation.perspective.gateway.api.LoggingContext.mdc(LoggingContext.java:54)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at com.inductiveautomation.perspective.gateway.model.ActionCollection$ActionSequence$ExecuteActionsTask.call(ActionCollection.java:252)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at com.inductiveautomation.perspective.gateway.model.ActionCollection$ActionSequence$ExecuteActionsTask.call(ActionCollection.java:221)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at com.inductiveautomation.perspective.gateway.threading.BlockingTaskQueue$TaskWrapper.run(BlockingTaskQueue.java:154)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at com.inductiveautomation.perspective.gateway.threading.BlockingWork$BlockingWorkRunnable.run(BlockingWork.java:58)
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	at java.base/java.lang.Thread.run(Unknown Source)
INFO   | jvm 1    | 2024/05/06 14:54:47 | Caused by: org.python.core.PyException: TypeError: character mapping must return integer, None or unicode
INFO   | jvm 1    | 2024/05/06 14:54:47 | 	... 43 common frames omitted

PGriffith · May 6, 2024, 10:02pm

Your best approach would probably be to use the Java standard library to do the unzip operation. Not nearly as straightforward, though, unfortunately. Look into Filesystems.newFileSystem; you can open a given path to a zip file as if it's an entire browseable filesystem.

hunterdg · May 7, 2024, 5:31pm

Well that was a fun little exercise. AI failed hard and repeatedly. Thanks!

hunterdg · May 8, 2024, 5:15pm

Edit - additional struggles

It appears a manually unzipped/rezipped gwbk file fails to restore with java.io.FileNotFoundException: C:\Program Files\Inductive Automation\Ignition\user-lib\pylib (Access is denied) despite being "identical" (internally) - which makes absolutely no sense to me.

However if I purposefully exclude empty subfolders from the recreated gwbk, I have no issues restoring it, though I imagine this could cause problems.

Clearly Ignition is somehow creating gwbks with empty folders that don't break on restore (I imagine in Java directly) - which means I should also be able to accomplish the same thing through jython right? Any thoughts/pointers? Might this be a "feature" to block modified gwbks?

are you suggesting I should be able to interact with (modify) the gwbk directly (avoiding extract/recompress) with this method? If so, I haven't been able to come up with a working connection string to db_backup_sqlite.idb for zxJDBC.connect

PGriffith · May 8, 2024, 5:24pm

Nothing like that exists.

The zip file contents are likely the same (or close to the same; zip is a fairly malleable format, with things like intermediate folders being more of a 'suggestion' than a requirement) - but are the permissions being carried into the zip file? Perhaps the output of the script you're running is inheriting the 'ownership' from the gateway system user.

I think this is a fundamental SQLite limitation. As far as I know, the underlying C API only allows loading from a 'real' file handle that it locates and reads itself. I've run into the same limitation with the raw JDBC driver in Kindling; I have to copy the .idb out to a temp file to actually load it.

hunterdg · May 8, 2024, 5:27pm

~~I started down this path until I (accidentally) realized just excluding the empty dirs worked - and pylib is not an empty directory, so the behavior should theoretically be unrelated.~~ Maybe I'll dig a bit more

thanks, this makes me feel better

PGriffith · May 8, 2024, 6:28pm

Yeah, I really can't explain it. I remember having issues with manually generated ZIPs restoring cleanly in the past, but I don't remember ever really finding a solution...
Maybe this post will nerd snipe Phil successfully.

A random GUI zip tool suggests the files have slightly different attributes:

unzip -vl, oddly, suggests that in 'original', folders take up 2 bytes?

Archive:  original.gwbk
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
 1716224  Defl:N   320411  81% 05-08-2024 10:14 3fd13bf5  db_backup_sqlite.idb
       0  Defl:N        2   0% 05-08-2024 10:14 00000000  certificates/
       0  Defl:N        2   0% 05-08-2024 10:14 00000000  certificates/supplemental/
       0  Defl:N        2   0% 05-08-2024 10:14 00000000  email-profiles/
       0  Defl:N        2   0% 05-08-2024 10:14 00000000  gateway-network/
       0  Defl:N        2   0% 05-08-2024 10:14 00000000  gateway-network/client/

Whereas in 'identical' they're totally empty:

Archive:  identical.gwbk
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
       0  Defl:N        0   0% 05-08-2024 10:52 00000000  modules/
       0  Defl:N        0   0% 05-08-2024 10:52 00000000  projects/
       0  Defl:N        0   0% 05-08-2024 10:52 00000000  user-lib/
       0  Defl:N        0   0% 05-08-2024 10:52 00000000  certificates/
       0  Defl:N        0   0% 05-08-2024 10:52 00000000  opcua/
       0  Defl:N        0   0% 05-08-2024 10:52 00000000  email-profiles/

But I can't see a way to get a more useful description than that.

hunterdg · May 9, 2024, 6:51am

Well thanks for pointing out attributes - and sending me down another rabbit hole

I've managed to duplicate the original attributes (and lack of permissions), including the peculiar user-lib (no D attribute) and 2-byte folder quirks - all while saving a few KBs!

semi-related: Kindling shows the empty email-profiles folder as a file icon. It also appears to hang on these gwbks (including original)

PGriffith · May 13, 2024, 3:20pm

What'd you change in your code, out of curiosity?

I'll have to figure out what's going on there.