Lots of good advice allowed me to get to a good solution on this one. Thank you @pturmel for confirming this is a real issue that we needed to handle, not just pilot error on the programmers’ part.
The key from @paul-griffith is that json data structures have built-in serialization with the gson adapters. So one can pass a PyDictionary as a JsonObject, a PyList as a JsonArray, etc. Note that nulls are passed as JsonNull, but all of these types derive from JsonElement, so you can check for nulls by checking the class type. Thank you @bmusson for directing me to the TypeUtilities.
The serialization method takes in Jython objects as parameters, converts them to the appropriate gson object, uses the default serialization/deserialization for gson, then converts the gson object to the native java objects for use in the modules. Example below.
protected JsonObject SafePyDictionaryToJsonObject(PyDictionary pyDictionary) {
JsonElement je = TypeUtilities.pyToGson(pyDictionary);
if (!(je instanceof JsonObject))
je = new JsonObject();
return (JsonObject)je;
}
The gson serialization is only ‘mostly’ free. It will incorrectly convert data types: int→long, long→bigint, long→double and likely others that I haven’t tripped over yet. With the strong type-checking in Java, you’ll likely have to cast something back. Thankfully, the boxed primitives all derive from Number and allow a simple conversion back to the appropriate type.
Long locationId = ((Number)o).longValue();
BigInteger required its own adapter. If there was a built-in path to get it, I did not discover it.
The 8.3 module update process is not complete, but the serialization seems solid. I am concerned that for conditions that really should fail, the new serialization paths do raise some different exception types than before, so there’s a potential that we’ll have to change handlers that catch specific Exception types.
The other serialization things we’ve learned:
- Passing custom objects requires custom serialization adapters
- You can’t overload calls through the serializer. You can keep them overloaded for scripting interfaces, but to make the jump to a remote environment, it now must be unique.
- Don’t use the PyObject serializer, you’ll lose money on that bet.
- We implemented a value serializer to pass a single simple value. Now that I know the gson method, I suspect JsonPrimitive would have worked. Might have to go back and try it…
- You can pass simple recordset/dataset objects. You’ll have to create an adapter class. Since StreamingDatasets are not supported by the built-in basic dataset serializer. Use something like this and add your own custom handling in the conversion routing.
public DatasetPB toProtobufMessage(Dataset dataset) throws ProtobufSerializationException {
if (dataset instanceof BasicStreamingDataset bsd) {
dataset = convertStreamingToBasic(bsd);
}
if (dataset instanceof BasicDataset bd) {
return DatasetPB.newBuilder().setBasicDataset(this.basicDatasetSerializer.toProtobufMessage(bd)).build();
} else {
throw new ProtobufSerializationException("No serializer available for dataset class '%s'".formatted(dataset.getClass().getName()));
}
}