Turning off the bat light, we fixed this.
There was a lab root cert that we didn't have installed on our machines. Once we added that everything is hunky dory.
Shout out to @Allen_Miller for the help
Thanks,
Nick
1 Like
Today we performed the first full auto test of the module wherein with no existing context, we install the module and it automatically gets and installs its own certificate so that was really cool to see after working for a while to get it up to that level. So now we have a clear path on creation and renewal of certificates.
A remaining area of uncertainty is in obsoletion. We have an API method that we can call that obsoletes ( literally deletes ) the cert on the provider side so that after the instance using it is gone, it is no longer sitting there asking to be renewed. At present we plan to handle this with a semi manual means wherein the API call is activated from a manual action taken in the gateway prior to tearing down the VM or uninstalling Ignition.
What I would like to ask is:
Other than the shutdown gateway hook, is there any way we can capture an event, say when Ignition is uninstalled from the machine or when the VM itself is torn down?
I am pretty sure I know the answer but I want to verify. Also just to be clear we do not want to use the shut down hook to obsolete a certificate because it could be called in undesirable cases (ex: when upgrading the module but not yet wanting to obsolete the certificate).
Thanks,
Nick
Adding: is there anyway to discriminate what type of shutdown event it is?
Specifically is there anyway to know when the module has been chosen for uninstall via this button and not a gateway restart or module upgrade?
What I am trying to do is satisfy this requirement that the CWS team has for using their API in production:
Create, Download, Renew and Obsoletion should be automated.
There is a clear path for create, download, and renew, but we are trying to figure out how to handle obsolete.
Thanks,
Nick
How would you imagine capturing that event from your module? In theory, an uninstall hook could be added to modules; it'd have to be a totally separate hook class with its own scope and environment, because an uninstall can happen with no gateway running. In practice no one's ever really asked for it and I don't see us adding it. We have absolutely no way to capture a VM shutdown or cleanup.
It sounds like you might be interested in a container orchestration tool like Kubernetes, which does offer (some) support for higher level lifecycle events:
No.
@PGriffith RE: kubernetes......your nose is correct
The team that provides this API (CWS) mainly serves our internal cloud service ( so called WCNP ). So whenever you start up a service in WCNP, SSL is just magically there. What makes that happen in the background is CWS.
We however are, unfortunately, operating on onsite VM's and being told we must use CWS to automate certificate lifecycle and see the gap regarding the obsolescence part.
In any case, your answer is still useful to us regarding the actual possibilities and what we are going to propose is a kind of semi-auto obsolete process that is activated from the settings page.
Thanks,
Nick
If your module adds a webhook/route in Ignition could your orchestration call that in some kind of pre-shutdown hook? You could use that opportunity to in turn obsolete the certificate.
Could one override the moduleRemoved() method of the ModuleObserver object to request and obsolete the cert? Perhaps the same thing could be done with moduleAdded() for certificate creation.
The module doesn't create the module observer, so, no. That's part of the platform and the module cannot affect it.
Also, the module's own shutdown() hook is called before any methods on the observer are called. So you'd have to have another module in place to watch for lifecycle events on the first.
Only you know when you are taking an instance down permanently. We can't know that.
In your module, you could expose an API or route or something which initiates this "permanent shutdown". The implementation could obsolete the cert, then invoke Gateway shutdown.
You'd then need to ensure that you only call this API when taking an instance out permanently.
You'll still likely want to have some kind of side process (automated or manual) which runs periodically somehow (outside of any of your Ignition processes) to scan through your certs in your CWS API and obsolete any certs for instances which no longer exist, in case an instance was terminated in a non standard way... You could argue that this "cleanup" process could be responsible for all obsoletion.
1 Like
Yeah, as long as you have the auth and server certs you can establish mTLS from another place so we could delete certs on the provider against a list of no longer existent VM's if needed. That would be a matter of having a guarantee that the machine is actually gone and not mistakenly removing a cert that is still needed.
The overarching theme here is that on-prem VM sucks for maintainability, especially when our company has a cloud service that automatically provides you a SSL cert whenever a pod is started......
We are beginning discussions of moving to all cloud deployed but its going to take several years I suspect so we'll have to use this module, however it ends up, in the meantime.....
Thank you guys for the constant and helpful dialog.
Nick