Since our EAM machine would hold all of the common names that need to be managed in one place, I am thinking to make a module that is installed on the EAM machine that does the following. Since java/module is inherently harder to tamper with that python code, that would be a key driver to have it be in a module since SSL is mission critical.
periodically checks expiration dates, if a renewal is needed it is scheduled (to prevent too many happening at once)
execute the renewal API and package the SSL cert and private key into a PKCS12 file
Using SSH, perform backup of the current cert, transfer new one, and execute GWCMD -g to load the new cert immediately
Perform a git call to the following to check gateway "https://{address}:8043/system/gwinfo"
There is an internal table called "WSCONNECTIONSETTINGS" that already has all of the EAM agent common names which is exactly what we would need to be managing.
My initial questions are:
is this the correct place to look to get all EAM agent common names?
If yes, would it be preferable to grab the info with HTTP calls ( /data/status/internal-db ) or with a query?
I'd think that this would save you the work of having to write and maintain a Module. Instead, you'd let an ACME client such as certbot which runs as another process alongside Ignition's java process deal with certificate automation. Each Gateway in your network in theory would have its own cert bot so that you can divide and conquer by delegating each host to deal with their own certs.
@jspecht thanks for pointing it out, we will consider it. One convenient thing about a module is that while we do have to handle the details, it is easy to distribute via EAM and would make logging and status checking much more visible from the gateway UI. If we have something running on the VM but is fully decoupled from Ignition, we won't be able to see if there is an issue without going to the VM itself which involves going in via RDP which is no issue, but I would prefer to have visibility in the gateway logs at least.
If we setup cert bot and cronjobs, we have to do it on each and every gateway and same again if we need to ever modify.
Gotcha, well if you roll your own, you could also consider implementing an ACME client in your module or pulling in a third party dependency which implements an ACME client in Java (for example, see: ACME Client Implementations - Let's Encrypt). That's something I want to add to the Ignition platform some day...not likely any time soon though.
Regarding your question about the common name...it depends. Is this certificate intended for the Ignition web server (default port 8043)? Which column are you looking at in WSCONNECTIONSETTINGS table?
It really depends on your network and DNS setup. Which hostname(s) are your TLS clients (i.e. web browsers, vision clients, designer instances, etc) using to connect to the target Gateway?
You might be better off inverting the design and going with the ACME client implementation I mentioned above or something like that - have each Gateway reach out to your central authority periodically when it determines that it is time to renew certs. Let each Gateway manage its own cert lifecycle instead of burdening one single Gateway. You could still expose status routes and tags in your module so that each agent can expose info to your central controller if that's what you'd like...
{ Joel's being awfully polite. Distributing the work with a certbot implementation is definitely the way to go. I'd also go through Venafi to monitor it. Distribute your "certbot module" with EAM. }
Yeah, also through internal discussion today we are leaning that way i.e. have the module installed on each individual gateway and get it there initially and update it as needed via EAM. Basically we've moved away from the central idea today as you both are eluding to.
A very useful product feature would be if Ignition had auto SSL cert renewal built in. Joel hints above that the idea already exists. IA did a fantastic job on the SAML cert auto download feature, something that may not have arrived had we not had the discussion.
Imagine if on the web server page there was an option to turn SSL cert auto-renew on/off and then the needed fields such as renewal threshold (days before expiry) and other necessary fields to be able to authenticate/communicate with the cert provider API. Set and forget for SSL would be awesome.
In any case, we have a consultation with CWS (cryptography web service) on Wednesday and I will update this thread as we progress this topic which is necessary for us to solve since the need to renew certs never goes away.....
Additional info I got this morning on a mail written by the head of infosec at our company is that in our version of Venafi, ACME is not supported or in place and there is no plan to do so.
So making a custom module to interact with the CWS API shown above is really the only option.
@jspecht I met with the CWS team yesterday and got some more context.
In our cloud platform (WCNP, Walmart Cloud Native Platform) whenever you start an app in kubernetes, its SSL cert is automatically there. You never apply, never renew, you never have to obsolete, its just magically there. Talking to some people who work directly with WCNP development, CWS (the API mentioned above) is what is doing that.
So then it seems security has said, this is the one method for the company and that is essentially why ACME is not used. What I have to figure out is how to make this work in the context of a windows VM .
We're going to proceed writing a module that takes care of these lifecycle requirements that they have in order to allow us to use the CWS API:
Initial cert application will no longer be allowed manually, but must originate from the app
Yearly cert renewal must be done by the app
Decommission/obsolescence request must originate from the app
For 2. I already have a clear idea. For 1 and 3 I'm imagining some sort of settings page for SSL auto management where the user will have the option to manually create and obsolete a cert. Will be thinking through how to best do it with our team.
I guess I wanted to post this here so that if IA is ever thinking about SSL cert management functionality in the future, you have one use case to look.
@jspecht I'm at the point with the CWS API that I'm ready to download a cert. We can download it in PKCS12 which will be what Ignition needs (SSL cert + private key). The thing I'm currently trying to wrap my head around how to do is related to the key store and private key passwords which Ignition defaults to "ignition".
In the API there are these requirements which are in code so we cannot bypass at the time of call:
So from the start we can't even get a cert back unless I use a more sophisticated password. Then if I am not mistaken we can change the key store password but we cannot change the private key password once it is issued. Correct me if I am wrong, but the only option seems like we have to add these parameters to the ignition.conf file.
Then, if we change the password by manually modifying the ignition config file, there are 2 issues:
If we do it on a gateway that is already running, it will break SSL
Since we can only specify it statically, we'd be using the same password everywhere, something security may bite us for in the future.
One workaround off the top of my head: generate random passwords for the purposes of obtaining the keystore, then store it using the passwords from system properties that Ignition uses...
So if we changed the key store password to match what was already in Ignition, we also have to change the private key password to be able to let Ignition access it, correct?
Yes, that is the plan, change the password on both the cert store and private key before install. Now working through the mechanics of doing so in Java.
.
