SSL Auto Renewal Module

I am currently thinking about how to implement this, how to make SSL certs auto-renew

Our cert provider (Venafi) has some entity in the company that provides an API which looks like this:

Since our EAM machine would hold all of the common names that need to be managed in one place, I am thinking to make a module that is installed on the EAM machine that does the following. Since java/module is inherently harder to tamper with that python code, that would be a key driver to have it be in a module since SSL is mission critical.

  1. periodically checks expiration dates, if a renewal is needed it is scheduled (to prevent too many happening at once)
  2. execute the renewal API and package the SSL cert and private key into a PKCS12 file
  3. Using SSH, perform backup of the current cert, transfer new one, and execute GWCMD -g to load the new cert immediately
  4. Perform a git call to the following to check gateway
    "https://{address}:8043/system/gwinfo"

There is an internal table called "WSCONNECTIONSETTINGS" that already has all of the EAM agent common names which is exactly what we would need to be managing.

My initial questions are:

  1. is this the correct place to look to get all EAM agent common names?
  2. If yes, would it be preferable to grab the info with HTTP calls ( /data/status/internal-db ) or with a query?

Thanks,

Nick

Have you looked into the ACME protocol? Looks like Venafi can act as an ACME server: What is ACME Protocol | Venafi

You could follow this guide except you swap "Let's Encrypt" with your Venafi as the ACME server: Let's Encrypt Guide for Ignition | Inductive Automation

I'd think that this would save you the work of having to write and maintain a Module. Instead, you'd let an ACME client such as certbot which runs as another process alongside Ignition's java process deal with certificate automation. Each Gateway in your network in theory would have its own cert bot so that you can divide and conquer by delegating each host to deal with their own certs.

2 Likes

@jspecht thanks for pointing it out, we will consider it. One convenient thing about a module is that while we do have to handle the details, it is easy to distribute via EAM and would make logging and status checking much more visible from the gateway UI. If we have something running on the VM but is fully decoupled from Ignition, we won't be able to see if there is an issue without going to the VM itself which involves going in via RDP which is no issue, but I would prefer to have visibility in the gateway logs at least.

If we setup cert bot and cronjobs, we have to do it on each and every gateway and same again if we need to ever modify.

Thanks,

Nick

Gotcha, well if you roll your own, you could also consider implementing an ACME client in your module or pulling in a third party dependency which implements an ACME client in Java (for example, see: ACME Client Implementations - Let's Encrypt). That's something I want to add to the Ignition platform some day...not likely any time soon though.

Regarding your question about the common name...it depends. Is this certificate intended for the Ignition web server (default port 8043)? Which column are you looking at in WSCONNECTIONSETTINGS table?

It really depends on your network and DNS setup. Which hostname(s) are your TLS clients (i.e. web browsers, vision clients, designer instances, etc) using to connect to the target Gateway?

You might be better off inverting the design and going with the ACME client implementation I mentioned above or something like that - have each Gateway reach out to your central authority periodically when it determines that it is time to renew certs. Let each Gateway manage its own cert lifecycle instead of burdening one single Gateway. You could still expose status routes and tags in your module so that each agent can expose info to your central controller if that's what you'd like...

1 Like

Hint, Nick. Hint, hint.

{ Joel's being awfully polite. Distributing the work with a certbot implementation is definitely the way to go. I'd also go through Venafi to monitor it. Distribute your "certbot module" with EAM. }

1 Like

Yeah, also through internal discussion today we are leaning that way i.e. have the module installed on each individual gateway and get it there initially and update it as needed via EAM. Basically we've moved away from the central idea today as you both are eluding to.

A very useful product feature would be if Ignition had auto SSL cert renewal built in. Joel hints above that the idea already exists. IA did a fantastic job on the SAML cert auto download feature, something that may not have arrived had we not had the discussion.

Imagine if on the web server page there was an option to turn SSL cert auto-renew on/off and then the needed fields such as renewal threshold (days before expiry) and other necessary fields to be able to authenticate/communicate with the cert provider API. Set and forget for SSL would be awesome.

In any case, we have a consultation with CWS (cryptography web service) on Wednesday and I will update this thread as we progress this topic which is necessary for us to solve since the need to renew certs never goes away.....

Cheers,

Nick

I would deploy certbot or whatever acme client you want to use via Ansible. Should be a relatively easy task.

Additional info I got this morning on a mail written by the head of infosec at our company is that in our version of Venafi, ACME is not supported or in place and there is no plan to do so.

So making a custom module to interact with the CWS API shown above is really the only option.

Rgds,

Nick

@jspecht I met with the CWS team yesterday and got some more context.

In our cloud platform (WCNP, Walmart Cloud Native Platform) whenever you start an app in kubernetes, its SSL cert is automatically there. You never apply, never renew, you never have to obsolete, its just magically there. Talking to some people who work directly with WCNP development, CWS (the API mentioned above) is what is doing that.

So then it seems security has said, this is the one method for the company and that is essentially why ACME is not used. What I have to figure out is how to make this work in the context of a windows VM :grinning:.

We're going to proceed writing a module that takes care of these lifecycle requirements that they have in order to allow us to use the CWS API:

  1. Initial cert application will no longer be allowed manually, but must originate from the app
  2. Yearly cert renewal must be done by the app
  3. Decommission/obsolescence request must originate from the app

For 2. I already have a clear idea. For 1 and 3 I'm imagining some sort of settings page for SSL auto management where the user will have the option to manually create and obsolete a cert. Will be thinking through how to best do it with our team.

I guess I wanted to post this here so that if IA is ever thinking about SSL cert management functionality in the future, you have one use case to look.

@Kyle_Chase

Should be a relatively easy task

Love the optimism :rofl:

Rgds,

Nick

1 Like

@jspecht I'm at the point with the CWS API that I'm ready to download a cert. We can download it in PKCS12 which will be what Ignition needs (SSL cert + private key). The thing I'm currently trying to wrap my head around how to do is related to the key store and private key passwords which Ignition defaults to "ignition".

In the API there are these requirements which are in code so we cannot bypass at the time of call:

So from the start we can't even get a cert back unless I use a more sophisticated password. Then if I am not mistaken we can change the key store password but we cannot change the private key password once it is issued. Correct me if I am wrong, but the only option seems like we have to add these parameters to the ignition.conf file.

Then, if we change the password by manually modifying the ignition config file, there are 2 issues:

  • If we do it on a gateway that is already running, it will break SSL
  • Since we can only specify it statically, we'd be using the same password everywhere, something security may bite us for in the future.

I may be mistaken in my understanding.

Thanks,

Nick

One workaround off the top of my head: generate random passwords for the purposes of obtaining the keystore, then store it using the passwords from system properties that Ignition uses...

So if we changed the key store password to match what was already in Ignition, we also have to change the private key password to be able to let Ignition access it, correct?

Nick

Why can you not put a new password on the private key, knowing the original password? (OpenSSL can do this for you.)

Yes, that is the plan, change the password on both the cert store and private key before install. Now working through the mechanics of doing so in Java.

Nick

Are there any methods in the SDK to execute GWCMD commands?

In particular:

gwcmd -g

I looked around in gateway context but didn't see it.

Thanks,

Nick

No. There's a non-public API to issue the reload keystore command. I'm not entirely sure why it's non public, but if you ask the context for its WebResourceManager, then (reflectively) ask that for a getSslManager() method, then ask the ssl manager object to invoke refresh you can programatically invoke the same operation. Just be aware that any Ignition version bump could change those method names.

It does refresh automatically every 15 minutes by default, or every ignition.ssl.refresh system property minutes. If you're doing your renewals plenty in advance you could just wait for that.

Yeah, what I want to do is place the new file and check immediately over port 8043 as proof that the cert is correctly installed per this conversation.

If there was a hook that let me know when the cert has been reloaded every 15 minutes I could use that as a time to check but I doubt such a thing exists so this script module would be unaware of the event.

Sounds like I just need to execute the command line from java.

Thanks,

Nick

In case anyone else in the future is wondering how to execute gwcmd -g from a module, functionally this does accomplish it and prints the output to the gateway logs:

    private void loadSSLKeyStoreImmediately() {
        String home = getGatewayHome();
        String drive = home.substring(0, 1) + ":";
        String command = String.format("%s && cd %s && gwcmd -g", drive, home);
        try {
            ProcessBuilder builder = new ProcessBuilder("cmd.exe", "/c", command);
            builder.redirectErrorStream(true);
            Process p = builder.start();
            BufferedReader r = new BufferedReader(new InputStreamReader(p.getInputStream()));
            String line;

            while(true) {
                line = r.readLine();
                if (line == null) { break; }
                logger.info(line);
            }
        } catch (Exception e) {
            logger.error("Unable to execute gwcmd -g", e);
        }
    }

Gateway Log Output

Rgds,

Nick

@jspecht after quite a learning journey, I'm to the point now where I am trying to hot load the new cert into the gateway. I'll state the problem here and then provide more detail below:

Problem
When loading the new PKCS12 in the gateway, the CMD output looks good and I can see this in the logs after I restore the backup. But when I first load the new cert and do gwcmd -g, the SSL lock stays on, but nothing will display in the gateway when you try to navigate, everything is blank. Restoring the back up cert puts things back to normal.

Here, entry 2 is the backup restore, 1 is when the new cert was loaded. Both look successful but the first one is seemingly not:

If I look at the new cert and the original (known working SSL) in key store explorer, both look the same and open with the same passwords:

New (doesn't behave right in the gateway)

Original ( works correctly )

In both cases, the command line didn't give any indication of issue

Background
When we get the cert back from API call, it is in a string. It is then converted to a key store and then the password and alias are changed:

// workflow
KeyStore ks = stringToKeyStore(certAsString, downloadPass);
ks = changePassword(ks, downloadPass);
saveKeyStore(ks, pass, filePath);

// Functions

    public static KeyStore stringToKeyStore (String certAsString, String password) {
        char[] certPass = password.toCharArray();
        byte[] decodedBytes = Base64.getDecoder().decode(certAsString);
        try {
            InputStream inputStream = new ByteArrayInputStream(decodedBytes);
            KeyStore ks = KeyStore.getInstance("PKCS12");
            ks.load(inputStream, certPass);
            return ks;
        } catch (Exception e) {
            System.out.println("Issue encountered converting string to PKCS12");
            return null;
        }
    }

    public static KeyStore changePassword(KeyStore ks, String currentPassword)  {
        Enumeration<String> aliases = ks.aliases();
        while (aliases.hasMoreElements()) {
            String alias = aliases.nextElement();
            Certificate cert = ks.getCertificate(alias);
            if (((X509Certificate) cert).getSubjectDN().toString().contains(commonName)) {
                Key key = ks.getKey(alias, currentPassword.toCharArray());
                Certificate[] chain = ks.getCertificateChain(alias);
                ks.setKeyEntry(pass, key, pass.toCharArray(), chain);
                ks.deleteEntry(alias);
            }
        }
        return  ks;
    }

    public static Boolean saveKeyStore(KeyStore ks, String password, String outputPath) {
        try {
            FileOutputStream fos = new FileOutputStream(outputPath, false);
            ks.store(fos, password.toCharArray());
            fos.close();
            return true;
        } catch (Exception e) {
            System.out.println(String.format("Issue encountered when trying to save [%s]", outputPath));
            System.out.println(e);
            return false;
        }
    }

Still working on it but the fact that when I load the new keystore into the gateway and it doesn't throw any error but the gateway then shows blank pages is where I'm a bit confused on how to fix.

Thanks,

Nick